XLoader Freezes on Large Files (>2GB) Due to Non-Chunked copy_expert

[cgoldshtein]

Problem
XLoader uses copy_expert to load entire files at once, causing the process to freeze/hang when processing very large files (>2GB). This makes the system unresponsive and may cause timeouts or memory exhaustion.

Current Behavior
Loads entire 2GB+ file into memory at once

System freezes for hours with no feedback

May crash due to memory exhaustion

Users cannot tell if process is working or stuck

Expected Behavior
Process files in small chunks (e.g., 100MB pieces)

Show progress updates ("Processing chunk 5 of 20")

Use consistent memory regardless of file size

System remains responsive during upload

Impact
System becomes unresponsive for hours

Memory exhaustion on large datasets

No way to track upload progress

Poor experience with big data files

Proposed Solution
Currently working on chunked processing with configurable ckanext.xloader.copy_chunk_size parameter to process files in smaller pieces (e.g., 100MB chunks) while maintaining performance for multi-gigabyte files.

Related to #255

[duttonw]

When you are doing the batch load. What ram is set in the batch server and on the web server? Did you also have swap disk installed and is it double the ram size?

[cgoldshtein]

RAM Configuration
100GB RAM on both batch and web servers (Google Cloud).
Swap Disk
No swap configured (Google Cloud default), but not needed with 100GB RAM.
Batch Load Clarification
I'm not doing batch loading - I implemented chunked processing in the XLoader upload file process itself.
Root Cause
The freezing isn't memory-related - it's XLoader's copy_expert blocking while loading entire files. PR #263's chunked processing should fix this.

[ThrawnCA]

I'm not doing batch loading

I think by "batch processing" Will simply meant "running a job in a CKAN worker queue".