Severe Memory Spike/OOM in export_query_history_task During Chat History Export

[ThomaciousD]

Hi,

There is a critical memory issue in the export_query_history_task defined in backend/ee/onyx/background/celery/apps/heavy.py

Triggering the query history export (via /admin/query-history/start-export) or clicking the UI admin "Kickoff Export" button in the "Query History" section

causes the entire history of chat sessions to be loaded and processed in memory, which results in:

• Massive memory spikes (up to 50 GB in our case)
• Consistent Out-Of-Memory crashes of the Celery background pod
• Re-triggering of this job hourly (likely due to Celery retries), compounding the problem if not flushed manually

Probable root cause is In the task export_query_history_task:

onyx/backend/ee/onyx/background/celery/apps/heavy.py

Line 59 in 312e3b9

fetch_and_process_chat_session_history(

• limit=None causes the entire database of chat sessions to be fetched.
• fetch_and_process_chat_session_history() calls create_chat_chain() for every chat session, which is explicitly noted in the code to be very slow and memory-heavy.
• It builds multiple large Python data structures before streaming to CSV.

We also suspect that:

If the task fails due to OOM, it is retried automatically by Celery every hour (based on logs and pod memory graphs).
This creates a loop of instability and memory crashes of the background process.

So currently, the "Query History" exports for large tenants are currently not feasible.

Temporary Mitigation:
For others encountering this issue, you can flush the csv_generation queue from Redis manually (if using Redis as Celery broker) to stop re-triggering. If the export task was already started, look at the unacked keys in Redis for the "export_query_history_task" and delete it.

[raunakab]

@ThomaciousD Sorry for the late update. Have a small PR that aims to reduce all the memory-intensive data materializations that we were performing during the query-history export process.

The idea here is super simple: I'm performing multiple smaller paginated fetches instead of one large fetch. Each paginated fetch is then turned into a CSV row and written to memory. When the pagination fetches finish, then we close the file and mark the shared-task as finished.

I also updated the fetching of chat-session fetching to be executed in parallel on a threadpool instead (this change may be under a bit more contention during review, I predict).

I'm reviewing this with the team right now, and we'll run this on some large internal datasets as well, but just wanted to share the update with you.

Furthermore, if you'd like to try these changes on your own dataset, compare the differences, and report back any improvements / regressions, then any such insight would be much appreciated!

[raunakab]

@ThomaciousD Just re-pinging here. If you're able to test this out, would love to hear any feedback that you have.

[emerzon]

@raunakab , I can confirm this patch solved the issue we were having!

[raunakab]

@emerzon That's nice to hear! Glad to know this helped.