perf: Change query-exporting to use generators instead of expanding fully into memory #4729
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎
|
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
There was a problem hiding this comment.
PR Summary
This PR optimizes query history export by implementing generator-based pagination and parallel processing to reduce memory usage.
- Refactored
fetch_and_process_chat_session_historyin/backend/ee/onyx/server/query_history/api.pyto use paginated fetching withPAGE_SIZE=100instead of loading all data at once - Added parallel processing of chat session snapshots using
parallel_yieldto improve performance while maintaining thread safety - Modified CSV writing in
/backend/ee/onyx/background/celery/apps/heavy.pyto process data incrementally as it's fetched rather than materializing entire dataset - Fixed pagination logic bug where the break condition was incorrectly checking for equal instead of less than
PAGE_SIZE - Removed redundant list comprehensions and memory-intensive operations throughout the codebase
2 file(s) reviewed, 1 comment(s)
Edit PR Review Bot Settings | Greptile
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
rkuo-danswer
approved these changes
May 19, 2025
ferdinandl007
pushed a commit
to ferdinandl007/onyx
that referenced
this pull request
May 27, 2025
…ully into memory (onyx-dot-app#4729) * Change query-exporting to use generators instead of expanding fully into memory * Fix pagination logic Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * Add type annotation * Add early break if list of chat_sessions is empty --------- Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
aronszanto
pushed a commit
to aronszanto/onyx
that referenced
this pull request
May 27, 2025
…ully into memory (onyx-dot-app#4729) * Change query-exporting to use generators instead of expanding fully into memory * Fix pagination logic Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * Add type annotation * Add early break if list of chat_sessions is empty --------- Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
ZhipengHe
pushed a commit
to ZhipengHe/onyx
that referenced
this pull request
Jun 6, 2025
…ully into memory (onyx-dot-app#4729) * Change query-exporting to use generators instead of expanding fully into memory * Fix pagination logic Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * Add type annotation * Add early break if list of chat_sessions is empty --------- Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
AnkitTukatek
pushed a commit
to TukaTek/onyx
that referenced
this pull request
Sep 23, 2025
…ully into memory (onyx-dot-app#4729) * Change query-exporting to use generators instead of expanding fully into memory * Fix pagination logic Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * Add type annotation * Add early break if list of chat_sessions is empty --------- Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Query history exporting fetches the entire history and populates it into memory. This causes significant pressure on memory, and can lead to OOMs for very large datasets.
This PR updates the logic to incrementally fetch pages of data, transform them, and write them to file, instead of fetching the entire dataset at once, transforming each row, and then writing it to file.
I.e., we don't fully collect / materialize the entire dataset, we do it in pages instead.
This PR also parallelizes the chat-session fetching logic (given that chat-session reading does not have any cross-thread dependencies [I think?]).
Addresses: https://linear.app/danswer/issue/DAN-1984/improve-performance-of-query-history-exporting.
How Has This Been Tested?
This is performance change, not a feature-addition. We don't have much (if any) performance testing / benchmarking in our code. Therefore, this is not tested.
The feature still outputs the same output for the same input.