-
Notifications
You must be signed in to change notification settings - Fork 2.1k
fix: attempt to fix parsing of tricky template files #5080
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
The latest updates on your projects. Learn more about Vercel for Git ↗︎
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Greptile Summary
This PR enhances error handling in the docx_to_text_and_images
function within the file processing module. The change expands the exception catching from just BadZipFile
to include ValueError
exceptions when processing DOCX files with python-docx
.
The modification addresses a specific edge case with Google Drive template files where there's a mismatch between the MIME type provided by Google Drive and the actual file content structure. When python-docx
encounters these inconsistencies, it raises a ValueError
instead of the previously handled BadZipFile
exception.
The fix follows the existing error handling pattern in the codebase - when either exception occurs, the system logs a warning and gracefully degrades to parsing the file as plain text using the read_text_file
function with proper encoding detection. This maintains the robustness of the document processing pipeline by ensuring files don't fail completely due to format inconsistencies, particularly important for the Google Drive connector integration described in the related documentation.
Confidence score: 4/5
• This is a safe defensive programming change that expands existing error handling without breaking current functionality
• The change follows established patterns in the codebase and maintains the same fallback behavior
• The file needs attention to ensure the ValueError
catching doesn't mask other legitimate validation errors that should bubble up
1 file reviewed, no comments
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
cubic analysis
No issues found across 1 file. Review in cubic
* fix bug in index swap (onyx-dot-app#5036) * Add PR labeller job (onyx-dot-app#4611) * fix: Fix Confluence pagination (onyx-dot-app#5035) * Re-implement pagination * Add note * Fix invalid integration test configs * Fix other failing test * Edit failing test * Revert test * Revert pagination size * Add comment on yielding style * Use fixture instead of manually initializing sql-engine * Fix failing tests * Move code back and copy-paste * fix: Have document show up before message starts streaming back (onyx-dot-app#5006) * Have document show up before message starts streaming back * Add docs * fix: Move around group-sync tests (since they require docker services to be running) (onyx-dot-app#5041) * Move around tests * Add missing fixtures + change directory structure up some more * Add env variables * remove chat session necessity from send message simple api (onyx-dot-app#5040) * Improve support for non-default postgres schemas (onyx-dot-app#5046) * fix: improve check for indexing status (onyx-dot-app#5042) * Improve check_for_indexing + check_for_vespa_sync_task * Remove unused * Fix * Simplify query * Add more logging * Address bot comments * Increase # of tasks generated since we're not going cc-pair by cc-pair * Only index 50 user files at a time * fix: improve assistant fetching efficiency (onyx-dot-app#5047) * Improve assistant fetching efficiency * More fix * Fix weird build stuff * Improve * feat: KG improvements (onyx-dot-app#5048) * improvements * drop views if SQL fails * mypy fix * feat: Search and Answer Quality Test Script (onyx-dot-app#4974) * aefads * search quality tests improvement Co-authored-by: wenxi-onyx <wenxi@onyx.app> * nits * refactor: config refactor * document context + skip genai fix * feat: answer eval * more error messages * mypy ragas * mypy * small fixes * feat: more metrics * fix * feat: grab content * typing * feat: lazy updates * mypy * all at front * feat: answer correctness * use api key so it works with auth enabled * update readme * feat: auto add path * feat: rate limit * fix: readme + remove rerank all * fix: raise exception immediately * docs: improved clarity * feat: federated handling * fix: mypy * nits --------- Co-authored-by: wenxi-onyx <wenxi@onyx.app> * Remove empty tooltip (onyx-dot-app#5050) * feat: Updated KG admin page (onyx-dot-app#5044) * Update KG admin UI * Styling changes * More changes * Make edits auto-save * Add more stylings / transitions * Fix opacity * Separate out modal into new component * Revert backend changes * Update styling * Add convenience / styling changes to date-picker * More styling / functional updates to kg admin-page * Avoid reducing opacity of active-toggle * Update backend APIs for new KG admin page * More updates of styling for kg-admin page * Remove nullability * Remove console log * Remove unused imports * Change type of `children` variable * Update web/src/app/admin/kg/interfaces.ts Co-authored-by: cubic-dev-ai[bot] <191113872+cubic-dev-ai[bot]@users.noreply.github.com> * Update web/src/components/CollapsibleCard.tsx Co-authored-by: cubic-dev-ai[bot] <191113872+cubic-dev-ai[bot]@users.noreply.github.com> * Remove null * Update web/src/components/CollapsibleCard.tsx Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * Force non-null * Fix failing test --------- Co-authored-by: cubic-dev-ai[bot] <191113872+cubic-dev-ai[bot]@users.noreply.github.com> Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * Make `from_.user` optional (use "Unknown User") if not found (onyx-dot-app#5051) * feat: connector indexing decoupling (onyx-dot-app#4893) * WIP * renamed and moved tasks (WIP) * minio migration * bug fixes and finally add document batch storage * WIP: can suceed but status is error * WIP * import fixes * working v1 of decoupled * catastrophe handling * refactor * remove unused db session in prep for new approach * renaming and docstrings (untested) * renames * WIP with no more indexing fences * robustness improvements * clean up rebase * migration and salesforce rate limits * minor tweaks * test fix * connector pausing behavior * correct checkpoint resumption logic * cleanups in docfetching * add heartbeat file * update template jsonc * deployment fixes * fix vespa httpx pool * error handling * cosmetic fixes * dumb * logging improvements and non checkpointed connector fixes * didnt save * misc fixes * fix import * fix deletion of old files * add in attempt prefix * fix attempt prefix * tiny log improvement * minor changes * fixed resumption behavior * passing int tests * fix unit test * fixed unit tests * trying timeout bump to see if int tests pass * trying timeout bump to see if int tests pass * fix autodiscovery * helm chart fixes * helm and logging * Tiny launch.json template improvement (onyx-dot-app#5055) * refactor: Update the error message that is logged when PR title fails Conventional Commits regex (onyx-dot-app#5062) * fix: Make pr-labeler run on edits too * fix: time discrepancy (onyx-dot-app#5056) * fix time discrepancy * remove log * remove log * handle empty doc batches (onyx-dot-app#5058) * fix: too many internet chunks (onyx-dot-app#5060) * minor internet search env vars * add limit to internet search chunks * note * nits * fix: remove extra group sync (onyx-dot-app#5061) * fix: remove extra group sync * second extra task * fix: regen api key (onyx-dot-app#5064) * feat: avoid full rerun (onyx-dot-app#5063) * fix: remove extra group sync * second extra task * minor improvement for non-checkpointed connectors * fix: explicit api_server dependency on minio in docker compose files (onyx-dot-app#5066) * fix: adjust template variable from .Chart.AppVersion to .Values.global.version to match versioning pattern. (onyx-dot-app#5069) * refactor: Update location of `sidebar` (onyx-dot-app#5067) * Use props instead of inline type def * Add new AppProvider * Remove unused component file * Move `sessionSidebar` to be inside of `components` instead of `app/chat` * Change name of `sessionSidebar` to `sidebar` * Remove `AppModeProvider` * Fix bug in how the cookies were set * fix: remove locks from indexing callback (onyx-dot-app#5070) * attempt fix for broken excel files (onyx-dot-app#5071) * fix: sharepoint lg files issue (onyx-dot-app#5065) * add SharePoint file size threshold check * Implement retry logic for SharePoint queries to handle rate limiting and server error * mypy fix * add content none check * remove unreachable code from retry logic in sharepoint connector * add library to fall back to for tokenizing (onyx-dot-app#5078) * fix: drive external links (onyx-dot-app#5079) * feat: support aspx files (onyx-dot-app#5068) * Support aspx files * Add fetching of site pages * Improve * Small enhancement * more improvements * Improvements * Fix tests * attempt to fix parsing of tricky template files (onyx-dot-app#5080) * typo (onyx-dot-app#5082) * fix: preserve error traces (onyx-dot-app#5083) * fix: sidebar ranges (onyx-dot-app#5084) * onyx metadata minio fix + permissive unstructured fail (onyx-dot-app#5085) * feat: pruning freq (onyx-dot-app#5097) * pruning frequency increase * add logs * [Vespa] Update to optimized configuration * Let's do this properly * [Vespa] Update to optimized configuration pt.2 (onyx-dot-app#5113) * Node option to avoid heap out of memory (#1) * set some resource limits that work (#2) * set some resource limits that work --------- Signed-off-by: nigel brown <nigel@stacklok.com> * Adds endpoint to the onyx API for using the search tool (#3) Usage: ```bash curl -X POST "http://localhost:8080/onyx-tools/search-tool" \ -H "Content-Type: application/json" \ -d '{"query": "key projects stacklok"}' | jq ``` * create a new node pool and used it and ECR (#4) Signed-off-by: nigel brown <nigel@stacklok.com> * Add document links to the output (#5) * Remove unused DocumentResult class (#6) * Enhance search API with filtering and structured response (#8) - Add time_cutoff parameter to filter results by date - Add document_sources parameter to filter by source types - Replace string response with structured FoundDocSearchTool objects - Update SearchToolRequest model with new optional parameters - Improve type hints using modern Python syntax (list[] vs List[]) 🤖 Generated with [Claude Code](https://claude.ai/code) Co-authored-by: Claude <noreply@anthropic.com> * Enables basic auth in Onyx (#9) * Enables basi auth in Onyx Before we we're not authenticating in Onyx. Now the requests should be authenticated for them to be successful * Pass token user to SearchTool --------- Signed-off-by: nigel brown <nigel@stacklok.com> Co-authored-by: Evan Lohn <evan@danswer.ai> Co-authored-by: Raunak Bhagat <r@rabh.io> Co-authored-by: Wenxi <wenxi@onyx.app> Co-authored-by: Chris Weaver <chris@onyx.app> Co-authored-by: joachim-danswer <joachim@danswer.ai> Co-authored-by: Rei Meguro <36625832+Orbital-Web@users.noreply.github.com> Co-authored-by: cubic-dev-ai[bot] <191113872+cubic-dev-ai[bot]@users.noreply.github.com> Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> Co-authored-by: Devin <49892118+Dcooley1350@users.noreply.github.com> Co-authored-by: PaulHLiatrio <100874415+PaulHLiatrio@users.noreply.github.com> Co-authored-by: SubashMohan <subashmohan75@gmail.com> Co-authored-by: justin-tahara <justintahara@gmail.com> Co-authored-by: Justin Tahara <105671973+justin-tahara@users.noreply.github.com> Co-authored-by: Nigel Brown <nigel@stacklok.com> Co-authored-by: Pankaj Telang <ptelang@gmail.com> Co-authored-by: Claude <noreply@anthropic.com>
Description
There are exceedingly rare cases where certain template files in drive have a mimetype provided by google drive that differs from the mimetype extracted from within the file that causes a ValueError to be raised when they are processed by python-docx. Now we catch these errors and attempt to parse as raw text.
How Has This Been Tested?
n/a, worst case is we get a different error
Backporting (check the box to trigger backport action)
Note: You have to check that the action passes, otherwise resolve the conflicts manually and tag the patches.
Summary by cubic
Fixed an error when parsing rare template files with mismatched mimetypes by catching ValueError and falling back to reading as raw text.