fix: drive external links #5079
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎
|
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
There was a problem hiding this comment.
Greptile Summary
This PR fixes a critical crash issue in DOCX file processing when files contain external links (such as images referenced from external sources). The change modifies the docx_to_text_and_images function in extract_file_text.py to handle external relationships gracefully instead of crashing.
The fix adds two defensive measures to the image extraction logic:
- External relationship detection: Uses
getattr(rel, "is_external", False)to identify and skip externally linked images that don't have internal blob data - Exception handling: Wraps the
rel.target_part.blobaccess in a try-catch block to handle edge cases where relationships lack an internal target_part
This change integrates well with the existing file processing pipeline by maintaining the same return format while preventing crashes that would completely block indexing of DOCX files containing external links. The approach aligns with the codebase's defensive programming patterns and ensures that processable content can still be extracted even when some elements cannot be handled.
Confidence score: 4/5
• This change is safe to merge and fixes a legitimate crash issue without breaking existing functionality
• The defensive programming approach properly handles edge cases and maintains existing behavior for normal DOCX files
• The extract_file_text.py file needs attention to ensure the fix works as expected in production scenarios
1 file reviewed, no comments
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
There was a problem hiding this comment.
cubic analysis
No issues found across 1 file. Review in cubic
aponcedeleonch
added a commit
to StacklokLabs/onyx
that referenced
this pull request
Jul 29, 2025
* fix bug in index swap (onyx-dot-app#5036) * Add PR labeller job (onyx-dot-app#4611) * fix: Fix Confluence pagination (onyx-dot-app#5035) * Re-implement pagination * Add note * Fix invalid integration test configs * Fix other failing test * Edit failing test * Revert test * Revert pagination size * Add comment on yielding style * Use fixture instead of manually initializing sql-engine * Fix failing tests * Move code back and copy-paste * fix: Have document show up before message starts streaming back (onyx-dot-app#5006) * Have document show up before message starts streaming back * Add docs * fix: Move around group-sync tests (since they require docker services to be running) (onyx-dot-app#5041) * Move around tests * Add missing fixtures + change directory structure up some more * Add env variables * remove chat session necessity from send message simple api (onyx-dot-app#5040) * Improve support for non-default postgres schemas (onyx-dot-app#5046) * fix: improve check for indexing status (onyx-dot-app#5042) * Improve check_for_indexing + check_for_vespa_sync_task * Remove unused * Fix * Simplify query * Add more logging * Address bot comments * Increase # of tasks generated since we're not going cc-pair by cc-pair * Only index 50 user files at a time * fix: improve assistant fetching efficiency (onyx-dot-app#5047) * Improve assistant fetching efficiency * More fix * Fix weird build stuff * Improve * feat: KG improvements (onyx-dot-app#5048) * improvements * drop views if SQL fails * mypy fix * feat: Search and Answer Quality Test Script (onyx-dot-app#4974) * aefads * search quality tests improvement Co-authored-by: wenxi-onyx <wenxi@onyx.app> * nits * refactor: config refactor * document context + skip genai fix * feat: answer eval * more error messages * mypy ragas * mypy * small fixes * feat: more metrics * fix * feat: grab content * typing * feat: lazy updates * mypy * all at front * feat: answer correctness * use api key so it works with auth enabled * update readme * feat: auto add path * feat: rate limit * fix: readme + remove rerank all * fix: raise exception immediately * docs: improved clarity * feat: federated handling * fix: mypy * nits --------- Co-authored-by: wenxi-onyx <wenxi@onyx.app> * Remove empty tooltip (onyx-dot-app#5050) * feat: Updated KG admin page (onyx-dot-app#5044) * Update KG admin UI * Styling changes * More changes * Make edits auto-save * Add more stylings / transitions * Fix opacity * Separate out modal into new component * Revert backend changes * Update styling * Add convenience / styling changes to date-picker * More styling / functional updates to kg admin-page * Avoid reducing opacity of active-toggle * Update backend APIs for new KG admin page * More updates of styling for kg-admin page * Remove nullability * Remove console log * Remove unused imports * Change type of `children` variable * Update web/src/app/admin/kg/interfaces.ts Co-authored-by: cubic-dev-ai[bot] <191113872+cubic-dev-ai[bot]@users.noreply.github.com> * Update web/src/components/CollapsibleCard.tsx Co-authored-by: cubic-dev-ai[bot] <191113872+cubic-dev-ai[bot]@users.noreply.github.com> * Remove null * Update web/src/components/CollapsibleCard.tsx Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * Force non-null * Fix failing test --------- Co-authored-by: cubic-dev-ai[bot] <191113872+cubic-dev-ai[bot]@users.noreply.github.com> Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * Make `from_.user` optional (use "Unknown User") if not found (onyx-dot-app#5051) * feat: connector indexing decoupling (onyx-dot-app#4893) * WIP * renamed and moved tasks (WIP) * minio migration * bug fixes and finally add document batch storage * WIP: can suceed but status is error * WIP * import fixes * working v1 of decoupled * catastrophe handling * refactor * remove unused db session in prep for new approach * renaming and docstrings (untested) * renames * WIP with no more indexing fences * robustness improvements * clean up rebase * migration and salesforce rate limits * minor tweaks * test fix * connector pausing behavior * correct checkpoint resumption logic * cleanups in docfetching * add heartbeat file * update template jsonc * deployment fixes * fix vespa httpx pool * error handling * cosmetic fixes * dumb * logging improvements and non checkpointed connector fixes * didnt save * misc fixes * fix import * fix deletion of old files * add in attempt prefix * fix attempt prefix * tiny log improvement * minor changes * fixed resumption behavior * passing int tests * fix unit test * fixed unit tests * trying timeout bump to see if int tests pass * trying timeout bump to see if int tests pass * fix autodiscovery * helm chart fixes * helm and logging * Tiny launch.json template improvement (onyx-dot-app#5055) * refactor: Update the error message that is logged when PR title fails Conventional Commits regex (onyx-dot-app#5062) * fix: Make pr-labeler run on edits too * fix: time discrepancy (onyx-dot-app#5056) * fix time discrepancy * remove log * remove log * handle empty doc batches (onyx-dot-app#5058) * fix: too many internet chunks (onyx-dot-app#5060) * minor internet search env vars * add limit to internet search chunks * note * nits * fix: remove extra group sync (onyx-dot-app#5061) * fix: remove extra group sync * second extra task * fix: regen api key (onyx-dot-app#5064) * feat: avoid full rerun (onyx-dot-app#5063) * fix: remove extra group sync * second extra task * minor improvement for non-checkpointed connectors * fix: explicit api_server dependency on minio in docker compose files (onyx-dot-app#5066) * fix: adjust template variable from .Chart.AppVersion to .Values.global.version to match versioning pattern. (onyx-dot-app#5069) * refactor: Update location of `sidebar` (onyx-dot-app#5067) * Use props instead of inline type def * Add new AppProvider * Remove unused component file * Move `sessionSidebar` to be inside of `components` instead of `app/chat` * Change name of `sessionSidebar` to `sidebar` * Remove `AppModeProvider` * Fix bug in how the cookies were set * fix: remove locks from indexing callback (onyx-dot-app#5070) * attempt fix for broken excel files (onyx-dot-app#5071) * fix: sharepoint lg files issue (onyx-dot-app#5065) * add SharePoint file size threshold check * Implement retry logic for SharePoint queries to handle rate limiting and server error * mypy fix * add content none check * remove unreachable code from retry logic in sharepoint connector * add library to fall back to for tokenizing (onyx-dot-app#5078) * fix: drive external links (onyx-dot-app#5079) * feat: support aspx files (onyx-dot-app#5068) * Support aspx files * Add fetching of site pages * Improve * Small enhancement * more improvements * Improvements * Fix tests * attempt to fix parsing of tricky template files (onyx-dot-app#5080) * typo (onyx-dot-app#5082) * fix: preserve error traces (onyx-dot-app#5083) * fix: sidebar ranges (onyx-dot-app#5084) * onyx metadata minio fix + permissive unstructured fail (onyx-dot-app#5085) * feat: pruning freq (onyx-dot-app#5097) * pruning frequency increase * add logs * [Vespa] Update to optimized configuration * Let's do this properly * [Vespa] Update to optimized configuration pt.2 (onyx-dot-app#5113) * Node option to avoid heap out of memory (#1) * set some resource limits that work (#2) * set some resource limits that work --------- Signed-off-by: nigel brown <nigel@stacklok.com> * Adds endpoint to the onyx API for using the search tool (#3) Usage: ```bash curl -X POST "http://localhost:8080/onyx-tools/search-tool" \ -H "Content-Type: application/json" \ -d '{"query": "key projects stacklok"}' | jq ``` * create a new node pool and used it and ECR (#4) Signed-off-by: nigel brown <nigel@stacklok.com> * Add document links to the output (#5) * Remove unused DocumentResult class (#6) * Enhance search API with filtering and structured response (#8) - Add time_cutoff parameter to filter results by date - Add document_sources parameter to filter by source types - Replace string response with structured FoundDocSearchTool objects - Update SearchToolRequest model with new optional parameters - Improve type hints using modern Python syntax (list[] vs List[]) 🤖 Generated with [Claude Code](https://claude.ai/code) Co-authored-by: Claude <noreply@anthropic.com> * Enables basic auth in Onyx (#9) * Enables basi auth in Onyx Before we we're not authenticating in Onyx. Now the requests should be authenticated for them to be successful * Pass token user to SearchTool --------- Signed-off-by: nigel brown <nigel@stacklok.com> Co-authored-by: Evan Lohn <evan@danswer.ai> Co-authored-by: Raunak Bhagat <r@rabh.io> Co-authored-by: Wenxi <wenxi@onyx.app> Co-authored-by: Chris Weaver <chris@onyx.app> Co-authored-by: joachim-danswer <joachim@danswer.ai> Co-authored-by: Rei Meguro <36625832+Orbital-Web@users.noreply.github.com> Co-authored-by: cubic-dev-ai[bot] <191113872+cubic-dev-ai[bot]@users.noreply.github.com> Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> Co-authored-by: Devin <49892118+Dcooley1350@users.noreply.github.com> Co-authored-by: PaulHLiatrio <100874415+PaulHLiatrio@users.noreply.github.com> Co-authored-by: SubashMohan <subashmohan75@gmail.com> Co-authored-by: justin-tahara <justintahara@gmail.com> Co-authored-by: Justin Tahara <105671973+justin-tahara@users.noreply.github.com> Co-authored-by: Nigel Brown <nigel@stacklok.com> Co-authored-by: Pankaj Telang <ptelang@gmail.com> Co-authored-by: Claude <noreply@anthropic.com>
aponcedeleonch
pushed a commit
to StacklokLabs/onyx
that referenced
this pull request
Jul 29, 2025
aponcedeleonch
pushed a commit
to StacklokLabs/onyx
that referenced
this pull request
Jul 29, 2025
wenxi-onyx
pushed a commit
that referenced
this pull request
Aug 11, 2025
AnkitTukatek
pushed a commit
to TukaTek/onyx
that referenced
this pull request
Sep 23, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Fixes https://linear.app/danswer/issue/DAN-2250/docx-external-links
We were previously erroring out on docx files with external links; we're temporarily going to skip them to allow indexing. In the future we should add some best-effort fetching of external links.
How Has This Been Tested?
n/a, should be totally benign
Backporting (check the box to trigger backport action)
Note: You have to check that the action passes, otherwise resolve the conflicts manually and tag the patches.
Summary by cubic
Docx files with external image links are now skipped during processing to prevent errors and allow successful indexing.