feat: Env var for community-supported connectors #5043

wenxi-onyx · 2025-07-17T23:22:59Z

Description

Reorganized connector page for clarity
Added new env var that shows/hides "Other" connectors supported by the community (not officially supported by Onyx)

How Has This Been Tested?

local

Backporting (check the box to trigger backport action)

Note: You have to check that the action passes, otherwise resolve the conflicts manually and tag the patches.

This PR should be backported (make sure to check that the backport attempt succeeds)
[Optional] Override Linear Check

vercel · 2025-07-17T23:23:04Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Project	Deployment	Preview	Comments	Updated (UTC)
internal-search	❌ Failed			Aug 11, 2025 11:18pm

greptile-apps

Greptile Summary

This PR introduces a new feature flag to control the visibility of community-supported connectors in Onyx's UI, along with a significant reorganization of connector categories for better clarity. The changes span across both frontend and backend, implementing a consistent way to distinguish between officially supported connectors and those maintained by the community.

Key changes:

Added SHOW_EXTRA_CONNECTORS environment variable (defaults to false)
Reorganized connectors into more specific categories (e.g., moving from generic 'Other' to specific categories like 'Sales & Marketing')
Updated category names to be more descriptive (e.g., 'Storage' → 'Web Crawl & File Storage')
Added settings integration to control connector visibility
Removed the NOT_APPLICABLE value from BlobType enum as part of cleanup

This change helps organizations better manage their connector offerings by providing control over which integrations are visible to users, particularly useful for enterprise deployments where limiting to officially supported connectors may be desired.

Confidence score: 4 /5

This PR is generally safe to merge with minimal risk of breaking existing functionality
The changes are well-structured and follow existing patterns, but the removal of BlobType.NOT_APPLICABLE warrants careful testing of internet search functionality
Files needing attention:
- backend/onyx/configs/constants.py: Verify internet search still works after BlobType enum change
- web/src/app/admin/add-connector/page.tsx: Ensure proper error handling for settings context

_{12 files reviewed, 7 comments}
_{Edit PR Review Bot Settings | Greptile}

deployment/docker_compose/env.prod.template

deployment/docker_compose/docker-compose.dev.yml

backend/onyx/server/settings/models.py

deployment/docker_compose/docker-compose.dev.yml

web/src/app/admin/settings/interfaces.ts

backend/onyx/server/settings/store.py

web/src/app/admin/add-connector/page.tsx

cubic-dev-ai

cubic reviewed 12 files and found no issues. Review PR in cubic.dev.

GeorgeGithiri5 · 2025-07-26T19:02:22Z

deployment/docker_compose/env.prod.template


 # NOTE: Generative AI configurations are done via the UI now
-
+EXA_API_KEY=


EXA_API_KEY=
Looks uncomplete

… to be running) (#5041) * Move around tests * Add missing fixtures + change directory structure up some more * Add env variables

* Improve check_for_indexing + check_for_vespa_sync_task * Remove unused * Fix * Simplify query * Add more logging * Address bot comments * Increase # of tasks generated since we're not going cc-pair by cc-pair * Only index 50 user files at a time

* Improve assistant fetching efficiency * More fix * Fix weird build stuff * Improve

* improvements * drop views if SQL fails * mypy fix

* aefads * search quality tests improvement Co-authored-by: wenxi-onyx <wenxi@onyx.app> * nits * refactor: config refactor * document context + skip genai fix * feat: answer eval * more error messages * mypy ragas * mypy * small fixes * feat: more metrics * fix * feat: grab content * typing * feat: lazy updates * mypy * all at front * feat: answer correctness * use api key so it works with auth enabled * update readme * feat: auto add path * feat: rate limit * fix: readme + remove rerank all * fix: raise exception immediately * docs: improved clarity * feat: federated handling * fix: mypy * nits --------- Co-authored-by: wenxi-onyx <wenxi@onyx.app>

* Update KG admin UI * Styling changes * More changes * Make edits auto-save * Add more stylings / transitions * Fix opacity * Separate out modal into new component * Revert backend changes * Update styling * Add convenience / styling changes to date-picker * More styling / functional updates to kg admin-page * Avoid reducing opacity of active-toggle * Update backend APIs for new KG admin page * More updates of styling for kg-admin page * Remove nullability * Remove console log * Remove unused imports * Change type of `children` variable * Update web/src/app/admin/kg/interfaces.ts Co-authored-by: cubic-dev-ai[bot] <191113872+cubic-dev-ai[bot]@users.noreply.github.com> * Update web/src/components/CollapsibleCard.tsx Co-authored-by: cubic-dev-ai[bot] <191113872+cubic-dev-ai[bot]@users.noreply.github.com> * Remove null * Update web/src/components/CollapsibleCard.tsx Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * Force non-null * Fix failing test --------- Co-authored-by: cubic-dev-ai[bot] <191113872+cubic-dev-ai[bot]@users.noreply.github.com> Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>

* WIP * renamed and moved tasks (WIP) * minio migration * bug fixes and finally add document batch storage * WIP: can suceed but status is error * WIP * import fixes * working v1 of decoupled * catastrophe handling * refactor * remove unused db session in prep for new approach * renaming and docstrings (untested) * renames * WIP with no more indexing fences * robustness improvements * clean up rebase * migration and salesforce rate limits * minor tweaks * test fix * connector pausing behavior * correct checkpoint resumption logic * cleanups in docfetching * add heartbeat file * update template jsonc * deployment fixes * fix vespa httpx pool * error handling * cosmetic fixes * dumb * logging improvements and non checkpointed connector fixes * didnt save * misc fixes * fix import * fix deletion of old files * add in attempt prefix * fix attempt prefix * tiny log improvement * minor changes * fixed resumption behavior * passing int tests * fix unit test * fixed unit tests * trying timeout bump to see if int tests pass * trying timeout bump to see if int tests pass * fix autodiscovery * helm chart fixes * helm and logging

… Conventional Commits regex (#5062)

* fix time discrepancy * remove log * remove log

* nit var clarity * maintain file names in connector config for display * remove unused util * migration draft * optional file names to not break existing instances * backwards compatible * backwards compatible * migration logging * update file ocnn tests * unncessary none * mypy + explanatory comments

* fix unsupported character error in minio migration * slash fix

* Add customization via env vars for curator role * Simplify * Simplify more * Address comments

* feat(infra): Adding helm directory * one more fix

* fix slash command slackbot to respond in private msg * rename confusing variable. fix slash message response in DMs

* feat(infra): Bump Vespa Helm Version * Adding the Chart.lock file

* WIP merged approach untested * tested custom configs * JT comments * fix unit test * CW comments * fix unit test

* fix: remove erroneous error case and add valid error * also address docfetching-docprocessing limbo

* support gpt5 models * gpt5mini visible

* switch to markitdown untested * passing tests * reset file * dotenv version * docs * add test file * add doc * fix integration test

* file processing refactor * mypy * CW comments * address CW

* sanitize slack payload before logging * nit

* fix(infra): Update github workflow to not tag latest * Cleaned up the code a bit

* fix: restrict user file access to current user only * fix: enhance user file access control for recent folder

* initial migration * getting metadata from tags * complete migration * migration override for cloud * fix: more robust structured tag gen * tag and indexing update * fix: move is_list to tags * migration rebase * test cases + bugfix on unique constraint * fix logging

* max tokens param * fix unit test * fix unit test

* sharepoint perm sync first draft * feat: Implement SharePoint permission synchronization * mypy fix * remove commented code * bot comments fixes and job failure fixes * introduce generic way to upload certificates in credentials * mypy fix * add checkpoiting to sharepoint connector * add sharepoint integration tests * Refactor SharePoint connector to derive tenant domain from verified domains and remove direct tenant domain input from credentials * address review comments * add permission sync to site pages * mypy fix * fix tests error * fix tests and address comments * Update file extraction behavior in SharePoint connector to continue processing on unprocessable files

wenxi-onyx added 3 commits July 17, 2025 14:53

minor internet search env vars

1c13975

clean up connector page and add new option for uncommon connectors

73258f2

vscode env template

10900c5

wenxi-onyx requested a review from a team as a code owner July 17, 2025 23:23

greptile-apps bot reviewed Jul 17, 2025

View reviewed changes

wenxi-onyx changed the title ~~Feat: Env var for community-supported connectors~~ feat: Env var for community-supported connectors Jul 17, 2025

cubic-dev-ai bot reviewed Jul 17, 2025

View reviewed changes

github-actions bot added the feat label Jul 17, 2025

vercel bot deployed to Preview July 17, 2025 23:27 View deployment

wenxi-onyx added 2 commits July 17, 2025 16:56

deployment fix and change default to false

8cde39a

greptile nit

9b92c2f

vercel bot deployed to Preview July 18, 2025 00:04 View deployment

GeorgeGithiri5 suggested changes Jul 26, 2025

View reviewed changes

raunakab and others added 16 commits August 11, 2025 16:12

fix: Move around group-sync tests (since they require docker services…

c1b706e

… to be running) (#5041) * Move around tests * Add missing fixtures + change directory structure up some more * Add env variables

remove chat session necessity from send message simple api (#5040)

8edcb69

Improve support for non-default postgres schemas (#5046)

28d5a63

fix: improve assistant fetching efficiency (#5047)

7dbe4ed

* Improve assistant fetching efficiency * More fix * Fix weird build stuff * Improve

feat: KG improvements (#5048)

d1d8626

* improvements * drop views if SQL fails * mypy fix

Remove empty tooltip (#5050)

44bee6f

Make from_.user optional (use "Unknown User") if not found (#5051)

83a3544

Tiny launch.json template improvement (#5055)

3612345

refactor: Update the error message that is logged when PR title fails…

b2bc730

… Conventional Commits regex (#5062)

fix: Make pr-labeler run on edits too

8e43ea4

fix: time discrepancy (#5056)

768e3e9

* fix time discrepancy * remove log * remove log

handle empty doc batches (#5058)

d245f53

wenxi-onyx and others added 28 commits August 11, 2025 16:13

fix unsupported character error in minio migration (#5145)

746e536

* fix unsupported character error in minio migration * slash fix

fix: preserve error traces (#5152)

5c0ddc4

feat: add customization via env vars for curator role (#5150)

1729928

* Add customization via env vars for curator role * Simplify * Simplify more * Address comments

feat(infra): Release Charts on Github Pages (#5155)

dd344c9

fix(infra): Adding helm directory (#5156)

c4d9009

* feat(infra): Adding helm directory * one more fix

no more scheduled stalling (#5154)

5172477

fix: slash command slackbot to respond in private msg (#5151)

1091b89

* fix slash command slackbot to respond in private msg * rename confusing variable. fix slash message response in DMs

feat(infra): Update Vespa Helm Chart Version

c4130c9

Revert that change. Let's do this properly

b0460f4

feat(infra): Bump Vespa Helm Version (#5161)

9a2a843

* feat(infra): Bump Vespa Helm Version * Adding the Chart.lock file

feat: salesforce custom query (#5158)

f0158ad

* WIP merged approach untested * tested custom configs * JT comments * fix unit test * CW comments * fix unit test

mask llm api key from logs

baaf765

fix: remove erroneous error case and add valid error (#5163)

c186ceb

* fix: remove erroneous error case and add valid error * also address docfetching-docprocessing limbo

feat: support gpt5 models (#5169)

2c923a7

* support gpt5 models * gpt5mini visible

Make starter messages visible on smaller screens (#5170)

14f1d85

feat: office parsing markitdown (#5115)

066172e

* switch to markitdown untested * passing tests * reset file * dotenv version * docs * add test file * add doc * fix integration test

refactor: file processing (#5136)

718c820

* file processing refactor * mypy * CW comments * address CW

fix: sanitize slack payload before logging (#5167)

dda8c8b

* sanitize slack payload before logging * nit

add gpt 5 display names (#5175)

d91fe11

fix(infra): Update github workflow to not tag latest (#5172)

66c02dd

* fix(infra): Update github workflow to not tag latest * Cleaned up the code a bit

fix(infra): Removing invalid helm version (#5176)

0f97f77

fix: restrict user file access to current user only (#5177)

ce840d8

* fix: restrict user file access to current user only * fix: enhance user file access control for recent folder

fix: max tokens param (#5174)

1ba4ce0

* max tokens param * fix unit test * fix unit test

clean up connector page and add new option for uncommon connectors

d3367b3

deployment fix and change default to false

15bca34

vercel bot had a problem deploying to Preview August 11, 2025 23:18 Failure

wenxi-onyx closed this Aug 11, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: Env var for community-supported connectors #5043

feat: Env var for community-supported connectors #5043

Uh oh!

wenxi-onyx commented Jul 17, 2025 •

edited

Loading

Uh oh!

vercel bot commented Jul 17, 2025 •

edited

Loading

Uh oh!

greptile-apps bot left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cubic-dev-ai bot left a comment

Uh oh!

GeorgeGithiri5 Jul 26, 2025

Uh oh!

Uh oh!


		# NOTE: Generative AI configurations are done via the UI now

		EXA_API_KEY=

feat: Env var for community-supported connectors #5043

feat: Env var for community-supported connectors #5043

Uh oh!

Conversation

wenxi-onyx commented Jul 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

How Has This Been Tested?

Backporting (check the box to trigger backport action)

Uh oh!

vercel bot commented Jul 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

greptile-apps bot left a comment

Choose a reason for hiding this comment

Greptile Summary

Confidence score: 4 /5

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cubic-dev-ai bot left a comment

Choose a reason for hiding this comment

Uh oh!

GeorgeGithiri5 Jul 26, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

wenxi-onyx commented Jul 17, 2025 •

edited

Loading

vercel bot commented Jul 17, 2025 •

edited

Loading