-
Notifications
You must be signed in to change notification settings - Fork 2.1k
feat: Knowledge graph full-stack implementation #4790
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
evan-onyx
approved these changes
Jun 16, 2025
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
one nit, otherwise looks good
task_prompt=KG_BETA_ASSISTANT_TASK_PROMPT, | ||
datetime_aware=False, | ||
include_citations=True, | ||
num_chunks=25, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This should be a constant
AnkitTukatek
pushed a commit
to TukaTek/onyx
that referenced
this pull request
Sep 23, 2025
* db setup * transfer 1 - incomplete * more adjustments * relationship table + query update * temp view creation * restructuring * nits * updates * separate read_only engine * extraction revamp * focus on metadata relatonships 1 * dev * migration downgrade fix * rebase migration change * a3+ * progress * base * new extraction * progress * fixed KG extraction * nits * updates * simplifications & cleanup * fixes * updates * more feature flag checks * fixes * extraction process fix * read-only user creation as part of setup * fix for missing entity attributes * kg read-only user creation as part of migration * typo * EL initial comments * initial Account/SF Connector chnges * SF Connector update - include account information * base w/ salesforce * evan updates + quite a bit more * kg-filtered search * EL changes pt 2 * migrations and env vars * quick migration fix * migration update * post_rebase fixes * mypy fixes * test fixes * test fix * test fix * read_only pool + misc * nf * env vars * test improvements * salesforce fix * test update * small changes * small adjustments * SF Connector fix & kg_stage removal for one table * mypy fix * small fixes * EL + RK (pt 1) comments * nit * setting updated * Salesforce test update * EL comments * read-only user replacement & cleanup * SQL View fix * converting entity type-name separators * sql view group ownership * view fix * SQL tweak * dealing with docs that were skipped by indexing * increased error handling * more error handling * Output formatting fix * kg-incremental-reindexing * 0-doc found improvement * celery * migration correction * timeout adjustments * nit * Updated migration * Entity Normalization for KG Dev 1 (onyx-dot-app#4746) * feat: trigrams column * fix: reranking and db * feat: v1 * fix: convert to orm * feat: parallel * fix: default to id_name * fix: renamed semantic_id and semantic_id_trigrams * fix: scalar subquery * fix: tuning + redundancy * fix: threshold * fix: typo * fix: shorten names * wip * fix: reverted * feat: config * feat: works but it was dumb * feat: clustering works * fix: mypy * normalization <-> language awareness for SQL generation * small type fixes --------- Co-authored-by: joachim-danswer <joachim@danswer.ai> * mypy * typo and dead code * kg_time_fencing * feat: remove temp views on migration downgrade * remove functions and triggers for now * rebase adjustments * EL code review results * quick fix + trigger/funcs for single tenant * fix: typo, mypy, dead code * fix: autoflake * small updatesd * nit * fix: typo * early + faster view creation * Extension creation in MT migration * nit changes to default ETs * Incremental Clustering and KG Refactor V1 (onyx-dot-app#4784) Optimized/restructured incremental clustering. New pipeline actually that moves vespa updates to clustering. Also, celery configuration has been updated. --------- Co-authored-by: joachim-danswer <joachim@danswer.ai> * Move file * Fix all prior imports * Clean sidebar items logic; add kg page * Add kg_processing celery background task * prompt tweak & ET extraction reset * more general hierarchical structure * feat: better vespa reset logic * Add basic knowledge graph configuration * Add configurations for KG entity-type * prompt optimization and entity replacemants * small prompt changes * Implement backend APIs * KG Refactor V2 (onyx-dot-app#4814) Clustering & Extraction improvements & various nits Co-authored-by: joachim-danswer <joachim@danswer.ai> * add connector-level coverage days * Update APIs to be more frontend ergonomic * Add simple test * Make config optional in test * fix: nit * initial EL responses * refactor: helper functions for formatting * fix: more helper fns & comments * fix: comment code that's been implemented elsewhere * Add entity-types APIs * Hook up frontend to backend * Finish hookup up entity-types to backend * Update ordering of entity-types and fix form submitting * Add backend API to get kg-exposed * Add kg-exposed to sidebar * Fix path * Use existing values, even if kg-enabled is false * Update what initial values are used * Add skeleton for kg resetting * Add return type * Add default entity-type population when fetching entity-types * Remove circular deps * Minor fixes to logic * Edit logic for default entity-types population * Add re-index API + skeleton * Update verbiage for KG * Remove templatization in favour of function * Address comments on PR * Pull call out into its own binding * Remove re-index API and revert implement of reset back to stub * Fix circular import error * Remove 'reindex' button * Edit how the empty vendor name list is handled * Edit how exposed is processed * Redirect if navigated to `/admin/kg` and kg is not exposed * Address comments on PR * reset + entity type table display & updating updates * Update fetching entity-types * Make KG entity types refresh when reset * Edit verbiage of reset button * Update package-lock.json file * Protect against overflowing * Re-implement refreshing table after reset * Edit message when nothing is shown. * UI enhancements * small fixes * remove form validation? * fix * nit * nit * nit * nit * fix configure max coverage days * EL comments for JR * refactor: moved functions where they belong to fix circular import * feat: intuitive coverage days * feat: intuitive coverage days * fix: safe date picker * fix: startdate * evan fixes * fix: evan comment on enable/disable * fix: style * fix: ui issues * fix: ui issues for reset too * fix: tests * fix: kg entity is not enabled * fix: entity type reload on enable --------- Co-authored-by: joachim-danswer <joachim@danswer.ai> Co-authored-by: Rei Meguro <36625832+Orbital-Web@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This PR contains the frontend + backend code for the knowledge graph.
Addresses: https://linear.app/danswer/issue/DAN-2017/knowledge-graph-full-stack-implementation.
How Has This Been Tested?
Frontend: Todo!
Backend: Tests have been added to
backend/tests/integration/tests/kg/test_kg.py
.Screenshots
This is the main configuration page (when KG is enabled):
This is the main KG configuration modal: