Kg batch clustering

[Orbital-Web]

Description

• Clustering no longer loads every staging entity, relationship, etc. into memory and clusters them
• Clustering does work in batches for memory efficiency and crash robustness
• Fixed issue where if a staging entity's parent is in the normalized table but not the staging table, a parent-child relationship will not be created
• Made parent child relationship generation parallel

How Has This Been Tested?

Locally, on GitHub, Jira, Linear, and Salesforce. Both incrementally and in one go
Checked vespa to ensure the right information is there

Backporting (check the box to trigger backport action)

Note: You have to check that the action passes, otherwise resolve the conflicts manually and tag the patches.

• This PR should be backported (make sure to check that the backport attempt succeeds)
• [Optional] Override Linear Check

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
internal-search	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Jun 8, 2025 7:54pm

[greptile-apps]

PR Summary

Major refactor of Knowledge Graph clustering to improve memory efficiency and crash resilience through batch processing and parallelization.

• Introduces batch processing in backend/onyx/kg/clustering/clustering.py to prevent loading all entities into memory at once
• Added parent_key column in backend/onyx/db/models.py for explicit parent-child relationships tracking
• Parallelized parent-child relationship generation in backend/onyx/kg/clustering/clustering.py for better performance
• Changed relationship handling in backend/onyx/db/relationships.py to work directly with normalized tables instead of staging
• Fixed edge case in parent-child relationship creation when parent exists in normalized but not staging tables

_{5 file(s) reviewed, 2 comment(s)
Edit PR Review Bot Settings | Greptile}

[joachim-danswer]

Tested locally