fix: don't skip ccpairs if embedding swap in progress

[wenxi-onyx]

Description

title

How Has This Been Tested?

locally

Backporting (check the box to trigger backport action)

Note: You have to check that the action passes, otherwise resolve the conflicts manually and tag the patches.

• This PR should be backported (make sure to check that the backport attempt succeeds)
• [Optional] Override Linear Check

---

Summary by cubic

Fixed a bug where connector credential pairs were skipped during indexing if an embedding swap was in progress. Now, all relevant pairs are collected and re-indexed when a swap is detected.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Preview	Comments	Updated (UTC)
internal-search	Ready	Preview	Comment	Aug 29, 2025 11:03pm

[greptile-apps]

Greptile Summary

This PR fixes a critical bug in the embedding model swap logic within the document processing pipeline. The issue was that during embedding model transitions, the system was incorrectly applying normal indexing limits (such as the USER_FILE_INDEXING_LIMIT of 100 user files) even when all connector-credential pairs needed to be re-indexed with the new embedding model.

The fix introduces logic to detect when an embedding swap is in progress by checking if any search settings have a FUTURE status. When this condition is detected, the system bypasses normal filtering and collection limits to ensure all indexable connector-credential pairs are gathered for re-indexing. This prevents the problematic scenario where some documents would retain old embeddings while others use new ones, which would lead to inconsistent search results.

The changes span two files:

1. backend/onyx/db/enums.py - Adds an is_future() helper method to the IndexModelStatus enum, following the same pattern as the existing is_current() method
2. backend/onyx/background/celery/tasks/docprocessing/tasks.py - Implements the core detection logic that checks for embedding swaps in progress and conditionally fetches all CC pairs instead of applying normal limits

The solution integrates well with the existing codebase architecture, particularly the search settings management system and the connector-credential pair indexing pipeline. The IndexModelStatus enum already had infrastructure for tracking different states (PAST, PRESENT, FUTURE), and this fix leverages that existing pattern to properly handle the transition states during embedding model swaps.

Confidence score: 4/5

• This PR addresses a clear functional bug with a targeted fix that should resolve embedding swap issues
• The logic is straightforward and well-contained, reducing risk of introducing new problems
• Pay close attention to the docprocessing tasks file to ensure the embedding swap detection logic works correctly in all scenarios

_{2 files reviewed, no comments}

_{Edit Code Review Bot Settings | Greptile}

[cubic-dev-ai]

cubic analysis

No issues found across 2 files. Review in cubic

[Weves]

For each search setting:

1. Fetch "all" relevant cc pairs (only fetch some max # of user files)
2. Apply should_index to filter out irrelevant ones
3. Package them as a tuple (search_settings, all_cc_pairs_to_index)

Next step just takes this in and executes the indexing, doesn't do any filtering

[Weves]

@cubic-dev-ai please review this

[cubic-dev-ai]

@cubic-dev-ai please review this

@Weves I've started the AI code review. It'll take a few minutes to complete.

[cubic-dev-ai]

@cubic-dev-ai please review this

@Weves I've started the AI code review. It'll take a few minutes to complete.

[cubic-dev-ai]

@cubic-dev-ai please review this

@Weves I've started the AI code review. It'll take a few minutes to complete.

[cubic-dev-ai]

No issues found across 4 files