fix re-processing of previously seen docs Confluence

[evan-onyx]

Description

Fixes https://linear.app/danswer/issue/DAN-1830/confluence-duplicate-docs

The confluence API's granularity is only to the minute, so we were re-processing docs from the last checkpoint. This change skips docs seen in the previous checkpoint to avoid this issue.

How Has This Been Tested?

Tested in UI + unit tests

Backporting (check the box to trigger backport action)

Note: You have to check that the action passes, otherwise resolve the conflicts manually and tag the patches.

• This PR should be backported (make sure to check that the backport attempt succeeds)
• [Optional] Override Linear Check

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
internal-search	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Apr 16, 2025 10:20pm

[greptile-apps]

PR Summary

This PR addresses duplicate document processing in the Confluence connector by implementing checkpoint-based tracking of previously seen documents.

• Added last_seen_doc_ids to ConfluenceCheckpoint to track and skip previously processed documents, preventing duplicates from API time fuzziness
• Refactored load_everything_from_checkpoint_connector into two functions to support custom initial checkpoints in /backend/tests/unit/onyx/connectors/utils.py
• Modified _fetch_document_batches to check document IDs against previous checkpoint before processing in /backend/onyx/connectors/confluence/connector.py
• Added comprehensive test coverage in test_confluence_checkpointing.py to verify documents aren't reprocessed when using checkpoints

The changes follow best practices with clear boundaries, explicit typing, and fail-fast error handling while maintaining simplicity in the implementation.

_{3 file(s) reviewed, no comment(s)
Edit PR Review Bot Settings | Greptile}