restructure to signal activity while processing

[rkuo-danswer]

Description

Fixes https://linear.app/danswer/issue/DAN-1977/mitigate-connector-timeout-in-salesforce-connector-when-initial

How Has This Been Tested?

[Describe the tests you ran to verify your changes]

Backporting (check the box to trigger backport action)

Note: You have to check that the action passes, otherwise resolve the conflicts manually and tag the patches.

• This PR should be backported (make sure to check that the backport attempt succeeds)
• [Optional] Override Linear Check

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
internal-search	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	May 14, 2025 9:24pm

[greptile-apps]

PR Summary

This PR restructures the Salesforce connector to prevent timeouts during CSV processing and improves error message handling in the image summarization module. - Refactored _fetch_from_salesforce in backend/onyx/connectors/salesforce/connector.py to yield empty document lists during CSV processing, preventing connector timeouts - Added memory management with gc.collect() calls after batch processing in backend/onyx/connectors/salesforce/connector.py to address potential memory leaks - Implemented byte size tracking for document batches in backend/onyx/connectors/salesforce/connector.py to prevent memory issues - Added error message truncation to 1024 chars in backend/onyx/file_processing/image_summarization.py for better log management - Improved CSV file cleanup by removing processed files immediately in backend/onyx/connectors/salesforce/connector.py

_{2 file(s) reviewed, 3 comment(s)
Edit PR Review Bot Settings | Greptile}

[greptile-apps]

style: Consider using a constant for the max error message length (1024) to make it configurable and maintain consistency

[greptile-apps]

logic: no error handling around update_from_csv call - could silently fail and continue processing

Suggested change

	new_ids = sf_db.update_from_csv(
	object_type=object_type,
	csv_download_path=csv_path,
	)
	updated_ids.update(new_ids)
	try:
	new_ids = sf_db.update_from_csv(
	object_type=object_type,
	csv_download_path=csv_path,
	)
	updated_ids.update(new_ids)
	except Exception as e:
	logger.error(f"Failed to update from CSV {csv_path}: {e}")
	continue

[greptile-apps]

style: file removal should be in a try/finally block to ensure cleanup even if update_from_csv fails

Suggested change

	os.remove(csv_path)
	try:
	new_ids = sf_db.update_from_csv(
	object_type=object_type,
	csv_download_path=csv_path,
	)
	updated_ids.update(new_ids)
	finally:
	os.remove(csv_path)