Enable async and distributed processing for the ML backend #910
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…void shadowing the FlatBugDetector model
✅ Deploy Preview for antenna-preview canceled.
|
vanessavmac
changed the base branch from
main
to
706-support-for-reprocessing-detections-and-skipping-detector
Base automatically changed from
706-support-for-reprocessing-detections-and-skipping-detector
to
main
August 16, 2025 01:36
…nc-distributed-ml-backend
f-PLT
changed the title
…use serially scheduled tasks instead of periodic task
This was referenced Sep 4, 2025
…ter the job start time; refactoring
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
The current batch image processing system runs MLJobs as a single celery task. This causes issues when processing large numbers of image (i.e. 100+ images) since the long running task can be interrupted or lost.
This PR uses celery as the task queue and rabbitmq as the message broker to send batches of images as individual
process_pipeline_requestsinto queues dedicated to a specific ML pipeline. Processing services can pick up tasks based on the pipelines they host. A periodic celery beat task listens for completedprocess_pipeline_requestsand enqueuessave_resultstasks.The planning of this feature was discussed in #515. See the comments beginning at #515 (comment)
List of Changes
process_pipeline_requesttask which takes aPipelineRequestand returns the model's results. This is defined on the processing service.Jobmodel to includesubtasksandinprogress_subtasksto track the celery tasks queued (these can be eitherprocess_pipeline_requestorsave_resultstaskscheck_ml_job_statuswhich checks the subtasks of anMLJob, updates the job status, and schedulessave_resultstasksMLTaskRecordmodel which stores the results and stats of a celery taskRelated Issues
Addresses (in part?) #515
Detailed Description
Potential side effects or risks associated with the changes...
How to Test the Changes
Instructions on how to test the changes Include references to automated and/or manual tests that were created/used to
test the changes.
Screenshots
If applicable, add screenshots to help explain this PR (ex. Before and after for UI changes).
Deployment Notes
Include instructions if this PR requires specific steps for its deployment (database migrations, config changes, etc.)
Checklist