Skip to content

Update data syncing to delete missing images #456

@mihow

Description

@mihow

Update the data sync method to delete source images in the DB if they are missing from the source. What are some ways we should handle this? Warnings to the user? Check if source images have already been processes and have detections, verifications. Do we soft delete or loose it for real?

Consider switching to s5cmd instead of boto3 to increase speed of sync? - OR first try to tune concurrency settings: https://boto3.amazonaws.com/v1/documentation/api/latest/guide/s3.html

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions