Skip to content
This repository was archived by the owner on Sep 30, 2024. It is now read-only.
This repository was archived by the owner on Sep 30, 2024. It is now read-only.

Partial data ingest #354

@esheehan-gsl

Description

@esheehan-gsl

When we have an error in the save step of our ingest pipeline—as we did with #353—we can end up with partially ingested data. For example, the Parquet write happens after the Zarr is written. If an error happens there, we end up with the data written to a new Zarr group, but not available in the database (or in the Parquet files, of course). So the data is not presented in the application.

We should set up our error handling so that if any part of the ingest pipeline fails, we clear out all of the data and log a useful message.

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions