Use Oxbow to parse some formats?

[jeromekelleher]

Oxbow looks like a great way of getting an efficient tabular view on a range of bioinfo formats. This could provide a neat way of doing the painful bit (getting the data out of the bioinfo format into an array-oriented form) and bio2zarr could provide the functionality of writing the actual Zarr.

The distinction between the two approaches (as I see it) is that Oxbow is about providing a high performance and useful interface to existing files, while bio2zarr is about translating these files to Zarr (both of which are useful in different contexts, I'd imagine).

[benjeffery]

Nice find! Seems they recently went through a refactor for 0.4, so there is some active development.

[nvictus]

Yes, 0.4 was a major refactor to support better access patterns and implement comprehensive Arrow data models.

[tomwhite]

This sounds like a great fit. Along with zarrs, I can imagine that the long-term goal could be a fully-rustified bio2zarr.

[nvictus]

Happy to chime in if there are any questions. cc @conradbzura

The distinction between the two approaches (as I see it) is that Oxbow is about providing a high performance and useful interface to existing files

Correct.

Note that for VCF/BCF, we can project genotype data nested as either the conventional sample > FORMAT field or as FORMAT field > sample, the latter of which should align better with the VCF Zarr data model.

Btw, we have the ability to partition BGZF-compressed files similar to your vcfpartition tool. Currently this is only exposed by making genomic range queries with an index file, but the functionality to use arbitrary virtual position ranges is there and something we could prioritize.