Enable pre-download subsetting of files

[orlando-code]

Hello all!

Thanks for a great package.

Would it be possible to facilitate subsetting (e.g. by coordinate range, levels etc.) of files before downloading? This would be useful both to decreasing downloading times, save space at the download destination, and speed up subsequent processing.

I can't imagine this would be a quick implementation (due to having to modify post-download file metadata to reflect the subsetting and all the downstream things this would effect) but wondered whether it might be within scope of the package at some point. Or whether I've missed something obvious!

Many thanks for your thoughts!

[AtefBN]

Hi @orlando-code this is indeed a very interesting feature that could make the lives of many users easier.
Unfortunately, this is not part of the roadmap for esgpull as they are other more pressing feature requests to respond to.

This could change in mid to long term if the ESGF catalogue supports kerchunk or virtual zarr across the federation of course but not something we can promise ourselves.

[orlando-code]

That's fair enough – thank you for your response!

My hope would be that given that some of the download options e.g. OPeNDAP, esgf pyclient appear to support subsetting, that this functionality might not be too much of a stretch! These two options have significant limitations: I believe that the first requires manual (via the metagrid) file exploration, while downloading by https via xarray (which could include subsetting) is throttled by download packet size.

I do understand that it's not currently the way that esgpull interacts with the ESGF catalogue. It's a shame, given that I'm not aware of any other package which is half as good at interacting with the ESGF catalogue.

Looking forward to seeing and supporting further development!

[orlando-code]

Just an advance on this for anyone who might find it useful:

I've thrown together some functionality which subsets and regrids (via xesmf) data from the Metagrid nodes. It does this at the cost of the sql file tracking.

N.B. since it's incomplete and I wrote it mostly to allow subsetting/regridding (not particularly in the theme of the main esgpull) I wouldn't be able to contribute it to the wider repo!

Hope it's useful for at least one other person though...