Non-CLI download pipeline

[orlando-code]

Brilliant package, thank you!

Wondering whether it's possible (before I sink too much more time into coding it) to use esgpull add/track/download pipeline via (e.g. Python) scripts?

I'd be keen on this functionality in order to be able to specify a catalogue of data I need (e.g. different ensembles, variables, timescales detailed in a json/yaml) for a particular project, call the script from within a larger pipeline and get all the relevant data downloaded into the right place. This would also for a more efficient way to e.g. reproject and subset data as it's downloaded, allowing the original (large) files to be deleted before they take up too much space.

I'd be grateful for any thoughts on this, thank you!

[AtefBN]

Hi @orlando-code @svenrdz has been working hard on this feature that should see the light soon. There will be event triggers (file downloaded, dataset downloaded etc.) that enable users to create custom scripts to pipeline into.

[orlando-code]

Evening! Thanks for your response.

I've thrown together some functionality which partially achieves this (it only handles the search/download process: not the sql file tracking) if it's useful. N.B. since it's incomplete and I wrote it mostly to allow subsetting/regridding (not particularly in the theme of the main esgpull) I wouldn't be able to contribute it to the wider repo!

Hope it's useful for at least one other person though...

[AtefBN]

FYI @orlando-code the PR for the pipeline feature is here albeit it's still not finalized #84

[svenrdz]

I've thrown together some functionality which partially achieves this (it only handles the search/download process: not the sql file tracking) if it's useful. N.B. since it's incomplete and I wrote it mostly to allow subsetting/regridding (not particularly in the theme of the main esgpull) I wouldn't be able to contribute it to the wider repo!

Your EsgpullAPI class is something I've been meaning to put together since I started working on esgpull, but it was never a priority. Nice work!

I don't think this new feature (plugins, event-based) will be useful to your particular use case at the moment, since you already have what's needed with esgpulllite (and you finished before I even published my PR).

[orlando-code]

Thanks for your kind words!

The method provides parallelised download of files using a .yaml setup file specifying model, institute, variables etc. via execution of a single python script.

At the moment, however, it downloading doesn't interact with the underlying database and otherwise doesn't handle the downloads in exactly the same way as the repo.

I'd love to feed back to the great esgpull project rather than creating yet another separate downloading method! With this in mind:

1. Is this scripted functionality something that would be interesting to the repo?
2. If so, how much work would need to be done to integrate my method with the rest of the repo?

Thanks in advance!