NVIDIA DALI pipelines to facilitate direct loading of data into GPU memory.

[negin513]

I have recently learned about NVIDIA DALI (Data Loading Library) that can be used for streaming data directly.

This can be potentially used for data streaming of our files directly to GPUs to avoid any CPU-GPU data transfer latency.

This discussion in Xarray Repository is relevant: pydata/xarray#9249

@weiji14 suggested that this can be integrated as a dedicated BackendEntrypoint as an alternative to the kvikio engine and I like this idea.

[weiji14]

A group of us are hacking on this together over at https://github.com/pangeo-data/ncar-hackathon-xarray-on-gpus. As mentioned at pangeo-data/ncar-hackathon-xarray-on-gpus#4 (comment), we know that DALI's ExternalSource operator works with zarr-python (no xarray) at least. Currently trying to see if this works with DALI's python_function operator too over at pangeo-data/ncar-hackathon-xarray-on-gpus#11. The developer/user experience isn't very ergonomic yet, but just mentioning that there are ideas floating around on what needs to be done upstream in various libraries to make this happen.