[WIP] Faster downloads/uploads with Xet storage

[lhoestq]

Over the past few weeks, Hugging Face’s Xet Team took a major step forward by migrating the first Model and Dataset repositories off LFS and to Xet storage.

See more information on the HF blog: https://huggingface.co/blog/xet-on-the-hub

You can already enable Xet on Hugging Face account to benefit from faster downloads and uploads :)

We’re finalizing an official integration with the huggingface_hublibrary that will mean you get the benefits of Xet without any significant changes to your current workflow. In the meantime you might see this warning in push_to_hub():

Uploading files as bytes or binary IO objects is not supported by Xet Storage.

This means the huggingface_hub + Xet integration isn't enabled for datasets yet. Stay tuned !