num_proc parallelization works only for first ~10s.

[pshishodiaa]

Describe the bug

When I try to load an already downloaded dataset with num_proc=64, the speed is very high for the first 10-20 seconds acheiving 30-40K samples / s, and 100% utilization for all cores but it soon drops to <= 1000 with almost 0% utilization for most cores.

Steps to reproduce the bug

// download dataset with cli
!huggingface-cli download --repo-type dataset timm/imagenet-1k-wds --max-workers 32

from datasets import load_dataset
ds = load_dataset("timm/imagenet-1k-wds", num_proc=64)

Expected behavior

100% core utilization throughout.

Environment info

Azure A100-80GB, 16 cores VM

[lhoestq]

Hi, can you check if the processes are still alive ? It's a bit weird because datasets does check if processes crash and return an error in that case

[pshishodiaa]

Thank you for reverting quickly. I digged a bit, and realized my disk's IOPS is also limited - which is causing this. will check further and report if it's an issue of hf datasets' side or mine.