Add IterableDataset.reshard()

[lhoestq]

To increase the number of shards of a dataset, you can use [IterableDataset.reshard]:

>>> dataset
IterableDataset({
    features: ['label', 'title', 'content'],
    num_shards: 4
})
>>> dataset.reshard()
IterableDataset({
    features: ['label', 'title', 'content'],
    num_shards: 3600
})

The resharding mechanism depends on the dataset file format.
For example for Parquet, it reshards using row groups instead of having one file per shard.

We can implement other formats later (e.g. JSON Lines, CSV can be split by recovering line boundaries from arbitrary locations)

Other details:

• fixed concatenate after shuffling, now it correctly shuffles the shards: close concatenate_datasets does not preserve shuffling state #7196
• fixed interleave after split_by_node: close Data duplication with split_dataset_by_node and interleaved_dataset #7868
• changed a bit how the effective seed works in multi node/worker/epoch situations

related to #7917

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.