Skip to content

Add num_proc parameter to push_to_hub #7591

Open
@SwayStar123

Description

@SwayStar123

Feature request

A number of processes parameter to the dataset.push_to_hub method

Motivation

Shards are currently uploaded serially which makes it slow for many shards, uploading can be done in parallel and much faster

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions