Skip to content

tokenizer is still being used in Trainer instead of processing_class #1711

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
arjunaskykok opened this issue Apr 23, 2025 · 2 comments
Closed

tokenizer is still being used in Trainer instead of processing_class #1711

arjunaskykok opened this issue Apr 23, 2025 · 2 comments

Comments

@arjunaskykok
Copy link

In the Fine-tuning a model with the Trainer API page, either in the documentation or the notebook, we are told to use the tokenizer parameter when initializing the Trainer class.

Image

But the tokenizer parameter has been deprecated. We should use the processing_class parameter.
@julien-c
Copy link
Member

Hi, can you open this in transformers instead?

@arjunaskykok
Copy link
Author

The issue has been moved to transformers:

huggingface/transformers#37734

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@julien-c @arjunaskykok