Skip to content

tokenizer is still being used in Trainer instead of processing_class #37734

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
arjunaskykok opened this issue Apr 24, 2025 · 3 comments · May be fixed by huggingface/notebooks#574
Open

tokenizer is still being used in Trainer instead of processing_class #37734

arjunaskykok opened this issue Apr 24, 2025 · 3 comments · May be fixed by huggingface/notebooks#574

Comments

@arjunaskykok
Copy link

In the Fine-tuning a model with the Trainer API page, either in the documentation or the notebook, we are told to use the tokenizer parameter when initializing the Trainer class.

Image

But the tokenizer parameter has been deprecated. We should use the processing_class parameter.

@stevhliu
@arjunaskykok arjunaskykok mentioned this issue Apr 24, 2025
Closed
@stevhliu
Copy link
Member

Good catch, would you like to open a PR to update the docs and course to use the processing_class parameter instead?

@arjunaskykok
Copy link
Author

Okay, I'll do it.

@arjunaskykok arjunaskykok linked a pull request Apr 25, 2025 that will close this issue
Open
@arjunaskykok
Copy link
Author

PRs:

huggingface/course#895
huggingface/notebooks#574

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

replace deprecated tokenizer with processing_class in chapter 3 arjunaskykok/notebooks
2 participants
@arjunaskykok @stevhliu