ChronosTokenizer Wrapper for Hugging Face T5 Integration?

[kushalkodn-db]

Hi! I am trying to integrate ChronosTokenizer into Hugging Face T5 tokenizer code, e.g. something like this:

from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("google-t5/t5-small")

1. Do you happen to have code to convert a ChronosTokenizer object into a transformers.T5TokenizerFast tokenizer (PreTrainedTokenizerFast) using a wrapper class for HF integration?
2. Do you know if it is possible to match the interface of the tokenizer returned by AutoTokenizer.from_pretrained with the MeanScaleUniformBins(ChronosTokenizer) tokenizer?

I am basically trying to maintain all the same tokenizer functionalities in the chronos-forecasting codebase, while also creating a new chronos tokenizer that inherits from the transformers.PreTrainedTokenizerFast class.

When I try to do the same thing with the chronos-t5 models:

tokenizer = AutoTokenizer.from_pretrained("amazon/chronos-t5-small")

I get this error:

OSError                                   Traceback (most recent call last)
Cell In[13], [line 1]
----> [1] tokenizer = AutoTokenizer.from_pretrained("amazon/chronos-t5-small")

File /mnt/workdisk/kushal/llm-foundry/llmfoundry-venv/lib/python3.11/site-packages/transformers/models/auto/tokenization_auto.py:880, 
in AutoTokenizer.from_pretrained(cls, pretrained_model_name_or_path, *inputs, **kwargs)
    [878] tokenizer_class_py, tokenizer_class_fast = TOKENIZER_MAPPING[type(config)]
    [879] if tokenizer_class_fast and (use_fast or tokenizer_class_py is None):
--> [880]     return tokenizer_class_fast.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs)
    [881] else:
    [882]     if tokenizer_class_py is not None:

File /mnt/workdisk/kushal/llm-foundry/llmfoundry-venv/lib/python3.11/site-packages/transformers/tokenization_utils_base.py:2073, 
in PreTrainedTokenizerBase.from_pretrained(cls, pretrained_model_name_or_path, cache_dir, force_download, local_files_only, 
token, revision, trust_remote_code, *init_inputs, **kwargs)
   [2067]     logger.info(
   [2068]         f"Can't load following files from cache: {unresolved_files} and cannot check if these "
   [2069]         "files are necessary for the tokenizer to operate."
   [2070]     )
   [2072] if all(full_file_name is None for full_file_name in resolved_vocab_files.values()):
-> [2073]     raise EnvironmentError(
   [2074]         f"Can't load tokenizer for '{pretrained_model_name_or_path}'. If you were trying to load it from "
   [2075]         "'https://huggingface.co/models', make sure you don't have a local directory with the same name. "
   [2076]         f"Otherwise, make sure '{pretrained_model_name_or_path}' is the correct path to a directory "
   [2077]         f"containing all relevant files for a {cls.__name__} tokenizer."
   [2078]     )
   [2080] for file_id, file_path in vocab_files.items():
   [2081]     if file_id not in resolved_vocab_files:

OSError: Can't load tokenizer for 'amazon/chronos-t5-small'. If you were trying to load it from 'https://huggingface.co/models', 
make sure you don't have a local directory with the same name. Otherwise, make sure 'amazon/chronos-t5-small' is the correct path 
to a directory containing all relevant files for a T5TokenizerFast tokenizer.

I hope this request makes sense. Please let me know if I can clarify anything. Thanks!

[abdulfatir]

ChronosTokenizer has no relation to any language model tokenizer such as T5Tokenizer. You could potentially create one that follows the same API as HF tokenizers but we haven't tried it.