Skip to content

Conversation

@dkalinowski
Copy link
Collaborator

When using older openvino-tokenizer during tokenizer export via convert_tokenizer, tokenizer produces different tokens than in the latest openvino-tokenizer package.

Example (2025.3):
29491, 781, 781, 5, 29560, 7567, 1629,

Now:
29491, 781, 781, 5, 1501, 7567, 1629,

which causes tokenized->detokenized text to miss the space

29560 was [, but 1501 is [

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants