fix: set clean_up_tokenization_spaces=False in Llama 3 tokenizer conversion

[maxsloef-goodfire]

What does this PR do?

The Llama3Converter in convert_llama_weights_to_hf.py hardcodes clean_up_tokenization_spaces=True (line 468). This causes tokenizer.decode() to silently strip spaces before punctuation for all converted Llama 3 models, producing incorrect decoded text.

clean_up_tokenization_spaces applies BERT-era string replacements ( . → ., ? → ?, etc.) that are destructive for Llama 3's BPE tokenizer. Llama 2's LlamaTokenizer explicitly set this to False, and Llama 4 ships with False. The True was introduced in #30334 by inheriting the library default, then explicitly hardcoded in #33778 for backward compatibility.

Minimal reproduction

from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-3.1-8B-Instruct")
text = "x != y and a.b == c"
ids = tokenizer.encode(text, add_special_tokens=False)
print(repr(tokenizer.decode(ids)))
# 'x!= y and a.b == c'  ← space before != silently dropped
print(repr(tokenizer.decode(ids, clean_up_tokenization_spaces=False)))
# 'x != y and a.b == c'  ← correct

Tested across every version of transformers from 4.40.0 through 5.3.0 — all produce incorrect decoded text.

Companion fix PRs have been opened on all 24 affected meta-llama model repos on the Hub: Llama-3.1-8B-Instruct discussion #356.

Fixes #35175
Fixes #31187

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline, Pull Request section?
• Was this discussed/approved via a Github issue or the forum? Please add a link to it if that's the case. — Detokenization discrepancy with Llama3.1 #35175, Original Llama-3 tokenizer behaves differently from transformers version #31187, Llama3 Tokenizer Decode Removing Space Character #32575
• Did you make sure to update the documentation with your changes?
• Did you write any new necessary tests?

Who can review?

@ArthurZucker @itazap — tokenizer maintainers. Arthur previously acknowledged this should be False in #35175.

[github-actions]

[For maintainers] Suggested jobs to run (before merge)

run-slow: llama

[ArthurZucker]

LGTM for the conversion, these are removed from releases so it won't change much

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.