`replace_vocabulary` crashes when `unk_token == pad_token` (CLIP case) — `ValueError: '<|endoftext|>' is not in list`

[rola93]

replace_vocabulary crashes when unk_token == pad_token (CLIP case) — ValueError: '<|endoftext|>' is not in list

Summary

When distilling a static embedding from a model whose tokenizer has the same string for both unk_token and pad_token (CLIP-style tokenizers frequently use "<|endoftext|>" for both), model2vec throws:

ValueError: '<|endoftext|>' is not in list

The exception occurs in tokenizer.py::_rename_added_token called from replace_vocabulary during distill_from_model(...) (and transitively via StaticEmbedding.from_distillation(...) in SentenceTransformers).

Why this happens (root cause)

Here replace_vocabulary(...) unconditionally calls:

added_tokens = _rename_added_token(unk_token, "[UNK]", added_tokens, pre_tokenized_tokens)
added_tokens = _rename_added_token(pad_token, "[PAD]", added_tokens, pre_tokenized_tokens)

_rename_added_token mutates the vocabulary list in-place:

vocabulary[idx] = new_form

If unk_token == pad_token (e.g., both are "<|endoftext|>"), the first call renames that entry to "[UNK]". The second call then tries to find the original form ("<|endoftext|>") in vocabulary, which now no longer exists, so vocabulary.index(form) raises ValueError.

Expected behavior

distill_from_model should handle the case unk_token == pad_token gracefully (e.g., skip the second rename, or make _rename_added_token idempotent) and proceed to build the static embedding.

###Actual behavior
It crashes with:

ValueError: '<|endoftext|>' is not in list

originating from model2vec/tokenizer/tokenizer.py::_rename_added_token.

Reproduce it!

This notebook reproduces the error in google colab with those dependencies:

Python: 3.12.11 (main, Jun  4 2025, 08:56:18) [GCC 11.4.0]
PyTorch: 2.8.0+cu126
Transformers: 4.56.1
Sentence-Transformers: 5.1.0
model2vec: 0.6.0

I tried a simple local patch by renaming the current replace_vocabulary function as follows:

def _replace_vocabulary(
    tokenizer: Tokenizer, new_vocabulary: list[Token], unk_token: str | None, pad_token: str | None
) -> Tokenizer:
  """
  Let this as is, just rename replace_vocabulary with _replace_vocabulary
  """
  pass


def replace_vocabulary(
    tokenizer: Tokenizer, new_vocabulary: list[Token], unk_token: str | None, pad_token: str | None
) -> Tokenizer:

  if unk_token is not None and unk_token == pad_token:
    # both token are the same! ignore pad token then
    return _replace_vocabulary(
    tokenizer=tokenizer, new_vocabulary=new_vocabulary, unk_token=unk_token, pad_token=None
  )
  else:
    # otherwise just call _replace_vocabulary as it was
    return _replace_vocabulary(
    tokenizer=tokenizer, new_vocabulary=new_vocabulary, unk_token=unk_token, pad_token=pad_token
    )

Then the error disappears, but I'm not sure this is the most effective way to prevent it (mainly because of potential consequences in the remainder of the execution and other use cases).

Since this is my first contribution to the repository, I’m not sure what the preferred way to address this issue is. I’m happy to implement the fix and open a PR, but I’d like to confirm the correct or recommended approach first.

[Pringled]

Hey @rola93, thanks for the detailed bug report! We only support text encoders, see #33, so this won't work even with the proposed fix (e.g. if I apply the proposed fix I get ValueError: You have to specify pixel_values). Did you encounter this issue with any text encoders? If so, I'd be happy to propose a fix, but otherwise I'm hestitant about adding this to the current tokenizer code.

[rola93]

Hello! I'm trying to tackle exactly this issue; I was able to workaround this error but yet all sentences collapse to same embedding, so, in practice it is not working yet. My best guess is that there is something wrong with the tokenizer and all tokens converge to "[UNK]", but not 100% sure.

I'll keep working on this today

[rola93]

This is what I mean: first, I load CLIP model to memory, then dump model and tokenizer to disk:

import os, json, torch
from huggingface_hub import snapshot_download
from transformers import CLIPTextModel, CLIPTokenizerFast
from sentence_transformers import SentenceTransformer, models
from sentence_transformers.models import StaticEmbedding

def _make_text_only_dir(src_dir: str, out_dir: str) -> str:
    os.makedirs(out_dir, exist_ok=True)
    # Evita re-crear si ya existe
    if os.path.exists(os.path.join(out_dir, "config.json")) and \
       os.path.exists(os.path.join(out_dir, "tokenizer.json")):
        return out_dir

    # Guarda solo torre de texto
    txt = CLIPTextModel.from_pretrained(src_dir, local_files_only=True)
    txt.save_pretrained(out_dir)

    tok = CLIPTokenizerFast.from_pretrained(src_dir, local_files_only=True, use_fast=True)
    tok.save_pretrained(out_dir)
    
    return out_dir


local_path = snapshot_download(
            repo_id="laion/CLIP-ViT-B-32-laion2B-s34B-b79K",
            local_files_only=False,
            ignore_patterns=["*.msgpack"],
        )

text_only_dir = _make_text_only_dir(local_path, './text_only_dir')

And now, I load it using StaticEmbedding:

se = StaticEmbedding.from_distillation(
    text_only_dir,
    device="cuda",
    pca_dims="auto", 
    apply_zipf=True
)

The output shows that all 50k tokens are encoded, as you can see:

The `use_subword` parameter is deprecated and will be removed in the next release. It doesn't do anything.
The `apply_zipf` parameter is deprecated and will be removed in the next release. Zipf weighting is applied based on the sif_coefficient parameter. If this is set to None, no weighting is applied.

Then wrap it to use as SentenceTransformer:

from sentence_transformers import SentenceTransformer
model = SentenceTransformer(modules=[se])

Up to this point everything looks ok; however, when using it to encode any string, it returns same embedding:

import numpy as np

np.isclose(model.encode(["Hello world"]), model.encode(["show me the bug!"])).all()

Output:

np.True_

[Pringled]

Thanks for digging further, but just to be clear: CLIP isn’t supported by Model2Vec (see #33), only text encoders are supported. So even if you fix this part about the embeddings collapsing, it still won't work as a static embedding model unfortunately. I think this is out of scope for the library, but if you do have an example of a text encoder where the UNK and PAD token are the same, let me know, and I can look into how to make it work.

[rola93]

I totally understand that Model2Vec only supports text-encoders, I'm ok with that.

However, I don't understand why this is out of scope: I mean, what I'm actually building is a CLIPTextModel which acccepts only text, and it can even be loaded as a SentenceTransformer, like this:

import torch
import torch.nn.functional as F
from typing import List, Dict, Union
from transformers import CLIPTextModelWithProjection, CLIPTokenizerFast
from sentence_transformers import SentenceTransformer
from sentence_transformers.models import Module

class CLIPTextEncoderST(Module):
    def __init__(self, model_name_or_path: str, max_seq_length: int | None = None, device: str | None = None):
        super().__init__()
        self.tokenizer = CLIPTokenizerFast.from_pretrained(model_name_or_path)
        self.model = CLIPTextModelWithProjection.from_pretrained(model_name_or_path)
        self.max_seq_length = max_seq_length or min(77, self.tokenizer.model_max_length)  # CLIP usa 77 tokens por defecto
        if device is not None:
            self.model.to(device)

    def get_sentence_embedding_dimension(self) -> int:
        # proyección de texto (espacio CLIP)
        return int(self.model.config.projection_dim)

    def tokenize(self, texts: List[str]) -> Dict[str, torch.Tensor]:
        return self.tokenizer(
            texts,
            padding=True,
            truncation=True,
            max_length=self.max_seq_length,
            return_tensors="pt",
        )

    def forward(self, features: Dict[str, torch.Tensor]) -> Dict[str, torch.Tensor]:
        # Espera input_ids / attention_mask (SentenceTransformer llama a tokenize() del primer módulo)
        outputs = self.model(input_ids=features["input_ids"], attention_mask=features.get("attention_mask"))
        # outputs.text_embeds ya incluye la proyección al espacio CLIP
        sent_emb = outputs.text_embeds
        # Normalizar como hace CLIP (útil para similitud coseno)
        features["sentence_embedding"] = F.normalize(sent_emb, p=2, dim=-1)
        return features

    def save(self, output_path: str): 
        self.tokenizer.save_pretrained(output_path)
        self.model.save_pretrained(output_path)
        with open(f"{output_path}/clip_text_encoder_config.json", "w") as f:
            f.write('{"type": "CLIPTextEncoderST"}')  # marca simple del tipo de módulo

repo_id = "laion/CLIP-ViT-B-32-laion2B-s34B-b79K"
clip_text_module = CLIPTextEncoderST(repo_id)
model = SentenceTransformer(modules=[clip_text_module])

print(model.encode(["Hello world!"]).shape)  # (1, 512) 

So, model from the example is a SentenceTransformer which can not be distilled: tokens somehow collapse (even after fixing the bug I reported here)

EDIT: Had to add save method since otherwise it does not work.

[Pringled]

It's out of scope because we only support sentence transformer-like models. CLIP style models have their own module in sentence transformers (see https://github.com/UKPLab/sentence-transformers/blob/master/sentence_transformers/models/CLIPModel.py). This is not trivial to support since the forward pass (that we have to call to compute static embeddings) calls both the text and image encoding functions in transformers, see https://github.com/huggingface/transformers/blob/main/src/transformers/models/clip/modeling_clip.py#L939. At the moment there's no plans to support this, but if you can provide some more information about your usecase I'd be happy to discuss whether it makes sense adding support for just the text part of CLIP style models. From my perspective, you break the text/image alignment if you were to only create static embeddings for the text part, and in general, CLIP style models are poor as standalone text encoders.

Another issue is that CLIP style models perform pooling over the EOS token (that might be why you're seeing weird results). Model2vec does mean pooling by default.

[rola93]

I totally agree with respect to CLIP Model, but I strongly disagree with respect to Clip text tower only.

First, I still don't see why this Sentence Transformer instance model that I shared (model = SentenceTransformer(modules=[clip_text_module])) is different from the usually supported Sentence Transformer instances.

Then, with respect to this:

the forward pass (that we have to call to compute static embeddings) calls both the text and image encoding functions in transformers

I It is true for the CLIPModel, and I agree with this. However, I'm using only the text tower of clip alone, CLIPTextModelWithProjection. In fact, you can use the model instance I shared to compute sentence embeddings for text only, and it works. What is more, it performs pretty well ( this is why I'm here at the end of the day: I benchmarked several models, similar size, and this turn out to be the best in my task, which is just classification and re-ranking). However, when I try to distill it with model2vec it fails and tokens collapse.

From my perspective, you break the text/image alignment if you were to only create static embeddings for the text part

I don't agree here. In my particular case, I don't even care about images, because I don't have them, but, more generally, a typical usage of CLIP is to compute image embeddings, store them in a vector database, and then query it with jut its tex embeddings. In this case, both images and texts embeddings are computed independently.

Yesterday I debugged it and found that tokenizer was changed while distilled, which may explain the source of the problem, today I'll keep digging a little more on this

Another issue is that CLIP style models perform pooling over the EOS token (that might be why you're seeing weird results). Model2vec does mean pooling by default.

I didn't consider this hypothesis yet, it looks reasonable for me, I'll also explore it.

Thanks for thoughts on this and your time, I really appreciate it.

[rola93]

Let me share some insigths.

The clip_tokenizer that is loaded with my _make_text_only_dir function, is this:

CLIPTokenizerFast(name_or_path='/home/melibuntu/.cache/huggingface/hub/models--laion--CLIP-ViT-B-32-laion2B-s34B-b79K/snapshots/1a25a446712ba5ee05982a381eed697ef9b435cf', vocab_size=49408, model_max_length=77, is_fast=True, padding_side='right', truncation_side='right', special_tokens={'bos_token': '<|startoftext|>', 'eos_token': '<|endoftext|>', 'unk_token': '<|endoftext|>', 'pad_token': '<|endoftext|>'}, clean_up_tokenization_spaces=False, added_tokens_decoder={
	49406: AddedToken("<|startoftext|>", rstrip=False, lstrip=False, single_word=False, normalized=True, special=True),
	49407: AddedToken("<|endoftext|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
}
)

and I can use it to tokenize:

clip_tokenizer.tokenize("Hello world")

output:

['hello</w>', 'world</w>']

However, the tokenizer loaded with StaticEmbedding.from_distillation is different:

from sentence_transformers.models import StaticEmbedding
from sentence_transformers import SentenceTransformer

se_fixed = StaticEmbedding.from_distillation(
    model_name=text_only_dir,
    device="cuda",
    pca_dims="auto",
    apply_zipf=True
)

model_fixed = SentenceTransformer(modules=[se_fixed])

model_fixed.tokenizer.encode("Hello world").tokens

Output:

The `use_subword` parameter is deprecated and will be removed in the next release. It doesn't do anything.
The `apply_zipf` parameter is deprecated and will be removed in the next release. Zipf weighting is applied based on the sif_coefficient parameter. If this is set to None, no weighting is applied.
Encoding tokens: 100%|██████████| 49407[/49407](https://otro_asymmetric-embeddings.labs-clients.furyapps.io/49407) [00:03<00:00, 12367.34 tokens[/s](https://otro_asymmetric-embeddings.labs-clients.furyapps.io/s)]
['ĠhelloĠworld']

So: tokenization is actually different. I didn't expected that, since StaticEmbedding.from_distillation docs says:

If you don't pass a vocabulary, we use the model's tokenizer directly.

I'm not providing a vocabulary, but my tokenizer is being modified. The tokenizer should have been built here, and used to call distill_from_model:

tokenizer = cast(
        PreTrainedTokenizerFast,
        AutoTokenizer.from_pretrained(model_name, trust_remote_code=trust_remote_code, use_fast=True),
    )

    return distill_from_model(
        model=model,
        tokenizer=tokenizer,
        vocabulary=vocabulary,
        device=device,
        pca_dims=pca_dims,
        apply_zipf=apply_zipf,
        token_remove_pattern=token_remove_pattern,
        sif_coefficient=sif_coefficient,
        quantize_to=quantize_to,
        use_subword=use_subword,
        vocabulary_quantization=vocabulary_quantization,
    )

If I copy-and-paste it on my notebook, I can run and load it properly:

from transformers.tokenization_utils_fast import PreTrainedTokenizerFast
from transformers import AutoModel, AutoTokenizer
from typing import cast


tokenizer = cast(
        PreTrainedTokenizerFast,
        AutoTokenizer.from_pretrained(text_only_dir, trust_remote_code=False, use_fast=True),
    )
tokenizer.tokenize("Hello world")

output:

['hello</w>', 'world</w>']

So, this is consistent with the clip tokenizer I loaded my self using _make_text_only_dir, and is the same type of object: transformers.models.clip.tokenization_clip_fast.CLIPTokenizerFast.

In summary: distill_from_model is being called with model and tokenizer as expected.

[rola93]

I keept executing and following the code with all the logic with respect to tokenizers but I get lost, so I decided to change my strategy and just go for something easier but usefull: instead of finding the cause, show clearly the syntom.

So, the problem is a missmatch in the tokenizar, it builds tokens that, for some reason, are out of vocabulary, see this (this assumes that the patch proposed in this issue was applied, I did it on my local copy):

import os, json, torch
from huggingface_hub import snapshot_download
from transformers import CLIPTextModel, CLIPTokenizerFast
from sentence_transformers import SentenceTransformer, models
from sentence_transformers.models import StaticEmbedding

def _make_text_only_dir(src_dir: str, out_dir: str) -> str:
    os.makedirs(out_dir, exist_ok=True)
   
    txt = CLIPTextModel.from_pretrained(src_dir, local_files_only=True)
    
    txt.save_pretrained(out_dir)

    tokenizer = CLIPTokenizerFast.from_pretrained(src_dir, local_files_only=True, use_fast=True)
    
    tokenizer.save_pretrained(out_dir)
    
    return out_dir, tokenizer

local_path = snapshot_download(
            repo_id="laion/CLIP-ViT-B-32-laion2B-s34B-b79K",
            local_files_only=False,
            ignore_patterns=["*.msgpack"],
        )

text_only_dir, clip_tokenizer = _make_text_only_dir(local_path, './text_only_model_dir')

# Distill CLIP model with vocabulary=None
static_model =  StaticEmbedding.from_distillation(
        text_only_dir,
        device="cuda",
        pca_dims="auto", 
        apply_zipf=True
    )

# Test tokenization mismatch with multi-word input
vocab = static_model.tokenizer.get_vocab()

print("=== Demonstrating tokenization mismatch ===")

for text in ["hello world", "this is a bug"]:
    encoding = static_model.tokenizer.encode(text)
    
    print(f"Text: '{text}'")
    print(f"  Tokenizer produces: {encoding.tokens}")
    
    # Check if produced tokens exist in vocabulary
    for token in encoding.tokens:
        exists = token in vocab
        print(f"  Token '{token}' exists in vocab: {exists}")
        
        if not exists:
            # Show that equivalent SentencePiece tokens DO exist
            sentencepiece_token = token.replace('Ġ', '▁')
            sp_exists = sentencepiece_token in vocab
            print(f"    But SentencePiece equivalent '{sentencepiece_token}' exists: {sp_exists}")
            
            # If it's a multi-word token, show individual words exist
            if '▁' in sentencepiece_token:
                words = sentencepiece_token.split('▁')[1:]  # Remove first empty element
                print(f"    Individual words in vocab:")
                for word in words:
                    individual_token = f"▁{word}"
                    individual_exists = individual_token in vocab
                    print(f"      '{individual_token}': {individual_exists}")
    print()

Output:

=== Demonstrating tokenization mismatch ===
Text: 'hello world'
  Tokenizer produces: ['ĠhelloĠworld']
  Token 'ĠhelloĠworld' exists in vocab: False
    But SentencePiece equivalent '▁hello▁world' exists: False
    Individual words in vocab:
      '▁hello': True
      '▁world': True

Text: 'this is a bug'
  Tokenizer produces: ['ĠthisĠisĠaĠbug']
  Token 'ĠthisĠisĠaĠbug' exists in vocab: False
    But SentencePiece equivalent '▁this▁is▁a▁bug' exists: False
    Individual words in vocab:
      '▁this': True
      '▁is': True
      '▁a': True
      '▁bug': True

So, my best guess is that Pretokenizer produces ByteLevel tokens (Ġ) that don't exist in SentencePiece vocabulary (▁)

(At this point, I think may be we should move this issue to a newer)

[Pringled]

We currently support Wordpiece, BPE and Unigram tokenizers with Model2Vec. However, the tokenizer you're using is a CLIPTokenizer (https://huggingface.co/laion/CLIP-ViT-B-32-laion2B-s34B-b79K/blob/main/tokenizer_config.json#L33), which we don't officially support. I'm not familar enough with CLIPTokenizers to know specifically what the issue is unfortunately. I could add a more informative error to explicitly check for this type of tokenizer and throw an error.

[rola93]

This explains why I could not reproduce the bug for other models 😓 Just for the record, this was the snippet I used (It may be usefull as a sanity check if some fix is eventually implemented):

from sentence_transformers.models import StaticEmbedding
from transformers import AutoTokenizer
import numpy as np

# Use a pure text-only sentence transformer based on RoBERTa (uses ByteLevel tokenizer)

def test_model(model_name):
    print(f"=== Testing with pure sentence transformer: {model_name} ===")
    
    # Check original tokenizer type
    original_tokenizer = AutoTokenizer.from_pretrained(model_name)
    print(f"Original tokenizer: {type(original_tokenizer)}")
    print(f"Original pretokenizer: {original_tokenizer.backend_tokenizer.pre_tokenizer}")
    
    # Test original tokenization
    test_text = "hello world"
    original_encoding = original_tokenizer.encode(test_text, add_special_tokens=False)
    original_tokens = original_tokenizer.convert_ids_to_tokens(original_encoding)
    print(f"Original tokenization of '{test_text}': {original_tokens}")
    
    # Distill the model (this will cause the bug)
    static_model = StaticEmbedding.from_distillation(
        model_name,
        vocabulary=None,  # Should preserve original tokenizer
        device="cuda",
        pca_dims=None,
        sif_coefficient=None
    )
    
    # Demonstrate the mismatch
    vocab = static_model.tokenizer.get_vocab()
    print(f"\n=== Tokenization Mismatch Demonstration ===")
    
    for text in ["hello world", "this is a test"]:
        encoding = static_model.tokenizer.encode(text)
        
        print(f"Text: '{text}'")
        print(f"  Distilled tokenizer produces: {encoding.tokens}")
        
        for token in encoding.tokens:
            exists = token in vocab
            print(f"  Token '{token}' exists in vocab: {exists}")
            
            if not exists and token.startswith('Ġ'):
                # Show SentencePiece equivalent exists
                sp_token = token.replace('Ġ', '▁')
                sp_exists = sp_token in vocab
                print(f"    SentencePiece equivalent '{sp_token}' exists: {sp_exists}")
    
    # Confirm identical embeddings due to UNK mapping
    from sentence_transformers import SentenceTransformer
    model_wrapper = SentenceTransformer(modules=[static_model])
    
    emb1 = model_wrapper.encode("hello world")
    emb2 = model_wrapper.encode("completely different text")
    print(f"\nAll embeddings identical (UNK issue): {np.allclose(emb1, emb2)}")

it worked ok for all of those:

models_to_test = [
    "sentence-transformers/LaBSE",
    "microsoft/DialoGPT-medium", 
    "sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2",
    "microsoft/BiomedNLP-PubMedBERT-base-uncased-abstract",
    "sentence-transformers/distiluse-base-multilingual-cased",
    "openai-gpt",
    "gpt2",
    "facebook/bart-base",
    "microsoft/DialoGPT-small", 
    "distilgpt2",
    "sentence-transformers/all-roberta-large-v1"
]

If you have any additional clue on how to fix it, I'm glad to keep digging in or trying alternative approaches

[Pringled]

Hi @rola93, I've just added support for configurable pooling in the latest Model2Vec release (see https://github.com/MinishLab/model2vec/releases/tag/v0.7.0, specifically #288). So you could try using "last" token pooling (EOS pooling), which I think will greatly benefit performance in this case.