enrich-formula Exception: data did not match any variant of untagged enum ModelWrapper

[wenboown]

Bug

https://www.mdpi.com/1099-4300/21/1/3

entropy-1.pdf

I try to convert the pdf of this paper into md. But docling output <!-- formula-not-decoded --> in normal mode. When I added the --enrich-formula flag, it gives the following error:

docling --enrich-formula entropy-1.pdf

Traceback (most recent call last):
File "/Users/bwen/.pyenv/versions/3.11.1/bin/docling", line 8, in
sys.exit(app())
^^^^^
File "/Users/bwen/.pyenv/versions/3.11.1/lib/python3.11/site-packages/typer/main.py", line 338, in call
raise e
File "/Users/bwen/.pyenv/versions/3.11.1/lib/python3.11/site-packages/typer/main.py", line 321, in call
return get_command(self)(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/bwen/.pyenv/versions/3.11.1/lib/python3.11/site-packages/click/core.py", line 1157, in call
return self.main(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/bwen/.pyenv/versions/3.11.1/lib/python3.11/site-packages/typer/core.py", line 665, in main
return _main(
^^^^^^
File "/Users/bwen/.pyenv/versions/3.11.1/lib/python3.11/site-packages/typer/core.py", line 197, in _main
rv = self.invoke(ctx)
^^^^^^^^^^^^^^^^
File "/Users/bwen/.pyenv/versions/3.11.1/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
return ctx.invoke(self.callback, **ctx.params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/bwen/.pyenv/versions/3.11.1/lib/python3.11/site-packages/click/core.py", line 783, in invoke
return __callback(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/bwen/.pyenv/versions/3.11.1/lib/python3.11/site-packages/typer/main.py", line 703, in wrapper
return callback(**use_params)
^^^^^^^^^^^^^^^^^^^^^^
File "/Users/bwen/.pyenv/versions/3.11.1/lib/python3.11/site-packages/docling/cli/main.py", line 593, in convert
export_documents(
File "/Users/bwen/.pyenv/versions/3.11.1/lib/python3.11/site-packages/docling/cli/main.py", line 167, in export_documents
for conv_res in conv_results:
File "/Users/bwen/.pyenv/versions/3.11.1/lib/python3.11/site-packages/docling/document_converter.py", line 243, in convert_all
for conv_res in conv_res_iter:
File "/Users/bwen/.pyenv/versions/3.11.1/lib/python3.11/site-packages/docling/document_converter.py", line 278, in _convert
for item in map(
File "/Users/bwen/.pyenv/versions/3.11.1/lib/python3.11/site-packages/docling/document_converter.py", line 324, in _process_document
conv_res = self._execute_pipeline(in_doc, raises_on_error=raises_on_error)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/bwen/.pyenv/versions/3.11.1/lib/python3.11/site-packages/docling/document_converter.py", line 345, in _execute_pipeline
pipeline = self._get_pipeline(in_doc.format)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/bwen/.pyenv/versions/3.11.1/lib/python3.11/site-packages/docling/document_converter.py", line 307, in _get_pipeline
self.initialized_pipelines[cache_key] = pipeline_class(
^^^^^^^^^^^^^^^
File "/Users/bwen/.pyenv/versions/3.11.1/lib/python3.11/site-packages/docling/pipeline/standard_pdf_pipeline.py", line 106, in init
CodeFormulaModel(
File "/Users/bwen/.pyenv/versions/3.11.1/lib/python3.11/site-packages/docling/models/code_formula_model.py", line 107, in init
self.code_formula_model = CodeFormulaPredictor(
^^^^^^^^^^^^^^^^^^^^^
File "/Users/bwen/.pyenv/versions/3.11.1/lib/python3.11/site-packages/docling_ibm_models/code_formula_model/code_formula_predictor.py", line 83, in init
self._tokenizer = AutoTokenizer.from_pretrained(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/bwen/.pyenv/versions/3.11.1/lib/python3.11/site-packages/transformers/models/auto/tokenization_auto.py", line 889, in from_pretrained
return tokenizer_class.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/bwen/.pyenv/versions/3.11.1/lib/python3.11/site-packages/transformers/tokenization_utils_base.py", line 2163, in from_pretrained
return cls._from_pretrained(
^^^^^^^^^^^^^^^^^^^^^
File "/Users/bwen/.pyenv/versions/3.11.1/lib/python3.11/site-packages/transformers/tokenization_utils_base.py", line 2397, in _from_pretrained
tokenizer = cls(*init_inputs, **init_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/bwen/.pyenv/versions/3.11.1/lib/python3.11/site-packages/transformers/models/gpt2/tokenization_gpt2_fast.py", line 99, in init
super().init(
File "/Users/bwen/.pyenv/versions/3.11.1/lib/python3.11/site-packages/transformers/tokenization_utils_fast.py", line 115, in init
fast_tokenizer = TokenizerFast.from_file(fast_tokenizer_file)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Exception: data did not match any variant of untagged enum ModelWrapper at line 255422 column 3

Steps to reproduce

run docling --enrich-formula entropy-1.pdf

Docling version

Docling version: 2.30.0
Docling Core version: 2.27.0
Docling IBM Models version: 3.4.1
Docling Parse version: 4.0.1
Python: cpython-311 (3.11.1)
Platform: macOS-15.5-x86_64-i386-64bit

[wenboown]

It seems that the problem is not with that particular PDF. I tried a few other PDFs. Got the same error.

PDFs:
https://arxiv.org/pdf/0705.2535
https://arxiv.org/pdf/1204.6481

I also tried to run Docling on another RHEL with 2 V100 GPUs. Got a different error:

docling Informatics\ Carnot\ Machine.pdf --enrich-formula

/home/bwen/.local/lib/python3.9/site-packages/numpy/_core/fromnumeric.py:3596: RuntimeWarning: Mean of empty slice.
return _methods._mean(a, axis=axis, dtype=dtype,
/home/bwen/.local/lib/python3.9/site-packages/numpy/_core/_methods.py:138: RuntimeWarning: invalid value encountered in scalar divide
ret = ret.dtype.type(ret / rcount)
/home/bwen/.local/lib/python3.9/site-packages/docling/pipeline/standard_pdf_pipeline.py:262: RuntimeWarning: Mean of empty slice
np.nanmean(
WARNING:docling.cli.main:Document /tmp/tmpscth_ka1/Informatics Carnot Machine.pdf failed to convert.

Docling version

Docling version: 2.34.0
Docling Core version: 2.32.0
Docling IBM Models version: 3.4.3
Docling Parse version: 4.0.1
Python: cpython-39 (3.9.21)
Platform: Linux-5.14.0-503.26.1.el9_5.x86_64-x86_64-with-glibc2.34

[wenboown]

Claude helped me debug this and found the problem:

Root Cause

This is a tokenizer compatibility issue between:

• transformers==4.42.4 (required by docling-ibm-models on macOS x86_64)
• tokenizers==0.19.1 (installed as dependency)
• The tokenizer.json file format in the ds4sd/CodeFormula model

The fast tokenizer JSON format in the CodeFormula model is incompatible with the specific combination of transformers/tokenizers versions constrained by the current dependencies.

Environment Details

• OS: macOS (darwin 24.5.0)
• Architecture: x86_64
• Python: 3.12
• docling: 2.34.0
• docling-ibm-models: 3.4.3
• transformers: 4.42.4 (constrained by platform-specific requirements)
• tokenizers: 0.19.1

Reproduction Steps

1. Install docling on macOS x86_64
2. Enable formula enrichment: pipeline_options.do_formula_enrichment = True
3. Process any PDF document
4. Error occurs when loading the CodeFormula tokenizer

Temporary Workaround

We found that removing the tokenizer.json file from the CodeFormula model cache forces the use of the slow tokenizer, which works correctly. However, this requires manual intervention and is not a sustainable solution.

Suggested Solutions

1. Update dependency constraints: Allow newer transformers/tokenizers versions that are compatible with the CodeFormula model's tokenizer format
2. Pin to compatible versions: Identify and pin to specific transformers/tokenizers versions that work with the current CodeFormula model
3. Force slow tokenizer: Modify the CodeFormula model loading to use use_fast=False by default
4. Update CodeFormula model: Regenerate the tokenizer.json file with a compatible format

Impact

This issue prevents the formula enrichment feature from working on macOS, which is a significant limitation for users processing mathematical/scientific documents on this platform.

Would you like me to format this differently or add any additional technical details?

[wenboown]

Here are the sources:

Issue Summary

Error: Exception: data did not match any variant of untagged enum ModelWrapper at line [large number] column 3

Root Cause: Compatibility issue between tokenizer file format and the tokenizers library version, specifically affecting models with newer tokenizer.json files when loaded with older tokenizers library versions.

Affected Versions & Combinations

Problematic Combinations:

• tokenizers==0.19.1 + transformers==4.42.4
• tokenizers==0.15.2 + transformers==4.38.2
• Generally affects tokenizers < 0.20.0 with newer model tokenizer files

Working Solutions Found:

1. Upgrade approach: transformers>=4.45 + tokenizers>=0.20.0
2. Downgrade approach: transformers==4.38.2 + tokenizers==0.15.2 (but may not work with all models)
3. Workaround: Use use_fast=False when loading tokenizers

Affected Models & Scenarios

Confirmed Affected Models:

• ds4sd/CodeFormula (our case)
• mistralai/Mistral-7B-v0.3
• meta-llama/Llama-3.1-8B-Instruct
• unsloth/Llama-3.2-3B-Instruct-bnb-4bit
• Custom trained tokenizers with newer format

Common Scenarios:

• Loading models in production environments (AWS SageMaker, vLLM, Docker containers)
• Fine-tuning workflows (Unsloth, custom training)
• Model conversion/quantization pipelines

GitHub Issues & Discussions Found

Primary Sources:

1. HuggingFace Tokenizers Issue Erroneous header classification in AsciiDoc #1562: "Unable to Load Custom GPT2 Tokenizer"

   • Status: Closed as expected behavior
   • Solution: Update to tokenizers==0.21.1

2. Unsloth Issue Question about cluster comparison function #1059: "[FIXED] Exception: data did not match any variant of untagged enum ModelWrapper"

   • Multiple users affected
   • Solutions: Update transformers>=4.45 or create new conda environment

3. HuggingFace Discussions:

   • Multiple forum posts with same error
   • Common in deployment scenarios

Key Insights from Issues:

1. Breaking Change: HuggingFace made a breaking change in tokenizers format that's not backward compatible
2. Environment Specific: Often works locally but fails in production/Docker environments
3. Model Specific: Newer models more likely to have incompatible tokenizer files
4. Platform Issues: Particularly common on macOS and in containerized environments

Technical Details

Error Location:

• Occurs in TokenizerFast.from_file() when parsing tokenizer.json
• Specific line numbers vary (255422, 1251003, etc.) indicating large tokenizer files
• Related to enum parsing in Rust tokenizers backend

Workarounds That Work:

1. Slow Tokenizer: AutoTokenizer.from_pretrained(model, use_fast=False)
2. Version Pinning: Specific compatible version combinations
3. Fresh Environment: New conda/venv often resolves conflicts
4. Manual Tokenizer Replacement: Download fresh tokenizer files from HuggingFace

[dolfim-ibm]

Thanks for the analysis. If we recap correctly:

1. You are targeting Python 3.11 on macOS x86_64, right?
2. From our understanding transformers 4.42.x was the last version compatible with macOS x86_64, specially because of the torch compatibility.
   • But you actually found a patch which works, right?
   • Which transformers version did you eventually use?

[wenboown]

Hi Michele,

1. You are targeting Python 3.11 on macOS x86_64, right?

Yes.

2. From our understanding transformers 4.42.x was the last version compatible with macOS x86_64, specially because of the torch compatibility.

Would a different Python version work better? I can try 3.12 or 3.13, please give me guidance if you know.

• But you actually found a patch which works, right?

I need to download docling source code and set use_fast=False, also need to use 'cpu' as device. So it becomes quite slow for larger PDF.

I tried on a RHEL with two V100, there's no version issue, but got a different type of error, seems to be memory issue. That one I don't know how to debug.

• Which transformers version did you eventually use?

On my laptop, the pip install docling resolved to the following versions

sentence-transformers                    3.2.0
tokenizers                               0.19.1
transformers                             4.42.4

I try to force change them but got conflict:

pip install --upgrade "transformers>=4.45"

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. Th
is behaviour is the source of the following dependency conflicts.
docling-ibm-models 3.4.1 requires transformers<4.43.0,>=4.42.0; sys_platform == "darwin" and platform_machine
 == "x86_64", but you have transformers 4.52.4 which is incompatible.
Successfully installed tokenizers-0.21.1 transformers-4.52.4

So in the end I gave up on trying to resolve the version issue. But used the slow mode (load CodeFormula tokenizer with use_fast=False).