Support for quantized ONNX

[Mihaiii]

Hello!

I noticed your ONNX script (also referenced in the readme), but I can't manage to get a quantized smaller ONNX file from it.
I tried multiple scripts, including the one from transformers.js repo, but still no success.

From what I get, there is some script already somewhere because we have such a quantized model here: https://huggingface.co/minishlab/M2V_base_output/blob/main/onnx/model_quantized.onnx .

Please considered adding support for the quantized ONNX files.

[stephantul]

Hello @Mihaiii!

These were added by Xenova, not by us. See here for the relevant PR: https://huggingface.co/minishlab/M2V_base_output/discussions/1/files

I actually don't know how they were made, but I'll investigate. As an aside: can you give some rationale as to what the quantized versions offer over the regular models? Is it just size, or do they actually work better/are they faster?

Stéphan

[Mihaiii]

@stephantul Thank you!

I'm interested in the quantized version to reduce disk size. I want to use a "model2vec" model as the default model in semantic-autocomplete, a component that downloads the model to the client side when the website user interacts with the web component for the first time. For this reason, I need to use very lightweight models. Currently, I'm using a model that's around 15 MB.

[stephantul]

@Mihaiii one option could be to use potion-base-4m, which is 15mb. I'm not sure how good the onnx quantized version is compared to 4m, but 4m is actually not that bad.

[stephantul]

Hello @Mihaiii ,

I didn't get around to checking the quantization, sorry. Did you check out potion-base-4m? That also might work well

Stéphan

[Mihaiii]

Hey @stephantul !

No, I did not try it - I already use a 15B model as default; I'm interested in smaller ones :)

[Tamnac]

Is this possible now that m2v supports quantization?

[stephantul]

Hello @Tamnac (and @Mihaiii, sorry never got back to you)

Yes, we support float16 to ONNX right now, see the script here: https://github.com/MinishLab/model2vec/blob/main/scripts/export_to_onnx.py

If I distill a bge-base model like this and save it:

d = distill("baai/bge-base-en-v1.5", pca_dims=128)
d.save_pretrained("temp")

and then run:

python3 scripts/export_to_onnx.py --model_path temp --save_path onnx_model

I get a model with 7mb on disk weights.

[thiswillbeyourgithub]

[decided to move my comment to a new issue: #256]

[stephantul]

Hello @thiswillbeyourgithub,

I'll try binary quantization. We've tried it before, but without much luck. For model2vec models, the magnitude of the individual dimensions varies a lot, so reducing a vector to the sign bits, which is what the link proposes, does not work well. What we can try is quantizing to int8 first, and then doing multiple bits per dimension. I'll investigate.

Stéphan

[thiswillbeyourgithub]

(Oh sorry I decided to move my comment to its own issue but you replied in the mean time, if you don't mind let's move it there!)