/v1/embeddings please

[yuhai-china]

when will /v1/embeddings API available?
Thank you

[zhuohan123]

Hi! Adding support to return embeddings is definitely on our road map. In addition, I believe the modifications to support embedding should not be very complicated and is a very good first issue. If you're interested, feel free to contribute!

[Vinno97]

I looked into it to maybe pick it up as a "good first issue", but did not find it to be straightforward to implement. I'm afraid any changes I would make, would just be hacks. If you have any pointers on how and where I could best add it, I'd be happy to give it a second look.

[bm777]

@zhuohan123 and @yuhai-china are you talking about a multilingual or the monolingual model?
I'm keen to contribute and would be interested in the sentence-transformer model from HF Zoo.

[bm777]

@Vinno97 are you still working on it? I would love to help because I'm interested to use it too.

[Vinno97]

No I haven't come back to it. I hoped I could just create a new endpoint that hooked into the model and returned the last hidden state. But I found that the LLMEngine was so written around text generation that I didn't see myself easily and cleanly adding embeddings into it.

But do give it a try! I must admit I spent less than an hour looking into it.

[zhuohan123]

@yuhai-china @Vinno97 @bm777 Thanks for your interest in this. I previously misunderstood this API to be getting the hidden states for the generated sequence, and that should be easy. However, it turns out that this API is for a completely different set of models (i.e., BERT-like embedding models).

The current vLLM mainly focuses on autoregressive generation. For embedding, both paged attention and continuous batching cannot help performance. Therefore, I think it's better to use other libraries for embedding for now. In the future, when we are extending the scope of vLLM, we will look into this again.

[bm777]

Completely understandable, and also embedding model generates only vectors from text, it doesn't swap operation, unlike it could increase the latency. Sentence transformers do already the job efficiently.

All this makes sense and @zhuohan123 thanks for reminding :)

[Vinno97]

I was actually looking into getting the hidden states for the input prompt. Some techniques like https://github.com/HazyResearch/TART use embeddings of generative LLMs. Especially for large models, it makes sense to only host them once, instead of hosting a separate instance just for getting the embeddings.

[LaVieEnRose365]

Hi there! Has the API of getting hidden states for the generated sequence been implemented? I think it's very helpful if we can get the hidden states.

[kheyer]

Can you give a sense of the technical challenges in implementing this?

Echoing comments on hosting, right now I need to have two serving frameworks with twice the GPU allocation to generate text and compute embeddings from the same model. Even if performance isn't optimum for embeddings, being able to use a single serving framework and deploy a model once would make life a lot easier.

[zhuohan123]

Move this issue to discussions as it's more of a longer future plan.

[lambda-science]

Does anyone have recommandation tools like vLLM for embedding models ?
vLLM is great for LLMs but I couldn't find anything for reliable hosting embedding models (with standardized API)

[bm777]

@lambda-science i asked that question, but it seems not to be in their current roadmap. So I'm using now Candle by hf.

[lambda-science]

I also found this: https://github.com/huggingface/text-embeddings-inference I guess it works

[mattmalcher]

For others looking for alternatives - have stumbled on https://github.com/michaelfeil/infinity

[kheyer]

+1 to the huggingface repo since they moved to Apache 2.0 in the 1.2.1 release

[dtrckd]

LLM2Vec: Large Language Models Are Secretly Powerful Text Encoders : https://github.com/McGill-NLP/llm2vec

[leoguillaume]

Waiting of this major feature of VLLM, I created a very simple merge version of VLLM and HuggingFace Text Embeddings Inference to have one API with full OpenAI endpoints : /v1/embeddings, /v1/chat/completion ... : https://github.com/leoguillaume/VLLMEmbeddings

[adrianlyjak]

this looks like its been implemented! #3734

[GeraudBourdin]

I don't sée any reference on this subject in documentation. That would be awesome

[yosefmih]

It is not salient, but there is an example in the docs https://docs.vllm.ai/en/latest/getting_started/examples/openai_embedding_client.html

[GeraudBourdin]

unfortunately this is not working with mistral7b.
thanks for pointed me the doc :)

[ccamacho]

@GeraudBourdin did you manage to get mistral7b on the embeddings endpoint?

[lauhaide]

Hi all, as output in the generate() method, I would like to also get the hidden_states associated to the generated sequences. As fas as I searched, this wasnt available, has this been implemented now?

[yuanzhiyong1999]

Have you found an alternative method?

[lauhaide]

no, not yet.

[Huyueeer]

So now you support it?

[sudarshan-kamath]

Is it supported? If possible, could this be added into the vllm_worker of Fastchat? Thanks

https://github.com/lm-sys/FastChat/blob/main/fastchat/serve/vllm_worker.py