You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Add support for list of prompts as an input for LLM.generate method, also add adjust the interface for the generate_streaming and generate with_metadata. I'm talking about something like this:
responses=awaitLLiteLLM.generate(["Tell me the meaning of life", ...])
Motivation
Currently there is no way to send multiple prompts in a single request to the LLM. There are many use cases that can take advantage of this feature, like LLM as judge evaluation pipeline or LLM reranker that need to process the same prompts with different input data.
Additional context
No response
The text was updated successfully, but these errors were encountered:
Feature description
Add support for list of prompts as an input for
LLM.generate
method, also add adjust the interface for thegenerate_streaming
andgenerate with_metadata
. I'm talking about something like this:It should also work for strings and other types:
Motivation
Currently there is no way to send multiple prompts in a single request to the LLM. There are many use cases that can take advantage of this feature, like LLM as judge evaluation pipeline or LLM reranker that need to process the same prompts with different input data.
Additional context
No response
The text was updated successfully, but these errors were encountered: