Skip to content

feat: add batch support for LLM.generate #442

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
micpst opened this issue Mar 24, 2025 · 0 comments
Open

feat: add batch support for LLM.generate #442

micpst opened this issue Mar 24, 2025 · 0 comments
Labels
feature New feature or request
Milestone

Comments

@micpst
Copy link
Collaborator

micpst commented Mar 24, 2025

Feature description

Add support for list of prompts as an input for LLM.generate method, also add adjust the interface for the generate_streaming and generate with_metadata. I'm talking about something like this:

responses = await LLiteLLM.generate([ImageDescriberPrompt(...), ...])

It should also work for strings and other types:

responses = await LLiteLLM.generate(["Tell me the meaning of life", ...])

Motivation

Currently there is no way to send multiple prompts in a single request to the LLM. There are many use cases that can take advantage of this feature, like LLM as judge evaluation pipeline or LLM reranker that need to process the same prompts with different input data.

Additional context

No response

@micpst micpst added the feature New feature or request label Mar 24, 2025
@micpst micpst added this to ragbits Mar 24, 2025
@micpst micpst moved this to Backlog in ragbits Mar 24, 2025
@micpst micpst self-assigned this Mar 24, 2025
@micpst micpst moved this from Backlog to Ready in ragbits Mar 24, 2025
@micpst micpst removed their assignment Mar 24, 2025
@mhordynski mhordynski added this to the Ragbits 1.1.0 milestone May 7, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature New feature or request
Projects
Status: Ready
Development

No branches or pull requests

2 participants