Offline inference benchmark #50
Conversation
ssraghavan-oci
commented
Apr 22, 2025
ssraghavan-oci
commented
- Docs for offline inference using vllm and sglang
- Has sample config file used for deployment
oracle-contributor-agreement
bot
added
the
OCA Verified
| ## Sample Config File (`example_sglang.yaml`) | ||
|
|
||
| ```yaml | ||
| benchmark_type: offline |
There was a problem hiding this comment.
I think this YAML / config approach is a great v0 for this. However, I do think we should add a field to the recipe json that accepts these parameters somehow so the user doesn't have to maintain the deployment JSON and the YAML.
I defer to @vishnukam3 on this one... can you weigh in here?
| @@ -0,0 +1,104 @@ | |||
| # Online Inference Blueprint (LLMPerf) | |||
|
|
|||
| This blueprint benchmarks **online inference performance** of large language models using **LLMPerf**, a standardized benchmarking tool. It is designed to evaluate LLM APIs served via platforms such as OpenAI-compatible interfaces, including self-hosted LLM inference endpoints. | |||
There was a problem hiding this comment.
Maybe we include a section on what offline vs online means here? And link between the offline version and the online version so people can choose after they understand the difference?
There was a problem hiding this comment.
yes. I do have a title When to use Offline inference in offline readme and this recipe helps section in online readme which outlines when to use when.
| "output_object_storage": [ | ||
| { | ||
| "bucket_name": "inference_output", | ||
| "mount_location": "/mlcommons_output", |
There was a problem hiding this comment.
maybe change this to benchmarking output? Is this specific to mlcommons? Looks like re-use perhaps.
There was a problem hiding this comment.
yea that makes sense. It can be anything that the customer wants to call. I just named it mlcommons output as metrics there and the json file here has the same data.
| "output_object_storage": [ | ||
| { | ||
| "bucket_name": "inference_output", | ||
| "mount_location": "/mlcommons_output", |
There was a problem hiding this comment.
similar here - maybe make /benchmarking_output
| ## Prerequisites | ||
|
|
||
| Before running this blueprint: | ||
| - You **must have an inference server already running**, compatible with the OpenAI API format. |
There was a problem hiding this comment.
I'd suggest adding something like:
"You can first deploy this blueprint for an online inference server to test the workflow".
Then, I would provide an example config + blueprint which utilizes this to test the workflow. That way, someone could have a pre-canned end-to-end workflow which we provide.
There was a problem hiding this comment.
Noted! will make the changes
…he PR for offline inference
