[RFC]: Response format extensions for structured outputs

[aarnphm]

Motivation.

Currently, users can provide additional constraints format via extra_body in OpenAI client:

from enum import Enum
from pydantic import BaseModel
from openai import OpenAI

simplified_sql_grammar = """
        root ::= select_statement

        select_statement ::= "SELECT " column " from " table " where " condition

        column ::= "col_1 " | "col_2 "

        table ::= "table_1 " | "table_2 "

        condition ::= column "= " number

        number ::= "1 " | "2 "
    """

prompt = (
        "Generate an SQL query to show the 'username' and 'email'"
        "from the 'users' table."
    )

completion = client.chat.completions.create(
        model=model,
        messages=[
            {
                "role": "user",
                "content": prompt,
            }
        ],
        extra_body={"guided_grammar": simplified_sql_grammar},

This also applies with guided_json, structural_tag, guided_regex.

While this is pretty convenient for most developers, these fields are still using v0 terminology wrt guided decoding.

With the upcoming v0 deprecation, I think it is the time to have a usage update with this pattern.

Proposed Change.

OpenAI already recommends users to use response_format with json_schema

Given that we already supports structural_tag via response_format (example), I propose an extension to response_format for the remainder of the fields

completion = client.chat.completions.create(
  model=model, messages=messages,
  response_format={
    "type": "vllm_regex",
    "regex": r"\w+@\w+\.com\n"
  }
)

completion = client.chat.completions.create(
  model=model, messages=messages,
  response_format={
    "type": "vllm_grammar",
    "grammar": """
root ::= select_statement
select_statement ::= "SELECT " column " from " table " where " condition
column ::= "col_1 " | "col_2 "
table ::= "table_1 " | "table_2 "
condition ::= column "= " number
number ::= "1 " | "2 "
"""
  }

completion = client.chat.completions.create(
  model=model, messages=messages,
  response_format={
    "type": "vllm_choice",
    "choice": ["Positive", "Negative", "Neutral"], 
  }
)

The previous json_schema + structural tag remains the same.

The field guided_* will still works previously, but will reserved only for more advance usage and won't be documented.

Feedback Period.

1 week for revision, 2-3 days for implementations plan (mostly frontend + protocol updates)

We can also add a debug log recommending using this new pattern for all existing usage of guided_* (so that we won't break production)

CC List.

@russellb @mgoin @simon-mo @hmellor

Any Other Things.

This is mostly frontend changes

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[russellb]

One concern I have about OpenAI-compatible API server extensions in general is that they may contain things that could conflict with future official additions to the API.

For example, OpenAI could add a type of regex with a different type of syntax and/or behavior.

A way to avoid this, and a suggestion I have for this proposal, is to namespace our additions with a vllm_ prefix. For example, "type": "vllm_regex". That way, we are explicit about what a vLLM-specific extension is, where we have defined the syntax and behavior, vs. cases where we are trying to implement compatibility with official APIs.

[simon-mo]

We should deprecate guided_* if it's lot too hard. The main issue is this is what happen if OpenAI change the interface and conflict with what we have (hence the reason we have been mostly extended top level fields). But overall I think the benefit outweighs the change as OpenAI seems to be moving towards Responses API

[russellb]

Other than the namespacing concern, this does seem like a better way to express this in the API.

[russellb]

We should deprecate guided_* if it's lot too hard. The main issue is this is what happen if OpenAI change the interface and conflict with what we have (hence the reason we have been mostly extended top level fields). But overall I think the benefit outweighs the change as OpenAI seems to be moving towards Responses API

You make a good point that response_format in whole is not our field to change.

An alternative is a new field vllm_response_format or something like that where we control its contents.

[aarnphm]

A way to avoid this, and a suggestion I have for this proposal, is to namespace our additions with a vllm_ prefix. For example, "type": "vllm_regex".

updated

[aarnphm]

We should deprecate guided_* if it's lot too hard. The main issue is this is what happen if OpenAI change the interface and conflict with what we have (hence the reason we have been mostly extended top level fields). But overall I think the benefit outweighs the change as OpenAI seems to be moving towards Responses API

+1 with deprecation. The reasoning behind what I have in the RFC is that because this has been existing since 0.6, I would assume a lot of production usage depends on this field, but we can proceed with our deprecation policy here.

[aarnphm]

The main issue is this is what happen if OpenAI change the interface and conflict with what we have (hence the reason we have been mostly extended top level fields). But overall I think the benefit outweighs the change as OpenAI seems to be moving towards Responses API

Initially I share this same sentiment. afaict response_format has been the usage pattern for specifying json_schema, hence this proposal.

[aarnphm]

An alternative is a new field vllm_response_format or something like that where we control its contents.

this would lives under extra_body. My preference is doing it via response_format to reduce depth level. But if we all agree on vllm_response_format for more explicit control, then I'm not opposed to this.

[russellb]

An alternative is a new field vllm_response_format or something like that where we control its contents.

this would lives under extra_body. My preference is doing it via response_format to reduce depth level. But if we all agree on vllm_response_format for more explicit control, then I'm not opposed to this.

The "pure" solution would be entirely new field that we control.

The more practical side of me thinks response_format using types that have a vllm_ prefix is fine, too. I'm happy either way.

That also means we should change structural_tag to vllm_structural_tag. The XGrammar team was planning a more significant definition for structural tag anyway, so that could all go under a new vllm_structural_tag and then we would deprecate the original structural_tag definition.