Add chat doc in quick start #21213
Add chat doc in quick start #21213
Conversation
|
👋 Hi! Thank you for contributing to the vLLM project. 💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels. Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging. To run CI, PR reviewers can either: Add 🚀 |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request adds a helpful note and code examples to the quickstart guide on how to use chat models with vLLM. The explanation is clear and valuable for users. I've found a bug in one of the new code examples and provided a suggestion to fix it. Once that's addressed, this should be good to merge.
| @@ -98,6 +98,41 @@ for output in outputs: | |||
| print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}") | |||
| ``` | |||
|
|
|||
| !!! note | |||
| The generate method does not automatically apply the corresponding model's chat template to the input prompt, as this method is designed to align with OpenAI's `completions` interface rather than the `chat/completions` interface. Therefore, if you are using an Instruct model or Chat model, you should manually apply the corresponding chat template to ensure the expected behavior. Alternatively, you can use the LLM.chat method and pass properly formatted data. | |||
There was a problem hiding this comment.
| The generate method does not automatically apply the corresponding model's chat template to the input prompt, as this method is designed to align with OpenAI's `completions` interface rather than the `chat/completions` interface. Therefore, if you are using an Instruct model or Chat model, you should manually apply the corresponding chat template to ensure the expected behavior. Alternatively, you can use the LLM.chat method and pass properly formatted data. | |
| The `llm.generate` method does not automatically apply the model's chat template to the input prompt. Therefore, if you are using an Instruct model or Chat model, you should manually apply the corresponding chat template to ensure the expected behavior. Alternatively, you can use the `llm.chat` method and pass a list of messages which have the same format as those passed to OpenAI's `client.chat.completions`: | |
| `` | |
| For quickstart, there is no need to provide much explanation. |
| !!! note | ||
| The generate method does not automatically apply the corresponding model's chat template to the input prompt, as this method is designed to align with OpenAI's `completions` interface rather than the `chat/completions` interface. Therefore, if you are using an Instruct model or Chat model, you should manually apply the corresponding chat template to ensure the expected behavior. Alternatively, you can use the LLM.chat method and pass properly formatted data. | ||
|
|
||
| ```python |
There was a problem hiding this comment.
Use a ??? code admoniton to collapse the code block
Essential Elements of an Effective PR Description Checklist
supported_models.mdandexamplesfor a new model.Purpose
Add documentation about the
chatinterface to the quickstart guide to align with OpenAI'schat/completionsAPI and prevent unexpected behavior when using thegeneratemethod withchat models.Test Plan
Test Result
(Optional) Documentation Update