[MODEL] New model support for naver-hyperclovax/HyperCLOVAX-SEED-Vision-Instruct-3B

[bigshanedogg]

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.

Purpose

• Add new model for naver-hyperclovax/HyperCLOVAX-SEED-Vision-Instruct-3B
  • FIX [New Model]: Support HCXVisionForCausalLM #19963
  • FIX [New Model]: HyperClova X SEED (ChatClova) #21275

Test Plan

offline inference

PYTHONPWD=${PWD} python examples/offline_inference/vision_language.py \
        --model-type hyperclovax_seed_vision \
        --modality image \
        --num-prompts 4

online serving

python -m vllm.entrypoints.openai.api_server \
    --model naver-hyperclovax/HyperCLOVAX-SEED-Vision-Instruct-3B \
    --trust_remote_code \
    --chat-template-content-format "openai" \
    --max-model-len 4096 \
    --max-num-seqs 8

Test Result

offline inference

--------------------------------------------------
A tower is in the background behind a tree with pink blossoms.
--------------------------------------------------
A tower is seen behind a tree with pink blossoms.
--------------------------------------------------
A tower is seen behind a tree with pink blossoms.
--------------------------------------------------
A tower is seen behind a tree with pink blossoms.
--------------------------------------------------

online serving

# client
messages = [
    {
        "role": "user",
        "content": [
            {
                "type": "image_url",
                "image_url": {"url": "https://upload.wikimedia.org/wikipedia/commons/d/da/2015_Kaczka_krzy%C5%BCowka_w_wodzie_%28samiec%29.jpg"},
            },
            {
                "type": "text", 
                "text": "Describe about the image.",
            },
        ],
    },
]

payload = {
    "messages": messages,
    "max_tokens": 256,
    "repetition_penalty": 1.0,
}

response = requests.post(
    f'{url}/v1/chat/completions', 
    headers=headers,
    json=payload,
)
response = response.json()
response["choices"][-1]["message"]["content"]

# result
This is a duck with its head up, floating on a water surface.  The duck is green and yellow in color.  The duck is located in a body of water.  The duck's reflection on the water's surface can be seen below it.  The floating duck overlaps a large area with light blue water.  The reflection of the duck being darker against the bright blue background water is clearly visible below the duck.  The duck's body takes up the space from the left to a little more than one-half way to the right.  The duck's back can be seen go all the way up toward the top of the image, and the duck's head is raised up to just above the top of the image.  The body of the duck slopes down from top to bottom of the image.  The duck is located towards the top left side of the image.  The reflection of the duck below the actual duck is a little shorter than the duck itself.

(Optional) Documentation Update

• updated supported_models.md
• updated examples/offline_inference/vision_language.py
• updated examples/offline_inference/vision_language_multi_image.py
• updated tests/models/multimodal/processing/test_common.py

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

[gemini-code-assist]

Summary of Changes

Hello @bigshanedogg, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces full support for the HyperCLOVAX-SEED-Vision-Instruct-3B multimodal model. It includes the necessary architectural components for processing both image and video inputs, integrating them with the language model, and handling various image resolutions. An example script is also provided to facilitate easy testing and usage of the new model.

Highlights

• New Model Support: Added comprehensive support for the naver-hyperclovax/HyperCLOVAX-SEED-Vision-Instruct-3B multimodal model, enabling it to be used within the system.
• Multimodal Processing Implementation: Implemented custom processing logic for images and videos, including HCXVisionProcessingInfo, HCXVisionMultiModalProcessor, and HCXVisionDummyInputsBuilder, to seamlessly integrate visual inputs into the language model's pipeline.
• Vision Encoder and Projector Integration: Integrated and initialized a vision tower (supporting CLIP or Siglip architectures) and a multimodal projector (HCXVisionMlp or HCXVisionCAbstractor) responsible for converting visual features into a format consumable by the language model.
• Any-Resolution Image Handling: Included utility functions such as unpad_image, select_best_resolution, and anyres_postprocessing to support flexible image input handling across various resolutions and aspect ratios.
• Example Usage Script: Provided a new offline inference example script (hyperclovax_seed_vision_3b_instruct.py) to demonstrate how to load and use the newly supported HyperCLOVAX-SEED-Vision-Instruct-3B model with multimodal inputs.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in issue comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist is currently in preview and may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments to provide feedback.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request adds support for the naver-hyperclovax/HyperCLOVAX-SEED-Vision-Instruct-3B model. The changes include the model implementation, an example script, and documentation updates. The implementation looks solid, but I've found a critical issue that would cause a runtime error, along with several high and medium severity issues related to correctness and maintainability in the new model file. Addressing these will ensure the new model is robust and well-integrated.

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @bigshanedogg.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[bigshanedogg]

I've addressed the feedback you left in the comments and updated the PR description accordingly.

• Removed unnecessary lines
• Moved the examples to a more appropriate location
• Rebased onto the latest main

(cc. @Isotr0py , @DarkLight1337)

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @bigshanedogg.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[DarkLight1337]

Can you push a new commit to this PR to trigger CI?