Add Arcee (AFM) model support to vLLM #21267

alyosha-swamy · 2025-07-20T22:05:09Z

[New Model] Support Arcee (Arcee Foundational Models)

1. Purpose (Why this PR?)

Add inference support for Arcee Foundational Model (AFM) so that users can serve it with vLLM in both Python and API-server workflows. AFM uses a unique ReLU² activation in its MLP layers, differentiating it from standard Llama-based models.

2. Model details

Field	Value / Reference
Source repo / HF id	huggingface.co/arcee-ai/AFM-4.5B-Base
Architecture	Llama-style decoder-only transformer with ReLU² MLP activation
Context length	64k tokens
Hidden size / #layers	4096 / 32
License	CC BY-NC 4.0
Special quirks	Uses ReLU² (squared ReLU) activation instead of SiLU in MLP layers

3. Implementation overview

Added ArceeForCausalLM class in vllm/model_executor/models/arcee.py with custom ArceeMLP using ReLU² activation
Registered model in _TEXT_GENERATION_MODELS in vllm/model_executor/models/registry.py
Updated docs/models/supported_models.md with Arcee entry in text generation table
Reused LlamaAttention from existing Llama implementation for attention layers
Implemented proper LoRA and Pipeline Parallelism support

4. Performance / sanity check

$ python -m vllm.entrypoints.openai.api_server --model arcee-ai/AFM-4.5B-Base --trust-remote-code
$ curl http://localhost:8000/v1/completions -H "Content-Type: application/json" -d '{
    "model": "arcee-ai/AFM-4.5B-Base",
    "prompt": "The future of artificial intelligence is",
    "max_tokens": 50
}'

Expected: Coherent completion about life's meaning

Observed: " a question that has been asked throughout the history of mankind. The search for an answer to this question has inspired countless works of art, literature, and philosophy. Whether we consider the existentialist ideas of Albert Camus or the religious perspectives of spiritual leaders"

5. Test plan ✔️

Test	Command	Expected
Unit	`pytest tests/models/test_arcee.py`	All tests pass
Model Loading	`python -c "from vllm import LLM; llm = LLM('arcee-ai/AFM-4.5B-Base')"`	Model loads without errors
Integration	`vllm serve arcee-ai/AFM-4.5B-Base --trust-remote-code`	Server starts, responds to requests
Generation	`curl localhost:8000/v1/completions`	200 OK + valid completions

6. Documentation

Added row to docs/models/supported_models.md under Text Generation models
Model listed as ArceeForCausalLM with example model arcee-ai/AFM-4.5B-Base
Marked as supporting LoRA (✅), Pipeline Parallel (✅), and V1 (✅)

Checklist

I ran pre-commit run --all-files (ruff formatting)
All CI tests pass locally (pytest -q)
The PR description follows vLLM's "Essential Elements" template
No breaking changes for existing model classes

Notes for reviewers

The key architectural difference from standard Llama models is the MLP activation function. Arcee uses ReLU² (squared ReLU) instead of SiLU:

ArceeMLP implements: x = torch.pow(torch.relu(x), 2)
No gating mechanism (no gate_proj), only up_proj and down_proj
All other components (attention, layer norm, etc.) reuse existing Llama implementations

The model has been tested with an internal HF repo during development, but the official model is arcee-ai/AFM-4.5B-Base.

Test result

seq	Prompt	vLLM Output
0	"The meaning of life is"	" a question that has been asked throughout the history of mankind. The search for an answer to this question has inspired countless works of art, literature, and philosophy. Whether we consider the existentialist ideas of Albert Camus or the religious perspectives of spiritual leaders"
1	"Climate change is primarily caused by"	" human activity, specifically the emission of greenhouse gases such as carbon dioxide (CO2) and methane (CH4). It leads to changes in average temperatures and weather patterns, impacting both nature and human society."
2	"Machine learning algorithms work by"	" training a predictive model using labeled training data: the model detects patterns in the training data and learns from it. That model is then tested using a test set, which it must predict to achieve a good accuracy rate."

All outputs are coherent and contextually appropriate.

Signed-off-by: alyosha-swamy <raghav@arcee.ai>

github-actions · 2025-07-20T22:05:17Z

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

gemini-code-assist

Code Review

This pull request introduces support for Arcee (AFM) models within vLLM. The implementation exhibits a well-organized structure, effectively incorporating a specialized ArceeMLP module to manage the ReLU² activation, while leveraging the existing LlamaAttention module. The modifications are clean, and the updates to documentation and the model registry are appropriate. A high-severity issue has been identified concerning a misleading comment within the weight loading logic. Addressing this issue is crucial to ensure clarity and maintainability of the codebase.

gemini-code-assist · 2025-07-20T22:06:36Z

vllm/model_executor/models/arcee.py

+        # No special weight name remapping needed (AFM uses standard LLaMA
+        # naming except no gate_proj)
+        return loader.load_weights(weights)


This comment is misleading and should be removed. While it's true that AFM uses standard LLaMA naming (except for the missing gate_proj), the AutoWeightLoader does perform weight name remapping, specifically for the QKV projection weights (fusing separate q_proj, k_proj, v_proj into a single qkv_proj). Leaving this comment as is could cause confusion during future maintenance, as it contradicts the actual code behavior.

jeejeelee · 2025-07-21T02:23:22Z

vllm/model_executor/models/arcee.py

+        "qkv_proj": ["q_proj", "k_proj", "v_proj"],
+    }
+    # Supported LoRA modules (same as LLaMA, minus gate_proj)
+    supported_lora_modules = [


We removed supported_lora_modules a long time ago. Please delete it.

jeejeelee · 2025-07-21T02:29:10Z

Please also update this model in https://github.com/vllm-project/vllm/blob/main/tests/models/registry.py

…ment - Remove deprecated supported_lora_modules attribute - Add ArceeForCausalLM to test registry Signed-off-by: alyosha-swamy <raghav@arcee.ai>

jeejeelee · 2025-07-21T05:11:50Z

vllm/model_executor/models/arcee.py

+    packed_modules_mapping = {
+        "qkv_proj": ["q_proj", "k_proj", "v_proj"],
+    }
+    # (No MLP prefix since there's no gate_proj)


Suggested change

# (No MLP prefix since there's no gate_proj)

jeejeelee · 2025-07-21T05:11:56Z

vllm/model_executor/models/arcee.py

+                self.unpadded_vocab_size += lora_config.lora_extra_vocab_size
+
+            # Import DEFAULT_VOCAB_PADDING_SIZE
+            from vllm.model_executor.layers.vocab_parallel_embedding import (


Why import here?

jeejeelee · 2025-07-21T05:12:54Z

Can you fix the pre-commit failure?

jeejeelee · 2025-07-21T05:39:02Z

It seems that this model is not available on HF

hmellor · 2025-07-21T07:31:57Z

If this is a private model, please either:

Write your vLLM modelling code and register the model as an out-of-tree model https://docs.vllm.ai/en/latest/contributing/model/registration.html
Write your Transformers modelling code according to https://docs.vllm.ai/en/latest/models/supported_models.html#transformers and use --model-impl transformers

adarshxs · 2025-07-21T07:45:52Z

@hmellor @jeejeelee the model is set to be released as open weights on HF soon, this is therefore for day zero support of the model. The architecture is the same as llama with relu^2 act func.

hmellor · 2025-07-21T08:08:52Z

If the architecture is the same as Llama with the activation function changed it probably will work with --model-impl transformers.

Meaning that it's not necessary to make any changes to vLLM.

- Set is_available_online=False in test registry for CI compatibility

alyosha-swamy added 4 commits July 20, 2025 16:31

Add Arcee (AFM) model support to vLLM

2823d64

Signed-off-by: alyosha-swamy <raghav@arcee.ai>

Update arcee.py with code review improvements and syntax fix

8d81278

Signed-off-by: alyosha-swamy <raghav@arcee.ai>

Fix linting issues: line length E501 errors

3c2ebeb

Signed-off-by: alyosha-swamy <raghav@arcee.ai>

Fix linting issues: line length E501 errors

a4ef76b

Signed-off-by: alyosha-swamy <raghav@arcee.ai>

alyosha-swamy requested a review from hmellor as a code owner July 20, 2025 22:05

mergify bot added documentation Improvements or additions to documentation new-model Requests to new models labels Jul 20, 2025

gemini-code-assist bot reviewed Jul 20, 2025

View reviewed changes

jeejeelee reviewed Jul 21, 2025

View reviewed changes

Address review comments: remove supported_lora_modules and update com…

679363f

…ment - Remove deprecated supported_lora_modules attribute - Add ArceeForCausalLM to test registry Signed-off-by: alyosha-swamy <raghav@arcee.ai>

alyosha-swamy requested review from DarkLight1337 and ywang96 as code owners July 21, 2025 04:19

jeejeelee reviewed Jul 21, 2025

View reviewed changes

- Inherit from LlamaForCausalLM for most functionality

d5b2eb1

- Set is_available_online=False in test registry for CI compatibility

alyosha-swamy closed this Jul 21, 2025

hmellor mentioned this pull request Jul 21, 2025

Add arcee model #21296

Merged

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

Add Arcee (AFM) model support to vLLM #21267

Add Arcee (AFM) model support to vLLM #21267

alyosha-swamy commented Jul 20, 2025 •

edited by github-actions bot

Loading

Uh oh!

github-actions bot commented Jul 20, 2025

Uh oh!

gemini-code-assist bot left a comment

Uh oh!

gemini-code-assist bot Jul 20, 2025

Uh oh!

jeejeelee Jul 21, 2025

Uh oh!

jeejeelee commented Jul 21, 2025

Uh oh!

jeejeelee Jul 21, 2025

Uh oh!

jeejeelee Jul 21, 2025 •

edited

Loading

Uh oh!

jeejeelee commented Jul 21, 2025

Uh oh!

jeejeelee commented Jul 21, 2025

Uh oh!

hmellor commented Jul 21, 2025

Uh oh!

adarshxs commented Jul 21, 2025

Uh oh!

hmellor commented Jul 21, 2025 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Add Arcee (AFM) model support to vLLM #21267

Add Arcee (AFM) model support to vLLM #21267

Conversation

alyosha-swamy commented Jul 20, 2025 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

[New Model] Support Arcee (Arcee Foundational Models)

1. Purpose (Why this PR?)

2. Model details

3. Implementation overview

4. Performance / sanity check

5. Test plan ✔️

6. Documentation

Checklist

Notes for reviewers

Test result

Uh oh!

github-actions bot commented Jul 20, 2025

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

gemini-code-assist bot Jul 20, 2025

Choose a reason for hiding this comment

Uh oh!

jeejeelee Jul 21, 2025

Choose a reason for hiding this comment

Uh oh!

jeejeelee commented Jul 21, 2025

Uh oh!

jeejeelee Jul 21, 2025

Choose a reason for hiding this comment

Uh oh!

jeejeelee Jul 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jeejeelee commented Jul 21, 2025

Uh oh!

jeejeelee commented Jul 21, 2025

Uh oh!

hmellor commented Jul 21, 2025

Uh oh!

adarshxs commented Jul 21, 2025

Uh oh!

hmellor commented Jul 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

alyosha-swamy commented Jul 20, 2025 •

edited by github-actions bot

Loading

jeejeelee Jul 21, 2025 •

edited

Loading

hmellor commented Jul 21, 2025 •

edited

Loading