[main] Fuse GroupedMatmul, Swiglu and DynamicQuant in `W8A8_DYNAMIC` quantized MoE layers #2275

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Merged

wangxiyuan merged 26 commits into vllm-project:main from zhoux77899:main_gmmswigluquant

Sep 4, 2025

Contributor

zhoux77899 commented Aug 8, 2025 •

edited by github-actions bot

Loading

What this PR does / why we need it?

Fuse GroupedMatmul, Swiglu and DynamicQuant into one fusion operation GroupedMatmulSwigluQuant.

extract common functions in w4a8_dynamic.py and w8a8_dynamic.py
if in supported occasion, use fusion operation npu_grouped_matmul_swiglu_quant

Does this PR introduce any user-facing change?

How was this patch tested?

Tested on W8A8 quantized Qwen3-235B-A22B model with bs=16

tp=8, dp=1, moe_tp=8, moe_ep=1, TPOP increased 21.54%, Output Token Throughput increased 27.35%

tp=8, dp=1, moe_tp=1, moe_ep=8, TPOP increased 17.38%, Output Token Throughput increased 6.86%

vLLM version: v0.10.1.1
vLLM main: vllm-project/vllm@6997a25

github-actions bot commented Aug 8, 2025

👋 Hi! Thank you for contributing to the vLLM Ascend project. The following points will speed up your PR merge:‌‌

A PR should do only one thing, smaller PRs enable faster reviews.
Every PR should include unit tests and end-to-end tests ‌to ensure it works and is not broken by other future PRs.
Write the commit message by fulfilling the PR description to help reviewer and future developers understand.

If CI fails, you can run linting and testing checks locally according Contributing and Testing.

github-actions bot added module:tests module:core module:quantization labels

codecov bot commented Aug 8, 2025 •

edited

Loading

Codecov Report

❌ Patch coverage is 96.52174% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 73.03%. Comparing base (24d4dad) to head (902b79a).
⚠️ Report is 20 commits behind head on main.

Files with missing lines	Patch %	Lines
vllm_ascend/ops/layers/moe_mlp.py	88.88%	3 Missing ⚠️
vllm_ascend/quantization/w8a8_dynamic.py	50.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2275      +/-   ##
==========================================
- Coverage   73.71%   73.03%   -0.69%     
==========================================
  Files         152      151       -1     
  Lines       21967    21533     -434     
==========================================
- Hits        16194    15726     -468     
- Misses       5773     5807      +34

Flag	Coverage Δ
unittests	`73.03% <96.52%> (-0.69%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

github-actions bot added module:tests merge-conflicts and removed module:tests module:core labels

github-actions bot commented Aug 11, 2025

This pull request has conflicts, please resolve those before we can evaluate the pull request.

github-actions bot removed the merge-conflicts label

ApsarasX reviewed

View reviewed changes

vllm_ascend/quantization/w8a8_dynamic.py Outdated Show resolved Hide resolved

vllm_ascend/quantization/w8a8_dynamic.py Outdated Show resolved Hide resolved

realliujiaxu reviewed

View reviewed changes

vllm_ascend/quantization/w8a8_dynamic.py Outdated Show resolved Hide resolved

github-actions bot added the merge-conflicts label

github-actions bot commented Aug 14, 2025

This pull request has conflicts, please resolve those before we can evaluate the pull request.

github-actions bot added module:ops module:core and removed merge-conflicts labels

zhoux77899 force-pushed the main_gmmswigluquant branch from a4bb618 to 4705afb Compare

August 14, 2025 06:58

github-actions bot removed module:ops module:core labels

zhoux77899 changed the title ~~[main] Support GroupedMatmulSwigluQuant in W8A8_DYNAMIC quantized MoE layers~~ [main] Fuse GroupedMatmul, Swiglu and DynamicQuant in W8A8_DYNAMIC quantized MoE layers

github-actions bot added the merge-conflicts label

github-actions bot commented Aug 20, 2025

This pull request has conflicts, please resolve those before we can evaluate the pull request.

github-actions bot removed the merge-conflicts label

weijinqian0 reviewed

View reviewed changes

vllm_ascend/quantization/w8a8_dynamic.py Outdated Show resolved Hide resolved

wangxiyuan approved these changes

View reviewed changes

Collaborator

wangxiyuan commented Aug 26, 2025

Sorry that after offline discuess, we'll merge #2500 first. You can then rebase then

github-actions bot commented Aug 27, 2025

This pull request has conflicts, please resolve those before we can evaluate the pull request.

github-actions bot added the merge-conflicts label

zhoux77899 and others added 7 commits

August 30, 2025 08:25


          Merge branch 'vllm-project:main' into main_gmmswigluquant

2be04b3


          Merge branch 'vllm-project:main' into main_gmmswigluquant

dde4126


          fix(ut): fix ut

e02305c

Signed-off-by: zhoux77899 <zhouxiang100@huawei.com>


          test(ci): trigger ci

af70693

Signed-off-by: zhoux77899 <zhouxiang100@huawei.com>


          Merge branch 'vllm-project:main' into main_gmmswigluquant

704c60d


          test(ci): trigger ci

34fdeae

Signed-off-by: zhoux77899 <zhouxiang100@huawei.com>


          test(ci): trigger ci

7c3cca9

Signed-off-by: zhoux77899 <zhouxiang100@huawei.com>

github-actions bot added the merge-conflicts label

github-actions bot commented Aug 30, 2025

This pull request has conflicts, please resolve those before we can evaluate the pull request.

zhoux77899 added 2 commits

August 30, 2025 23:04


          Merge branch 'main' into main_gmmswigluquant

83f9187

Signed-off-by: zhoux77899 <zhouxiang100@huawei.com>


          refactor(ops): adapt fused_moe refactoring

8dee31e

Signed-off-by: zhoux77899 <zhouxiang100@huawei.com>

github-actions bot removed the merge-conflicts label

zhoux77899 and others added 4 commits

August 30, 2025 23:30


          fix(ut): fix broken ut

1a5e16e

Signed-off-by: zhoux77899 <zhouxiang100@huawei.com>


          Merge branch 'vllm-project:main' into main_gmmswigluquant

538032e


          Merge branch 'vllm-project:main' into main_gmmswigluquant

e2b3f0e


          Merge branch 'vllm-project:main' into main_gmmswigluquant

cbdab8b

weijinqian0 reviewed

View reviewed changes

vllm_ascend/quantization/w8a8_dynamic.py Show resolved Hide resolved

weijinqian0 reviewed

View reviewed changes

vllm_ascend/quantization/w8a8_dynamic.py Show resolved Hide resolved

zhoux77899 and others added 5 commits

September 3, 2025 09:50


          Merge branch 'vllm-project:main' into main_gmmswigluquant

76a0bac


          fix(bug): remove unused torchair condition judgement

79f5935

Signed-off-by: zhoux77899 <zhouxiang100@huawei.com>


          fix(bug): restore w1_scale.dtype judgement

dc897d9

Signed-off-by: zhoux77899 <zhouxiang100@huawei.com>


          Merge branch 'vllm-project:main' into main_gmmswigluquant

6e58eb0


          Merge branch 'vllm-project:main' into main_gmmswigluquant

902b79a

wangxiyuan merged commit aff5189 into vllm-project:main

25 checks passed

zhoux77899 deleted the main_gmmswigluquant branch

September 4, 2025 03:46

Angazenn pushed a commit to Angazenn/vllm-ascend that referenced this pull request


          [main] Fuse GroupedMatmul, Swiglu and DynamicQuant in W8A8_DYNAMIC …

9546bc8

…quantized MoE layers (vllm-project#2275)

### What this PR does / why we need it?

Fuse `GroupedMatmul`, `Swiglu` and `DynamicQuant` into one fusion
operation `GroupedMatmulSwigluQuant`.

1. extract common functions in `w4a8_dynamic.py` and `w8a8_dynamic.py`
2. if in supported occasion, use fusion operation
`npu_grouped_matmul_swiglu_quant`

### Does this PR introduce _any_ user-facing change?

### How was this patch tested?

Tested on W8A8 quantized Qwen3-235B-A22B model with `bs=16`

1. `tp=8`, `dp=1`, `moe_tp=8`, `moe_ep=1`, TPOP increased 21.54%, Output
Token Throughput increased 27.35%
<img width="3443" height="211" alt="image"
src="https://github.com/user-attachments/assets/a1a9c14d-2310-41be-9a03-36125dabae6e"
/>

3. `tp=8`, `dp=1`, `moe_tp=1`, `moe_ep=8`, TPOP increased 17.38%, Output
Token Throughput increased 6.86%
<img width="3443" height="211" alt="image"
src="https://github.com/user-attachments/assets/1ce92e92-720d-40c0-8b4d-c493e5cb10a6"
/>


- vLLM version: v0.10.1.1
- vLLM main:
vllm-project/vllm@6997a25

---------

Signed-off-by: Ruri <33858552+zhoux77899@users.noreply.github.com>
Signed-off-by: zhoux77899 <zhouxiang100@huawei.com>

offline893 pushed a commit to offline893/vllm-ascend that referenced this pull request


          [main] Fuse GroupedMatmul, Swiglu and DynamicQuant in W8A8_DYNAMIC …

7f65a0c

…quantized MoE layers (vllm-project#2275)

### What this PR does / why we need it?

Fuse `GroupedMatmul`, `Swiglu` and `DynamicQuant` into one fusion
operation `GroupedMatmulSwigluQuant`.

1. extract common functions in `w4a8_dynamic.py` and `w8a8_dynamic.py`
2. if in supported occasion, use fusion operation
`npu_grouped_matmul_swiglu_quant`

### Does this PR introduce _any_ user-facing change?

### How was this patch tested?

Tested on W8A8 quantized Qwen3-235B-A22B model with `bs=16`

1. `tp=8`, `dp=1`, `moe_tp=8`, `moe_ep=1`, TPOP increased 21.54%, Output
Token Throughput increased 27.35%
<img width="3443" height="211" alt="image"
src="https://github.com/user-attachments/assets/a1a9c14d-2310-41be-9a03-36125dabae6e"
/>

3. `tp=8`, `dp=1`, `moe_tp=1`, `moe_ep=8`, TPOP increased 17.38%, Output
Token Throughput increased 6.86%
<img width="3443" height="211" alt="image"
src="https://github.com/user-attachments/assets/1ce92e92-720d-40c0-8b4d-c493e5cb10a6"
/>

- vLLM version: v0.10.1.1
- vLLM main:
vllm-project/vllm@6997a25

---------

Signed-off-by: Ruri <33858552+zhoux77899@users.noreply.github.com>
Signed-off-by: zhoux77899 <zhouxiang100@huawei.com>
Signed-off-by: offline0806 <z00858301@china.huawei.com>

Yikun mentioned this pull request

[Bug]: Remove outofdate commits to improve perf test #3051

Open

wangxiaoteng888 pushed a commit to LCAIZJ/vllm-ascend that referenced this pull request


          [main] Fuse GroupedMatmul, Swiglu and DynamicQuant in W8A8_DYNAMIC …

71a5c25

…quantized MoE layers (vllm-project#2275)

### What this PR does / why we need it?

Fuse `GroupedMatmul`, `Swiglu` and `DynamicQuant` into one fusion
operation `GroupedMatmulSwigluQuant`.

1. extract common functions in `w4a8_dynamic.py` and `w8a8_dynamic.py`
2. if in supported occasion, use fusion operation
`npu_grouped_matmul_swiglu_quant`

### Does this PR introduce _any_ user-facing change?

### How was this patch tested?

Tested on W8A8 quantized Qwen3-235B-A22B model with `bs=16`

1. `tp=8`, `dp=1`, `moe_tp=8`, `moe_ep=1`, TPOP increased 21.54%, Output
Token Throughput increased 27.35%
<img width="3443" height="211" alt="image"
src="https://github.com/user-attachments/assets/a1a9c14d-2310-41be-9a03-36125dabae6e"
/>

3. `tp=8`, `dp=1`, `moe_tp=1`, `moe_ep=8`, TPOP increased 17.38%, Output
Token Throughput increased 6.86%
<img width="3443" height="211" alt="image"
src="https://github.com/user-attachments/assets/1ce92e92-720d-40c0-8b4d-c493e5cb10a6"
/>


- vLLM version: v0.10.1.1
- vLLM main:
vllm-project/vllm@6997a25

---------

Signed-off-by: Ruri <33858552+zhoux77899@users.noreply.github.com>
Signed-off-by: zhoux77899 <zhouxiang100@huawei.com>

chopper0126 pushed a commit to chopper0126/vllm-ascend that referenced this pull request


          [main] Fuse GroupedMatmul, Swiglu and DynamicQuant in W8A8_DYNAMIC …

f042f21

…quantized MoE layers (vllm-project#2275)

### What this PR does / why we need it?

Fuse `GroupedMatmul`, `Swiglu` and `DynamicQuant` into one fusion
operation `GroupedMatmulSwigluQuant`.

1. extract common functions in `w4a8_dynamic.py` and `w8a8_dynamic.py`
2. if in supported occasion, use fusion operation
`npu_grouped_matmul_swiglu_quant`

### Does this PR introduce _any_ user-facing change?

### How was this patch tested?

Tested on W8A8 quantized Qwen3-235B-A22B model with `bs=16`

1. `tp=8`, `dp=1`, `moe_tp=8`, `moe_ep=1`, TPOP increased 21.54%, Output
Token Throughput increased 27.35%
<img width="3443" height="211" alt="image"
src="https://github.com/user-attachments/assets/a1a9c14d-2310-41be-9a03-36125dabae6e"
/>

3. `tp=8`, `dp=1`, `moe_tp=1`, `moe_ep=8`, TPOP increased 17.38%, Output
Token Throughput increased 6.86%
<img width="3443" height="211" alt="image"
src="https://github.com/user-attachments/assets/1ce92e92-720d-40c0-8b4d-c493e5cb10a6"
/>


- vLLM version: v0.10.1.1
- vLLM main:
vllm-project/vllm@6997a25

---------

Signed-off-by: Ruri <33858552+zhoux77899@users.noreply.github.com>
Signed-off-by: zhoux77899 <zhouxiang100@huawei.com>

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

module:ops module:quantization module:tests