[Feat] Unquantized linear nz support #2619

anon189Ty · 2025-08-28T14:34:36Z

What this PR does / why we need it?

Currently, when executing to the Linear layer of the model in vLLM-Ascend, the weights input format is ND in unquantized case and skipped ascend case, which is slower than FRACTAL_NZ.
This PR supplements the execution logic for Linear layer. When VLLM_ASCEND_ENABLE_MLP_OPTIMIZE=1 and CANN version is 8.3, the weights of the Linear layer will be converted to FRACTAL_NZ, in both unquantized case and skipped ascend case.

Does this PR introduce any user-facing change?

How was this patch tested?

vLLM version: main
vLLM main: vllm-project/vllm@267c80d

github-actions · 2025-08-28T14:34:44Z

👋 Hi! Thank you for contributing to the vLLM Ascend project. The following points will speed up your PR merge:‌‌

A PR should do only one thing, smaller PRs enable faster reviews.
Every PR should include unit tests and end-to-end tests ‌to ensure it works and is not broken by other future PRs.
Write the commit message by fulfilling the PR description to help reviewer and future developers understand.

If CI fails, you can run linting and testing checks locally according Contributing and Testing.

gemini-code-assist

Code Review

This pull request introduces an optimization for unquantized linear layers on Ascend hardware by converting weights to the FRACTAL_NZ format for CANN 8.3. This is implemented through a new AscendUnquantizedLinearMethod. The changes correctly propagate this new method to various linear layers, including a new AscendQKVParallelLinear for QKV projections, and the tests are updated accordingly. However, I've found a couple of critical typos in type hints that would lead to NameErrors, and an indentation error in a test file that would cause a SyntaxError. These issues need to be addressed before merging.

gemini-code-assist · 2025-08-28T14:35:40Z

vllm_ascend/ops/linear.py

+            self.quant_method: Qptional[
+                QuantizeMethodBase] = AscendUnquantizedLinearMethod()


There is a typo in the type hint. Qptional should be Optional. This will cause a NameError at runtime as Qptional is not defined.

Suggested change

self.quant_method: Qptional[

QuantizeMethodBase] = AscendUnquantizedLinearMethod()

self.quant_method: Optional[

QuantizeMethodBase] = AscendUnquantizedLinearMethod()

gemini-code-assist · 2025-08-28T14:35:41Z

vllm_ascend/ops/linear.py

+            self.quant_method: Qptional[
+                QuantizeMethodBase] = AscendUnquantizedLinearMethod()


There is a typo in the type hint. Qptional should be Optional. This will cause a NameError at runtime as Qptional is not defined.

Suggested change

self.quant_method: Qptional[

QuantizeMethodBase] = AscendUnquantizedLinearMethod()

self.quant_method: Optional[

QuantizeMethodBase] = AscendUnquantizedLinearMethod()

gemini-code-assist · 2025-08-28T14:35:41Z

tests/ut/ops/test_linear.py

+            expect_data = torch_npu.npu_format_cast(
+                expect_data, ACL_FORMAT_FRACTAL_NZ)
+        self.assertTrue(torch.equal(layer.weight.data, expect_data))
+


This line contains only whitespace, but it's indented, which will cause an IndentationError. Please remove this line.

github-actions · 2025-08-29T03:47:50Z

This pull request has conflicts, please resolve those before we can evaluate the pull request.

momo609 · 2025-08-29T08:16:22Z

vllm_ascend/ops/linear.py

+
+    def process_weights_after_loading(self, layer: torch.nn.Module) -> None:
+        super().process_weights_after_loading(layer)
+        if torch.version.cann.startswith("8.3"):


Is a cudagraph check necessary?

github-actions · 2025-08-29T15:26:02Z

This pull request has conflicts, please resolve those before we can evaluate the pull request.

github-actions · 2025-09-07T02:35:54Z

This pull request has conflicts, please resolve those before we can evaluate the pull request.

github-actions · 2025-09-08T14:59:20Z

This pull request has conflicts, please resolve those before we can evaluate the pull request.

Signed-off-by: anon189Ty <Stari_Falcon@outlook.com>

yiz-liu · 2025-09-11T03:08:03Z

@MengqingCao All clear, except for some known issues which is not introduced by this PR, please merge this ASAP.

### What this PR does / why we need it? Currently, when executing to the Linear layer of the model in vLLM-Ascend, the weights input format is ND in unquantized case and skipped ascend case, which is slower than FRACTAL_NZ. This PR supplements the execution logic for Linear layer. When VLLM_ASCEND_ENABLE_MLP_OPTIMIZE=1 and CANN version is 8.3, the weights of the Linear layer will be converted to FRACTAL_NZ, in both unquantized case and skipped ascend case. - vLLM version: main - vLLM main: vllm-project/vllm@267c80d Signed-off-by: anon189Ty <Stari_Falcon@outlook.com> Signed-off-by: Yizhou Liu <liu_yizhou@outlook.com>

This reverts commit 7b2ecc1.

This reverts commit 7b2ecc1. Signed-off-by: Yikun Jiang <yikunkero@gmail.com>

This reverts commit 7b2ecc1. Signed-off-by: anon189Ty <Stari_Falcon@outlook.com>

### What this PR does / why we need it? This reverts commit 7b2ecc1. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? CI passed - vLLM version: main - vLLM main: vllm-project/vllm@64d90c3 Closes: #2890 Closes: #2887 Closes: #2885 Signed-off-by: Yikun Jiang <yikunkero@gmail.com>

### What this PR does / why we need it? Currently, when executing to the Linear layer of the model in vLLM-Ascend, the weights input format is ND in unquantized case and skipped ascend case, which is slower than FRACTAL_NZ. This PR supplements the execution logic for Linear layer. When VLLM_ASCEND_ENABLE_MLP_OPTIMIZE=1 and CANN version is 8.3, the weights of the Linear layer will be converted to FRACTAL_NZ, in both unquantized case and skipped ascend case. - vLLM version: main - vLLM main: vllm-project/vllm@267c80d Signed-off-by: anon189Ty <Stari_Falcon@outlook.com> Signed-off-by: offline0806 <z00858301@china.huawei.com>

…lm-project#2896) ### What this PR does / why we need it? This reverts commit 7b2ecc1. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? CI passed - vLLM version: main - vLLM main: vllm-project/vllm@64d90c3 Closes: vllm-project#2890 Closes: vllm-project#2887 Closes: vllm-project#2885 Signed-off-by: Yikun Jiang <yikunkero@gmail.com> Signed-off-by: offline0806 <z00858301@china.huawei.com>

### What this PR does / why we need it? Currently, when executing to the Linear layer of the model in vLLM-Ascend, the weights input format is ND in unquantized case and skipped ascend case, which is slower than FRACTAL_NZ. This PR supplements the execution logic for Linear layer. When VLLM_ASCEND_ENABLE_MLP_OPTIMIZE=1 and CANN version is 8.3, the weights of the Linear layer will be converted to FRACTAL_NZ, in both unquantized case and skipped ascend case. - vLLM version: main - vLLM main: vllm-project/vllm@267c80d Signed-off-by: anon189Ty <Stari_Falcon@outlook.com>

…lm-project#2896) ### What this PR does / why we need it? This reverts commit 7b2ecc1. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? CI passed - vLLM version: main - vLLM main: vllm-project/vllm@64d90c3 Closes: vllm-project#2890 Closes: vllm-project#2887 Closes: vllm-project#2885 Signed-off-by: Yikun Jiang <yikunkero@gmail.com>

### What this PR does / why we need it? Currently, when executing to the Linear layer of the model in vLLM-Ascend, the weights input format is ND in unquantized case and skipped ascend case, which is slower than FRACTAL_NZ. This PR supplements the execution logic for Linear layer. When VLLM_ASCEND_ENABLE_MLP_OPTIMIZE=1 and CANN version is 8.3, the weights of the Linear layer will be converted to FRACTAL_NZ, in both unquantized case and skipped ascend case. - vLLM version: main - vLLM main: vllm-project/vllm@267c80d Signed-off-by: anon189Ty <Stari_Falcon@outlook.com>

…lm-project#2896) ### What this PR does / why we need it? This reverts commit 7b2ecc1. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? CI passed - vLLM version: main - vLLM main: vllm-project/vllm@64d90c3 Closes: vllm-project#2890 Closes: vllm-project#2887 Closes: vllm-project#2885 Signed-off-by: Yikun Jiang <yikunkero@gmail.com>

github-actions bot added module:tests module:ops module:core module:quantization labels Aug 28, 2025

gemini-code-assist bot reviewed Aug 28, 2025

View reviewed changes

github-actions bot added merge-conflicts and removed merge-conflicts labels Aug 29, 2025

anon189Ty changed the title ~~Unquantized linear nz support~~ [Feat] Unquantized linear nz support Aug 29, 2025

momo609 reviewed Aug 29, 2025

View reviewed changes

wangxiyuan approved these changes Aug 29, 2025

View reviewed changes

anon189Ty force-pushed the unquantized_linear_nz_support branch from 2ac497c to 24773fd Compare August 29, 2025 15:25

github-actions bot added the merge-conflicts label Aug 29, 2025

anon189Ty force-pushed the unquantized_linear_nz_support branch from 24773fd to 11e14dd Compare August 29, 2025 16:02

github-actions bot removed the merge-conflicts label Aug 29, 2025

anon189Ty force-pushed the unquantized_linear_nz_support branch 13 times, most recently from 864f9ae to c30eed9 Compare August 31, 2025 12:56

anon189Ty force-pushed the unquantized_linear_nz_support branch 4 times, most recently from e5f110f to 66e6579 Compare September 3, 2025 02:26

github-actions bot added merge-conflicts and removed merge-conflicts labels Sep 7, 2025

github-actions bot removed merge-conflicts module:core labels Sep 9, 2025

[Feat]Unquant NZ support

0bb2a32

Signed-off-by: anon189Ty <Stari_Falcon@outlook.com>

anon189Ty force-pushed the unquantized_linear_nz_support branch from 53fdccd to 0bb2a32 Compare September 10, 2025 07:02

MengqingCao added ready read for review ready-for-test start test by label for PR labels Sep 11, 2025

MengqingCao approved these changes Sep 11, 2025

View reviewed changes

MengqingCao merged commit 7b2ecc1 into vllm-project:main Sep 11, 2025
34 of 35 checks passed

Yikun added a commit to Yikun/vllm-ascend that referenced this pull request Sep 12, 2025

Revert "[Feat] Unquantized linear nz support (vllm-project#2619)"

0b00b9a

This reverts commit 7b2ecc1.

Yikun added a commit to Yikun/vllm-ascend that referenced this pull request Sep 12, 2025

Revert "[Feat] Unquantized linear nz support (vllm-project#2619)"

cea2888

This reverts commit 7b2ecc1. Signed-off-by: Yikun Jiang <yikunkero@gmail.com>

anon189Ty added a commit to anon189Ty/vllm-ascend that referenced this pull request Sep 12, 2025

Revert "[Feat] Unquantized linear nz support (vllm-project#2619)"

b3db04e

This reverts commit 7b2ecc1. Signed-off-by: anon189Ty <Stari_Falcon@outlook.com>

Yikun mentioned this pull request Sep 20, 2025

[Bug]: Remove outofdate commits to improve perf test #3051

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[Feat] Unquantized linear nz support #2619

[Feat] Unquantized linear nz support #2619

Uh oh!

anon189Ty commented Aug 28, 2025 •

edited by github-actions bot

Loading

Uh oh!

github-actions bot commented Aug 28, 2025

Uh oh!

gemini-code-assist bot left a comment

Uh oh!

gemini-code-assist bot Aug 28, 2025

Uh oh!

gemini-code-assist bot Aug 28, 2025

Uh oh!

gemini-code-assist bot Aug 28, 2025

Uh oh!

github-actions bot commented Aug 29, 2025

Uh oh!

momo609 Aug 29, 2025

Uh oh!

github-actions bot commented Aug 29, 2025

Uh oh!

github-actions bot commented Sep 7, 2025

Uh oh!

github-actions bot commented Sep 8, 2025

Uh oh!

yiz-liu commented Sep 11, 2025

Uh oh!

Uh oh!

Uh oh!

		self.quant_method: Qptional[
		QuantizeMethodBase] = AscendUnquantizedLinearMethod()

[Feat] Unquantized linear nz support #2619

[Feat] Unquantized linear nz support #2619

Uh oh!

Conversation

anon189Ty commented Aug 28, 2025 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What this PR does / why we need it?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

github-actions bot commented Aug 28, 2025

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

gemini-code-assist bot Aug 28, 2025

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist bot Aug 28, 2025

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist bot Aug 28, 2025

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Aug 29, 2025

Uh oh!

momo609 Aug 29, 2025

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Aug 29, 2025

Uh oh!

github-actions bot commented Sep 7, 2025

Uh oh!

github-actions bot commented Sep 8, 2025

Uh oh!

yiz-liu commented Sep 11, 2025

Uh oh!

Uh oh!

Uh oh!

anon189Ty commented Aug 28, 2025 •

edited by github-actions bot

Loading