Significant inference speed difference between vllm-ascend and MindIE

[Fly-Pluche]

Your current environment

Environment: Two machines, each with 8 * Ascend910B-64GB GPUs.

MindIE image: mindie_2.0.T18.B010-800I-A2-py3.11-openeuler24.03-lts-aarch64

Vllm-ascend image: 0.8.5rc1-torch_npu2.5.1-cann8.1.rc1-python3.10-oe2203lts

Model weights: DeepSeek-V3-0324-w8a8-modelers (https://modelers.cn/models/Modelers_Park/DeepSeek-V3-0324-w8a8/tree/main)

Describe the problem

When testing inference speeds using different inference frameworks.

Inference speeds observed:
vllm-ascend: Approximately 1.4 Tokens/s
MindIE: Approximately 14 Tokens/s

[Yikun]

Please try latest version, we will provide the E2E installation docs ASAP

[Yikun]

Please try v0.9.1-dev or main https://quay.io/repository/ascend/vllm-ascend?tab=tags&tag=latest

[Fly-Pluche]

Test in v0.9.1-dev with one request：
Total tokens: 24, Total time: 18.78s, infer speed: 1.28 tokens/s

CLI command:
vllm serve --model="/data2/nfsdata/DeepSeek-V3-0324-w8a8" --trust-remote-code --distributed_executor_backend "ray" --tensor-parallel-size 8 --pipeline-parallel-size 2 --disable-frontend-multiprocessing --port 12345 --quantization ascend --max-model-len 1024

[Yikun]

Recomand to use DP+TP+EP, you can refer to latest doc: https://vllm-ascend.readthedocs.io/en/latest/tutorials/multi_node.html , and the PP are still working in progress, see also: #1700 #1751 #1754

And here is a e2e test on DSv3 here: #1742 (comment)

[Fly-Pluche]

Thanks.
Solve the problem in v0.9.2rc1.