Skip to content

[WIP][Prefill Performance] Parallel Strategy Optimizations (VRAM-for-Speed Tradeoff) #1687

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 27 commits into
base: v0.9.1-dev
Choose a base branch
from
Open

[WIP][Prefill Performance] Parallel Strategy Optimizations (VRAM-for-Speed Tradeoff) #1687

wants to merge 27 commits into from
+70 −56

Conversation

SlightwindSec
Copy link

  1. Optimized MoE Expert All2All: Rewritten for better throughput with increased memory cost.
  2. Shared Expert Sharding Strategy Update: Switched from TP-aligned to pure DP for shared experts, enabling more efficient execution.
  3. O_Proj AllReduce → ReduceScatter: Reduced communication overhead by using ReduceScatter, made possible by pure DP sharding.
  4. AllGather Postponed: Delayed to after QKV down projection to reduce synchronization impact during prefill.

SlightwindSec and others added 3 commits July 9, 2025 10:51
@SlightwindSec
Prefill optimization draft
23a7889 
Signed-off-by: SlightwindSec <slightwindsec@gmail.com>
@SlightwindSec
fix mc2 mask
a175c70
qkv dp
687dd2d 
Signed-off-by: angazenn <zengyanjia@huawei.com>
@github-actions GitHub Actions
Copy link

github-actions bot commented Jul 9, 2025

This pull request has conflicts, please resolve those before we can evaluate the pull request.

SlightwindSec and others added 4 commits July 10, 2025 22:10
@SlightwindSec
fix mc2 mask
2fb03eb
fix qkv dp with mtp
925f65e 
Signed-off-by: angazenn <zengyanjia@huawei.com>
Merge remote-tracking branch 'upstream/v0.9.1-dev' into upstream_v0.9…
e8f9733 
….1-dev_dev0
resolve conficts with chunked mc2
2adcc11 
Signed-off-by: angazenn <zengyanjia@huawei.com>
angazenn and others added 2 commits July 14, 2025 20:54
resolve conficts with chunked mc2
9956a30 
Signed-off-by: angazenn <zengyanjia@huawei.com>
@SlightwindSec
feat: add flag to toggle prefill performance improvements
ae401bb 
Signed-off-by: SlightwindSec <slightwindsec@gmail.com>
SlightwindSec and others added 3 commits July 15, 2025 14:47
@SlightwindSec
remove unused import
0c7db37 
Signed-off-by: SlightwindSec <slightwindsec@gmail.com>
@kunpengW-code
[bugfix] prefill optimization support torchair graph mode
6029cde 
Signed-off-by: Wang Kunpeng <1289706727@qq.com>
async h2d
92994fb 
Signed-off-by: angazenn <zengyanjia@huawei.com>
@Angazenn Angazenn force-pushed the upstream_v0.9.1-dev_dev0 branch from e4f0f20 to 92994fb Compare July 17, 2025 01:50
angazenn and others added 3 commits July 17, 2025 10:05
fix unpadding
9cd37ca 
Signed-off-by: angazenn <zengyanjia@huawei.com>
fix unpadding
8024f73 
Signed-off-by: angazenn <zengyanjia@huawei.com>
@kunpengW-code
[prefill] access ATBMLAPrefill operator
d27702f 
Signed-off-by: Wang Kunpeng <1289706727@qq.com>
@github-actions GitHub Actions
Copy link

This pull request has conflicts, please resolve those before we can evaluate the pull request.

angazenn added 3 commits July 17, 2025 18:30
fix unpadding
7d43361 
Signed-off-by: angazenn <zengyanjia@huawei.com>
Merge remote-tracking branch 'upstream/v0.9.1-dev' into upstream_v0.9…
64183a2 
….1-dev_dev0
support eplb with prefill optimization
802fc2a 
Signed-off-by: angazenn <zengyanjia@huawei.com>
@ApsarasX ApsarasX mentioned this pull request Jul 21, 2025
Merged
@github-actions GitHub Actions
Copy link

This pull request has conflicts, please resolve those before we can evaluate the pull request.

@SlightwindSec
merge upstream_v0.9.1-dev
f581038 
Signed-off-by: SlightwindSec <slightwindsec@gmail.com>
@jianzs
Copy link
Collaborator

jianzs commented Jul 23, 2025

This pull request can't be merged. It's better to submit these features as separate pull requests. @Yikun @wangxiyuan @ganyi1996ppo

@github-actions GitHub Actions
Copy link

This pull request has conflicts, please resolve those before we can evaluate the pull request.

@kunpengW-code
Merge branch 'v0.9.1-dev' of https://github.com/vllm-project/vllm-ascend
427ee25 
 into upstream_v0.9.1-dev_dev0

# Conflicts:
#	vllm_ascend/ops/fused_moe.py
@kunpengW-code
[bugfix] prefill optimization support prefix_cache
3615190 
Signed-off-by: Wang Kunpeng <1289706727@qq.com>
@kunpengW-code
[bugfix] support floating
38e73c8 
Signed-off-by: Wang Kunpeng <1289706727@qq.com>
@github-actions GitHub Actions
Copy link

This pull request has conflicts, please resolve those before we can evaluate the pull request.

@SlightwindSec
fix lint
ed1aeee 
Signed-off-by: SlightwindSec <slightwindsec@gmail.com>
@SlightwindSec
fix lint
88c56fb 
Signed-off-by: SlightwindSec <slightwindsec@gmail.com>
@SlightwindSec
fix lint
b93e6a0 
Signed-off-by: SlightwindSec <slightwindsec@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
module:quantization
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@SlightwindSec @jianzs @kunpengW-code