Skip to content

Fixes for dp + ep + tp combinations #78

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 171 commits into
base: modular-fused-experts
Choose a base branch
from
Open

Fixes for dp + ep + tp combinations #78

wants to merge 171 commits into from

Conversation

varun-sundar-rabindranath
Copy link

@varun-sundar-rabindranath varun-sundar-rabindranath commented May 6, 2025 •
edited by github-actions bot
Loading

vllm server command :
vllm serve ${model}
--enforce-eager
--trust-remote-code
--tensor-parallel-size ${tp_size}
--data-parallel-size ${dp_size}
${EP_ARGS}
--no-enable-prefix-caching
--port ${server_port}

lm_eval command :
lm_eval --model local-completions
--tasks gsm8k
--model_args model=deepseek-ai/DeepSeek-V2-Lite,base_url=http://127.0.0.1:${SERVER_PORT}/v1/completions,num_concurrent=5,max_retries=3,tokenized_requests=False
--limit 100

verified correctness using lm-eval for the following combinations:

  • DP=1 TP=1 EP=False
  • DP=1 TP=2 EP=False
  • DP=2 TP=1 EP=False
  • DP=2 TP=2 EP=False
  • DP=2 TP=1 EP=True
  • DP=2 TP=2 EP=True ## Works only with VLLM_MLA_DISABLE=1

bnellnm added 30 commits April 30, 2025 16:53
@bnellnm
moe refactoring
9a6ee6b 
Signed-off-by: Bill Nell <bnell@redhat.com>
@bnellnm
module deepgemm moe working
82188dd 
Signed-off-by: Bill Nell <bnell@redhat.com>
@bnellnm
working deep gemm, wip cutlass
fea4fbf 
Signed-off-by: Bill Nell <bnell@redhat.com>
@bnellnm
working cutlass
d78f1c8 
Signed-off-by: Bill Nell <bnell@redhat.com>
@bnellnm
deepgemm working again
8e3e6a9 
Signed-off-by: Bill Nell <bnell@redhat.com>
@bnellnm
cutlass working again
e41e4bf 
Signed-off-by: Bill Nell <bnell@redhat.com>
@bnellnm
cutlass working again
0b877ba 
Signed-off-by: Bill Nell <bnell@redhat.com>
@bnellnm
fix inplace, format and name cleanups
5d76ee9 
Signed-off-by: Bill Nell <bnell@redhat.com>
@bnellnm
fix inplace, format + name cleanups
c2ce01a 
Signed-off-by: Bill Nell <bnell@redhat.com>
@bnellnm
test improvements
b52b50d 
Signed-off-by: Bill Nell <bnell@redhat.com>
@bnellnm
make modular triton classes, fix edge cases
49a9d11 
Signed-off-by: Bill Nell <bnell@redhat.com>
@bnellnm
fix outplace bug
bcac19a 
Signed-off-by: Bill Nell <bnell@redhat.com>
@bnellnm
refactor dispatch/combine stuff
e2ab4f5 
Signed-off-by: Bill Nell <bnell@redhat.com>
@bnellnm
initial pplx dispatch/combine class
ecaca4e 
Signed-off-by: Bill Nell <bnell@redhat.com>
@bnellnm
merge triton dispatch into standard, add some comments
9bcbde0 
Signed-off-by: Bill Nell <bnell@redhat.com>
@bnellnm
format
73847e0 
Signed-off-by: Bill Nell <bnell@redhat.com>
@bnellnm
comments
b136032 
Signed-off-by: Bill Nell <bnell@redhat.com>
@bnellnm
fix linter
62584bf 
Signed-off-by: Bill Nell <bnell@redhat.com>
@bnellnm
fix more linter stuff
cbdc471 
Signed-off-by: Bill Nell <bnell@redhat.com>
@bnellnm
cleanup for review
8e2c5b2 
Signed-off-by: Bill Nell <bnell@redhat.com>
@bnellnm
review comments
cef98ab 
Signed-off-by: Bill Nell <bnell@redhat.com>
@bnellnm
forgot return
13da7ea 
Signed-off-by: Bill Nell <bnell@redhat.com>
@bnellnm
add dp_rank_num_tokens to DPMetadata
fb39d50 
Signed-off-by: Bill Nell <bnell@redhat.com>
@bnellnm
better check for fp8 in _fp8_permute
9bac87a 
Signed-off-by: Bill Nell <bnell@redhat.com>
@bnellnm
updates
9882f97 
Signed-off-by: Bill Nell <bnell@redhat.com>
@bnellnm
fix merge issues
cfcdb70 
Signed-off-by: Bill Nell <bnell@redhat.com>
@bnellnm
fix lint
4664e0f 
Signed-off-by: Bill Nell <bnell@redhat.com>
@bnellnm
add pplx tests
42f12d7 
Signed-off-by: Bill Nell <bnell@redhat.com>
@bnellnm
lint
dc0a640 
Signed-off-by: Bill Nell <bnell@redhat.com>
@bnellnm
undo random lint changes
64acde9 
Signed-off-by: Bill Nell <bnell@redhat.com>
bnellnm and others added 17 commits April 30, 2025 16:53
@bnellnm
fixes
be22c57 
Signed-off-by: Bill Nell <bnell@redhat.com>
@bnellnm
pplx update
9018df8 
Signed-off-by: Bill Nell <bnell@redhat.com>
@bnellnm
varun's fixes
3433b73 
Signed-off-by: Bill Nell <bnell@redhat.com>
@bnellnm
varun's fixes
800dde1 
Signed-off-by: Bill Nell <bnell@redhat.com>
@bnellnm
tweak bound_m
918e62b 
Signed-off-by: Bill Nell <bnell@redhat.com>
@bnellnm
run linter
b6ae861 
Signed-off-by: Bill Nell <bnell@redhat.com>
@bnellnm
more lint stuff
448658a 
Signed-off-by: Bill Nell <bnell@redhat.com>
@bnellnm
add guards for pplx import
c7ddca4 
Signed-off-by: Bill Nell <bnell@redhat.com>
@bnellnm
fix forward_chunked
22b988a 
Signed-off-by: Varun Sundar Rabindranath <varun@neuralmagic.com>
Signed-off-by: Bill Nell <bnell@redhat.com>
@bnellnm
fix more lint
c09cefd 
Signed-off-by: Bill Nell <bnell@redhat.com>
@bnellnm
cleanups
938c516 
Signed-off-by: Bill Nell <bnell@redhat.com>
@bnellnm
cleanups + lint, layer.py wip
c0fc027 
Signed-off-by: Bill Nell <bnell@redhat.com>
@bnellnm
fix parallel_state lint
f74ab61 
Signed-off-by: Bill Nell <bnell@redhat.com>
@bnellnm
fix M=1 pplx test
3e8a0e3 
Signed-off-by: Bill Nell <bnell@redhat.com>
@bnellnm
fix M=1 pplx test
886045e 
Signed-off-by: Bill Nell <bnell@redhat.com>
@bnellnm
fix M=1 pplx test
5d960df 
Signed-off-by: Bill Nell <bnell@redhat.com>
fixes
1014679 
Signed-off-by: Varun Sundar Rabindranath <varun@neuralmagic.com>
@github-actions GitHub Actions
Copy link

github-actions bot commented May 6, 2025

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

varun-sundar-rabindranath
varun-sundar-rabindranath commented May 6, 2025
View reviewed changes
vllm/model_executor/layers/fused_moe/layer.py
@@ -50,6 +50,112 @@
MOE_DP_CHUNK_SIZE = 256


@dataclass
class FusedMoEParallelConfig:
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

move tp / dp / ep computation here and out of FusedMoe .

varun-sundar-rabindranath
varun-sundar-rabindranath commented May 6, 2025
View reviewed changes
vllm/model_executor/models/deepseek_v2.py
# With EP and the pplx kernels - this is no longer viable
# as all GPU ranks in DP, produce the complete set of hidden_states.
# Therefore reduce the shared experts early.
reduce_results=self.experts.must_reduce_shared_outputs(),
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

reduce_results must be True when using pplx dispatch combine.

varun-sundar-rabindranath
varun-sundar-rabindranath commented May 6, 2025
View reviewed changes
tests/kernels/moe/test_pplx_moe.py
@@ -325,8 +325,9 @@ def pplx_dispatch_combine(pgi, dp_size, a, topk_weight, topk_ids, num_experts):
ata,
max_num_tokens,
world_size,
dp_size,
rank,
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cosmetics - rearrange args

zero out attn outputs during profile run
ba8f478 
Signed-off-by: Varun Sundar Rabindranath <varun@neuralmagic.com>
@bnellnm bnellnm force-pushed the modular-fused-experts branch from 5d960df to b04e5d3 Compare May 7, 2025 15:24
@bnellnm bnellnm requested a review from mgoin as a code owner May 7, 2025 15:24
@bnellnm bnellnm force-pushed the modular-fused-experts branch 2 times, most recently from f5bcc22 to 5ba84d2 Compare May 9, 2025 16:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@tlrmchlsmth tlrmchlsmth Awaiting requested review from tlrmchlsmth tlrmchlsmth is a code owner

@robertgshaw2-redhat robertgshaw2-redhat Awaiting requested review from robertgshaw2-redhat robertgshaw2-redhat is a code owner

@njhill njhill Awaiting requested review from njhill njhill is a code owner

@alexm-redhat alexm-redhat Awaiting requested review from alexm-redhat alexm-redhat is a code owner

@mgoin mgoin Awaiting requested review from mgoin mgoin is a code owner

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@varun-sundar-rabindranath @bnellnm @tlrmchlsmth @ilmarkov @yma11