Skip to content

[BugFix] Fix dummy_run memory explosion in eager mode #3132

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Sep 25, 2025
+10 −0
Merged

[BugFix] Fix dummy_run memory explosion in eager mode #3132

Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
cae6215
[BugFix] Fixi single-operator dummy_run memory explosion
dragondream-chen Sep 23, 2025
5da2849
fix pre-commit
dragondream-chen Sep 23, 2025
6651347
tiny fix
dragondream-chen Sep 25, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions vllm_ascend/ops/moe/fused_moe_prepare_and_finalize.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -186,6 +186,12 @@ def finalize(self, hidden_states: torch.Tensor,
self.moe_config.tp_group.device_group)
hidden_states = torch.cat(self.split_hidden_states, dim=0)

# TODO: It is a quick bugfix for the single-operator memory explosion issue
# that requires further restructuring.
# If the cache is not cleared after `self.split_hidden_states` is created,
# it can lead to the single-operator memory explosion.
del self.split_hidden_states
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consider not assigning self.split_hidden_states during the prepare phase. Instead, use get_tp_group().allgather() in the finalize phase, which internally employs allgather_into_tensor to eliminate tensor move overhead.


# Unpad if necessary
if self.num_tokens < hidden_states.shape[0]:
hidden_states = hidden_states[:self.num_tokens]
Expand Down Expand Up @@ -270,6 +276,12 @@ def finalize(self, hidden_states: torch.Tensor,
self.moe_config.tp_group.device_group)
hidden_states = torch.cat(self.split_hidden_states, dim=0)

# TODO: It is a quick bugfix for the single-operator memory explosion issue
# that requires further restructuring.
# If the cache is not cleared after `self.split_hidden_states` is created,
# it can lead to the single-operator memory explosion.
del self.split_hidden_states
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

While this change correctly fixes the memory leak, I noticed that the finalize method in FusedMoEPrepareAndFinalizeWithAll2All is identical to the one in FusedMoEPrepareAndFinalizeWithMC2. This code duplication, including the bug fix you are applying, increases maintenance overhead and the risk of introducing inconsistencies in the future. Since you've already noted that further restructuring is required, I recommend abstracting this common finalize logic into a shared base class as part of that effort. This will improve maintainability.


if self.num_tokens < hidden_states.shape[0]:
hidden_states = hidden_states[:self.num_tokens]

Expand Down
Loading