Skip to content

[BugFix] Fix dummy_run memory explosion in eager mode #3132

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Sep 25, 2025
Merged

[BugFix] Fix dummy_run memory explosion in eager mode #3132

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
cae6215
[BugFix] Fixi single-operator dummy_run memory explosion
dragondream-chen Sep 23, 2025
5da2849
fix pre-commit
dragondream-chen Sep 23, 2025
6651347
tiny fix
dragondream-chen Sep 25, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions vllm_ascend/ops/moe/fused_moe_prepare_and_finalize.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -186,6 +186,11 @@ def finalize(self, hidden_states: torch.Tensor,
self.moe_config.tp_group.device_group)
hidden_states = torch.cat(self.split_hidden_states, dim=0)

# TODO: It is a quick bugfix for the memory explosion issue in eager mode.
# If the cache is not cleared after `self.split_hidden_states` is created,
# it can lead to the memory explosion in eager mode.
del self.split_hidden_states
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consider not assigning self.split_hidden_states during the prepare phase. Instead, use get_tp_group().allgather() in the finalize phase, which internally employs allgather_into_tensor to eliminate tensor move overhead.


# Unpad if necessary
if self.num_tokens < hidden_states.shape[0]:
hidden_states = hidden_states[:self.num_tokens]
Expand Down Expand Up @@ -270,6 +275,11 @@ def finalize(self, hidden_states: torch.Tensor,
self.moe_config.tp_group.device_group)
hidden_states = torch.cat(self.split_hidden_states, dim=0)

# TODO: It is a quick bugfix for the memory explosion issue in eager mode.
# If the cache is not cleared after `self.split_hidden_states` is created,
# it can lead to the memory explosion in eager mode.
del self.split_hidden_states

if self.num_tokens < hidden_states.shape[0]:
hidden_states = hidden_states[:self.num_tokens]

Expand Down
Loading