[AMD] dsv4-fp4-mi355x-atom: enable DPA at high concurrency, update image to atom0.1.4 by seungrokj · Pull Request #1717 · SemiAnalysisAI/InferenceX

seungrokj · 2026-06-12T12:41:18Z

Summary

Update image to rocm/atom:rocm7.2.4_ubuntu24.04_py3.12_pytorch_release_2.10.0_atom0.1.4_20260612
Enable on top of DPA+TP8 at high concurrency:
- ISL=1024 / OSL=1024
- ISL=8192 / OSL=1024
Update ISL=8192 search-space: TP8-only conc=4–64, DPA conc=128–1024 (previously conc=1–64 and DPA conc=64–512)
Add perf-changelog entry

Motivation

Based on Pareto frontier analysis from ATOM runs 27367309656 and 27030375093,

Test plan

Verify bash -n syntax check passes on dsv4_fp4_mi355x_atom.sh

🤖 Generated with Claude Code

Note

Medium Risk
Larger concurrency sweeps and new serving flags (TBO, index cache, higher mem util) can shift stability/OOM behavior on MI355X; the dev nightly image pin adds release drift versus the changelog’s atom0.1.4 note.

Overview
Updates dsv4-fp4-mi355x-atom to drop the day-0 “CONC=1 only” guard and move the container image to rocm/atom-dev:nightly_202606161823. The fixed-seq-len search space now sweeps higher concurrency with DPA through conc 2048 on 1k/1k, and on 8k/1k adds explicit TP4/TP8 conc lists plus DPA from conc 128–2048 (replacing the narrower TP8 + DPA 64–512 grid).

dsv4_fp4_mi355x_atom.sh turns on DPA+TBO (--enable-tbo, GPU_MAX_HW_QUEUES=5) at high concurrency for those ISL/OS pairs, raises static GPU memory to 0.9, passes CUDA graph capture sizes, disables prefix caching, and adds use_index_cache / index_topk_freq HF overrides. Eval-only runs can set EVAL_MAX_MODEL_LEN via compute_eval_context_length.

perf-changelog.yaml documents the config change (note: its image string references atom0.1.4, which does not match the atom-dev nightly pin in the YAML diff).

^{Reviewed by Cursor Bugbot for commit 4364ef9. Bugbot is set up for automated code reviews on this repo. Configure here.}

…e image to atom0.1.4 - Enable --enable-tbo for ISL=1024/OSL=1024 at CONC>=1024 and ISL=8192/OSL=1024 at CONC>=256 - Update image to atom0.1.4_20260612 - Update ISL=8192 search-space to start at conc=4 and use DPA from conc=128 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

github-actions · 2026-06-12T12:41:28Z

Thanks for the contribution! For vLLM & SGLang, please ensure that your recipes is similar to the official vLLM recipes and/or the SGLang cookbook

If it is not, please create a PR first before we can merge your single node PR into the master branch. Let's ensure that the documentation is first class such that the entire ML community can benefit from your hard work! Thank you

PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. If re-running failed jobs is attempted, PR authors are responsible for ensuring it passes. See GitHub's docs on re-running failed jobs: https://docs.github.com/en/actions/how-tos/manage-workflow-runs/re-run-workflows-and-jobs#re-running-failed-jobs-in-a-workflow

As a rule of thumb, generally, PR authors should request a review & get a PR approval from the respective companies' CODEOWNERS before requesting a review from core maintainers.

If additional help is needed, PR authors can reach out to core maintainers over Slack.

github-actions · 2026-06-12T12:41:28Z

Thanks for the contribution! For vLLM & SGLang, please ensure that your recipes is similar to the official vLLM recipes and/or the SGLang cookbook

If it is not, please create a PR first before we can merge your single node PR into the master branch. Let's ensure that the documentation is first class such that the entire ML community can benefit from your hard work! Thank you

PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. If re-running failed jobs is attempted, PR authors are responsible for ensuring it passes. See GitHub's docs on re-running failed jobs: https://docs.github.com/en/actions/how-tos/manage-workflow-runs/re-run-workflows-and-jobs#re-running-failed-jobs-in-a-workflow

As a rule of thumb, generally, PR authors should request a review & get a PR approval from the respective companies' CODEOWNERS before requesting a review from core maintainers.

If additional help is needed, PR authors can reach out to core maintainers over Slack.

github-actions · 2026-06-12T12:41:28Z

Thanks for the contribution! For vLLM & SGLang, please ensure that your recipes is similar to the official vLLM recipes and/or the SGLang cookbook

If it is not, please create a PR first before we can merge your single node PR into the master branch. Let's ensure that the documentation is first class such that the entire ML community can benefit from your hard work! Thank you

PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. If re-running failed jobs is attempted, PR authors are responsible for ensuring it passes. See GitHub's docs on re-running failed jobs: https://docs.github.com/en/actions/how-tos/manage-workflow-runs/re-run-workflows-and-jobs#re-running-failed-jobs-in-a-workflow

As a rule of thumb, generally, PR authors should request a review & get a PR approval from the respective companies' CODEOWNERS before requesting a review from core maintainers.

If additional help is needed, PR authors can reach out to core maintainers over Slack.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

github-actions · 2026-06-12T12:59:36Z

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=27416363770
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=27416363770

…onc range - Pass --max-model-len to server using SERVE_MAX_MODEL_LEN - Add EVAL_ONLY path: compute eval context length via compute_eval_context_length - Extend conc-end to 8192 (isl=1024) and 4096 (isl=8192) in amd-master.yaml Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

github-actions · 2026-06-12T13:18:19Z

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=27417198968
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=27417198968

github-actions · 2026-06-12T14:23:22Z

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=27418122192
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=27418122192

github-actions · 2026-06-12T17:08:40Z

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=27418122192
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=27418122192

…sable max-model-len Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…m-seqs Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

functionstackx · 2026-06-13T16:20:20Z

can u rebase it so the sweep triggers?

…e, enable max-model-len Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

… remove stale entries Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

github-actions · 2026-06-16T16:44:12Z

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=27612720147
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=27612720147

github-actions · 2026-06-16T18:28:49Z

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=27612720147
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=27612720147

…x mem frac Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…add TP4/TP8 conc lists for ISL8192 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

cursor

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 421313c. Configure here.}

github-actions · 2026-06-17T05:55:49Z

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=27668862004
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=27668862004

github-actions · 2026-06-17T05:57:19Z

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=27668923188
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=27668923188

github-actions · 2026-06-17T05:58:57Z

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=27668950324
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=27668950324

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

github-actions · 2026-06-17T06:01:45Z

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=27669004948
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=27669004948

github-actions · 2026-06-17T08:20:34Z

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=27669101609
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=27669101609

…nch override Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

github-actions · 2026-06-17T08:25:14Z

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=27675672700
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=27675672700

…/TP8/DPA Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

github-actions · 2026-06-17T11:21:40Z

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=27676739575
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=27676739575

github-actions · 2026-06-17T11:56:21Z

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=27676739575
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=27676739575

seungrokj · 2026-06-17T13:02:17Z

@functionstackx @cquil11 can you please merge this ?

chunfangamd

LGTM

seungrokj · 2026-06-17T13:06:23Z

/reuse-sweep-run

seungrokj · 2026-06-17T13:06:26Z

/merge-prs

seungrokj and others added 2 commits June 12, 2026 21:38

[AMD] perf-changelog: dsv4-fp4-mi355x-atom DPA TBO + image atom0.1.4

c566e28

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

seungrokj requested a review from a team June 12, 2026 12:41

seungrokj requested review from 1am9trash, billishyahao, chunfangamd and yctseng0211 as code owners June 12, 2026 12:41

github-project-automation Bot added this to InferenceMAX Board Jun 12, 2026

[AMD] perf-changelog: add PR link #1717

7e1aa06

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

seungrokj added AMD full-sweep-enabled labels Jun 12, 2026

[AMD] dsv4_fp4_mi355x_atom.sh: disable prefix caching

65e0fa3

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

seungrokj removed the full-sweep-enabled label Jun 13, 2026

[AMD] dsv4-fp4-mi355x-atom: narrow eval to single conc=1024 point, di…

c3b3289

…sable max-model-len Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

cursor Bot reviewed Jun 13, 2026

View reviewed changes

Comment thread .github/configs/amd-master.yaml Outdated

Comment thread benchmarks/single_node/fixed_seq_len/dsv4_fp4_mi355x_atom.sh Outdated

[AMD] dsv4_fp4_mi355x_atom.sh: add cudagraph-capture-sizes and max-nu…

7ffa976

…m-seqs Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

cursor Bot reviewed Jun 13, 2026

View reviewed changes

Comment thread benchmarks/single_node/fixed_seq_len/dsv4_fp4_mi355x_atom.sh

[AMD] dsv4-fp4-mi355x-atom: bump to nightly image, expand search spac…

f2677b2

…e, enable max-model-len Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

cursor Bot reviewed Jun 15, 2026

View reviewed changes

Comment thread .github/configs/amd-master.yaml Outdated

[AMD] set GPU_MAX_HW_QUEUES=5 in dsv4_fp4_mi355x_atom.sh

f5f0d66

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

seungrokj changed the title ~~[AMD] dsv4-fp4-mi355x-atom: enable DPA TBO at high concurrency, update image to atom0.1.4~~ [AMD] dsv4-fp4-mi355x-atom: enable DPA at high concurrency, update image to atom0.1.4 Jun 15, 2026

seungrokj and others added 3 commits June 16, 2026 19:53

[AMD] perf-changelog: revert dsv4-fp4-mi355x-atom image/search-space,…

f56f877

… remove stale entries Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Merge branch 'main' into amd/dsv4_atom_0612

f7c9de8

[AMD] perf-changelog: add dsv4-fp4-mi355x-sglang entry for PR #1762

a4828cb

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

seungrokj and others added 4 commits June 17, 2026 13:26

update dsv4-fp4-mi355x-atom: bump image, enable TBO conditionally, fi…

19b8757

…x mem frac Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

expand dsv4-fp4-mi355x-atom search space: restore ISL1024 scenarios, …

03aaa6b

…add TP4/TP8 conc lists for ISL8192 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Merge branch 'main' into amd/dsv4_atom_0612

cf3962f

Update perf-changelog.yaml

421313c

cursor Bot reviewed Jun 17, 2026

View reviewed changes

Comment thread benchmarks/single_node/fixed_seq_len/dsv4_fp4_mi355x_atom.sh

Update perf-changelog.yaml

ae77233

Update perf-changelog.yaml

a8f6bd0

Update perf-changelog.yaml

5fbd068

update perf-changelog: move dsv4-fp4-mi355x-atom entry to end

d080faa

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

narrow dsv4-fp4-mi355x-atom to DPA conc=256-2048 ISL8192, fix TBO bra…

91f6277

…nch override Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

seungrokj removed the full-sweep-enabled label Jun 17, 2026

restore full dsv4-fp4-mi355x-atom search space: ISL1024 + ISL8192 TP4…

4364ef9

…/TP8/DPA Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

seungrokj added the full-sweep-enabled label Jun 17, 2026

chunfangamd approved these changes Jun 17, 2026

View reviewed changes

Conversation

seungrokj commented Jun 12, 2026 • edited by cursor Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Motivation

Test plan

Uh oh!

github-actions Bot commented Jun 12, 2026

Uh oh!

github-actions Bot commented Jun 12, 2026

Uh oh!

github-actions Bot commented Jun 12, 2026

Uh oh!

github-actions Bot commented Jun 12, 2026

Uh oh!

github-actions Bot commented Jun 12, 2026

Uh oh!

github-actions Bot commented Jun 12, 2026

Uh oh!

github-actions Bot commented Jun 12, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

functionstackx commented Jun 13, 2026

Uh oh!

Uh oh!

github-actions Bot commented Jun 16, 2026

Uh oh!

github-actions Bot commented Jun 16, 2026

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

github-actions Bot commented Jun 17, 2026

Uh oh!

github-actions Bot commented Jun 17, 2026

Uh oh!

github-actions Bot commented Jun 17, 2026

Uh oh!

github-actions Bot commented Jun 17, 2026

Uh oh!

github-actions Bot commented Jun 17, 2026

Uh oh!

github-actions Bot commented Jun 17, 2026

Uh oh!

github-actions Bot commented Jun 17, 2026

Uh oh!

github-actions Bot commented Jun 17, 2026

Uh oh!

seungrokj commented Jun 17, 2026

Uh oh!

chunfangamd left a comment

Choose a reason for hiding this comment

Uh oh!

seungrokj commented Jun 17, 2026

Uh oh!

seungrokj commented Jun 17, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

seungrokj commented Jun 12, 2026 •

edited by cursor Bot

Loading