[AMD][AgentX] Add DeepSeek-V4-Pro FP4 MI355X ATOM MTP / 新增 DeepSeek-V4-Pro ATOM MTP - #2668
[AMD][AgentX] Add DeepSeek-V4-Pro FP4 MI355X ATOM MTP / 新增 DeepSeek-V4-Pro ATOM MTP#2668yhl-amd wants to merge 18 commits into
Conversation
Add the MI355X TP8 AgentX sweep for DeepSeek-V4-Pro on ATOM with the validated FP8 cache, prefix-caching, checkpointing, and three-token MTP settings. Use golden synthetic acceptance for throughput and real MTP acceptance for evaluation. 新增 DeepSeek-V4-Pro 在 MI355X TP8 ATOM 上的 AgentX sweep，采用已验证的 FP8 缓存、前缀缓存、状态检查点和三 token MTP 配置。吞吐测试使用 golden synthetic acceptance，评测使用真实 MTP acceptance。
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
Backfill the performance changelog with the submitted InferenceX pull request URL. 在性能变更日志中回填已提交的 InferenceX PR 链接。
|
Could a maintainer please approve the fork workflow run and add the 请有权限的 maintainer 批准 fork workflow run，并添加 |
|
Dispatched trusted sweep run for approved external revision 已为获批的外部提交 |
Use a 3600-second warmup grace period for the ATOM C32 and C48 saturation arms while keeping C1 through C16 at the shared 1800-second default.\n\n中文：将 ATOM C32 和 C48 饱和并发点的 warmup grace period 调整为 3600 秒，C1 至 C16 继续使用共享的 1800 秒默认值。
|
Updated the ATOM recipe only / 仅更新 ATOM 配方：
Validation / 验证：Bash syntax, YAML parse, exact-key 7-row matrix generation, New head / 新提交： Because the head SHA changed, please remove and re-add |
|
Dispatched trusted sweep run for approved external revision 已为获批的外部提交 |
seungrokj
added
evals-only
|
Dispatched trusted sweep run for approved external revision 已为获批的外部提交 |
|
Dispatched trusted sweep run for approved external revision 已为获批的外部提交 |
|
Dispatched trusted sweep run for approved external revision 已为获批的外部提交 |
|
Dispatched trusted sweep run for approved external revision 已为获批的外部提交 |
|
Dispatched trusted sweep run for approved external revision 已为获批的外部提交 |
There was a problem hiding this comment.
As a PR reviewer and CODEOWNER, I have reviewed this and have:
- Verified that as of the moment of typing this, this is the latest version of PR_REVIEW_CHECKLIST.md
- Verified that the general code quality meets the InferenceX standard and does not make the code quality any worse.
- Verified that this PR has passed PR validation. Please link to GitHub Action workflow that shows this.
- Verified that this PR passes evals. Please link to GitHub Action workflow that shows this.
- Verified that speculative decoding PRs uses chat templates to align the AL distribution to real world
- For agentic workloads: verified that speculative-decoding configs (EAGLE / MTP / draft models) run with simulated synthetic acceptance, with the acceptance-length value taken from the committed golden AL curve in golden_al_distribution/ for that model, thinking mode, and draft length. A submission may choose any supported draft length, but it may not substitute a different acceptance target.
- Verified against the current MODELS.md that this PR does not submit a deprecated model, scenario, or model-scenario combination.
- Verified that the model architecture isn't changed with benchmark hacks like using --hf-overrides to skipping indexer for every x layers on models that don't natively support this. As a general rule, we won't accept optimizations that reduces the number of model architecture FLOPs. Anything that makes that same computation run faster is fair game; FLOPs at lower precisions is fine, given that the config passes private evals. As an general north star princple, we should only use optimizations which is used in production by customers that care about accuracy
- If an company claims that they support vLLM/SGLang as first class LLM inference engines on their hardware, I have verified that the respective vLLM submission made using upstream https://hub.docker.com/u/vllm docker repo, upstream SGLang https://hub.docker.com/u/lmsysorg docker repo. The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet as supported by vLLM/SGLang community maintainers
- If an company claims that they support vLLM/SGLang as first class upstream in-tree LLM inference engines on their hardware, I have have verified that the respective vLLM/SGLang submission has been made before additional frameworks (TRT-LLM, ATOM, etc.). The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet.
- Verified that every single-node vLLM/SGLang recipe in this PR is documented in the official vLLM recipes and/or the SGLang cookbook:
- I linked the corresponding upstream PR in the vLLM recipe repo or SGLang repo and verified that it is MERGED before this InferenceX PR merges. An opened, draft, or closed-without-merge upstream PR does not satisfy this requirement. If the matching recipe was already published, I linked the published recipe/cookbook page in the additional detail section below.
- Verified that this PR does not patch the inference engine or serving stack — the pinned image must run as shipped. This covers .patch files / git apply / patch, inline patches embedded in benchmark scripts (e.g. a python3/sed heredoc that rewrites installed engine sources before serving), in-place edits of site-packages, monkey-patching, overwriting container files, and installing forked/rebuilt engine wheels on top of the pinned image. The only exception is a patch covered by a filled-out waiver at docs/waiver/
<PR_NUMBER>.md— named after the PR that introduces the patch and filed in that same PR, stating what is patched, why the unmodified upstream image cannot run this benchmark, the upstream PR/issue link, and the removal plan — which I have linked below in the additional detail section. - If this PR uses
append-only: true, verified that it only adds generated points or recipe variants inside a selected existing config/scenario and existing same-image visual curve: every previously generated point remains present with the same recipe, no prior point is removed or rerun, and every benchmark-affecting change in the complete diff can affect only the corresponding newly appended points (never an existing point), regardless of which file contains it. - If any of the above criteria cannot reasonably be satisfied, I have provided additional reasoning below.
Additional detail section:
- insert any additional info here
recipe at https://github.com/ROCm/ATOM/blob/main/recipes/DeepSeek-V4-Agentic-Benchmark.md
Signed: seungrokj
|
/reuse-sweep-run |
|
superseded by #2698 |
* [AMD][AgentX] Add DeepSeek-V4-Pro FP4 MI355X ATOM MTP Internalizes fork PR #2668 (yhl-amd) so full-sweep CI can run on it. Adds the MI355X DeepSeek-V4-Pro FP4 ATOM AgentX MTP recipe plus its amd-master.yaml config entry and perf-changelog entry. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Use ATOM --spec-decode-acceptance-length flag; bump image and pr-link Switch the MTP throughput arm to --spec-decode-acceptance-length 2.49 (ROCm/ATOM#1948) instead of the derived --spec-decode-acceptance-rate, bump the ATOM image to nightly_202608201032, and point the changelog pr-link at #2698. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
3 participants
English
Summary
dsv4-fp4-mi355x-atom-agentic-mtpfor DeepSeek-V4-Pro FP4 on one 8x MI355X node with ATOM, TP8, no KV offload, and AgentX concurrency[1, 2, 4, 8, 16, 32, 48].max-num-seqs = 2 * concurrency, FP8 KV/index caches, prefix caching, 32K state checkpoints, 16K batched-token/prefill-chunk limits, compilation level 3, and FULL cudagraph mode.thinking_ongolden AL 2.49 (--spec-decode-acceptance-rate 0.4966666667); eval-only runs omit synthetic acceptance and use real MTP acceptance.rocm/atom-dev:nightly_202608181633(manifest digestsha256:fdc5650f2b6d13c22f1f1b3873ee6dc61278e6b0b90dc24d0f6c55810895032f).This refresh supersedes the stale #2346 submission: it uses the current AgentX/AIPerf harness, a current ATOM image, complete server/runtime settings from the validated baseline, and the now-supported golden synthetic acceptance path. It does not modify or close #2346.
This recipe does not patch or check out ATOM at runtime. It is independent of ROCm/ATOM#1937 and does not use changes from that PR.
Validation
bash -n benchmarks/single_node/agentic/dsv4_fp4_mi355x_atom_mtp.shconfigs/amd-master.yaml,configs/runners.yaml, andperf-changelog.yamltest-configmatrix generation: 7 intended AgentX entries onlyfull-sweepgeneration fordsv4/atom/fp4/cluster:mi355x-amdsgit diff --checkGPU sweep/eval will run under
full-sweep-fail-fastafter the changelog PR URL is backfilled.中文
概要
dsv4-fp4-mi355x-atom-agentic-mtp：在单节点 8x MI355X 上用 ATOM 运行 DeepSeek-V4-Pro FP4，采用 TP8、无 KV offload，AgentX 并发为[1, 2, 4, 8, 16, 32, 48]。max-num-seqs = 2 * concurrency，启用 FP8 KV/index cache、prefix caching、32K state checkpoint、16K batched-token/prefill-chunk 上限、编译 level 3 和 FULL cudagraph。thinking_on的 golden AL 2.49（--spec-decode-acceptance-rate 0.4966666667）；eval-only 不注入 synthetic acceptance，使用模型真实的 MTP acceptance。rocm/atom-dev:nightly_202608181633（manifest digest：sha256:fdc5650f2b6d13c22f1f1b3873ee6dc61278e6b0b90dc24d0f6c55810895032f）。这个 PR 是对陈旧 #2346 的更新替代：使用当前 AgentX/AIPerf harness、当前 ATOM 镜像、实测基线中的完整 server/runtime 参数，以及 ATOM 现已支持的 golden synthetic acceptance 路径。本 PR 不修改或关闭 #2346。
本 recipe 不会在运行时 patch 或 checkout ATOM；它与 ROCm/ATOM#1937 无关，也不使用该 PR 的改动。
验证
bash -nconfigs/amd-master.yaml、configs/runners.yaml和perf-changelog.yaml均通过 YAML 解析test-config只生成预期的 7 个 AgentX 点dsv4/atom/fp4/cluster:mi355x-amds的过滤full-sweep生成通过git diff --check通过回填 changelog 中的真实 PR URL 后，将添加
full-sweep-fail-fast运行 GPU sweep/eval。