feat(agentx): add Kimi-K3 FP4 MI355X ATOM DSpark recipe - #2697
Conversation
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
1 similar comment
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| - "Set the simulated acceptance length to 2.54, giving spec-decode-acceptance-rate (2.54 - 1) / 2 = 0.77, which is the value the submitted ATOM numbers were measured under. This is 1.2% above the committed golden 2.51 that the vLLM arm feeds to synthetic_acceptance_length, and the script carries a review note saying so; set it to 2.51 for a strictly like-for-like sweep. Evaluations use real acceptance." | ||
| - "Divide the aggregate TOTAL_CPU_DRAM_GB budget by TP for LMCACHE_MAX_LOCAL_CPU_SIZE, which is a per-rank setting; dram-utilization 0.085 lands that at roughly 32 GB per rank." |
There was a problem hiding this comment.
🟡 perf-changelog.yaml (lines 6207-6208) claims the ATOM numbers used simulated acceptance length 2.54 (rate 0.77) with a script review note about a 2.54-vs-2.51 discrepancy, and dram-utilization 0.085. Neither matches what's shipped: kimik3_fp4_mi355x_atom_mtp.sh sets SIMULATE_ACC_LEN=2.51 (rate 0.755, matching the PR description) with no such review note, and configs/amd-master.yaml sets dram-utilization: 0.086 (with its own comment explaining why 0.086 was picked over 0.085). The changelog prose should be corrected to describe the values actually shipped.
Extended reasoning...
perf-changelog.yaml is meant to be the authoritative human-facing record of what a submitted benchmark measured, but the new entry for kimik3-fp4-mi355x-atom-agentic-mtp (lines 6207-6208) describes methodology that does not match either of the two files it is documenting.
Line 6207 states: "Set the simulated acceptance length to 2.54, giving spec-decode-acceptance-rate (2.54 - 1) / 2 = 0.77, which is the value the submitted ATOM numbers were measured under ... and the script carries a review note saying so; set it to 2.51 for a strictly like-for-like sweep." The shipped script, benchmarks/single_node/agentic/kimik3_fp4_mi355x_atom_mtp.sh, sets SIMULATE_ACC_LEN=2.51 for CONC != 1 (with NUM_SPEC_TOKENS=2), which computes to SPEC_ACCEPTANCE_RATE=(2.51-1)/2=0.755 — not 0.77. There is no 2.54 anywhere in the script, and no review-note comment discussing a 2.54-vs-2.51 discrepancy; the only related comment cites the committed golden acceptance length of 2.51. The PR description itself independently confirms 2.51/0.755 as the value the numbers were measured under, so the changelog is the outlier, not the script.
Line 6208 states dram-utilization is 0.085, giving roughly 32 GB per rank. The shipped configs/amd-master.yaml entry for kimik3-fp4-mi355x-atom-agentic-mtp sets dram-utilization: 0.086, and the adjacent comment in that same file explicitly explains the choice: "dram-utilization 0.086 puts the aggregate budget at 257 GB, so the script's divide-by-TP lands on exactly the 32 GB per rank the recipe was measured with (0.085 gives 254, which floors to 31)." So 0.085 was the value considered and rejected, and 0.086 is what actually ships.
Nothing in the reviewed code paths prevents this kind of drift: perf-changelog.yaml is free-text prose maintained by hand alongside the config/script changes, and there's no validation tying its claimed constants back to the actual shipped values, so a stale draft value (2.54/0.085, perhaps from an earlier iteration of the PR) survived into the final changelog entry after the script and config were finalized at 2.51/0.086.
Step-by-step proof: (1) grep benchmarks/single_node/agentic/kimik3_fp4_mi355x_atom_mtp.sh for SIMULATE_ACC_LEN — only 2.51 (non-conc-1) and 3.75 (conc-1) appear, never 2.54. (2) Compute (2.51-1)/2 = 0.755, matching the PR description's stated rate, not the changelog's 0.77. (3) grep the script for "review note" or "discrepancy" — no match; the only spec-decode comment references the golden 2.51 value directly with no mention of 2.54. (4) grep configs/amd-master.yaml for the new kimik3-fp4-mi355x-atom-agentic-mtp block — dram-utilization: 0.086, not 0.085, with its own comment justifying 0.086 over 0.085.
This is documentation-only: the benchmark itself runs correctly using the actual shipped 2.51/0.086 values regardless of what perf-changelog says, so there's no functional or measurement impact. But since perf-changelog is the record future readers will use to understand what the ATOM numbers represent, it should be corrected to state 2.51 / 0.755 / 0.086 and drop the nonexistent review-note claim.
39bbc96 to
a2de999
Compare
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32373121203 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32373307590 |
1 participant
Recipe PR: ROCm/ATOM#1941
Adds the MI355X Kimi-K3 ATOM AgentX submission alongside the existing vLLM arm: same checkpoint, same runner, same concurrency points, so the two engines are directly comparable.
TP8 at concurrency 1 and 4 is GPU-resident. Concurrency 8 and 10 switch on the LMCache DRAM tier and ATOM's CPU state-offload tier together, because Kimi-K3 is a hybrid -- Kimi Delta Attention carries a per-request recurrent state alongside the paged KV, so a resumed agentic turn needs the state back and the paged KV tier alone cannot restore one.
spec-decode-acceptance-rate is derived from the committed golden acceptance length 2.51 at num_speculative_tokens 2 as (2.51 - 1) / 2 = 0.755, the same golden the vLLM arm feeds to synthetic_acceptance_length, so both engines are held to one acceptance assumption.
LMCACHE_MAX_LOCAL_CPU_SIZE is per rank, so the aggregate TOTAL_CPU_DRAM_GB is divided by TP as the agentic README requires; dram-utilization 0.085 lands that at roughly 32 GB per rank.
Reopening #2670. Original author: @zejunchen-zejun. cc @seungrokj.