[NV] GB300 DeepSeek-V4-Pro AgentX Dynamo-SGLang session-routing upgrade / [NV] GB300 DeepSeek-V4-Pro AgentX Dynamo-SGLang 会话路由升级

[csahithi]

Summary

Update six DeepSeek-V4-Pro FP4 GB300 Dynamo-SGLang AgentX MTP recipe topologies covering nine concurrency points:

• Upgrade Dynamo from 1.3.0.dev1 to 1.3.0.dev20260718; remove the DYN_ROUTER_TEMPERATURE=10000000 override and use the deterministic default.
• Move AgentX session affinity from legacy nvext.session_control actions to X-Dynamo-Session-ID headers. The recipes set AIPERF_HTTP_X_DYNAMO_SESSION_ID_FROM_CORRELATION_ID=true; nginx affinity and the Dynamo router use the same header with a 3600-second TTL. Existing Dynamo recipes retain the legacy CLI path when the header opt-in is absent.
• Pin the AIPerf submodule to 655792405980c5211722bc45a5f8401f3bad304a, the merge commit of SemiAnalysisAI/aiperf#19, which re-landed the session-header support on cquil11/aiperf-agentx-v1.0.
• Upgrade SGLang from nightly-dev-cu13-20260711-7de33ce8 to upstream lmsysorg/sglang:nightly-dev-cu13-20260719-99f5a6f4.
• Disable only the pathological DSV4 EAGLE draft-extend CUDA-graph capture for the aggregate TP4 and disaggregated 1P1D TP4-decode recipes while retaining target/decode graphs.
• Add cluster:gb300-nv CPU-DRAM and GPU-per-node metadata required by the AgentX DRAM-offload matrix logic.

Scope and validation

• Scope: AgentX agentic-coding replay only; AGG and DISAGG; MTP with synthetic AL 2.49; trace-derived ISL/OSL rather than fixed-sequence 1k1k or 8k1k.
• Run Sweep 30051155312, attempt 2 completed successfully on in-PR commit 41407f30b2886e2501f12f671d63fd90925a324d; all nine applicable multi-node AgentX jobs passed with the current SGLang image and Dynamo version.
• The swept AIPerf pin 6c84373b1d638a7a9ac077ee8b90490b9e93e7d9 and current pin 655792405980c5211722bc45a5f8401f3bad304a have identical file trees; the intervening commits are the revert and re-land sequence.

中文说明

更新 DeepSeek-V4-Pro FP4 GB300 Dynamo-SGLang AgentX MTP 的 6 个配方拓扑，共覆盖 9 个并发点：

• 将 Dynamo 从 1.3.0.dev1 升级到 1.3.0.dev20260718；移除 DYN_ROUTER_TEMPERATURE=10000000 覆盖项，恢复确定性的默认行为。
• 将 AgentX 会话亲和性从旧版 nvext.session_control 操作迁移到 X-Dynamo-Session-ID 请求头。配方设置 AIPERF_HTTP_X_DYNAMO_SESSION_ID_FROM_CORRELATION_ID=true，nginx 亲和性与 Dynamo 路由器统一使用该请求头，并将 TTL 设为 3600 秒。未启用该请求头的既有 Dynamo 配方仍保留旧版 CLI 路径。
• 将 AIPerf 子模块固定到 655792405980c5211722bc45a5f8401f3bad304a，即已合并的 SemiAnalysisAI/aiperf#19 的合并提交；该 PR 在 cquil11/aiperf-agentx-v1.0 上重新引入了会话请求头支持。
• 将 SGLang 从 nightly-dev-cu13-20260711-7de33ce8 升级到上游镜像 lmsysorg/sglang:nightly-dev-cu13-20260719-99f5a6f4。
• 仅针对聚合 TP4 与分离式 1P1D TP4 解码配方禁用异常耗时的 DSV4 EAGLE draft-extend CUDA Graph 捕获，同时保留 target/decode CUDA Graph。
• 补充 cluster:gb300-nv 的 CPU DRAM 容量与单节点 GPU 数量元数据，供 AgentX DRAM 卸载矩阵逻辑使用。

范围与验证

• 范围：仅 AgentX agentic-coding 回放；同时覆盖聚合与分离式推理；使用 MTP 与合成接受长度(AL) 2.49；ISL/OSL 由轨迹数据决定，不属于固定序列长度的 1k1k 或 8k1k。
• Run Sweep 30051155312，第 2 次尝试 已在本 PR 的提交 41407f30b2886e2501f12f671d63fd90925a324d 上成功完成；9 个适用的多节点 AgentX 任务全部通过，并使用当前 SGLang 镜像与 Dynamo 版本。
• 扫描时使用的 AIPerf 提交 6c84373b1d638a7a9ac077ee8b90490b9e93e7d9 与当前提交 655792405980c5211722bc45a5f8401f3bad304a 的文件树完全一致；中间提交仅对应回滚与重新引入流程。

[github-actions]

Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase As a PR reviewer and CODEOWNER, I have reviewed this and have.

For PR verification, add the full-sweep-fail-fast label (strongly recommended) to this PR — the benchmark sweep only runs on labeled PRs. Use full-sweep-enabled only if you need matrix jobs to keep running past a failure.

PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs

---

感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 As a PR reviewer and CODEOWNER, I have reviewed this and have。

如需进行 PR 验证，请为此 PR 添加 full-sweep-fail-fast 标签（强烈推荐）— 基准测试 sweep 仅在带有标签的 PR 上运行。仅当需要矩阵任务在失败后继续运行时才使用 full-sweep-enabled。

PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档

[github-actions]

Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase As a PR reviewer and CODEOWNER, I have reviewed this and have.

For PR verification, add the full-sweep-fail-fast label (strongly recommended) to this PR — the benchmark sweep only runs on labeled PRs. Use full-sweep-enabled only if you need matrix jobs to keep running past a failure.

PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs

---

感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 As a PR reviewer and CODEOWNER, I have reviewed this and have。

如需进行 PR 验证，请为此 PR 添加 full-sweep-fail-fast 标签（强烈推荐）— 基准测试 sweep 仅在带有标签的 PR 上运行。仅当需要矩阵任务在失败后继续运行时才使用 full-sweep-enabled。

PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30051155312
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=30051155312

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30051155312
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=30051155312

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30315392821
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=30315392821

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30315392821
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=30315392821

[cquil11]

@csahithi the AgentX/AIPerf harness has been updated, please merge origin/main into your branch and refresh your submission. Additional tuning may be necessary depending on the config. I apologize for any inconvenience.

This is an automated message.

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30586991521
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=30586991521

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=31143712931
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=31143712931

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=31143712931
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=31143712931

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=31143712931
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=31143712931

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=31275542474
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=31275542474

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=31275542474
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=31275542474

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=31275542474
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=31275542474

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=31275542474
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=31275542474

[xinli-sw]

/stage-results 31275542474

[github-actions]

@xinli-sw staged run 31275542474: https://inferencemax-app-git-staging-semianalysisai.vercel.app/inference?i_dates=2026-08-08~r31275542474

This run remains available across future /stage-results requests. Staging the same run ID again updates its staged data. Staging workflow

@xinli-sw 已将运行 31275542474 发布到预发布环境：https://inferencemax-app-git-staging-semianalysisai.vercel.app/inference?i_dates=2026-08-08~r31275542474

后续的 /stage-results 请求不会移除此运行；再次发布相同的运行 ID 会更新其预发布数据。预发布工作流

[xinli-sw]

/reuse-sweep-run

[cquil11]

@csahithi Please rebase.

[Ankur-singh]

/reuse-sweep-run 31275542474

[Ankur-singh]

As a PR reviewer and CODEOWNER, I have reviewed this and have:

• Verified that as of the moment of typing this, this is the latest version of PR_REVIEW_CHECKLIST.md
• Verified that the general code quality meets the InferenceX standard and does not make the code quality any worse.
• Verified that this PR has passed PR validation. Please link to GitHub Action workflow that shows this. https://github.com/SemiAnalysisAI/InferenceX/actions/runs/31275542474
• Verified that this PR passes evals. Please link to GitHub Action workflow that shows this.
• Verified that speculative decoding PRs uses chat templates to align the AL distribution to real world
• For agentic workloads: verified that speculative-decoding configs (EAGLE / MTP / draft models) run with simulated synthetic acceptance, with the acceptance-length value taken from the committed golden AL curve in golden_al_distribution/ for that model, thinking mode, and draft length. A submission may choose any supported draft length, but it may not substitute a different acceptance target.
• Verified against the current MODELS.md that this PR does not submit a deprecated model, scenario, or model-scenario combination.
• Verified that the model architecture isn't changed with benchmark hacks like using --hf-overrides to skipping indexer for every x layers on models that don't natively support this. As a general rule, we won't accept optimizations that reduces the number of model architecture FLOPs. Anything that makes that same computation run faster is fair game; FLOPs at lower precisions is fine, given that the config passes private evals. As an general north star princple, we should only use optimizations which is used in production by customers that care about accuracy
• If an company claims that they support vLLM/SGLang as first class LLM inference engines on their hardware, I have verified that the respective vLLM submission made using upstream https://hub.docker.com/u/vllm docker repo, upstream SGLang https://hub.docker.com/u/lmsysorg docker repo. The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet as supported by vLLM/SGLang community maintainers
• If an company claims that they support vLLM/SGLang as first class upstream in-tree LLM inference engines on their hardware, I have have verified that the respective vLLM/SGLang submission has been made before additional frameworks (TRT-LLM, ATOM, etc.). The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet.
• Verified that every single-node vLLM/SGLang recipe in this PR is documented in the official vLLM recipes and/or the SGLang cookbook:
  • I linked the corresponding upstream PR in the vLLM recipe repo or SGLang repo and verified that it is MERGED before this InferenceX PR merges. An opened, draft, or closed-without-merge upstream PR does not satisfy this requirement. If the matching recipe was already published, I linked the published recipe/cookbook page in the additional detail section below.
• Verified that this PR does not patch the inference engine or serving stack — the pinned image must run as shipped. This covers .patch files / git apply / patch, inline patches embedded in benchmark scripts (e.g. a python3/sed heredoc that rewrites installed engine sources before serving), in-place edits of site-packages, monkey-patching, overwriting container files, and installing forked/rebuilt engine wheels on top of the pinned image. The only exception is a patch covered by a filled-out waiver at docs/waiver/<PR_NUMBER>.md — named after the PR that introduces the patch and filed in that same PR, stating what is patched, why the unmodified upstream image cannot run this benchmark, the upstream PR/issue link, and the removal plan — which I have linked below in the additional detail section.
• If any of the above criteria cannot reasonably be satisfied, I have provided additional reasoning below.

Additional detail section:

Validation evidence and artifact reuse. The linked sweep 31275542474 ran at head b33ee5f8 with 9/9 AgentX jobs green, covering exactly the submitted grid (agg c2/c4/c8/c16; disagg c80/c256/c512/c768/c1536) on the submitted image lmsysorg/sglang:nightly-dev-cu13-20260719-99f5a6f4. The current head 54ed9c15 is a Merge origin/main commit whose sweep was skipped under /reuse-sweep-run, which I have pinned to that run ID. I verified the reuse is sound rather than assuming it:

• configs/nvidia-master.yaml — the two dsv4-fp4-gb300-dynamo-sglang-agentic-* entries are byte-identical at both heads (md5 8635926044ef1f26c3ec1dcd18da2323). Published artifacts derive from these entries, so master↔recipe parity is unchanged by the merge.
• benchmarks/benchmark_lib.sh (the AgentX/AIPerf harness) is unchanged between the two heads.
• runners/launch_gb300-nv.sh changed by comment rewording only; the NVIDIA/srt-slurm v1.0.38 pin is identical.
• The merge added three recipe fields, none of which alter the measured workload:
  • AIPERF_USE_DYNAMO_CONV_AWARE_ROUTING: "0" is a no-op for these recipes. benchmark_lib.sh only appends --use-dynamo-conv-aware-routing when AIPERF_HTTP_X_DYNAMO_SESSION_ID_FROM_CORRELATION_ID != "true", and that flag was already "true" at b33ee5f8, so the routing flag was never applied in either revision.
  • AIPERF_REQUIRED_SERVER_METRIC_PREFIX: "sglang:" is a post-run artifact guard (validate_required_agentic_server_metrics) that greps the metrics export and changes no measured value. enable-metrics: true was already set on prefill and decode at b33ee5f8, so the guard would have passed on the linked run.
  • DYN_TCP_REQUEST_TIMEOUT: "60" (12P4D/c1536 only) relaxes Dynamo's 10s frontend request-plane default. The linked run passed under the stricter default, so the measurement was not rescued by the longer timeout; it does not affect scheduling, parallelism, or KV offload.

DISAGG master↔recipe parity. Re-derived independently for all five disagg variants. Worker/TP topology matches every CONFIG_FILE= recipe name on GB300's 4-GPU nodes (e.g. c1536: prefill 6 workers x TP8 = 48 GPUs = 12P; decode 1 x TP16 = 4D). Offload settings agree: master kv-offloading: dram + kv-offload-backend: {name: hicache} is realized in each recipe as enable-hierarchical-cache: true, hicache-write-policy: write_back, hicache-ratio: 1, hicache-io-backend: direct. Hierarchical cache is enabled on the prefill engines, which is where the session-routed prefix reuse this PR targets actually lands.

Speculative decoding / acceptance length. MTP with speculative-num-steps: 3, speculative-eagle-topk: 1, speculative-num-draft-tokens: 4. Acceptance is simulated (SGLANG_SIMULATE_ACC_METHOD: match-expected, SGLANG_SIMULATE_ACC_TOKEN_MODE: real-draft-token) with SGLANG_SIMULATE_ACC_LEN: 2.49, which is exactly golden_al_distribution/dsv4_mtp.yaml deepseek-v4-pro.thinking_on[3]. Thinking mode matches the curve's measurement conditions (SGLANG_DEFAULT_THINKING: '1', SGLANG_DSV4_REASONING_EFFORT: high; the curve was captured with {"thinking": true, "reasoning_effort": "high"}). No substituted acceptance target. On the chat-template item: these recipes use the AgentX path (benchmark.type: custom -> agentic_srt.sh, AIPerf replaying real agentic-coding trajectories), not the random-token benchmark_serving.py path, and the AL is pinned to a golden curve that was itself measured through the model's chat template, so the AL distribution is aligned to real-world usage.

Evals — not applicable, left unchecked. Both config keys declare only scenarios: agentic-coding: with no evals entry, so search-space-config.agentic_evals is empty and run-sweep.yml's sweep-agentic-evals job cannot be dispatched. There is therefore no eval workflow to link for this submission; the eval jobs are skipped by construction rather than by omission. SGLANG_DISABLE_DRAFT_EXTEND_CUDA_GRAPH: "1" is a CUDA-graph capture fallback to eager for the EAGLE draft-extend step (capture exceeded an hour at TP4/bs128); it changes no model FLOPs and no numerics.

Single-node recipe documentation — not applicable. All six recipes in this PR are multi-node srt-slurm recipes under benchmarks/multi_node/; the PR adds no single-node vLLM/SGLang recipe, so the upstream vLLM-recipes / SGLang-cookbook publication requirement is vacuous here.

Engine patching — none. The pinned upstream lmsysorg/sglang image runs as shipped; no .patch, git apply, site-packages edit, monkey-patch, or rebuilt engine wheel. dynamo.install: true with wheel: "1.3.0.dev20260718" installs the Dynamo router, which is the declared serving frontend for this config (router: {name: dynamo-router, version: "1.3.0.dev20260718"}) and is the standard install path for every dynamo-* framework entry in the repo; PIP_BREAK_SYSTEM_PACKAGES: "1" exists only because the SGLang image is PEP 668 externally-managed. The SGLang engine itself is unmodified.

Signed: Ankur-singh

[Klaud-Cold]

✅✅✅ Verdict: PASS ✅✅✅

✅ Check 0 (CODEOWNER): PASS — Ankur-singh is a named owner of configs/nvidia-master.yaml; the remaining paths carry only the * @InferenceX/core catch-all, which a recognized CODEOWNER covers.
✅ Check 1 (passing sweep on in-PR commit): PASS — run 31275542474 executed on in-PR commit b33ee5f8 with all 9 multi-node agentic / jobs success, covering exactly the submitted grid (agg c2/c4/c8/c16; disagg c80/c256/c512/c768/c1536) on the submitted image lmsysorg/sglang:nightly-dev-cu13-20260719-99f5a6f4.
➖ Check 2 (evals pass): N/A — both config keys declare only scenarios: agentic-coding with no evals entry, so agentic_evals is empty and run-sweep.yml skips eval jobs by construction (no agentic entry in the master config defines evals); the sign-off left the eval item unchecked with this explanation.
➖ Check 3 (recipe linked & merged): N/A — disaggregated/multi-node submission (all recipes under benchmarks/multi_node/srt-slurm-recipes/**, entries multinode: true, framework dynamo-sglang); the recipe-link requirement applies to single-node recipes only.
✅ Check 4 (reuse command): PASS — /reuse-sweep-run 31275542474 posted by COLLABORATOR Ankur-singh (plus a bare /reuse-sweep-run from COLLABORATOR xinli-sw).
✅ Check 5 (latest checklist): PASS — all current-template items present; the eval item is unchecked with reasoning in the additional detail section, which the template's final item permits.
✅ Check 6 (upstream image & ordering): PASS — image is upstream lmsysorg/sglang:nightly-dev-cu13-20260719-99f5a6f4; no new framework entry added (existing entries modified), the dynamo-sglang entries run the upstream SGLang engine, and every cluster:gb300-nv entry is necessarily a Dynamo-orchestrated multi-node config.
✅ Check 7 (deprecated models): PASS — dsv4 agentic-coding MTP is the active published arm per MODELS.md as of 2026-08-10 (the deprecated arm is the non-MTP one).
✅ Check 8 (no architecture hacks): PASS — no --hf-overrides or FLOP-reducing knobs; SGLANG_DISABLE_DRAFT_EXTEND_CUDA_GRAPH=1 only falls back to eager for the draft-extend step, changing no computation.
✅ Check 9 (chat template): PASS — the AgentX replay drives /v1/chat/completions with --endpoint-type chat (benchmarks/benchmark_lib.sh:1866).
✅ Check 10 (no engine patches): PASS — no patch/heredoc/site-packages edits; the Dynamo wheel install is the declared dynamo-router frontend for the dynamo-sglang framework, and the pinned SGLang image runs as shipped.
✅ Check 11 (golden AL): PASS — SGLANG_SIMULATE_ACC_LEN=2.49 with match-expected/real-draft-token on all engines equals golden_al_distribution/dsv4_mtp.yaml deepseek-v4-pro.thinking_on[3] for speculative-num-steps: 3, with thinking mode matching the curve's measurement conditions.