feat(glm52-agentx): Add B300 Dynamo+TRT-LLM AgentX recipes - #2666
feat(glm52-agentx): Add B300 Dynamo+TRT-LLM AgentX recipes#2666RohitNagraj wants to merge 9 commits into
Conversation
添加 GLM-5.2 B300 Dynamo-TRT AgentX MTP 配置。
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
1 similar comment
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
补充 GLM-5.2 B300 配置提交的拉取请求链接。
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
Additional findings (outside current diff — PR may have been updated during review):
-
🟡
perf-changelog.yaml:6174-6181— The new perf-changelog.yaml entry for glm5.2-fp4-b300-dynamo-trt-agentic-mtp setspr-linktohttps://github.com/SemiAnalysisAI/InferenceX/pull/XXX, a leftover placeholder that was never filled in. Every other entry in this append-only file (e.g. pull/2576, pull/2590, pull/2656, pull/2658) links to its real PR number; since this is PR #2666, it should readpull/2666.Extended reasoning...
What the bug is. The diff appends a new entry to
perf-changelog.yaml(lines 6174-6181) forglm5.2-fp4-b300-dynamo-trt-agentic-mtp. Itspr-linkfield is set to a literal placeholder:pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/XXX
XXXwas clearly meant to be substituted with the actual PR number before the PR was opened, and that substitution never happened.How it manifests / code path.
perf-changelog.yamlis an append-only audit trail: every prior entry in the file (visible right above the new one, e.g. lines 6163-6173) links to a real, resolvable PR —pull/2576,pull/2590,pull/2656,pull/2658, etc. This PR is #2666 per the PR metadata, so the convention clearly calls forpull/2666here. Nothing in the tooling (YAML/Bash syntax checks, config generation, matrix validation) checks thatpr-linkresolves to a real, existing PR, so this placeholder slipped through all the PR's stated validation steps undetected.Why existing checks don't catch it. The changelog is metadata only — it's not consumed by any runtime code path (benchmark execution, config generation, or scoring). There's no schema validation enforcing that
pr-linkmatches a real PR number, so a literalXXXstring passes YAML parsing just as well as a real integer would.Impact. Purely a traceability/documentation issue: anyone who later opens
https://github.com/SemiAnalysisAI/InferenceX/pull/XXXwhile auditing changelog history gets a 404 instead of the PR that introduced this config. It does not affect benchmark execution, sweep validation, or config generation in any way.Fix. Change line 6181 from:
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/XXX
to:
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2666
Step-by-step proof.
- Open
perf-changelog.yamland scroll to the last entry (added by this PR), lines 6174-6181. - Observe
config-keys: [glm5.2-fp4-b300-dynamo-trt-agentic-mtp]— this is the new entry from this PR. - Observe its
pr-linkvalue:https://github.com/SemiAnalysisAI/InferenceX/pull/XXX. - Compare against the immediately preceding entry (lines 6163-6173, for
glm5.2-fp4-b200-sglang-agentic-mtpper PR #2576), whosepr-linkishttps://github.com/SemiAnalysisAI/InferenceX/pull/2576— a real, filled-in PR number. - Check the PR metadata for this PR: it is PR #2666, opened 2026-08-19.
- Conclude that
XXXshould have been2666, and was left as an unfilled template placeholder.
All verifier passes independently reached the same conclusion with no refutations, so this is a confirmed, uncontested finding. Given it's changelog-only metadata with no functional impact, it's a nit rather than a blocking issue — trivial to fix (a one-line string substitution) but should be corrected before merge to keep the changelog's traceability links valid.
- Open
| check_env_vars \ | ||
| MODEL MODEL_PREFIX FRAMEWORK PRECISION CONC RESULT_FILENAME DURATION \ | ||
| SRT_FRONTEND_HOST SRT_FRONTEND_PORT | ||
|
|
||
| export AIPERF_SERVER_URL="http://${SRT_FRONTEND_HOST}:${SRT_FRONTEND_PORT}" |
There was a problem hiding this comment.
🔴 benchmarks/multi_node/agentic_srt.sh is the shared benchmark.type=custom client sourced by ~80 srt-slurm multi-node agentic recipes (dsv4, qwen3.5, minimax-m3, kimi-k3, glm5.2-sglang, etc.), each pinning a different srt-slurm branch/fork/commit. This PR makes SRT_FRONTEND_HOST/SRT_FRONTEND_PORT a hard check_env_vars requirement with no MODEL_PREFIX/FRAMEWORK guard, and switches AIPERF_SERVER_URL/wait_for_agentic_servers_idle to depend on them instead of the previous http://localhost:$PORT. These two vars appear nowhere else in the repo, so they must come from srt-slurm at runtime, and only the new glm5.2 dynamo-trt path (v1.0.36) was validated here — if any of the other pinned srt-slurm versions do not export them into the custom command env, every other agentic recipe fails immediately at check_env_vars on its next sweep, and this PR is not the one that would surface it since its own CI only exercises the new glm5.2 configs.
Extended reasoning...
What the bug is. benchmarks/multi_node/agentic_srt.sh (lines 12-16) is the shared benchmark.type=custom client invoked by roughly 80 srt-slurm multi-node agentic recipe YAMLs across dsv4, qwen3.5, minimax-m3, kimi-k3, and glm5.2-sglang, not just the new glm5.2 dynamo-trt recipes added in this PR. Before this change the script required only MODEL MODEL_PREFIX FRAMEWORK PRECISION CONC RESULT_FILENAME DURATION and talked to http://localhost:$PORT, relying on the client being co-located with the frontend (the script's own header comment says it runs 'against the already-ready frontend on the head node'). This PR appends SRT_FRONTEND_HOST SRT_FRONTEND_PORT to check_env_vars unconditionally and rewires AIPERF_SERVER_URL / wait_for_agentic_servers_idle's metrics URL to depend on them, with no MODEL_PREFIX/FRAMEWORK guard around the new requirement.\n\nWhy it can regress unrelated recipes. SRT_FRONTEND_HOST and SRT_FRONTEND_PORT do not appear anywhere else in the repository — they are not exported by any launcher script or recipe YAML env block — so the only place they can come from is the srt-slurm tool injecting them into the custom benchmark command's environment at runtime. Different agentic recipes pin different srt-slurm branches/forks/commits (main, sa-submission-q2-2026, cam/sa-submission-q2-2026, various pinned SHAs, and now v1.0.36 for glm5.2 dynamo-trt). Only the new glm5.2 dynamo-trt path was dry-run validated in this PR's checklist; nothing establishes that the older/other srt-slurm versions also inject these two vars into the custom command environment.\n\nStep-by-step proof of the failure mode. (1) A pre-existing recipe, e.g. a dsv4 dynamo-sglang agentic YAML, still has benchmark.type: custom with command: bash .../agentic_srt.sh. (2) Its launcher pins an srt-slurm branch other than v1.0.36 (e.g. main or sa-submission-q2-2026). (3) On the next scheduled sweep, srt-slurm starts the frontend/workers and then execs the custom benchmark command. (4) agentic_srt.sh sources benchmark_lib.sh and immediately calls check_env_vars ... SRT_FRONTEND_HOST SRT_FRONTEND_PORT. (5) If that srt-slurm version's custom-command environment does not set those two vars (plausible, since the pre-PR script never needed or used them and instead worked purely via localhost:$PORT co-location), check_env_vars prints the missing-vars error and exit 1 fires before any benchmark work happens — the whole job fails instantly regardless of whether the server itself came up healthy.\n\nWhy nothing else in the PR catches this. The PR's own CI/validation (YAML/bash syntax checks, master/recipe parity, and the v1.0.36 dry-run) only exercises the seven new glm5.2 dynamo-trt configs added here. None of the ~80 other agentic recipes sharing this script are re-run as part of this PR's validation, so a break in them would only surface on their own regularly scheduled sweep, after merge — exactly the kind of latent regression code review is supposed to catch.\n\nTelling asymmetry with the sibling fix. The companion change in benchmark_lib.sh's build_replay_cmd is defensive: --url ${AIPERF_SERVER_URL:-http://localhost:$PORT}, i.e. it falls back to the old behavior if AIPERF_SERVER_URL isn't set. agentic_srt.sh does the opposite — it hard-fails via check_env_vars before that fallback could ever apply, even though AIPERF_SERVER_URL is itself derived from the two new required vars just above it.\n\nSuggested fix. Scope the new requirement to the glm5.2 dynamo-trt case (mirroring the $FRAMEWORK/$MODEL_PREFIX guards already used elsewhere in launch_b300-nv.sh for this same feature), or make the two vars optional with a fallback consistent with the benchmark_lib.sh change: export AIPERF_SERVER_URL="http://${SRT_FRONTEND_HOST:-localhost}:${SRT_FRONTEND_PORT:-$PORT}", and drop SRT_FRONTEND_HOST/SRT_FRONTEND_PORT from the unconditional check_env_vars list.
恢复 GLM-5.2 Dynamo-TRT 的集群 UCX 挂载，并将 AgentX 结果文件名与工作流生成的名称保持一致。
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32214847694 |
为 c227 和 c259 配方补充 CUDA UCX 传输，并将 AgentX 结果文件名对齐逻辑保持在最小范围。恢复 GLM Dynamo-TRT 使用容器内 UCX。
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32268233175 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32274841492 |
hshrivastava-droid
changed the title
为六个 GLM-5.2 B300 AgentX 配方恢复 CUDA UCX 传输，使 NIXL 能正确识别并注册显存。
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32306801520 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32322877658 |
2 participants
Description
Adds
glm5.2-fp4-b300-dynamo-trt-agentic-mtpfor B300 using Dynamo with TensorRT-LLM disaggregated serving. The submission includes seven checked-in srt-slurm recipes with NIXL KV transfer and MTP speculative decoding.The B300 launcher stages the local-NVMe checkpoint, pins srt-slurm
v1.0.36, passes--no-preflightfor the compute-node-only model path, uses the image UCX stack for NIXL, and configures root remapping for PMIx. Throughput keeps the frontend onfirst_decodeand the benchmark client onlast_decode; eval-only jobs co-locate the frontend with lm-eval on the allocation head. The AgentX client and metrics collector use the frontend endpoint supplied by srt-slurm.Validation:
v1.0.36dry-run for all seven recipes中文说明
新增
glm5.2-fp4-b300-dynamo-trt-agentic-mtp配置，在 B300 上使用 Dynamo 与 TensorRT-LLM 分离式服务。该提交包含七个已纳入仓库的 srt-slurm 配方，使用 NIXL 传输 KV 缓存并启用 MTP 推测解码。B300 启动脚本使用本地 NVMe 模型路径，固定 srt-slurm
v1.0.36，并为计算节点可见的模型路径传入--no-preflight；NIXL 使用镜像内的 UCX，同时为 PMIx 配置 root remapping。吞吐测试保持前端位于first_decode、基准客户端位于last_decode；仅评估任务则将前端与分配头节点上的 lm-eval 放在同一节点。AgentX 客户端与指标采集器使用 srt-slurm 提供的前端地址。验证：
v1.0.36dry-runRelated Issue
N/A
Type of Change
Checklist
perf-changelog.yaml/reuse-sweep-runafter a final green sweep