Skip to content

[DO NOT MERGE][Test] DSV4 FP4 B300 SGLang AgentX DEP8 c512 on nightly (HiCache + MegaMoE FP4-act) #2680

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
yhyang201 wants to merge 2 commits into
base: main
Choose a base branch
from
+62 −4
Open

[DO NOT MERGE][Test] DSV4 FP4 B300 SGLang AgentX DEP8 c512 on nightly (HiCache + MegaMoE FP4-act) #2680

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
52612a9
Add test-only B300 sglang AgentX c512 nightly config
yhyang201 Aug 19, 2026
c10394a
Add perf-changelog entry
yhyang201 Aug 19, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
39 changes: 35 additions & 4 deletions benchmarks/single_node/agentic/dsv4_fp4_b300_sglang_mtp.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -127,10 +127,21 @@ if [ "$DP_ATTENTION" = "true" ]; then
)
MEM_FRACTION_STATIC=0.95
if [ "$CONC" -ge 512 ]; then
# Leave room for FlashInfer's transient MoE workspace at the DEP8 tail.
MEM_FRACTION_STATIC=0.94
# DEP8 c512: MegaMoE FP4-activation A2A (same DeepGEMM path as the
# vLLM amxf4 recipe) plus a larger per-step prefill budget so the
# 5,677-request warmup finishes inside the job budget. The DSV4
# indexer's fp32 MQA logits transient scales with the per-rank chunk
# and has no budget cap in this image, so static memory must stay at
# 0.85 (the only value that has completed c512 end to end).
PARALLEL_ARGS+=(--moe-a2a-backend megamoe)
export SGLANG_OPT_DEEPGEMM_MEGA_MOE_USE_FP4_ACTS=1
export SGLANG_OPT_DEEPGEMM_MEGA_MOE_USE_MXF4_KIND=1
export SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK=16384
MEM_FRACTION_STATIC=0.85
CHUNKED_PREFILL_SIZE=131072
else
CHUNKED_PREFILL_SIZE=16384
fi
CHUNKED_PREFILL_SIZE=16384
else
PARALLEL_ARGS+=(
Comment on lines 127 to 146

@claude claude Bot Aug 19, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 The new CONC>=512 branch in dsv4_fp4_b300_sglang_mtp.sh (megamoe A2A backend, SGLANG_OPT_DEEPGEMM_MEGA_MOE_* env vars, --enable-deepseek-v4-fp4-indexer, mem-fraction 0.85, chunked-prefill 131072) is gated only on DP_ATTENTION and CONC, with no check on image or config key. The existing production config dsv4-fp4-b300-sglang-agentic-hicache-mtp (pinned to v0.5.17-cu130) already has a DEP8 arm with dp-attn: true and conc-list including 512, so it will also hit this nightly-only branch — contradicting the PR description's claim that "Existing entries untouched" and risking a startup failure or silently altered baseline on that production c512 point.

Extended reasoning...

The bug: benchmarks/single_node/agentic/dsv4_fp4_b300_sglang_mtp.sh is the single shared launcher for both the pre-existing dsv4-fp4-b300-sglang-agentic-hicache-mtp config (image lmsysorg/sglang:v0.5.17-cu130) and the new dsv4-fp4-b300-sglang-agentic-hicache-mtp-nightly config (image lmsysorg/sglang:nightly-dev-cu13-20260818-c0b6474b) added in this PR. Both keys resolve to this script because they share model-prefix: dsv4, precision: fp4, runner: b300-nv, framework: sglang, and the agentic-coding scenario.

Code path: In configs/nvidia-master.yaml, the existing dsv4-fp4-b300-sglang-agentic-hicache-mtp config's last arm is { tp: 8, ep: 8, dp-attn: true, kv-offloading: none, spec-decoding: mtp, conc-list: [52, 72, 100, 128, 144, 196, 512], router: {...} } — note dp-attn: true and 512 already in its conc-list. When this arm's c512 point runs, the script receives DP_ATTENTION=true and CONC=512. The new branch added at lines 129-146 (if [ "$CONC" -ge 512 ]) and the indexer flag at lines 157-159 (if [ "$DP_ATTENTION" = "true" ] && [ "$CONC" -ge 512 ]) check nothing about which config or image is invoking the script — they fire purely on the env vars DP_ATTENTION/CONC set by whatever caller runs them.

Why nothing prevents this: The script has no notion of "which config key called me" or "which image am I running under" — it's purely parameterized by env vars (TP, CONC, DP_ATTENTION, KV_OFFLOADING, etc.), and both the old and new config's DEP8/c512 arms produce the identical DP_ATTENTION=true, CONC=512 combination. There's no image-version check, no dedicated flag (e.g. checking KV_OFFLOADING=dram — which does differ between the two configs, since the existing arm uses kv-offloading: none while the nightly arm uses kv-offloading: dram), so the gate is broader than intended.

Impact: Before this PR, the existing v0.5.17 c512 point ran with MEM_FRACTION_STATIC=0.94, CHUNKED_PREFILL_SIZE=16384, no --moe-a2a-backend megamoe, and no --enable-deepseek-v4-fp4-indexer. After this PR merges, that same production entry's c512 point will additionally receive --moe-a2a-backend megamoe, three SGLANG_OPT_DEEPGEMM_MEGA_MOE_* env vars, --enable-deepseek-v4-fp4-indexer, MEM_FRACTION_STATIC dropped to 0.85, and CHUNKED_PREFILL_SIZE raised to 131072. The PR description itself frames megamoe/the fp4-indexer flag as nightly-only capabilities being validated on the new nightly image — on the pinned v0.5.17-cu130 image these flags may not be recognized by sglang.launch_server's argparse, causing an immediate startup failure, or if they happen to exist, they'd silently change the production baseline's launch recipe. Either way this directly contradicts the PR description's explicit claim "Existing entries untouched."

Step-by-step proof:

  1. configs/nvidia-master.yaml (pre-existing, line ~1174): dsv4-fp4-b300-sglang-agentic-hicache-mtp → arm { tp: 8, ep: 8, dp-attn: true, kv-offloading: none, spec-decoding: mtp, conc-list: [..., 512] }.
  2. The sweep driver launches the shared script with DP_ATTENTION=true, CONC=512, KV_OFFLOADING=none.
  3. Script line 127: if [ "$DP_ATTENTION" = "true" ]; then → true, enters DP-attn branch, sets MEM_FRACTION_STATIC=0.95.
  4. Script line ~133 (new): if [ "$CONC" -ge 512 ]; then → true (512 >= 512) — this is new logic added by this PR with no image/config check.
  5. Sets PARALLEL_ARGS+=(--moe-a2a-backend megamoe), exports the three SGLANG_OPT_DEEPGEMM_MEGA_MOE_* vars, sets MEM_FRACTION_STATIC=0.85, CHUNKED_PREFILL_SIZE=131072.
  6. Script line ~157 (new): if [ "$DP_ATTENTION" = "true" ] && [ "$CONC" -ge 512 ]; then MODEL_ARGS+=(--enable-deepseek-v4-fp4-indexer); fi → true, adds the indexer flag.
  7. sglang.launch_server is invoked on image v0.5.17-cu130 with these nightly-only flags/env vars it was never validated against — the production c512 point's recipe has changed from what it was before this PR, or fails to launch outright.

Fix: Scope the new branch to the nightly config specifically — e.g., gate on KV_OFFLOADING = "dram" (which does differ: the existing arm uses kv-offloading: none, the nightly arm uses kv-offloading: dram) or introduce an explicit env var/flag set only by the new nightly config key, rather than keying purely off CONC -ge 512.

--moe-runner-backend flashinfer_mxfp4
Expand All @@ -143,6 +154,9 @@ MODEL_ARGS=(
--page-size 256
--disable-shared-experts-fusion
)
if [ "$DP_ATTENTION" = "true" ] && [ "$CONC" -ge 512 ]; then
MODEL_ARGS+=(--enable-deepseek-v4-fp4-indexer)
fi

# AgentX concurrency counts live session trees, not individual requests.
# Allow subagent fan-out to exceed CONC without clipping request bursts.
Expand Down Expand Up @@ -240,6 +254,23 @@ wait_for_ready \

if [ "$USE_SGLANG_ROUTER" = "true" ]; then
echo "Starting SGLang router on port $PORT for $TP DP ranks..."
# Wave hardening, gated on the pinned router's flag surface: bounded
# retries with backoff absorb transient upstream failures during the
# cache-hit flood so a single 5xx cannot abort a root warmup request,
# no circuit breaker (its fast-fail window can strike root requests),
# and idle-pool pruning drops connections before the server's 900 s
# keep-alive closes them under us.
ROUTER_HARDENING_ARGS=(--disable-retries)
ROUTER_HELP="$("${SGLANG_ROUTER_CMD[@]}" --help 2>&1 || true)"
if grep -q -- "--retry-max-retries" <<<"$ROUTER_HELP"; then
ROUTER_HARDENING_ARGS=(--retry-max-retries 8 --retry-initial-backoff-ms 250 --retry-max-backoff-ms 4000)
fi
if grep -q -- "--disable-circuit-breaker" <<<"$ROUTER_HELP"; then
ROUTER_HARDENING_ARGS+=(--disable-circuit-breaker)
fi
if grep -q -- "--pool-idle-timeout-secs" <<<"$ROUTER_HELP"; then
ROUTER_HARDENING_ARGS+=(--pool-idle-timeout-secs 300)
fi
"${SGLANG_ROUTER_CMD[@]}" \
--worker-urls "http://localhost:$SGLANG_BACKEND_PORT" \
--policy consistent_hashing \
Expand All @@ -252,7 +283,7 @@ if [ "$USE_SGLANG_ROUTER" = "true" ]; then
--connect-timeout-secs 900 \
--request-timeout-secs 14400 \
--disable-health-check \
--disable-retries > "$ROUTER_LOG" 2>&1 &
"${ROUTER_HARDENING_ARGS[@]}" > "$ROUTER_LOG" 2>&1 &
ROUTER_PID=$!
echo "Router PID: $ROUTER_PID"
wait_for_ready \
Expand Down
19 changes: 19 additions & 0 deletions configs/nvidia-master.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1173,6 +1173,25 @@ dsv4-fp4-b300-sglang-agentic-hicache-mtp:
- { tp: 4, ep: 4, dp-attn: true, kv-offloading: dram, kv-offload-backend: { name: hicache }, spec-decoding: mtp, conc-list: [32, 40, 48, 56, 64, 72, 80, 88, 96, 128], router: { name: sglang-router, version: "0.3.2" } }
- { tp: 8, ep: 8, dp-attn: true, kv-offloading: none, spec-decoding: mtp, conc-list: [52, 72, 100, 128, 144, 196, 512], router: { name: sglang-router, version: "0.3.2" } }

# Test-only DEP8 c512 arm on the current sglang nightly: HiCache host offload
# behind the DP-attention arm (the running set saturates the GPU KV pool at
# c512, so follow-up prefix reuse needs the host tier), MegaMoE FP4-activation
# A2A, and 16384-token/rank prefill chunks. Not intended to replace the
# v0.5.17 entry above.
dsv4-fp4-b300-sglang-agentic-hicache-mtp-nightly:
image: lmsysorg/sglang:nightly-dev-cu13-20260818-c0b6474b
model: deepseek-ai/DeepSeek-V4-Pro
model-prefix: dsv4
runner: cluster:b300-nv
precision: fp4
framework: sglang
multinode: false
scenarios:
agentic-coding:
- dram-utilization: 0.80
search-space:
- { tp: 8, ep: 8, dp-attn: true, kv-offloading: dram, kv-offload-backend: { name: hicache }, spec-decoding: mtp, conc-list: [512], router: { name: sglang-router, version: "0.3.2" } }

# DeepSeek-V4-Pro on B300 with EAGLE/MTP speculative decoding. Recipe is
# selected inside benchmarks/single_node/dsv4_fp4_b300_sglang_mtp.sh by
# DP_ATTENTION:
Expand Down
8 changes: 8 additions & 0 deletions perf-changelog.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6203,3 +6203,11 @@
description:
- "Add new B200 multi-node configs with PP2, DCP8"
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2672

- config-keys:
- dsv4-fp4-b300-sglang-agentic-hicache-mtp-nightly
scenario-type:
- agentic-coding
description:
- "Test-only: measure sglang nightly on B300 AgentX DEP8 c512 with HiCache DRAM offload, MegaMoE FP4-activation A2A, 131072-token chunked prefill, and mem-fraction 0.85."
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2680
Loading