Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion benchmarks/benchmark_lib.sh
Original file line number Diff line number Diff line change
Expand Up @@ -1996,7 +1996,7 @@ build_replay_cmd() {
# rolling TTFT/ITL/throughput block and emit it every 30 seconds.
export AIPERF_UI_REALTIME_METRICS_ENABLED=true
REPLAY_CMD="$AIPERF_CLI profile --scenario inferencex-agentx-mvp"
REPLAY_CMD+=" --url http://localhost:$PORT"
REPLAY_CMD+=" --url ${AIPERF_SERVER_URL:-http://localhost:$PORT}"
REPLAY_CMD+=" --endpoint /v1/chat/completions"
REPLAY_CMD+=" --endpoint-type chat"
REPLAY_CMD+=" --streaming"
Expand Down
9 changes: 7 additions & 2 deletions benchmarks/multi_node/agentic_srt.sh
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,12 @@ set -x
INFMAX_CONTAINER_WORKSPACE="${INFMAX_CONTAINER_WORKSPACE:-/infmax-workspace}"
source "$INFMAX_CONTAINER_WORKSPACE/benchmarks/benchmark_lib.sh"

check_env_vars MODEL MODEL_PREFIX FRAMEWORK PRECISION CONC RESULT_FILENAME DURATION
check_env_vars \
MODEL MODEL_PREFIX FRAMEWORK PRECISION CONC RESULT_FILENAME DURATION \
SRT_FRONTEND_HOST SRT_FRONTEND_PORT

export AIPERF_SERVER_URL="http://${SRT_FRONTEND_HOST}:${SRT_FRONTEND_PORT}"
echo "Using srt-slurm frontend endpoint: $AIPERF_SERVER_URL"
Comment on lines +12 to +17

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 This script is invoked by ~80 pre-existing multi-node agentic recipes (kimi-k3, minimax-m3, glm5.2, qwen3.5, deepseek-v4 vllm/sglang, etc.), none of which set SRT_FRONTEND_HOST/SRT_FRONTEND_PORT. Adding these to check_env_vars (which hard-exits with no fallback) means any recipe whose srt-slurm pin doesn't export them will fail immediately, before ever reaching resolve_trace_source. The fix is to mirror the safe ${AIPERF_SERVER_URL:-http://localhost:$PORT} fallback pattern already used in the sibling change to benchmark_lib.sh.

Extended reasoning...

benchmarks/multi_node/agentic_srt.sh is the shared client entrypoint invoked by command: bash /infmax-workspace/benchmarks/multi_node/agentic_srt.sh from roughly 80 pre-existing multi-node agentic recipes across vllm/sglang/trtllm (kimi-k3, minimax-m3, glm5.2, qwen3.5, deepseek-v4, etc.), spanning gb200-nv, gb300-nv, and mi355x launchers. This PR adds SRT_FRONTEND_HOST and SRT_FRONTEND_PORT to the check_env_vars call at line 14, and check_env_vars (benchmarks/benchmark_lib.sh:328) exits the whole job with status 1 the moment any listed variable is unset. Line 16 then builds AIPERF_SERVER_URL exclusively from those two new variables, with no fallback in this script.

Before this PR, the frontend URL was always http://localhost:$PORT, which worked because the benchmark client and frontend were implicitly co-located for every existing recipe. A repo-wide grep shows SRT_FRONTEND_HOST/SRT_FRONTEND_PORT have exactly one in-repo consumer — this script — so they must be injected by srt-slurm itself at runtime, and only if the pinned srt-slurm version supports it. The existing ~80 agentic recipes clone many different srt-slurm branches/pins (default branch, v1.0.31, v1.0.36, a pinned fork SHA, etc.) — none of them set these vars in their recipe yaml, and there's no evidence any of those pins export them. Only this PR's new dsv4 gb300 dynamo-trt recipes pin srt-slurm v1.0.50 and use orchestrator_placement/client_placement fields that differ from the frontend's node — that's the actual motivation for moving off localhost, but it was applied unconditionally to the shared script rather than being scoped to the new recipe.

The result: for every existing recipe whose srt-slurm pin doesn't export SRT_FRONTEND_HOST/SRT_FRONTEND_PORT, the job hits check_env_vars and exits with "required environment variables are not set: SRT_FRONTEND_HOST SRT_FRONTEND_PORT" before ever reaching resolve_trace_source/install_agentic_deps. This regression is latent with respect to this PR's own CI — the PR only sweeps its own new dsv4-trt recipe, so a breakage in the other ~80 recipes would not surface until they're run (e.g. on the next scheduled sweep or a re-run of an existing recipe).

Proof by example: take any pre-existing recipe, e.g. benchmarks/multi_node/srt-slurm-recipes/vllm/kimi-k3/agentic/*.yaml, which invokes this same shared agentic_srt.sh. Its srt-slurm pin (an older branch than v1.0.50) does not export SRT_FRONTEND_HOST/SRT_FRONTEND_PORT. When the job runs: (1) agentic_srt.sh sources benchmark_lib.sh and calls check_env_vars with the full list including the two new vars; (2) check_env_vars iterates the list and finds SRT_FRONTEND_HOST unset; (3) it prints "required environment variables are not set: SRT_FRONTEND_HOST SRT_FRONTEND_PORT" and calls exit 1; (4) the job dies before running any workload, with no fallback to the old localhost:$PORT behavior.

The sibling change in this same PR, benchmark_lib.sh:1999, demonstrates the fix is trivial and was already the codebase's chosen idiom: it changed --url http://localhost:$PORT to --url ${AIPERF_SERVER_URL:-http://localhost:$PORT}. agentic_srt.sh should apply the same fallback pattern — build AIPERF_SERVER_URL from SRT_FRONTEND_HOST/SRT_FRONTEND_PORT only when both are set, falling back to http://localhost:$PORT otherwise — and drop the two new vars from the hard-required list in check_env_vars. That preserves the new dsv4-trt recipe's need for an explicit frontend host while not breaking the ~80 existing recipes that rely on the old co-located default.


BASE_RESULT_DIR="${RESULT_DIR:-/logs/agentic}"
BASE_RESULT_FILENAME="$RESULT_FILENAME"
Expand All @@ -32,7 +37,7 @@ install_agentic_deps
wait_for_agentic_servers_idle() {
local timeout_seconds="${AIPERF_DRAIN_TIMEOUT_SECONDS:-1800}"
local poll_seconds="${AIPERF_DRAIN_POLL_SECONDS:-10}"
local frontend_metrics_url="http://localhost:${PORT}/metrics"
local frontend_metrics_url="${AIPERF_SERVER_URL%/}/metrics"

"$AIPERF_PYTHON" - \
"$timeout_seconds" \
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,187 @@
name: dynamo-disagg-gb300-1p1d-dep8-dep32-c388-b4-mtp
model:
path: deepseek-ai/DeepSeek-V4-Pro
container: nvcr.io#nvidia/tensorrt-llm/release:1.3.0rc24
precision: fp4
identity:
model:
repo: deepseek-ai/DeepSeek-V4-Pro
container:
image: nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc24
frameworks:
dynamo: 1.4.0.dev20260807
tensorrt_llm: 1.3.0rc24
dynamo:
install: true
wheel: 1.4.0.dev20260807
request_plane: tcp
health_check:
max_attempts: 1440
interval_seconds: 10
resources:
gpu_type: gb300
gpus_per_node: 4
prefill_nodes: 2
prefill_workers: 1
gpus_per_prefill: 8
decode_nodes: 8
decode_workers: 1
gpus_per_decode: 32
backend:
type: trtllm
prefill_environment:
TLLM_LOG_LEVEL: INFO
TRTLLM_SERVER_DISABLE_GC: '1'
TRTLLM_WORKER_DISABLE_GC: '1'
NCCL_GRAPH_MIXING_SUPPORT: '0'
TLLM_SPEC_DECODE_FORCE_NUM_ACCEPTED_TOKENS: '1.49'
UCX_TLS: cuda_ipc,cuda_copy,sm,self,tcp
DYN_ENGINE_CONV_AFFINITY: '1'
DYN_DEFAULT_THINKING_MODE: disabled
MIMALLOC_ARENA_RESERVE: '0'
TRTLLM_PINNED_WEIGHT_STAGING: '1'
PYTORCH_CUDA_ALLOC_CONF: expandable_segments:True
CUDA_SCALE_LAUNCH_QUEUES: 4x
decode_environment:
TLLM_LOG_LEVEL: INFO
TRTLLM_SERVER_DISABLE_GC: '1'
TRTLLM_WORKER_DISABLE_GC: '1'
NCCL_GRAPH_MIXING_SUPPORT: '0'
TLLM_SPEC_DECODE_FORCE_NUM_ACCEPTED_TOKENS: '1.49'
UCX_TLS: cuda_ipc,cuda_copy,sm,self,tcp
DYN_ENGINE_CONV_AFFINITY: '1'
DYN_DEFAULT_THINKING_MODE: disabled
MIMALLOC_ARENA_RESERVE: '0'
TRTLLM_PINNED_WEIGHT_STAGING: '1'
trtllm_config:
prefill:
attention_dp_config:
kv_cache_routing_conversation_affinity: true
cache_transceiver_config:
backend: NIXL
kv_transfer_timeout_ms: 600000
transceiver_runtime: PYTHON
cuda_graph_config: null
custom_tokenizer: deepseek_v4
disable_overlap_scheduler: false
enable_attention_dp: true
enable_chunked_prefill: true
kv_cache_config:
dtype: fp8
enable_block_reuse: true
event_buffer_max_size: 0
free_gpu_memory_fraction: 0.8
host_cache_size: 193273528320
pool_ratio:
- 0.55
- 0.22
- 0.23
tokens_per_block: 128
block_reuse_config:
policy: per_conversation
max_num_turns: 5
max_batch_size: 256
max_num_tokens: 16384
max_seq_len: 990016
moe_config:
backend: TRTLLM
moe_expert_parallel_size: 8
pipeline_parallel_size: 1
print_iter_log: true
return_perf_metrics: false
scheduler_config:
capacity_scheduler_policy: MAX_UTILIZATION
speculative_config:
decoding_type: MTP
max_draft_len: 3
tensor_parallel_size: 8
decode:
cache_transceiver_config:
backend: NIXL
kv_transfer_timeout_ms: 600000
transceiver_runtime: PYTHON
cuda_graph_config:
batch_sizes:
- 1
- 2
- 4
enable_padding: true
custom_tokenizer: deepseek_v4
enable_attention_dp: true
enable_lm_head_tp_in_adp: true
kv_cache_config:
avg_seq_len: 200000
dtype: fp8
enable_block_reuse: false
event_buffer_max_size: 0
free_gpu_memory_fraction: 0.7
host_cache_size: 0
tokens_per_block: 128
max_batch_size: 4
max_num_tokens: 16
max_seq_len: 996595
moe_config:
backend: MEGAMOE_DEEPGEMM
use_low_precision_moe_combine: true
moe_expert_parallel_size: 32
num_postprocess_workers: 4
pipeline_parallel_size: 1
print_iter_log: true
return_perf_metrics: false
sparse_attention_config:
algorithm: deepseek_v4
enable_heuristic_topk: true
speculative_config:
decoding_type: MTP
max_draft_len: 3
stream_interval: 20
tensor_parallel_size: 32
publish_events_and_metrics: false
frontend:
type: dynamo
enable_multiple_frontends: false
env:
MODEL: deepseek-ai/DeepSeek-V4-Pro
MODEL_PREFIX: dsv4
FRAMEWORK: dynamo-trt
PRECISION: fp4
CONC: '388'
DURATION: '3600'
KV_OFFLOADING: none
ETCD_LEASE_TTL: '120'
DYN_ROUTER_QUEUE_THRESHOLD: None
DYN_TOKENIZER_CACHE: '1'
DYN_TOKENIZER_CACHE_BYTES: '8000000000'
DYN_TOKENIZER: fastokens
DYN_ROUTER_SESSION_AFFINITY_TTL_SECS: '14400'
args:
router-mode: kv
no-kv-events: true
active-decode-blocks-threshold: None
active-prefill-tokens-threshold: None
active-prefill-tokens-threshold-frac: None
orchestrator_placement: first_decode
benchmark:
client_placement: last_decode
type: custom
use_chat_template: true
command: bash /infmax-workspace/benchmarks/multi_node/agentic_srt.sh
env:
MODEL: deepseek-ai/DeepSeek-V4-Pro
MODEL_PREFIX: dsv4
FRAMEWORK: dynamo-trt
PRECISION: fp4
CONC: '388'
DURATION: '3600'
KV_OFFLOADING: none
INFMAX_CONTAINER_WORKSPACE: /infmax-workspace
RESULT_DIR: /logs/agentic
PORT: '8000'
IS_MULTINODE: 'true'
AIPERF_DYNAMO_SESSION_TIMEOUT_SECONDS: '14400'
AIPERF_HTTP_X_DYNAMO_SESSION_ID_FROM_CORRELATION_ID: 'true'
AIPERF_USE_DYNAMO_CONV_AWARE_ROUTING: '0'
AIPERF_DATASET_MMAP_CACHE_DIR: /aiperf_mmap_cache
HF_HUB_CACHE: /hf_hub_cache
WEKA_LOADER_OVERRIDE: semianalysis_cc_traces_weka_062126
SERVED_MODEL_NAME: DeepSeek-V4-Pro
Loading
Loading