Skip to content

Add MiniMax-M3 NVFP4 GB300 vLLM Disaggregated AgentX with EAGLE3-GQA MTP / 新增 MiniMax-M3 NVFP4 GB300 vLLM 分离式 AgentX EAGLE3-GQA MTP 配置 #2663

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
hshrivastava-droid wants to merge 5 commits into
base: main
Choose a base branch
from
+1,738 −1
Open

Add MiniMax-M3 NVFP4 GB300 vLLM Disaggregated AgentX with EAGLE3-GQA MTP / 新增 MiniMax-M3 NVFP4 GB300 vLLM 分离式 AgentX EAGLE3-GQA MTP 配置 #2663

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
c0f60f3
feat(minimaxm3): add gb300 disaggregated agentx config
hshrivastava-droid Aug 18, 2026
1fbad05
chore(changelog): link PR #2663
hshrivastava-droid Aug 18, 2026
a717638
chore: merge main and resolve PR #2663 conflicts
Oseltamivir Aug 21, 2026
e4b0828
fix(config): run MiniMax-M3 GB300 AgentX thinking-on to match golden …
Ankur-singh Aug 21, 2026
d30a369
fix(config): cover the eval twin with dyn-default-thinking-mode / 用 d…
Ankur-singh Aug 21, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
163 changes: 163 additions & 0 deletions ...ti_node/srt-slurm-recipes/vllm/minimax-m3/agentic/gb300-fp4/1p1d-tep4-tp4-c1-agentic.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,163 @@
name: "minimax-m3-vllm-disagg-gb300-1p1d-tep4-tp4-c1-fp4-agentic"

model:
path: "nvidia/MiniMax-M3-NVFP4"
container: "vllm/vllm-openai:nightly-5e35a6f4f9bbc217c599692157ca985c894373f7"
precision: "fp4"

identity:
model:
repo: "nvidia/MiniMax-M3-NVFP4"
container:
image: "vllm/vllm-openai:nightly-5e35a6f4f9bbc217c599692157ca985c894373f7"
frameworks:
dynamo: "1.4.0.dev20260730"

dynamo:
version: "1.4.0.dev20260730"
install: true

health_check:
max_attempts: 2160
interval_seconds: 10

resources:
gpu_type: "gb300"
gpus_per_node: 4
het_jobs: false
spread_workers: false
prefill_nodes: 1
decode_nodes: 1
prefill_workers: 1
decode_workers: 1
gpus_per_prefill: 4
gpus_per_decode: 4

infra:
etcd_nats_dedicated_node: false
nats_max_payload_mb: 32

environment:
PYTHONHASHSEED: "0"

frontend:
type: dynamo
enable_multiple_frontends: false
nginx_session_affinity: true
nginx_session_affinity_header: "X-Dynamo-Session-ID"
args:
router-mode: "least-loaded"
router-session-affinity-ttl-secs: 1800
env:
DYN_TOKENIZER: "fastokens"
DYN_TOKENIZER_CACHE_BYTES: "8589934592"
DYN_TCP_CONNECT_TIMEOUT: "120"

backend:
type: vllm
connector: null
dp_launch_mode: per_node
mooncake_kv_store:
env:
MC_ENABLE_DEST_DEVICE_AFFINITY: "1"
MC_STORE_CLIENT_METRIC: "1"
MC_STORE_CLIENT_METRIC_INTERVAL: "5"
MC_TE_METRIC: "0"
store_config:
metadata_server: "P2PHANDSHAKE"
global_segment_size: "200GB"
local_buffer_size: "4GB"
protocol: "rdma"
device_name: "mlx5_0,mlx5_1,mlx5_2,mlx5_3"
mode: "embedded"
enable_offload: true
vllm_config:
prefill:
served-model-name: "nvidia/MiniMax-M3-NVFP4"
kv-cache-dtype: "fp8"
block-size: 128
trust-remote-code: true
enable-prefix-caching: true
language-model-only: true
no-enable-flashinfer-autotune: true
reasoning-parser: "minimax_m3"
dyn-tool-call-parser: "minimax_m3"
dyn-reasoning-parser: "minimax_m3"
max-cudagraph-capture-size: 512
Comment on lines +76 to +86

@claude claude Bot Aug 18, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 All 10 new gb300-fp4 MiniMax-M3 agentic/eval-agentic recipes set only reasoning-parser: minimax_m3 in prefill and decode, but omit dyn-tool-call-parser: minimax_m3 and dyn-reasoning-parser: minimax_m3 — args every existing MiniMax-M3 dynamo+vllm agentic recipe sets (e.g. gb200-fp4/agentic/disagg-1p1d-dep8-dep4-agentic.yaml lines 71-73, 97-99). Without these, the Dynamo frontend falls back to default tool-call/reasoning parsing for an agentic-coding workload that depends on correct tool-call extraction, which will misparse MiniMax-M3 output (especially in the real-verification -eval twins). Please add both dyn-tool-call-parser: "minimax_m3" and dyn-reasoning-parser: "minimax_m3" to the prefill and decode vllm_config blocks in all 10 new files.

Extended reasoning...

What the bug is: The 10 new benchmarks/multi_node/srt-slurm-recipes/vllm/minimax-m3/agentic/gb300-fp4/*.yaml recipes configure vLLM's engine-side reasoning-parser: "minimax_m3" in both the prefill and decode vllm_config blocks, but never set dyn-tool-call-parser or dyn-reasoning-parser. These two dyn-* keys are distinct, Dynamo-frontend-side directives — they tell the Dynamo frontend (not the vLLM engine) how to extract tool-call and reasoning content out of the raw model output stream before it's returned to the client/benchmark harness. reasoning-parser alone only configures vLLM's own internal parsing; it does not populate the Dynamo frontend's extraction path.\n\nWhere this diverges from precedent: Every existing MiniMax-M3 dynamo+vllm agentic recipe in this repo sets all three keys together, in both prefill and decode blocks. For example, gb200-fp4/agentic/disagg-1p1d-dep8-dep4-agentic.yaml sets reasoning-parser, dyn-tool-call-parser, and dyn-reasoning-parser, all to "minimax_m3", at lines 71-73 (prefill) and 97-99 (decode). Grepping all six gb200-fp4/agentic/*.yaml files (the agg-* and disagg-* siblings) confirms the same triad in every one. Grepping the 10 new gb300-fp4 files for dyn-tool-call-parser/dyn-reasoning-parser returns zero matches — only reasoning-parser (at line 83 prefill / line 103 decode in the representative 1p1d-tep4-tp4-c1-agentic.yaml) is present.\n\nWhy nothing else catches this: There's no schema validation on these recipe YAMLs enforcing that a given reasoning-parser value must be paired with matching dyn-* frontend args — the pairing is purely a convention followed by hand in every prior recipe. Nothing in launch_gb300-nv.sh or the srt-slurm framework injects a default derived from reasoning-parser; omitting the dyn-* keys just means the frontend uses whatever its built-in default parser is (likely none, or a generic one), not MiniMax-M3's actual tool-call format.\n\nImpact: These recipes exist specifically to benchmark an agentic-coding workload, where the benchmark harness needs to correctly extract tool calls from the model's output to measure/verify tool use. Without dyn-tool-call-parser: minimax_m3, the Dynamo frontend will fail to correctly delimit and extract MiniMax-M3's tool-call blocks, corrupting the agentic benchmark's traces. This is doubly important for the five *-eval-agentic.yaml twins, which run real MTP verification and rely on correctly-parsed tool calls to score correctness — a frontend parsing mismatch there would silently produce wrong eval results rather than an obvious crash.\n\nStep-by-step proof:\n1. Open gb200-fp4/agentic/disagg-1p1d-dep8-dep4-agentic.yaml (an existing, presumably-correct MiniMax-M3 dynamo+vllm agentic recipe) and look at the prefill vllm_config block: it sets reasoning-parser: "minimax_m3", dyn-tool-call-parser: "minimax_m3", and dyn-reasoning-parser: "minimax_m3" together (lines 71-73); the decode block repeats the same triad (lines 97-99).\n2. Now open the new 1p1d-tep4-tp4-c1-agentic.yaml added by this PR: the prefill block (around line 83) has only reasoning-parser: "minimax_m3"; the decode block (around line 103) has only the same single key. dyn-tool-call-parser and dyn-reasoning-parser do not appear anywhere in the file.\n3. Repeating this diff across the other 9 new gb300-fp4 files (both agentic and eval-agentic twins for each of the 5 topologies) shows the identical pattern: zero occurrences of either dyn-* key.\n4. Since reasoning-parser is vLLM-engine-scoped and the dyn-* keys are Dynamo-frontend-scoped (as evidenced by every other MiniMax-M3 recipe setting them independently and together), the frontend in these 10 new recipes has no MiniMax-M3-specific tool-call/reasoning extraction configured and falls back to its default behavior — which will misparse this model's actual output format.\n\nFix: Add dyn-tool-call-parser: "minimax_m3" and dyn-reasoning-parser: "minimax_m3" alongside reasoning-parser: "minimax_m3" in both the prefill and decode vllm_config blocks of all 10 new files, matching the established pattern in the gb200-fp4 agentic recipes.

max-num-batched-tokens: 16384
stream-interval: 20
gpu-memory-utilization: 0.9
tensor-parallel-size: 4
enable-expert-parallel: true
all2all-backend: "flashinfer_nvlink_one_sided"
attention-config: '{"backend":"FLASHINFER","use_trtllm_attention":true,"indexer_kv_dtype":"fp8"}'
speculative-config: '{"method":"eagle3","model":"Inferact/MiniMax-M3-EAGLE3-GQA","num_speculative_tokens":3,"attention_backend":"FLASH_ATTN","rejection_sample_method":"synthetic","synthetic_acceptance_length":2.78}'
kv-transfer-config: '{"kv_connector":"MultiConnector","kv_role":"kv_both","kv_load_failure_policy":"recompute","kv_connector_extra_config":{"connectors":[{"kv_connector":"NixlConnector","kv_role":"kv_both","kv_connector_extra_config":{"backends":["UCX"],"enforce_handshake_compat":false,"read_validation_timeout":30.0}},{"kv_connector":"MooncakeStoreConnector","kv_role":"kv_both","kv_connector_extra_config":{"lookup_async":true}}]}}'
enable-cumem-allocator: true
decode:
served-model-name: "nvidia/MiniMax-M3-NVFP4"
kv-cache-dtype: "fp8"
block-size: 128
trust-remote-code: true
enable-prefix-caching: true
language-model-only: true
no-enable-flashinfer-autotune: true
reasoning-parser: "minimax_m3"
dyn-tool-call-parser: "minimax_m3"
dyn-reasoning-parser: "minimax_m3"
dyn-default-thinking-mode: "enabled"
max-cudagraph-capture-size: 512
max-num-batched-tokens: 16384
max-num-seqs: 1024
stream-interval: 20
gpu-memory-utilization: 0.9
tensor-parallel-size: 4
enable-expert-parallel: false
attention-config: '{"backend":"FLASHINFER","use_trtllm_attention":true,"indexer_kv_dtype":"fp8","minimax_m3_msa_decode_backend":"cutlass"}'
speculative-config: '{"method":"eagle3","model":"Inferact/MiniMax-M3-EAGLE3-GQA","num_speculative_tokens":3,"attention_backend":"FLASH_ATTN","rejection_sample_method":"synthetic","synthetic_acceptance_length":2.78}'
kv-transfer-config: '{"kv_connector":"MultiConnector","kv_role":"kv_both","kv_load_failure_policy":"fail","kv_connector_extra_config":{"connectors":[{"kv_connector":"NixlConnector","kv_role":"kv_both","kv_connector_extra_config":{"backends":["UCX"],"enforce_handshake_compat":false,"read_validation_timeout":30.0}},{"kv_connector":"MooncakeStoreConnector","kv_role":"kv_consumer","kv_connector_extra_config":{"enable_lookup":false}}]}}'
enable-cumem-allocator: true
prefill_environment: &worker_environment
HF_HUB_CACHE: "/hf_hub_cache"
HUGGINGFACE_HUB_CACHE: "/hf_hub_cache"
TRANSFORMERS_CACHE: "/hf_hub_cache"
VLLM_ENGINE_READY_TIMEOUT_S: "3600"
DYN_TCP_CONNECT_TIMEOUT: "120"
VLLM_FLOAT32_MATMUL_PRECISION: "high"
VLLM_FLASHINFER_ALLREDUCE_BACKEND: "trtllm"
VLLM_USE_NCCL_SYMM_MEM: "0"
VLLM_ALLREDUCE_USE_SYMM_MEM: "0"
VLLM_MOONCAKE_LOAD_RECV_THREADS: "20"
UCX_MEMTYPE_CACHE: "n"
UCX_CUDA_IPC_ENABLE_MNNVL: "y"
UCX_MODULE_DIR: "/usr/local/lib/python3.12/dist-packages/nixl_cu13.libs/ucx"
UCX_RNDV_PIPELINE_ERROR_HANDLING: "y"
NCCL_CUMEM_ENABLE: "1"
NCCL_MNNVL_ENABLE: "1"
NCCL_NVLS_ENABLE: "1"
NCCL_IB_HCA: "mlx5_0,mlx5_1,mlx5_2,mlx5_3"
decode_environment:
<<: *worker_environment
VLLM_FLASHINFER_ALLREDUCE_BACKEND: "mnnvl"

sbatch_directives:
cpus-per-task: "72"
mem: "0"

srun_options:
container-remap-root: ""

benchmark:
type: custom
command: "bash /infmax-workspace/benchmarks/multi_node/agentic_srt.sh"
env:
INFMAX_CONTAINER_WORKSPACE: "/infmax-workspace"
RESULT_DIR: "/logs/agentic"
PORT: "8000"
IS_MULTINODE: "true"
AIPERF_HTTP_X_DYNAMO_SESSION_ID_FROM_CORRELATION_ID: "true"
AIPERF_EXTRA_INPUTS: "thinking:true"
AIPERF_DATASET_MMAP_CACHE_DIR: "/aiperf_mmap_cache"
AIPERF_SERVER_METRICS_COLLECTION_INTERVAL: "1.0"
HF_HUB_CACHE: "/hf_hub_cache"
WEKA_LOADER_OVERRIDE: "semianalysis_cc_traces_weka_062126"
Comment on lines +152 to +163

@claude claude Bot Aug 18, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 All 10 new MiniMax-M3 GB300 agentic recipes under gb300-fp4/ omit AIPERF_EXTRA_INPUTS: "thinking:true" from benchmark.env, unlike every existing MiniMax-M3 agentic recipe (e.g. the GB200 sibling disagg-1p1d-dep8-dep4-agentic.yaml line 113, and all agg-*-agentic.yaml files). Since MiniMax-M3 is a reasoning model and this flag is what tells aiperf to exercise thinking mode, these runs will silently benchmark with thinking disabled or defaulted, producing output-length/acceptance-length/latency numbers that are not comparable to the model's real agentic-coding behavior or its own GB200 curve; add the flag to all 10 recipes' benchmark.env blocks before merging.

Extended reasoning...

The bug. Every one of the 10 new recipes under benchmarks/multi_node/srt-slurm-recipes/vllm/minimax-m3/agentic/gb300-fp4/ (5 throughput + 5 eval variants) defines a benchmark.env block that ends at WEKA_LOADER_OVERRIDE and never sets AIPERF_EXTRA_INPUTS. Every pre-existing MiniMax-M3 agentic recipe does set it — confirmed by grep, AIPERF_EXTRA_INPUTS: "thinking:true" appears in all 6 files under minimax-m3/gb200-fp4/agentic/ (the 4 agg-*-agentic.yaml files and disagg-1p1d-dep8-dep4-agentic.yaml line 113), and nowhere else in the repo except benchmarks/benchmark_lib.sh (the consumer).

How the flag is used. benchmark_lib.sh:2052-2053 shows the mechanism directly:

if [ -n "${AIPERF_EXTRA_INPUTS:-}" ]; then
    REPLAY_CMD+=" --extra-inputs $AIPERF_EXTRA_INPUTS"
fi

This is a plain conditional with no fallback/default — if the recipe's benchmark.env doesn't set AIPERF_EXTRA_INPUTS, aiperf never receives --extra-inputs thinking:true and there is no other code path that injects it. This is exactly why every existing MiniMax-M3 recipe sets it explicitly: thinking mode is not on by default for this client-side aiperf flag.

Why the shared dataset doesn't save it. One might assume the WEKA_LOADER_OVERRIDE: "semianalysis_cc_traces_weka_062126" dataset implicitly drives thinking behavior, but the direct GB200 sibling recipe (disagg-1p1d-dep8-dep4-agentic.yaml) uses the exact same dataset value AND still sets thinking:true explicitly — so the dataset alone doesn't enable it. reasoning-parser: "minimax_m3" (present in both GB200 and GB300 recipes) is a separate, server-side vLLM response-parsing setting; it does not make aiperf send requests requesting thinking mode.

Why this looks like an authoring accident rather than an intentional change. The new GB300 benchmark.env blocks are structurally different from the GB200 ones — they drop AIPERF_USE_DYNAMO_CONV_AWARE_ROUTING, AIPERF_DYNAMO_SESSION_TIMEOUT_SECONDS, and AIPERF_REQUIRED_SERVER_METRIC_PREFIX, while adding AIPERF_SERVER_METRICS_COLLECTION_INTERVAL. This is consistent with the recipes having been templated from the DeepSeek-V4 agentic vLLM recipes (a non-reasoning model family that never sets this flag) rather than from the MiniMax-M3 GB200 agentic pattern. Nothing in the PR description documents disabling thinking mode as an intended change.

Impact. These recipes exist specifically to produce a GB300 curve comparable to the GB200 MiniMax-M3 agentic curve, and to power a "real-verification" eval twin plus a tuned synthetic_acceptance_length: 2.78 for the thinking workload. Benchmarking with thinking silently off means: (1) the throughput numbers (output length, acceptance length, latency) reflect a different workload than intended and aren't comparable to the GB200 sibling; (2) the eval twin verifies accuracy under the wrong mode; (3) the synthetic acceptance length tuning (2.78) was presumably chosen for the thinking workload and may not apply.

Proof walkthrough.

  1. Recipe gb300-fp4/1p1d-tep4-tp4-c1-agentic.yaml sets benchmark.env (lines 147-157) with WEKA_LOADER_OVERRIDE as the last key — no AIPERF_EXTRA_INPUTS.
  2. runners/launch_gb300-nv.sh exports every benchmark.env key as an environment variable for the agentic benchmark job (bash /infmax-workspace/benchmarks/multi_node/agentic_srt.sh).
  3. Inside that flow, benchmark_lib.sh:2052 checks [ -n "${AIPERF_EXTRA_INPUTS:-}" ] — since the var was never exported, this is false, and --extra-inputs thinking:true is never appended to REPLAY_CMD.
  4. aiperf therefore issues requests without the thinking-mode extra input, while the GB200 sibling recipe (same dataset, same model) does append it and does exercise thinking mode.
  5. Result: the GB300 run's output-length/acceptance-length/latency distribution reflects non-thinking (or default) behavior, not comparable to the GB200 curve it's meant to sit alongside.

Fix. Add AIPERF_EXTRA_INPUTS: "thinking:true" to the benchmark.env block of all 10 new gb300-fp4 recipe files, matching the GB200 pattern.

163 changes: 163 additions & 0 deletions ...de/srt-slurm-recipes/vllm/minimax-m3/agentic/gb300-fp4/1p1d-tep4-tp4-c1-eval-agentic.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,163 @@
name: "minimax-m3-vllm-disagg-gb300-1p1d-tep4-tp4-c1-fp4-eval-agentic"

model:
path: "nvidia/MiniMax-M3-NVFP4"
container: "vllm/vllm-openai:nightly-5e35a6f4f9bbc217c599692157ca985c894373f7"
precision: "fp4"

identity:
model:
repo: "nvidia/MiniMax-M3-NVFP4"
container:
image: "vllm/vllm-openai:nightly-5e35a6f4f9bbc217c599692157ca985c894373f7"
frameworks:
dynamo: "1.4.0.dev20260730"

dynamo:
version: "1.4.0.dev20260730"
install: true

health_check:
max_attempts: 2160
interval_seconds: 10

resources:
gpu_type: "gb300"
gpus_per_node: 4
het_jobs: false
spread_workers: false
prefill_nodes: 1
decode_nodes: 1
prefill_workers: 1
decode_workers: 1
gpus_per_prefill: 4
gpus_per_decode: 4

infra:
etcd_nats_dedicated_node: false
nats_max_payload_mb: 32

environment:
PYTHONHASHSEED: "0"

frontend:
type: dynamo
enable_multiple_frontends: false
nginx_session_affinity: true
nginx_session_affinity_header: "X-Dynamo-Session-ID"
args:
router-mode: "least-loaded"
router-session-affinity-ttl-secs: 1800
env:
DYN_TOKENIZER: "fastokens"
DYN_TOKENIZER_CACHE_BYTES: "8589934592"
DYN_TCP_CONNECT_TIMEOUT: "120"

backend:
type: vllm
connector: null
dp_launch_mode: per_node
mooncake_kv_store:
env:
MC_ENABLE_DEST_DEVICE_AFFINITY: "1"
MC_STORE_CLIENT_METRIC: "1"
MC_STORE_CLIENT_METRIC_INTERVAL: "5"
MC_TE_METRIC: "0"
store_config:
metadata_server: "P2PHANDSHAKE"
global_segment_size: "200GB"
local_buffer_size: "4GB"
protocol: "rdma"
device_name: "mlx5_0,mlx5_1,mlx5_2,mlx5_3"
mode: "embedded"
enable_offload: true
vllm_config:
prefill:
served-model-name: "nvidia/MiniMax-M3-NVFP4"
kv-cache-dtype: "fp8"
block-size: 128
trust-remote-code: true
enable-prefix-caching: true
language-model-only: true
no-enable-flashinfer-autotune: true
reasoning-parser: "minimax_m3"
dyn-tool-call-parser: "minimax_m3"
dyn-reasoning-parser: "minimax_m3"
max-cudagraph-capture-size: 512
max-num-batched-tokens: 16384
stream-interval: 20
gpu-memory-utilization: 0.9
tensor-parallel-size: 4
enable-expert-parallel: true
all2all-backend: "flashinfer_nvlink_one_sided"
attention-config: '{"backend":"FLASHINFER","use_trtllm_attention":true,"indexer_kv_dtype":"fp8"}'
speculative-config: '{"method":"eagle3","model":"Inferact/MiniMax-M3-EAGLE3-GQA","num_speculative_tokens":3,"attention_backend":"FLASH_ATTN"}'
kv-transfer-config: '{"kv_connector":"MultiConnector","kv_role":"kv_both","kv_load_failure_policy":"recompute","kv_connector_extra_config":{"connectors":[{"kv_connector":"NixlConnector","kv_role":"kv_both","kv_connector_extra_config":{"backends":["UCX"],"enforce_handshake_compat":false,"read_validation_timeout":30.0}},{"kv_connector":"MooncakeStoreConnector","kv_role":"kv_both","kv_connector_extra_config":{"lookup_async":true}}]}}'
enable-cumem-allocator: true
decode:
served-model-name: "nvidia/MiniMax-M3-NVFP4"
kv-cache-dtype: "fp8"
block-size: 128
trust-remote-code: true
enable-prefix-caching: true
language-model-only: true
no-enable-flashinfer-autotune: true
reasoning-parser: "minimax_m3"
dyn-tool-call-parser: "minimax_m3"
dyn-reasoning-parser: "minimax_m3"
dyn-default-thinking-mode: "enabled"
max-cudagraph-capture-size: 512
max-num-batched-tokens: 16384
max-num-seqs: 1024
stream-interval: 20
gpu-memory-utilization: 0.9
tensor-parallel-size: 4
enable-expert-parallel: false
attention-config: '{"backend":"FLASHINFER","use_trtllm_attention":true,"indexer_kv_dtype":"fp8","minimax_m3_msa_decode_backend":"cutlass"}'
speculative-config: '{"method":"eagle3","model":"Inferact/MiniMax-M3-EAGLE3-GQA","num_speculative_tokens":3,"attention_backend":"FLASH_ATTN"}'
kv-transfer-config: '{"kv_connector":"MultiConnector","kv_role":"kv_both","kv_load_failure_policy":"fail","kv_connector_extra_config":{"connectors":[{"kv_connector":"NixlConnector","kv_role":"kv_both","kv_connector_extra_config":{"backends":["UCX"],"enforce_handshake_compat":false,"read_validation_timeout":30.0}},{"kv_connector":"MooncakeStoreConnector","kv_role":"kv_consumer","kv_connector_extra_config":{"enable_lookup":false}}]}}'
enable-cumem-allocator: true
prefill_environment: &worker_environment
HF_HUB_CACHE: "/hf_hub_cache"
HUGGINGFACE_HUB_CACHE: "/hf_hub_cache"
TRANSFORMERS_CACHE: "/hf_hub_cache"
VLLM_ENGINE_READY_TIMEOUT_S: "3600"
DYN_TCP_CONNECT_TIMEOUT: "120"
VLLM_FLOAT32_MATMUL_PRECISION: "high"
VLLM_FLASHINFER_ALLREDUCE_BACKEND: "trtllm"
VLLM_USE_NCCL_SYMM_MEM: "0"
VLLM_ALLREDUCE_USE_SYMM_MEM: "0"
VLLM_MOONCAKE_LOAD_RECV_THREADS: "20"
UCX_MEMTYPE_CACHE: "n"
UCX_CUDA_IPC_ENABLE_MNNVL: "y"
UCX_MODULE_DIR: "/usr/local/lib/python3.12/dist-packages/nixl_cu13.libs/ucx"
UCX_RNDV_PIPELINE_ERROR_HANDLING: "y"
NCCL_CUMEM_ENABLE: "1"
NCCL_MNNVL_ENABLE: "1"
NCCL_NVLS_ENABLE: "1"
NCCL_IB_HCA: "mlx5_0,mlx5_1,mlx5_2,mlx5_3"
decode_environment:
<<: *worker_environment
VLLM_FLASHINFER_ALLREDUCE_BACKEND: "mnnvl"

sbatch_directives:
cpus-per-task: "72"
mem: "0"

srun_options:
container-remap-root: ""

benchmark:
type: custom
command: "bash /infmax-workspace/benchmarks/multi_node/agentic_srt.sh"
env:
INFMAX_CONTAINER_WORKSPACE: "/infmax-workspace"
RESULT_DIR: "/logs/agentic"
PORT: "8000"
IS_MULTINODE: "true"
AIPERF_HTTP_X_DYNAMO_SESSION_ID_FROM_CORRELATION_ID: "true"
AIPERF_EXTRA_INPUTS: "thinking:true"
AIPERF_DATASET_MMAP_CACHE_DIR: "/aiperf_mmap_cache"
AIPERF_SERVER_METRICS_COLLECTION_INTERVAL: "1.0"
HF_HUB_CACHE: "/hf_hub_cache"
WEKA_LOADER_OVERRIDE: "semianalysis_cc_traces_weka_062126"
Loading