[Benchmark] 3× RTX 2080 Ti: CUDA1+CUDA2 expert caches vs a 3-card layer split — short-chat decode +25%, prefill −35%, long-context decode −6% #4429

Reproduction status

Measured locally

What did you measure, or what went wrong

docs/SECOND_GPU.md says the helper caches are measured on two cards and that three and four cards are not validated. This is a three-card measurement on Turing, comparing the helper layout against the layer split the same rig runs daily.

Short version: with two helper caches (CUDA1 + CUDA2) the short-chat decode is +24–25% over a 3-card layer split, but a 4.3k-token prompt decodes 6% slower and its prefill is 35% slower (prefill runs on CUDA0 alone). One helper is worse than the split on two of the three short chats and on long-context decode.

Hardware: 3× RTX 2080 Ti 22 GB (modded, 210/167/200 W), i9-7980XE (18c/36t, AVX-512 without VNNI/VBMI → AVX-2 expert path), 125.5 GiB DDR4-3600, kernel 7.0.0-38, driver 580.178.04, CUDA 13.0. No P2P (GPU0–1 x16, GPU2 x8).

Build: 0.2.7 (5e8a4cca, engine 0.1.52), source, -DCMAKE_CUDA_ARCHITECTURES=75.

Model and common arguments: Qwen3.8-Flash-Next GSQ-RCO IQ3_S (native gsq_iq3s pack), --prefill auto --spec 3 --spec-min-p 0.54 --mtp mtp/rt --max-context 262144 --kv int8 --kv-resident 32768 --trim-stage-weights --batch 2, STRATA_SM75_PERF=1 STRATA_NO_LARGEPAGES=1 STRATA_BF16_TC=1. Greedy sampling (temperature 0, top_k 1).

Arms (one server per arm, two rounds, order reversed in round 2, unique prompt tag per request so nothing is reused):

• A — "gpu": [0,1,2], "layer_split": "16,32", --expert-cache auto (the daily config)
• B — no gpu key, CUDA_VISIBLE_DEVICES=0,1,2, --expert-cache auto --expert-cache-device1 auto --expert-cache-device2 auto --remote-expert-opt
• C — same as B with CUDA_VISIBLE_DEVICES=0,1 and only --expert-cache-device1 auto

Numbers (median of 2 rounds, token counts identical across arms)

Startup lines in B: remote ranking: 24576 profiled pairs, 0 other pairs to fill CUDA1..3, CUDA1: 9025 additional experts, 17.22 GiB, CUDA2: 9025 additional experts, 17.23 GiB. In C: CUDA1: 10736 additional experts, 20.38 GiB. Per request in B: CUDA1 ~83k expert entries / 1,439 MiB with full rows, CUDA2 ~50k entries / 1,477 MiB.

What I take from it

• The helper layout is a short-prompt/decode win and a long-prompt loss on this rig: prefill stays on CUDA0, so it reads at one card's rate (708 vs 1,095 tok/s across three), and long-context decode pays the pinned-host round trips.
• Two helpers is materially better than one (85.9 vs 71.4 on chat2, 81.2 vs 65.6 on the 4.3k prompt), so on a 3-card rig the second helper is worth having — but only if the workload is short-prompt.
• Nothing crashed, answers stayed coherent (Paris smoke test passes in all arms), load times 66–93 s.

What I did not test

• 4 cards (no hardware), and --peer-device beside a layer split ([Engine] RFE: allow --peer-device alongside a layer split (small 4th GPU as a silent expert-cache tier) #3834 asks for that).
• Long context: the prompt here is 4.3k tokens; the prefill gap should widen with length, but I did not measure 32k–131k.
• stripe vs layer placement, explicit expert counts instead of auto, and --remote-expert-opt off (I only ran it on in B/C).
• Identity: the doc says the helper path and --remote-expert-opt change summation order; I did not run a KL or --window-hashes check.