diff --git a/.agents/issue-index.md b/.agents/issue-index.md index cf724c0f3..b902e2e22 100644 --- a/.agents/issue-index.md +++ b/.agents/issue-index.md @@ -383,6 +383,7 @@ rather than merged. `scripts/check-agent-record.py` gates both. | [#1189](https://github.com/mudler/vllm.cpp/issues/1189) | `VT-QUANT-FP8-GROUP` | Block-wise (128x128) FP8 so `Qwen/Qwen3.8-27B-FP8` runs instead of being refused. `weight_block_size` appears nowhere in `src/` or `include/`, so `469f38395` refuses the arm by name (#1166). Six independently landable milestones; M1 lands here. M1 is `vt::QuantFp8Group`, the dynamic per-token per-group activation quant, CPU and CUDA. The numerics mirror the kernel that ACTUALLY EXECUTES on a CUDA-alike platform with a contiguous input, which is the C++ custom op at `csrc/libtorch_stable/quantization/w8a8/fp8/per_token_group_quant.cu:42-96` and NOT the Triton kernel at `fp8_utils.py:95-150`: `fp8_utils.py:635-650` calls the former and returns before the latter. The two arms differ, and the difference is measurable rather than cosmetic. The CUDA kernel divides twice, `local_absmax / max_8bit` at `:68` and `static_cast(src) / y_s` at `:85`; the Triton kernel multiplies by `(1.0 / fp8_max)` at `fp8_utils.py:145` under a comment that names the 1-ULP gap. Upstream's own test tolerates the gap with `rtol=0.15` (`test_block_fp8.py:112-114`), so a value comparison cannot tell the two apart and only a byte comparison against a spelled-out reference can. `eps` is the reduction's INITIAL value (`:47`), not a post-clamp, which is what keeps an all-zero group from dividing by zero. Scope refused here and owed to later milestones: the block-scaled GEMM (M2), `Fp8BlockWeight` and the loader (M3), `Fp8BlockLinearMethod` and the Qwen3.5 wiring (M4), the mainloop-scaled CUTLASS kernel and the column-major/TMA-aligned scale layouts (M5), merged `gate_up`/QKV (M6). M1 lands UNREACHED: no production entry point dispatches `vt::QuantFp8Group` at its merge commit, M4 owns the wiring, and `.agents/specs/vt-quant-fp8-group.md` lists it under `## Owed`. The CUDA arm compiles and its on-hardware leg is owed too, because the row took no GPU lease by design: the CPU arm is the gateable one | feature | | [#1190](https://github.com/mudler/vllm.cpp/issues/1190) | `ENV-AGNOSTIC-CAMPAIGN` | One operator's hosts, share paths and addresses are written into 227 tracked files where a `.env` placeholder belongs, so a second developer who follows the protocol documents is told to reach a box on one home network. Re-derived at `fd64c76ee` with `git grep -cIE 'dgx\.casa\|nas_share\|192\.168\.\|thor:gpu0'`: `dgx.casa` 203 files, `nas_share` 32, `192.168.` 24, `thor:gpu0` 11. The mechanism was already there and almost unused: `.env.example` is tracked, `.env` and `.agents/developer-preferences.md` are ignored, `scripts/agent-onboard.py --env-set` already records one answered value and refuses an undeclared key, and `${VLLM_SOURCE}` and `${GPU_LOCK}` already resolve in 59 and 50 files while `${DEVICE_ARCH}`, `${DEVICE_TOOLKIT_ROOT}` and `${DEVICE_COMPILER}` resolve in none. This SCOPING row lands the rule and the mechanism, not the sweep. The rule is a substitution test: replace the literal with a second developer's value and if the sentence stays true it is CONFIGURATION and becomes `${KEY}`, and if it becomes false it is PROVENANCE and stays literal. That reverses the issue's own ranking, because all 34 hits in the densest guide file `.agents/environment.md` are provenance or named-profile definition and none is configuration, so density does not predict the defect and a blind `sed` would falsify records. Landed: three new keys `GATE_CHECKOUT`, `SHARED_STORAGE_ROOT` and `GATE_DEVICE` derived from the literals that recur and map to no existing key; the create-on-first-use route in `scripts/agent-start.py`, which printed `environment: missing` as a status label and then listed next actions that never mentioned it, so the fallback in practice was a host name copied from a document; the matching obligation in `AGENTS.md`; and one worked example. The worked example `scripts/dgx-bringup.sh` found a live defect rather than a cosmetic one: it defaulted `CUTLASS_DIR` to `$HOME/cutlass_probe` while `.agents/environment.md:389` records `$HOME/cutlass-4.5.0` as mandatory on the same box, and a configure that misses CUTLASS silently drops the sm120a NVFP4 GEMM and FlashAttention-2, which that file measures as moving the SACRED `test_qwen27_paged_engine` from 235/235 to 234/235 with the source untouched, so a stale hard-coded default is a false green. Its new test also caught that `set -a; . ./.env; set +a` over an `.env.example` copy blanks a value the caller exported, so the process-environment contract is now executable. Waves `ENV-AGNOSTIC-W1-TOOLING` through `ENV-AGNOSTIC-W5-LEDGERS` own the sweep, partition all 227 files, and open their own issues. Spec [`env-agnostic.md`](specs/env-agnostic.md) | bug | | [#1193](https://github.com/mudler/vllm.cpp/issues/1193) | `SPEC-DSPARK-QWEN3-ROUTING` | A Qwen3 DSpark draft declaring `architectures=["DSparkDraftModel"]` with `model_type` `qwen3` has no route. The pin forces every DSpark draft that is not `Qwen3DSparkModel` or `Gemma4DSparkModel` onto `model_type` `deepseek_v4` (`vllm/config/speculative.py:934-944` @ `555967922`), and vLLM PR 52197 (merged 2026-08-17 at `7075ddac`) replaced that with a leading branch normalizing the pair to `Qwen3DSparkModel`. We diverge from BOTH: the forced rewrite was never ported, so nothing in `src/vllm/entrypoints/model_loader.cpp` reads a draft config's `architectures` key at all, and `SpeculativeConfig::IsDsparkDraft` (`include/vllm/config/speculative.h:120-136`) has no production caller — every reference outside its header is in `tests/vllm/config/test_speculative_dspark.cpp:132-140`, and `ResolveSpecConfig` branches on `cli.method` alone. The checkpoint is real and gateable here: `RadixArk/Qwen3.8-27B-DSpark` at revision `85ef153be924f17ce4bf62726954eeaa4a73e854` carries exactly that config shape in one 2718576122-byte shard, drafting five layers for a 64-layer Qwen3.8-27B target | bug | +| [#1194](https://github.com/mudler/vllm.cpp/issues/1194) | `ENG-EXPERT-STREAM` | `Qwen3.8-2.4T-A95B UD-Q1_0` serves on one 119.631 GiB GB10 and no user-facing document said how. AGENTS.md binds the checkpoint pin to [`docs/USAGE.md`](../docs/USAGE.md): file name, size, repo AND revision, sha256 for a quantized artifact. Revision `567d3e6ac26c5474b18311e619c04350fb9a5556` appeared only in this index (row #1123) and in [`expert-streaming.md`](specs/expert-streaming.md), and `USAGE.md` named the checkpoint three times and pinned it zero times. Nor was there an end-to-end recipe: the `Streaming routed experts from disk (capacity mode)` section is a MECHANISM reference (schema, precedence, #1135 reach limits, statistics line, the per-device limits) and never says which file `--model` takes, what the load costs, what decode costs, or where the ceiling is, while its four sibling per-model recipes in the same file (Muse Glimmer 30B, Nemotron-3.5-Lightning-30B, MiniMax-H3, MiniMax-Music3) all do. FIXED as a fifth sibling recipe that LINKS the mechanism section rather than restating it, so each fact keeps one home; `docs/examples/` was NOT created, because `examples/` in this tree is runnable C++ CMake targets and `docs/` is one topic per file listed in the README index, so a new markdown genre would have had one member and duplicated facts that already have owners. The pin is settled against the ARTIFACT rather than against a document or the HuggingFace tree API, whose `lfs.oid` is fabricated for a gated repo: **ten** shards `UD-Q1_0/Qwen3.8-2.4T-A95B-UD-Q1_0-000{01..10}-of-00010.gguf` summing to exactly **397,256,393,248 B (369.97 GiB)**, shard 1 declaring `split.count = 10` and `split.tensors.count = 1702` in its own metadata and carrying zero tensors, and shard 1's sha256 recomputed from the mirrored copy as `b7770552...0334d` (10,943,264 B), equal to the download manifest that also gives shard 2 `2765517f...3dcd` (48,759,636,544 B). That settles [#1420](https://github.com/mudler/vllm.cpp/issues/1420), whose two copy-paste commands named `-00001-of-00008.gguf`, a file that does not exist at that revision, and both now name `-00001-of-00010.gguf`. The section publishes no new measurement and carries only figures already recorded: the CPU arm's **11.05 s/token at 4000 expert-stream slots** (`ENG-EXPERT-STREAM-DEVICE` W0e), the 8000-slot 39.98 to 45.40 s/token memory result, and the 66.7 s/token streaming-off run, with NO ratio between them, because they were taken on different source trees. No `--device cuda` speed number appears, because that arm's token gate FAILS and `docs/BENCHMARKS.md` carries G0-SPEED as VOID. The unit arithmetic was re-derived rather than transcribed: 2790 slices x 2,490,368 B is **6.95 GB (6.47 GiB)** per token, so a ~5 GB/s NVMe floor is **1.39 s/token (~0.72 tok/s)**, not the 1.29 s / 0.77 tok/s a GiB-for-GB substitution gives | doc | | [#1202](https://github.com/mudler/vllm.cpp/issues/1202) | — | `Ltx2FuseLoraIntoTensor` (`src/vllm/model_executor/models/ltx2_lora.cpp:321-334`) computes the `(B * strength) @ A` LoRA product with a scalar single-threaded triple loop: one thread, no blocking, no SIMD, a non-inlined `vt::BF16ToF32` per multiply, and an inner operand `pair->a[k * cols + i]` striding by `cols` so every load in the innermost loop is its own cache line. Measured on `dgx` (GB10, 20 cores) loading the full/dev transformer (21,004,025,600 params) with the shipped 8.9 GB distilled adapter: three `gdb` stacks all reading `vt::BF16ToF32` <- `Ltx2FuseLoraIntoTensor` <- `Ltx2LoadDitFromSafetensors` <- `Ltx2VideoEngine::Load`, one thread at 99.9% of one core with 19 idle, and an f32 working set growing 9.432 -> 10.235 GiB over 300-629 s = **2.3% of one pass in 10.4 minutes**, cross-checked against the sum of `out*in*rank` over the 1660 targeted modules = 8.53e12 MAC, consistent with ~0.53 GFLOP/s. The operation is a rank-`r` GEMM and belongs on the `vt::` GEMM seam like every other projection in the tree; the arithmetic is already a correct mirror of `fuse_loras.py:103-116` (`B * strength` rounds to bf16 BEFORE the product, f32 accumulation, bf16 store) and only the execution strategy is wrong, so a replacement has a bit-exact oracle rather than a tolerance. Blocks every LoRA-bearing pipeline kind on the full model; `one_stage` is unaffected because upstream marks it `Full` with no adapter, which is why it is the only full-model arm that currently reaches generation. Owed by [`ltx25-decode-speed.md`](specs/ltx25-decode-speed.md) `## Owed`, whose §5 already frames "why the decode is single-threaded and on the host" | perf | | [#1208](https://github.com/mudler/vllm.cpp/issues/1208) | — | `Linear` in the LTX-2.5 text tower (`src/vllm/model_executor/models/ltx2_text_encoder.cpp:60-71`) is a scalar single-threaded triple loop with a `double` accumulator and both operands widened per multiply, so it cannot reach the f32 FMA path even if the compiler vectorised it. On the full model's caption projection (in_features on the order of 1.9e5) cost is `rows * out_features * in_features` and it dominates the pre-generation wall: measured on `dgx`, resident memory went flat at t≈150 s and stayed **byte-identical for 1073 s** with stacks in the text feature extractor. Because `ltx2-gen` prints nothing between load and completion ([#1010](https://github.com/mudler/vllm.cpp/issues/1010) owns that), this presents to a user as a hang rather than as slow arithmetic, and it is why an earlier probe's "reached Generate" claim was retracted — the trace does not support it. TWO separate defects: the execution strategy, which belongs on the `vt::` GEMM seam; and the `double` accumulator, which is NOT a mirror — `torch.nn.functional.linear` on f32 inputs accumulates in f32, and the comment directly above this function cites `F.linear` as its reference, so the widening diverges from the oracle it names, cannot be bit-compared against upstream, and hides reduction-order differences an f32 accumulator would expose. Per the dtype-polarity rule an f64 accumulator kept deliberately needs a one-line reason beside it. Every LTX-2.5 pipeline kind goes through the text tower, so this is on the critical path of all of them including `one_stage`. Owed by [`ltx25-decode-speed.md`](specs/ltx25-decode-speed.md) `## Owed` | perf | | [#1210](https://github.com/mudler/vllm.cpp/issues/1210) | — | The two-stage LoRA rebind cost that `src/vllm/multimodal/ltx2_video.cpp:2843-2851` records as "UNMEASURED on real weights", with a later perf row owning the number, is now measured. A two-stage recipe loads FUSED, phase 0 asks `Ltx2PhaseLoraScope::kNoAdapters` and `Ltx2RebindDitLoras` un-fuses, phase 1 asks `kAllAdapters` and re-fuses — so the load-time fusion is **provably wasted**, undone before any denoise step runs, and the DiT is left fused so the next render pays the same two again. At [#1202](https://github.com/mudler/vllm.cpp/issues/1202)'s measured ~0.53 GFLOP/s each pass is hours, and a two-stage full-model render pays three of them before the first step. TWO independent fixes, not one change: making the fusion fast (#1202) shrinks the constant but leaves the wasted round trip; separately, `Ltx2PipelineRecipe::phases` is available before `Load` runs, so the load could honour phase 0's scope and skip the fuse/un-fuse entirely. The terminal fused state is chosen rather than forced and is worth revisiting in the same change. Affects `a2vid_two_stage` and `ti2vid_two_stage`; `one_stage` pays nothing. NOT CLAIMED: the wall-clock figures are a rate measured over a 10.4-minute window and extrapolated, not a completed pass — no two-stage full-model render has completed, so the end-to-end number stays open; the rate, thread count and stack attribution are measured. Owed by [`ltx25-decode-speed.md`](specs/ltx25-decode-speed.md) `## Owed` | perf | diff --git a/.agents/specs/expert-streaming.md b/.agents/specs/expert-streaming.md index 92e7cce03..bb577d99a 100644 --- a/.agents/specs/expert-streaming.md +++ b/.agents/specs/expert-streaming.md @@ -1586,6 +1586,39 @@ into other suites. | a fitting GGUF still loads | the SAME call with a generous budget: the throw is a LATER, different one (the synthetic file has no tokenizer), which is what proves the check let it through rather than that it never ran | | the CPU arm is untouched | the same file with `device=cpu` never refuses, whatever the budget | +## The user-facing recipe and the checkpoint pin ([#1194](https://github.com/mudler/vllm.cpp/issues/1194)) + +`docs/USAGE.md` named `Qwen3.8-2.4T-A95B UD-Q1_0` three times and pinned it zero +times. AGENTS.md binds the pin to that file: file name, size, repo AND revision, +and a sha256 for a quantized artifact. The streaming section there is a MECHANISM +reference. It never said which file `--model` takes, what the load costs, what +decode costs, or where the ceiling is, while the four sibling per-model recipes +in the same file all do. + +Landed as a fifth sibling recipe, `Qwen3.8-2.4T-A95B UD-Q1_0: 370 GiB served from +a 119 GiB box`, which LINKS the mechanism section rather than restating it, so +each fact keeps one home. Three facts are repeated on purpose and the section +says which ones and why: which device to use, the expert bytes a token reads, and +the two streaming decode figures. Both places quote `ENG-EXPERT-STREAM-DEVICE` +W0e, so the re-measure that row owes has to change both. + +The pin is settled against the ARTIFACT rather than against a document or the +HuggingFace tree API, whose `lfs.oid` is fabricated for a gated repo: TEN shards +`UD-Q1_0/Qwen3.8-2.4T-A95B-UD-Q1_0-000{01..10}-of-00010.gguf` summing to exactly +397,256,393,248 B (369.97 GiB), with shard 1 declaring `split.count = 10` and +`split.tensors.count = 1702` in its own metadata and carrying no tensors. That +also answers [#1420](https://github.com/mudler/vllm.cpp/issues/1420), whose two +copy-paste commands named `-of-00008`, a file that does not exist at that +revision. + +The section publishes no new measurement. Every figure in it was already +recorded, it carries no ratio between runs taken on different source trees, and +it carries no `--device cuda` speed number, because that arm's token gate fails. +The figures come from the W0e harness `benchmarks/expert_stream_device_w0e.cpp` +and not from the `vllm-server` command the section publishes, and the section +discloses that difference beside the prompt and environment differences it +already disclosed. + ## Owed Carried debt for this row. Each item names why it is not closed here. diff --git a/docs/USAGE.md b/docs/USAGE.md index 1c02ad949..232964aeb 100644 --- a/docs/USAGE.md +++ b/docs/USAGE.md @@ -4711,7 +4711,7 @@ save. ```sh VT_MOE_EXPERT_STREAM=1 \ VT_MOE_EXPERT_STREAM_SLOTS=4000 \ - ./build/examples/vllm-cli --model /models/Qwen3.8-2.4T-A95B-UD-Q1_0-00001-of-00008.gguf \ + ./build/examples/vllm-cli --model /models/Qwen3.8-2.4T-A95B-UD-Q1_0-00001-of-00010.gguf \ --prompt "The capital of France is" --max-tokens 16 ``` @@ -4821,7 +4821,7 @@ weights from the device to host RAM, and the `vllm_cpp` key governs the tier below that, where weights stay borrowed out of the file mapping. ```sh -./build/examples/vllm-server --model /models/Qwen3.8-2.4T-A95B-UD-Q1_0-00001-of-00008.gguf \ +./build/examples/vllm-server --model /models/Qwen3.8-2.4T-A95B-UD-Q1_0-00001-of-00010.gguf \ --offload-config '{"vllm_cpp":{"mmap":{"enabled":true,"prefault":false}, "expert_stream":{"enabled":true,"slots":4000}}}' ``` @@ -5058,6 +5058,249 @@ on a GB10, because host and device share one pool. `cudaMemGetInfo` answers honestly, and its `total` is EXACTLY `/proc/meminfo MemTotal` (125442340 kB) times 1024. Do not size this from `nvidia-smi`. +## Qwen3.8-2.4T-A95B `UD-Q1_0`: 370 GiB served from a 119 GiB box + +A 2.4-trillion-parameter mixture-of-experts checkpoint, three times the size of +the machine's memory, loads and answers on one DGX Spark. This section is the +recipe. The mechanism it drives is the previous section, +[Streaming routed experts from disk](#streaming-routed-experts-from-disk-capacity-mode), +which owns the config schema, the precedence rule, the statistics line, the slot +count warning and what each device can serve. This section links them rather than +restating them. It repeats three of their facts on purpose: which device to use, +the expert bytes a token reads, and the two streaming decode figures in +[What decode costs](#what-decode-costs-and-why-the-ceiling-is-where-it-is). A +recipe that leaves those out is not a recipe. Each of the three has one record, +so a correction has to change both places. The decode figures are +`ENG-EXPERT-STREAM-DEVICE` W0e in +[`.agents/benchmark-record.md`](../.agents/benchmark-record.md). + +**Read the speed before you spend the download.** Steady decode on the recipe +below is measured in seconds per token, and the floor under it is storage rather +than this implementation. This is a capacity result, not an interactive one. +[What decode costs](#what-decode-costs-and-why-the-ceiling-is-where-it-is) gives +the figure and the arithmetic behind it. + +**Use `--device cpu` for this checkpoint.** `--device cuda` loads and decodes it +too, and its token gate against the CPU arm does not pass, so every number below +was measured on the CPU arm. The previous section states that arm's six limits. + +### The exact weights + +| Field | Value | +|---|---| +| Repo and revision | `unsloth/Qwen3.8-2.4T-A95B-GGUF` @ `567d3e6ac26c5474b18311e619c04350fb9a5556` | +| Publisher | Unsloth, a third-party quantization rather than a first-party release | +| Arm | `UD-Q1_0`, which stores the expert towers at 1.1875 bits per weight | +| Files | `UD-Q1_0/Qwen3.8-2.4T-A95B-UD-Q1_0-000{01..10}-of-00010.gguf`, ten shards | +| Bytes | 397 256 393 248 over the ten files, that is 369.97 GiB | +| Tensor records | 1702, equal to the `split.tensors.count` the shards declare | +| sha256, shard 1 | `b7770552b2ac24e7334c917bc92e90e218e87cfe29484db65e62e8ef2a60334d` for `-00001-of-00010.gguf`, 10 943 264 B | +| sha256, shard 2 | `2765517f833c736338d3ab34354e1c10eb8d79e62325f998285b435e5cf03dcd` for `-00002-of-00010.gguf`, 48 759 636 544 B | + +**Ten shards, and the count is part of every file name.** A GGUF split writes +the total into each member's name, so `-of-00008` and `-of-00010` name different +files, and the wrong one gives a file-not-found after a 370 GiB download. The +two recipes in the previous section carried `-of-00008` until this change +([#1420](https://github.com/mudler/vllm.cpp/issues/1420)). The count is settled +against the artifact and not against a document: shard 1's own metadata declares +`split.count = 10` and `split.tensors.count = 1702`, its sha256 recomputed from +the mirrored copy equals the value above, and the ten files sum to exactly the +byte total above. Shard 1 holds **no tensors at all**. It is the metadata and the +split declaration, so it is the file that says what the other nine are. + +**A repo id alone is not a pin**, because a quantized checkpoint gets +re-quantized in place under an unchanged name. Shard 1's digest was recomputed +from the mirrored copy. Shard 2's is the download manifest's, and its byte count +was recomputed. + +```sh +hf download unsloth/Qwen3.8-2.4T-A95B-GGUF \ + --revision 567d3e6ac26c5474b18311e619c04350fb9a5556 \ + --include "UD-Q1_0/*" \ + --local-dir ./qwen3.8-2.4t-a95b-gguf +``` + +The files land under a `UD-Q1_0/` subdirectory of `--local-dir`, because that is +where they live in the repo. Point `--model` at a copy on **local NVMe**. A +network filesystem puts an uncontrolled variable in front of the expert reads +that every token makes. + +**The encoding has no upstream reference.** `UD-Q1_0` stores its expert towers as +`IQ1_XXXS`, which upstream llama.cpp does not define. The encoding exists only in +the `unslothai/llama.cpp` fork, pinned as a secondary oracle in +[`.agents/oracles/llama-cpp-unsloth.md`](../.agents/oracles/llama-cpp-unsloth.md). +That fork is recorded `gateable = no`, because it has not been shown to build and +run this model, and [#933](https://github.com/mudler/vllm.cpp/issues/933) owes +the measurement. There is therefore no token-exact denominator for anything below. + +### Build and serve + +A plain CPU build is enough. No CUDA is involved on this path. + +```sh +cmake -S . -B build +cmake --build build -j +``` + +```sh +./build/examples/vllm-server \ + --model ./qwen3.8-2.4t-a95b-gguf/UD-Q1_0/Qwen3.8-2.4T-A95B-UD-Q1_0-00001-of-00010.gguf \ + --offload-config '{"vllm_cpp":{"mmap":{"enabled":true,"prefault":false}, + "expert_stream":{"enabled":true,"slots":4000}}}' \ + --device cpu \ + --max-num-seqs 1 \ + --max-model-len 512 \ + --port 8899 +``` + +Five things in that command are load-bearing. + +- **`--model` takes shard 1, not the directory.** A directory sends the loader + down the HuggingFace branch, which fatals on a missing `config.json` before it + looks for a GGUF. Given shard 1 the reader finds its nine siblings from the + `-NNNNN-of-MMMMM.gguf` naming and cross-checks `split.count`. +- **`prefault: false` is the setting that decides whether this works.** + Pre-faulting is **on** by default, and it is the right default for a model that + fits: it walks every borrowed span at load, so the first-touch faults do not + land inside the timed prefill. For 335.62 GiB of expert towers that cannot fit, + it reads the whole checkpoint to populate a page cache that cannot hold it. +- **`mmap: true` confirms the default rather than enabling it.** It is already on + wherever the weights stay quantized, and it is what makes the checkpoint fit at + all: an expert tower is borrowed from the file mapping and costs zero anonymous + bytes, so only the dense remainder becomes resident. +- **`expert_stream` is off by default, and this recipe turns it on at 4000 + slots.** That count is the one the published decode figure was measured at, and + the previous section explains why 8000 is worse rather than better. +- **`--device cpu`.** The note at the top of this section says why. + +`--max-num-seqs 1` and a small `--max-model-len` keep the KV cache out of the +way. Nothing is batched at this speed, and the capacity argument itself holds +only at low concurrency: at high concurrency every step touches most of the +experts and the working set stops being one. + +The recorded runs set the equivalent environment variables rather than the +config document: `VT_GGUF_PREFAULT=0`, `VT_MOE_EXPERT_STREAM=1` and +`VT_MOE_EXPERT_STREAM_SLOTS=4000`. The two forms are the same switches, and a +variable beats a config field wherever both are set. + +They also ran a different binary. Every figure below comes from +`benchmarks/expert_stream_device_w0e.cpp`, a purpose-built C ABI client that +reports the token ids, a per-step timestamp and the expert-stream counters +together, which no shipped command does. The command above starts `vllm-server` +over the same engine. At seconds per token, the server's HTTP and SSE framing +sits far below the run-to-run spread recorded below. + +### What the load costs + +Expect to wait. Two runs of this arm are recorded on `dgx:gpu0`, a GB10 with +119.631 GiB of unified memory reading the checkpoint from local NVMe, with the +page cache dropped before each one (`ENG-EXPERT-STREAM-DEVICE` W0e, 18 and +19 August 2026, [`.agents/benchmark-record.md`](../.agents/benchmark-record.md)): + +| Axis | Run 1 | Run 2 | +|---|---|---| +| load | 271.1 s | 255.7 s | +| first token | 85.90 s | 79.09 s | +| peak resident set | 86.5 GiB | 86.5 GiB | +| peak swap | not sampled | 6 883 MiB | + +Resident memory after the load settles at about **62 GiB** of 119 GiB, measured +at 62.45 GiB on the same-lease CPU control run of `ENG-EXPERT-STREAM-DEVICE` +W0f. That is the +dense remainder plus the KV cache and the runtime, and it agrees with what the +checkpoint's own tensor table predicts: 21.56 GiB of `attn_qkv` and 17.25 GiB of +`ssm_out` expanded to bf16, plus 5.81 GiB of embeddings and F32 norms, so +44.6 GiB before the KV cache and the runtime. The other 335.62 GiB is mapped, +not copied. **The model does not fit because of streaming. It fits because of +borrowing.** + +Check readiness against the model list rather than against the process: + +```sh +curl -sf http://127.0.0.1:8899/v1/models +``` + +```sh +curl -s http://127.0.0.1:8899/v1/completions -H 'Content-Type: application/json' \ + -d '{"model":"Qwen3.8-2.4T-A95B-UD-Q1_0-00001-of-00010.gguf", + "prompt":"Q: What is the capital of France? A:","max_tokens":4}' +``` + +The 16 August 2026 run, which served with streaming off, answered +` Paris. Q: What`. That is the whole point: the output is coherent, so the +one-bit encoding and the borrowed-tower path are both faithful enough to serve. +The four W0e runs drive a fixed prompt of token ids instead of this request, and +all four returned the same 32 ids, which detokenize to ` Paris. Paris is a city +located in the northern part of France, on the Seine River. It is the largest +city in France and is known for its iconic`. + +### What decode costs, and why the ceiling is where it is + +Every figure here comes from the box named above. + +| Arm | Steady decode | Where it comes from | +|---|---|---| +| streaming on, 4000 slots | **11.05 s/token** | W0e rep 2, median over steps 4 to 32 | +| streaming on, 8000 slots | 39.98 and 45.40 s/token | W0e, the medians of two reps | +| streaming off | 66.7 s/token | 16 August 2026, streaming not yet enabled | + +That 4000-slot figure has a min of 9.43 and a max of 13.25 over its window, and +rep 1 of the same arm gives 11.22, which is 1.54% above it. **The +streaming-off row carries no ratio against the other two**, because it was taken +on a different source tree on a different date. The two slot counts came from one +binary on one lease and are comparable with each other; the previous section +carries that comparison. + +**A bigger cache came out slower**, which is why this recipe sets 4000 slots. +The previous section states the reason and its evidence. + +Do not quote a first-token time as a decode number. Token 1 carries the prefill +and the cold expert set. From token 2 onward you are watching steady state. The +complete measurement record is [docs/BENCHMARKS.md](BENCHMARKS.md). + +The arithmetic behind those seconds is short, and it decides everything. The +first three rows are read from the checkpoint's own metadata: + +| Quantity | Value | +|---|---| +| blocks (`qwen35moe.block_count`) | 93 | +| experts routed per block, of `qwen35moe.expert_count` | 10 of 512 | +| projections per routed expert | 3 | +| expert slices per token | 2790 | +| bytes per slice | 2 490 368, that is 2.375 MiB | +| expert working set per token | **6.95 GB**, that is 6.47 GiB | +| slots this recipe reserves | 4000, a 9.28 GiB arena | + +That figure is a working set and not an I/O rate, because the slot cache serves +part of it from memory. The recorded 32-token run at 4000 slots counted 37 096 +hits against 58 538 misses. + +**The floor is storage, not software.** 6.95 GB at the roughly 5 GB/s an NVMe of +this class sustains is 1.39 s/token whatever the code does, which is 0.72 tok/s. +Reaching 3 tok/s would demand about 21 GB/s of expert bandwidth, so most of those +reads would have to come from memory instead. The arena holds 4000 slices against +the 2790 a token needs, under one and a half tokens of working set, and +top-10-of-512 routing does not give consecutive tokens enough reuse to close the +rest. **If you need conversational speed from this model you need more memory or +fewer active parameters, not better software.** + +### What this does not establish + +- **The quantization is extreme.** The expert towers hold 1.1875 bits per weight, + and they are about 97% of the parameters. The output is coherent; this is not + the configuration to judge the model's quality by. +- **There is no oracle.** No entry in the oracle table runs this checkpoint on + this hardware, so there is no token-exact and no throughput denominator. Every + figure above is an absolute measurement of this implementation, compared + against nothing. +- **One request at a time.** Nothing here says anything about concurrency, and + the capacity argument stops holding as concurrency rises. +- **One box.** Every number was taken on one DGX Spark GB10 with the checkpoint + on local NVMe. Different storage or a different host changes them. +- **Nothing here is a `--device cuda` number.** That arm decodes this checkpoint + and its token gate against the CPU arm fails. The previous section's sixth + limit states what follows for its speed axis. + ## Turning CUDA graph capture off, including the break seam `VLLM_CPP_CUDAGRAPH=0` disables CUDA graph capture. It reached the six batched