From dc89215d1054e651056f1412a03d754b476d3d2e Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Tue, 18 Aug 2026 08:00:11 +0000 Subject: [PATCH 1/4] docs: pin and reproduce Qwen3.8-2.4T-A95B UD-Q1_0 on one GB10 (#1194) The capability landed across #912, #1119 and #1132, and nothing a user reads said how to reach it. AGENTS.md binds the checkpoint pin to docs/USAGE.md, and revision 567d3e6ac26c5474b18311e619c04350fb9a5556 lived only in the agent record: USAGE.md named this checkpoint three times and pinned it zero times, so nobody outside .agents/ could reproduce a single number the project publishes about it. This lands the recipe as a fifth per-model section beside Muse Glimmer 30B, Nemotron-3.5-Lightning-30B, MiniMax-H3 and MiniMax-Music3, and it LINKS the "Streaming routed experts from disk" section rather than restating it, so the schema, the precedence rule, the #1135 reach limits and the #1123 CUDA refusal each keep one home. No docs/examples/ directory. `examples/` in this tree is runnable C++ CMake targets and `docs/` is one topic per file listed in the README index, so a new markdown genre would have had exactly one member and would have duplicated facts that already have owners. Every pin was verified against the HuggingFace API at that revision rather than transcribed: ten shards under UD-Q1_0/, 397,256,393,248 bytes on disk, which reconciles with the 397,245,341,184-byte weight footprint gguf_device_fit.h records, and two X-Linked-ETag sha256 digests. Shard 1 is pinned as well as shard 2 because shard 1 carries no weights - it is the split declaration, so it is the file that decides what the other nine are. Two defects found while reading the section this one sits beside, fixed here. Both streaming examples named -00001-of-00008.gguf, a file that does not exist at that revision. And two invocations wrote the server as ./build/vllm-server; examples/ is added with add_subdirectory and nothing overrides RUNTIME_OUTPUT_DIRECTORY, so it is build/examples/vllm-server, which is the form the same file already used at lines 1401 and 2634. The arithmetic was re-derived rather than copied, because the draft this came from carried GiB values wearing GB labels. 2790 slices at 2,490,368 bytes is 6.95 GB (6.47 GiB) per token, so 44.2 s/token implies 157 MB/s and a ~5 GB/s NVMe puts the ceiling at 1.39 s/token, about 0.72 tok/s - not the 1.29 s and 0.77 tok/s a GiB-for-GB substitution produces. No new measurement is published. The warm 667.0 s TTFT and 44.2 s/token are the figures #1123 already records; the cold 3318 s and 66.7 s/token are what BENCHMARKS.md and the spec record; and the expert slot cache's steady-decode figure stays VOID per #912 F1, stated as void rather than promoted to a loss. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code] --- .agents/issue-index.md | 1 + docs/USAGE.md | 215 ++++++++++++++++++++++++++++++++++++++++- 2 files changed, 213 insertions(+), 3 deletions(-) diff --git a/.agents/issue-index.md b/.agents/issue-index.md index 38997a380..0118dd6b5 100644 --- a/.agents/issue-index.md +++ b/.agents/issue-index.md @@ -367,3 +367,4 @@ rather than merged. `scripts/check-agent-record.py` gates both. | [#1181](https://github.com/mudler/vllm.cpp/issues/1181) | `FIX-READ-F32-SCALAR-GUARD` | `ReadF32Scalar` (`src/vllm/model_executor/models/qwen3_5_weights.cpp:312-318` @ `ab6e65216`) bounds its input with `t.data != nullptr && t.nbytes >= sizeof(float)`, a LOWER bound, and then `memcpy`s four bytes into a `float`. Two silent wrong-value paths follow and neither fails: an ARRAY is reduced to element 0, so a block-wise FP8 scale grid of shape `[ceil(N/128), ceil(K/128)]` passes and stands in for the whole weight (measured under [#1166](https://github.com/mudler/vllm.cpp/issues/1166) on `Qwen/Qwen3.8-27B-FP8` @ `017b9c7af6b5689d5dd426a76e0bc077eb5ca20a`, `q_proj.weight_scale_inv` is `[96, 40]`), and ANY dtype is reinterpreted, since that same tensor is `BF16` and its four bytes are two bf16 values read as one float. Both return a finite plausible float, so the output is fluent, plausible and wrong, which is what a token gate cannot see. Upstream makes both facts structural rather than optional: a per-tensor scale is a distinct parameter TYPE that asserts `loaded_weight.shape[0] == 1` (`vllm/model_executor/parameter.py:260-272,304-309` @ `555967922`, plus the `_assert_and_load` shape assert at `:93-96`), the slot is allocated `torch.float32` so a narrow on-disk dtype is VALUE-converted rather than reinterpreted (`utils/fp8_utils.py:1276`), and the declared strategy TENSOR/CHANNEL/BLOCK picks the parameter type before a byte is read (`compressed_tensors/schemes/compressed_tensors_w8a8_fp8.py:63,128`). The AUDIT corrects the issue's own framing twice. The 27 grep hits across five files are 5 definitions, 20 call sites and 2 comment references, and both counts are short: `ReadCtF32Scalar` (`include/vllm/model_executor/models/dense_weight_loaders.h:376`) is a SIXTH copy of the same defect under another name, reached from a SIXTH model file (`src/vllm/model_executor/models/qwen3_weights.cpp:100,126-128` through `LoadCtNvfp4W4A16`). Of the six, three check nothing, `LnReadF32Scalar`/`ShReadF32Scalar` check dtype but not count, and only `nemotron_h_weights.cpp:557-573` is correct, which makes it the model the shared guard generalizes. No call site legitimately passes a multi-element or non-F32 tensor, and every existing fixture emits rank-0 or `{1}` `F32`, so nothing in the tree needed the leniency. It is NOT merely latent: `dense_weight_loaders.h:73-74` and `docs/BENCHMARKS.md:52` both record `unsloth/Qwen3.6-27B-NVFP4` @ `ccdaab7e` as FP8 W8A8 throughout with BF16 PER-OUTPUT-CHANNEL scales, and `LoadAttnDense` branches on the weight dtype alone (`qwen3_5_dense_weights.cpp:478-480`), so those projections enter the per-tensor arm and hit both defects at once under the tensor name the loader actually asked for, with no misspelling to stop them. Fixed in flow by one `dense_loaders::ReadF32Scalar(get, name)` that refuses `numel != 1` naming the shape, refuses a non-`F32` dtype naming the dtype, and requires exactly four readable bytes, with the other five copies deleted onto it and `nemotron_h`'s `Loader`-based twin kept as the one tracked exception. A narrow dtype is refused rather than converted, because a one-element BF16 scale has never been read correctly here and the BF16 layout that IS shipped is per-channel, which the count check refuses first. Per-channel FP8, block-wise FP8 and any explicit narrow-dtype conversion stay owed. Spec [`read-f32-scalar-guard.md`](specs/read-f32-scalar-guard.md) | bug | | [#1185](https://github.com/mudler/vllm.cpp/issues/1185) | `ENV-ORACLE-WHEEL-IN-LEASE` | The pinned vLLM oracle BUILDS, installs, imports and sees the GPU inside an `rc` lease on `dgx:gpu0`, measured 2026-08-18, which falsifies the `nvcc` clause four records carried. [`lease-runtime-staging.md`](specs/lease-runtime-staging.md) said the oracle "needs `nvcc`, which the worker still lacks", and `.agents/environment.md`, [`mtp-k-gt-1.md`](specs/mtp-k-gt-1.md) and [`gpu-lease-methodology.md`](specs/gpu-lease-methodology.md) each derived a blocker from it. The build job (`buildvllm.sh`, staged sha256 prefix `15e140d41f44e7c2`) asserted the checkout against the pin BEFORE compiling, printing `PIN CONFIRMED` at `5559679229bc961848b121ccdeaa8fa5d79bec98` and aborting otherwise, took `nvcc` from the toolkit row `MODEL-NEMOTRON-H-ABI-A3-E2E` staged (`NVCC_RC=0`, CUDA `release 13.3, V13.3.73`) and produced `WHEEL_RC=0`, `PERSIST_RC=0` and a 434 MiB `vllm-0.1.dev1+g555967922.cu133-cp312-cp312-linux_aarch64.whl`, sha256 `7c58b339741a288fbb313f4f5196c9c92a9e3b3c3ebe2ea970b0ff50bb9bcba4`. The identity job (`oracleenv.sh`, prefix `6119f5223f5d818c`) asserted from `cd /`, outside any source tree: `vllm.__version__ = 0.1.dev1+g555967922`, `IDENTITY_RC=0`, `cuda True NVIDIA GB10`, `CUDA_RC=0`. SCOPE, and it carries the same weight as the result: RUNNING A MODEL IS UNTESTED. Only build, install, import and `torch.cuda.is_available()` are measured, and [`mtp-k-gt-1.md`](specs/mtp-k-gt-1.md) records that the last time an oracle reached this far it consumed the host in the step AFTER `torch.compile` and REBOOTED the box, at `gpu_memory_utilization` 0.75 and again at 0.30, so the fraction is not the lever. The version string is an OPEN discrepancy: `.agents/upstream-sync.md` records `vllm_runtime_version = 0.23.1rc1.dev1511+g555967922`, the commit segment matches and satisfies the pin's binding `+g` rule, and the prefix differs because a shallow fetch stops `setuptools_scm` counting commits since the last tag, so a full-string gate needs a deeper fetch or a recorded pretend-version. The venv is NOT staged, because that job was killed at a 90-minute ceiling mid-copy and its partial tree was removed, so only the WHEEL is durable. Four staging walls, all artifacts of the NAS rather than of CUDA: `cp -a` preserves `file_mode=0664` so `nvcc` exited 126. CIFS `nounix` stores no symlink so `include` and `lib64` vanished and CMake reported `Could NOT find CUDA (missing: CUDA_INCLUDE_DIRS CUDA_CUDART_LIBRARY) (found version "13.3")`, naming the version and denying the toolkit in one line. 32 library links `libfoo.so` and `libfoo.so.MAJOR` had to be rebuilt because only the `libfoo.so.X.Y.Z` real files survived. And `markupsafe` existed as a dist-info with NO package files from a `pip --target` killed at a 35-minute ceiling, so Marlin codegen died on `ModuleNotFoundError`. The `rc` worker container is REUSED between jobs, so a repair inside a staging branch is skipped on the next run (`nvcc already in place`) and an environment repair must be unconditional and assert its postcondition. CONSEQUENCE for the rows #1129 blocked, [#1003](https://github.com/mudler/vllm.cpp/issues/1003), [#915](https://github.com/mudler/vllm.cpp/issues/915), [#821](https://github.com/mudler/vllm.cpp/issues/821) and [#81](https://github.com/mudler/vllm.cpp/issues/81): UNBLOCKED FOR THE BUILD STEP and STILL BLOCKED FOR A MODEL RUN. None can take a measurement until a model run is demonstrated. Job details, walls and non-claims in [`oracle-wheel-in-lease.md`](specs/oracle-wheel-in-lease.md) | verification | | [#1193](https://github.com/mudler/vllm.cpp/issues/1193) | `SPEC-DSPARK-QWEN3-ROUTING` | A Qwen3 DSpark draft declaring `architectures=["DSparkDraftModel"]` with `model_type` `qwen3` has no route. The pin forces every DSpark draft that is not `Qwen3DSparkModel` or `Gemma4DSparkModel` onto `model_type` `deepseek_v4` (`vllm/config/speculative.py:934-944` @ `555967922`), and vLLM PR 52197 (merged 2026-08-17 at `7075ddac`) replaced that with a leading branch normalizing the pair to `Qwen3DSparkModel`. We diverge from BOTH: the forced rewrite was never ported, so nothing in `src/vllm/entrypoints/model_loader.cpp` reads a draft config's `architectures` key at all, and `SpeculativeConfig::IsDsparkDraft` (`include/vllm/config/speculative.h:120-136`) has no production caller — every reference outside its header is in `tests/vllm/config/test_speculative_dspark.cpp:132-140`, and `ResolveSpecConfig` branches on `cli.method` alone. The checkpoint is real and gateable here: `RadixArk/Qwen3.8-27B-DSpark` at revision `85ef153be924f17ce4bf62726954eeaa4a73e854` carries exactly that config shape in one 2718576122-byte shard, drafting five layers for a 64-layer Qwen3.8-27B target | bug | +| [#1194](https://github.com/mudler/vllm.cpp/issues/1194) | `ENG-EXPERT-STREAM` | `Qwen3.8-2.4T-A95B UD-Q1_0` serves on one 119.631 GiB GB10 and no user-facing document said how. AGENTS.md binds the checkpoint pin to [`docs/USAGE.md`](../docs/USAGE.md) — file name, size, repo AND revision, sha256 for a quantized artifact — and revision `567d3e6ac26c5474b18311e619c04350fb9a5556` appeared only in this index (row #1123) and in [`expert-streaming.md`](specs/expert-streaming.md); `USAGE.md` named the checkpoint three times and pinned it zero times. Nor was there an end-to-end recipe: the `Streaming routed experts from disk (capacity mode)` section is a MECHANISM reference (schema, precedence, #1135 reach limits, statistics line, the #1123 CUDA refusal) and never says which file `--model` takes, what the load costs, what decode costs, or where the ceiling is, while its four sibling per-model recipes in the same file (Muse Glimmer 30B, Nemotron-3.5-Lightning-30B, MiniMax-H3, MiniMax-Music3) all do. FIXED as a fifth sibling recipe that LINKS the mechanism section rather than restating it, so each fact keeps one home; `docs/examples/` was NOT created, because `examples/` in this tree is runnable C++ CMake targets and `docs/` is one topic per file listed in the README index, so a new markdown genre would have had one member and duplicated facts that already have owners. Pins verified against the HF API at that revision, not copied: **ten** shards `UD-Q1_0/Qwen3.8-2.4T-A95B-UD-Q1_0-000{01..10}-of-00010.gguf`, **397,256,393,248 B (369.97 GiB)** on disk — reconciling with the 397,245,341,184 B weight footprint `gguf_device_fit.h` records — and `X-Linked-ETag` sha256 `b7770552...0334d` (shard 1, 10,943,264 B) and `2765517f...3dcd` (shard 2, 48,759,636,544 B). TWO DEFECTS FOUND AND FIXED in the same flow: both streaming examples named `-00001-of-00008.gguf`, a file that does not exist at that revision; and two invocations wrote the server as `./build/vllm-server` when `examples/` is added with `add_subdirectory` and nothing overrides `RUNTIME_OUTPUT_DIRECTORY`, so it is `build/examples/vllm-server` — the form the same file already used at lines 1401/2634 and the form `expert-streaming.md` records as the working invocation. The unit arithmetic was re-derived rather than transcribed: 2790 slices x 2,490,368 B is **6.95 GB (6.47 GiB)** per token, so 44.2 s/token implies **157 MB/s** and a ~5 GB/s NVMe ceiling is **1.39 s/token (~0.72 tok/s)**, not the 1.29 s/0.77 tok/s a GiB-for-GB substitution gives. No new measurement is published: the streaming arm's decode figure stays VOID per [#912](https://github.com/mudler/vllm.cpp/issues/912) F1 | doc | diff --git a/docs/USAGE.md b/docs/USAGE.md index 410df88e6..325d870d4 100644 --- a/docs/USAGE.md +++ b/docs/USAGE.md @@ -2351,7 +2351,7 @@ straight at the file; the config comes from the GGUF's own metadata, so no `config.json` is needed: ```sh -./build/vllm-server --model /path/to/muse-glimmer-30B-kquant-17gb.gguf +build/examples/vllm-server --model /path/to/muse-glimmer-30B-kquant-17gb.gguf ``` Both published k-quants load (`muse-glimmer-30B-kquant-17gb.gguf` and the mixed @@ -3832,7 +3832,7 @@ save. ```sh VT_MOE_EXPERT_STREAM=1 \ VT_MOE_EXPERT_STREAM_SLOTS=8000 \ - ./build/vllm-cli --model /models/Qwen3.8-2.4T-A95B-UD-Q1_0-00001-of-00008.gguf \ + ./build/vllm-cli --model /models/Qwen3.8-2.4T-A95B-UD-Q1_0-00001-of-00010.gguf \ --prompt "The capital of France is" --max-tokens 16 ``` @@ -3845,7 +3845,7 @@ weights from the device to host RAM, and the `vllm_cpp` key governs the tier below that, where weights stay borrowed out of the file mapping. ```sh -./build/vllm-server --model /models/Qwen3.8-2.4T-A95B-UD-Q1_0-00001-of-00008.gguf \ +build/examples/vllm-server --model /models/Qwen3.8-2.4T-A95B-UD-Q1_0-00001-of-00010.gguf \ --offload-config '{"vllm_cpp":{"mmap":{"enabled":true,"prefault":false}, "expert_stream":{"enabled":true,"slots":8000}}}' ``` @@ -4052,6 +4052,215 @@ on a GB10, because host and device share one pool. `cudaMemGetInfo` answers honestly, and its `total` is EXACTLY `/proc/meminfo MemTotal` (125442340 kB) times 1024. Do not size this from `nvidia-smi`. +## Qwen3.8-2.4T-A95B `UD-Q1_0`: 370 GiB served from a 119 GiB box + +A 2.4-trillion-parameter mixture-of-experts checkpoint, three times the size of +the machine's memory, loads and answers on one DGX Spark with no GPU offload and +no swapping. This section is the recipe. The mechanism it drives is the previous +section, [Streaming routed experts from disk](#streaming-routed-experts-from-disk-capacity-mode), +which owns the config schema, the precedence rule, the statistics line and the +`--device cuda` refusal; nothing here repeats them. + +**Read the speed before you spend the download.** Steady decode is tens of +seconds per token, and the ceiling is set by storage rather than by this +implementation — see [What decode costs](#what-decode-costs-and-why-the-ceiling-is-where-it-is) +below. This is a capacity result, not an interactive one. + +### The exact weights (so the run is reproducible) + +| field | value | +|---|---| +| repo | [unsloth/Qwen3.8-2.4T-A95B-GGUF](https://huggingface.co/unsloth/Qwen3.8-2.4T-A95B-GGUF) — third party (Unsloth), see the note below | +| revision | `567d3e6ac26c5474b18311e619c04350fb9a5556` | +| arm | `UD-Q1_0`, roughly one bit per weight; the repo carries other arms in sibling directories | +| weights | `UD-Q1_0/Qwen3.8-2.4T-A95B-UD-Q1_0-000{01..10}-of-00010.gguf`, **ten** shards | +| on-disk total | 397 256 393 248 bytes (369.97 GiB) | +| tensor records | 1702, matching the `split.tensors.count` both tensor tables declare | +| sha256 (shard 1) | `b7770552b2ac24e7334c917bc92e90e218e87cfe29484db65e62e8ef2a60334d` for `-00001-of-00010.gguf` (10 943 264 B) | +| sha256 (shard 2) | `2765517f833c736338d3ab34354e1c10eb8d79e62325f998285b435e5cf03dcd` for `-00002-of-00010.gguf` (48 759 636 544 B) | + +**A repo id alone is not a pin**, because a quantized checkpoint gets +re-quantized in place under an unchanged name. Both digests are that revision's +own LFS records, read from the `X-Linked-ETag` the resolve endpoint returns. +Shard 1 is pinned as well as shard 2 because shard 1 carries no weights — it is +the metadata and the split declaration, so it is the file that decides what the +other nine are. + +```sh +hf download unsloth/Qwen3.8-2.4T-A95B-GGUF \ + --revision 567d3e6ac26c5474b18311e619c04350fb9a5556 \ + --include "UD-Q1_0/*" \ + --local-dir ./qwen3.8-2.4t-a95b-gguf +``` + +The files land under a `UD-Q1_0/` subdirectory of `--local-dir`, because that is +where they live in the repo. Point `--model` at the copy on **local NVMe**: a +network filesystem puts an uncontrolled variable in front of every one of the +6.95 GB of expert reads each token makes. + +**The encoding has no upstream reference.** `UD-Q1_0` stores its expert towers as +`IQ1_XXXS`, which upstream llama.cpp does not define; it exists only in the +`unslothai/llama.cpp` fork, pinned as a secondary oracle in +[`.agents/oracles/llama-cpp-unsloth.md`](../.agents/oracles/llama-cpp-unsloth.md). +The decode is sealed against golden vectors produced by that fork. The fork itself is +recorded `gateable = no` — it has not been shown to build and run this model, and +[#933](https://github.com/mudler/vllm.cpp/issues/933) owes that measurement — so +there is no token-exact denominator for anything below. + +### Build and serve + +A plain CPU build is enough; no CUDA is involved on this path. + +```sh +cmake -S . -B build +cmake --build build -j +``` + +```sh +build/examples/vllm-server \ + --model ./qwen3.8-2.4t-a95b-gguf/UD-Q1_0/Qwen3.8-2.4T-A95B-UD-Q1_0-00001-of-00010.gguf \ + --offload-config '{"vllm_cpp":{"mmap":{"enabled":true,"prefault":false}}}' \ + --device cpu \ + --max-num-seqs 1 \ + --max-model-len 512 \ + --port 8899 +``` + +Four things in that line are load-bearing. + +- **`--model` takes shard 1, not the directory.** A directory sends the loader + down the HuggingFace branch, which fatals on a missing `config.json` before it + looks for a GGUF. Given shard 1 the reader finds its nine siblings from the + `-NNNNN-of-MMMMM.gguf` naming and cross-checks `split.count`. +- **`prefault: false` is the setting that decides whether this works.** + Pre-faulting is **on** by default (`VT_GGUF_PREFAULT`), and it is the right + default for a model that fits: it walks every borrowed span at load so the + first-touch faults do not land inside the timed prefill. For 335.62 GiB of + expert towers that cannot fit, it reads the whole checkpoint to populate a page + cache that cannot hold it. +- **`mmap: true` confirms the default rather than enabling it** — it is already + on wherever weights stay quantized. It is what makes the checkpoint fit at all: + an expert tower is borrowed from the file mapping and costs zero anonymous + bytes, so only the dense remainder becomes resident. +- **`--device cpu`.** `--device cuda` is refused at load, by name and with the + arithmetic, because the larger-than-memory lane is host-only. That refusal is + documented in the previous section. + +`--max-num-seqs 1` and a small `--max-model-len` keep the KV cache out of the +way. At this speed nothing is being batched, and the capacity argument itself +only holds at low concurrency: at high concurrency every step touches most of the +experts and the working set stops being one. + +`--offload-config` does **not** reach `vllm-cli`, nor the server's +pooling/embedding and transcription-only paths +([#1135](https://github.com/mudler/vllm.cpp/issues/1135)). Use `VT_GGUF_PREFAULT=0` +on those. + +### What the load costs + +Expect to wait. Two runs are recorded, and the spread is the page cache rather +than noise: **13 minutes** to a serving state on the first `--device cpu` run +(16 August 2026), and **26 minutes** on the run that produced the CUDA refusal +([#1123](https://github.com/mudler/vllm.cpp/issues/1123)). + +Resident anonymous memory settles at **62 GiB** of 119 GiB and stays there. That +is the dense remainder plus the KV cache and runtime, and it matches what the +checkpoint's own tensor table predicts (21.56 GiB of `attn_qkv` and 17.25 GiB of +`ssm_out` expanded to bf16, plus 5.81 GiB of embeddings and F32 norms, so 44.6 +GiB before KV and runtime). The other 335.62 GiB is mapped, not copied. **The +model does not fit because of streaming; it fits because of borrowing.** + +Check readiness against the model list rather than the process: + +```sh +curl -sf http://127.0.0.1:8899/v1/models +``` + +```sh +curl -s http://127.0.0.1:8899/v1/completions -H 'Content-Type: application/json' \ + -d '{"model":"Qwen3.8-2.4T-A95B-UD-Q1_0-00001-of-00010.gguf", + "prompt":"Q: What is the capital of France? A:","max_tokens":4}' +``` + +It answers ` Paris. Q: What`, which is the whole point: the output is coherent, +so the one-bit encoding and the borrowed-tower path are both faithful enough to +serve. + +### What decode costs, and why the ceiling is where it is + +| | measured | +|---|---| +| TTFT, warm page cache | 667.0 s ([#1123](https://github.com/mudler/vllm.cpp/issues/1123)) | +| TTFT, cold | 3318 s | +| steady decode, warm | **44.2 s/token** ([#1123](https://github.com/mudler/vllm.cpp/issues/1123)) | +| steady decode, cold | 66.7 s/token (66.5, 66.9, 66.8) | + +Do not quote a TTFT as a decode number: token 1 carries prefill and the cold +expert set. From token 2 onward you are watching steady state. The full +measurement record is [docs/BENCHMARKS.md](BENCHMARKS.md). + +The arithmetic behind those seconds is short and it decides everything. Every +output token routes 93 blocks through the top 10 of 512 experts, and each expert +contributes three projections: + +| | | +|---|---| +| expert slices per token | 93 x 10 x 3 = 2790 | +| bytes per slice | 2 490 368 (2.375 MiB) | +| read per token | **6.95 GB** (6.47 GiB) | +| implied rate at 44.2 s/token | 157 MB/s | +| NVMe sequential, for reference | ~5 GB/s | + +Two things follow, and the second one is the important one. + +**There is real headroom.** 157 MB/s is what single-queue synchronous reads +give; issuing those 2790 reads concurrently should approach storage speed, which +is roughly a 32x decode improvement still on the table. + +**And that improvement lands near 0.72 tok/s, not higher.** 6.95 GB at 5 GB/s is +1.39 s/token whatever the software does. Reaching 3 tok/s would demand about 21 +GB/s of expert bandwidth, which means serving most of those reads from RAM — and +with 335.62 GiB of experts against at most tens of GB spendable on cache, top-10-of-512 +routing does not give consecutive tokens enough reuse to close that gap. **If you +need conversational speed from this model you need more memory or fewer active +parameters, not better software.** + +### The expert slot cache is off, and it has not yet earned being on + +`vllm_cpp.expert_stream` copies routed slices into a fixed slot arena instead of +re-faulting them through the mapping. It is off by default and this checkpoint is +the reason the default has not moved. + +In the recorded run (8000 slots, `resident=18.55 GiB` of arena) it was +**token-identical** to the mapped path and cut TTFT 4.5x, from 3318.1 s to +733.4 s — prefill touches a wide expert set once, where sequential order and slot +reuse both help. Both of those are COLD-cache figures against the cold baseline +in the table above; do not read 733.4 s against the warm 667.0 s, which is a +different regime. **Its steady-decode figure is recorded VOID**, not as a win or a +loss: the step clock had no caller, so the cache stopped serving in token 3 and +the run measured the mapped path wearing the cache's name +([#912](https://github.com/mudler/vllm.cpp/issues/912) F1). The re-measure is +owed. + +Two bounds are already known and neither needs that re-measure. The fill copies +from a pointer into the mmap, so it still takes the page fault it exists to +avoid; and 6.95 GB per token against a ~20 GiB arena is under three tokens of +working set. Before believing any number you take here, read the +`[expert-stream]` final statistics line: `steps` must advance and `exhausted` +must stay 0, or the run measured something else. + +### What this does not establish + +- **The quantization is extreme.** Roughly one bit per weight. The output is + coherent; this is not the configuration to judge the model's quality by. +- **There is no oracle.** No entry in the oracle table runs this checkpoint on + this hardware, so there is no token-exact or throughput denominator — every + figure above is an absolute measurement of this implementation, compared to + nothing. +- **One request at a time.** Nothing here says anything about concurrency, and + the capacity argument stops holding as concurrency rises. +- **Timings are warm unless labelled cold**, and the two differ by 5x on TTFT. + ## SSE keepalives on long prefill Async chat/completion streams can emit SSE **comment** frames (`:\n\n`) while From cc3775cbc953f4ac1aa1db05eea253ea72fc0ee8 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Thu, 20 Aug 2026 05:03:07 +0000 Subject: [PATCH 2/4] docs(ENG-EXPERT-STREAM): name the run behind each Qwen3.8-2.4T figure in the recipe Four sentences in the new section pointed at a measurement without saying which one. The answer that a reader can reproduce came from the streaming-off run of 16 August 2026, the 62.45 GiB residency came from the W0f CPU control, and the environment-variable form of the recipe sat inside parentheses as a required fact. Each now names its run. The comparison rule is also narrowed. The streaming-off row is the only one that carries no ratio, because it was taken on a different source tree; the two slot counts came from one binary on one lease and the previous section already compares them. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code] --- docs/USAGE.md | 42 ++++++++++++++++++++++-------------------- 1 file changed, 22 insertions(+), 20 deletions(-) diff --git a/docs/USAGE.md b/docs/USAGE.md index a14b9eafb..cfe6e475f 100644 --- a/docs/USAGE.md +++ b/docs/USAGE.md @@ -4998,10 +4998,10 @@ way. Nothing is batched at this speed, and the capacity argument itself holds only at low concurrency: at high concurrency every step touches most of the experts and the working set stops being one. -The recorded runs set the equivalent environment variables -(`VT_GGUF_PREFAULT=0 VT_MOE_EXPERT_STREAM=1 VT_MOE_EXPERT_STREAM_SLOTS=4000`) -rather than the config document. The two are the same switches, and a variable -beats a config field wherever both are set. +The recorded runs set the equivalent environment variables rather than the +config document: `VT_GGUF_PREFAULT=0`, `VT_MOE_EXPERT_STREAM=1` and +`VT_MOE_EXPERT_STREAM_SLOTS=4000`. The two forms are the same switches, and a +variable beats a config field wherever both are set. ### What the load costs @@ -5018,7 +5018,8 @@ page cache dropped before each one (`ENG-EXPERT-STREAM-DEVICE` W0e, 18 and | peak swap | not sampled | 6 883 MiB | Resident memory after the load settles at about **62 GiB** of 119 GiB, measured -at 62.45 GiB on the same-lease CPU control of the following run. That is the +at 62.45 GiB on the same-lease CPU control run of `ENG-EXPERT-STREAM-DEVICE` +W0f. That is the dense remainder plus the KV cache and the runtime, and it agrees with what the checkpoint's own tensor table predicts: 21.56 GiB of `attn_qkv` and 17.25 GiB of `ssm_out` expanded to bf16, plus 5.81 GiB of embeddings and F32 norms, so @@ -5038,12 +5039,13 @@ curl -s http://127.0.0.1:8899/v1/completions -H 'Content-Type: application/json' "prompt":"Q: What is the capital of France? A:","max_tokens":4}' ``` -The 16 August 2026 run answered ` Paris. Q: What`, which is the whole point: the -output is coherent, so the one-bit encoding and the borrowed-tower path are both -faithful enough to serve. The four W0e runs drive a fixed prompt of token ids -instead, and all four returned the same 32 ids, which detokenize to -` Paris. Paris is a city located in the northern part of France, on the Seine -River. It is the largest city in France and is known for its iconic`. +The 16 August 2026 run, which served with streaming off, answered +` Paris. Q: What`. That is the whole point: the output is coherent, so the +one-bit encoding and the borrowed-tower path are both faithful enough to serve. +The four W0e runs drive a fixed prompt of token ids instead of this request, and +all four returned the same 32 ids, which detokenize to ` Paris. Paris is a city +located in the northern part of France, on the Seine River. It is the largest +city in France and is known for its iconic`. ### What decode costs, and why the ceiling is where it is @@ -5056,10 +5058,11 @@ Every figure here was measured on `--device cpu`, on the box named above. | streaming off | 66.7 s/token | 16 August 2026, streaming not yet enabled | The 11.05 s/token figure has a min of 9.43 and a max of 13.25 over its window, -and rep 1 of the same arm gives 11.22, which agrees within 1.5%. **No ratio -between these three rows is published.** The streaming-off row was taken on a -different source tree on a different date, and the two slot counts differ in more -than the arena size, as the previous section records. +and rep 1 of the same arm gives 11.22, which agrees within 1.5%. **The +streaming-off row carries no ratio against the other two**, because it was taken +on a different source tree on a different date. The two slot counts came from one +binary on one lease and are comparable with each other; the previous section +carries that comparison. **A bigger cache came out slower, and 8000 slots is the count to avoid.** The second 8000-slot rep consumed all 30 625 MiB of the box's swap. The @@ -5091,8 +5094,8 @@ hits against 58 538 misses. **The floor is storage, not software.** 6.95 GB at the roughly 5 GB/s an NVMe of this class sustains is 1.39 s/token whatever the code does, which is 0.72 tok/s. Reaching 3 tok/s would demand about 21 GB/s of expert bandwidth, so most of those -reads would have to come from memory instead. The arena holds 4000 of the 2790 -slices a token needs, under one and a half tokens of working set, and +reads would have to come from memory instead. The arena holds 4000 slices against +the 2790 a token needs, under one and a half tokens of working set, and top-10-of-512 routing does not give consecutive tokens enough reuse to close the rest. **If you need conversational speed from this model you need more memory or fewer active parameters, not better software.** @@ -5108,9 +5111,8 @@ fewer active parameters, not better software.** against nothing. - **One request at a time.** Nothing here says anything about concurrency, and the capacity argument stops holding as concurrency rises. -- **One box.** Every number was taken on `dgx:gpu0` with the checkpoint on local - NVMe and the page cache dropped first. Different storage or a different host - changes them. +- **One box.** Every number was taken on one DGX Spark GB10 with the checkpoint + on local NVMe. Different storage or a different host changes them. - **Nothing here is a `--device cuda` number.** That arm decodes this checkpoint, its token gate against the CPU arm fails, and `docs/BENCHMARKS.md` therefore carries its speed axis as `VOID`. From f62675f8fb31dcf667e4fa55ca43a38a620b0010 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Thu, 20 Aug 2026 05:14:24 +0000 Subject: [PATCH 3/4] docs(ENG-EXPERT-STREAM): point at the 8000-slot finding instead of restating it The decode subsection repeated the previous section's account of why 8000 expert-stream slots measure slower than 4000, page-cache displacement and the swap column included. One fact belongs in one place, and the mechanism section owns this one, so the recipe now keeps the measured row and links the reason. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code] --- docs/USAGE.md | 7 ++----- 1 file changed, 2 insertions(+), 5 deletions(-) diff --git a/docs/USAGE.md b/docs/USAGE.md index bd1d5b9b2..7562f16a6 100644 --- a/docs/USAGE.md +++ b/docs/USAGE.md @@ -5160,11 +5160,8 @@ on a different source tree on a different date. The two slot counts came from on binary on one lease and are comparable with each other; the previous section carries that comparison. -**A bigger cache came out slower, and 8000 slots is the count to avoid.** The -second 8000-slot rep consumed all 30 625 MiB of the box's swap. The -best-supported reading of the recorded columns is page-cache displacement: the -borrowed 370 GiB mapping is served out of whatever memory is free, and the arena -takes that memory. That is a reading of those columns and not a proven mechanism. +**A bigger cache came out slower**, which is why this recipe sets 4000 slots. +The previous section states the reason and its evidence. Do not quote a first-token time as a decode number. Token 1 carries the prefill and the cold expert set. From token 2 onward you are watching steady state. The From ae152110ac989c15fe846228f16cc96a86dc1df5 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Thu, 20 Aug 2026 06:04:16 +0000 Subject: [PATCH 4/4] docs(ENG-EXPERT-STREAM): say what the recipe repeats, and stop claiming it repeats nothing The recipe section opened by saying "Nothing here repeats them" of the mechanism section above it, and then repeated five of its facts. The copies agree today. They will not after the W0e re-measure this row owes, and a fix applied to one copy leaves the other reading true and stale, which is the shape that put an 8000-slot figure under a 4000-slot heading on the first revision of this change. Each fact now has one home in the section, and the opener names the three the recipe repeats on purpose and says why: which device to use, the expert bytes a token reads, and the two streaming decode figures. A recipe that leaves those out is not a recipe. The other copies are gone, because the reader loses nothing by following one link. Three smaller repairs ride with it. The published command starts vllm-server while every figure came from the W0e C-ABI harness, so the section discloses the binary beside the prompt and environment differences it already disclosed. Rep 1's 11.22 against 11.05 is 1.54% and was recorded as "agrees within 1.5%", a bound the numbers do not meet. And #1194 now appears in the owning row's spec, which AGENTS.md requires alongside the index row and the pull request body. No verified fact moves. The shard count stays ten, no retired figure returns, no CUDA speed number appears, and the step-7 divergence still has no asserted cause. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code] --- .agents/specs/expert-streaming.md | 33 ++++++++++++++++++++++++ docs/USAGE.md | 43 +++++++++++++++++++++---------- 2 files changed, 62 insertions(+), 14 deletions(-) diff --git a/.agents/specs/expert-streaming.md b/.agents/specs/expert-streaming.md index 92e7cce03..bb577d99a 100644 --- a/.agents/specs/expert-streaming.md +++ b/.agents/specs/expert-streaming.md @@ -1586,6 +1586,39 @@ into other suites. | a fitting GGUF still loads | the SAME call with a generous budget: the throw is a LATER, different one (the synthetic file has no tokenizer), which is what proves the check let it through rather than that it never ran | | the CPU arm is untouched | the same file with `device=cpu` never refuses, whatever the budget | +## The user-facing recipe and the checkpoint pin ([#1194](https://github.com/mudler/vllm.cpp/issues/1194)) + +`docs/USAGE.md` named `Qwen3.8-2.4T-A95B UD-Q1_0` three times and pinned it zero +times. AGENTS.md binds the pin to that file: file name, size, repo AND revision, +and a sha256 for a quantized artifact. The streaming section there is a MECHANISM +reference. It never said which file `--model` takes, what the load costs, what +decode costs, or where the ceiling is, while the four sibling per-model recipes +in the same file all do. + +Landed as a fifth sibling recipe, `Qwen3.8-2.4T-A95B UD-Q1_0: 370 GiB served from +a 119 GiB box`, which LINKS the mechanism section rather than restating it, so +each fact keeps one home. Three facts are repeated on purpose and the section +says which ones and why: which device to use, the expert bytes a token reads, and +the two streaming decode figures. Both places quote `ENG-EXPERT-STREAM-DEVICE` +W0e, so the re-measure that row owes has to change both. + +The pin is settled against the ARTIFACT rather than against a document or the +HuggingFace tree API, whose `lfs.oid` is fabricated for a gated repo: TEN shards +`UD-Q1_0/Qwen3.8-2.4T-A95B-UD-Q1_0-000{01..10}-of-00010.gguf` summing to exactly +397,256,393,248 B (369.97 GiB), with shard 1 declaring `split.count = 10` and +`split.tensors.count = 1702` in its own metadata and carrying no tensors. That +also answers [#1420](https://github.com/mudler/vllm.cpp/issues/1420), whose two +copy-paste commands named `-of-00008`, a file that does not exist at that +revision. + +The section publishes no new measurement. Every figure in it was already +recorded, it carries no ratio between runs taken on different source trees, and +it carries no `--device cuda` speed number, because that arm's token gate fails. +The figures come from the W0e harness `benchmarks/expert_stream_device_w0e.cpp` +and not from the `vllm-server` command the section publishes, and the section +discloses that difference beside the prompt and environment differences it +already disclosed. + ## Owed Carried debt for this row. Each item names why it is not closed here. diff --git a/docs/USAGE.md b/docs/USAGE.md index 7562f16a6..5259d5fd8 100644 --- a/docs/USAGE.md +++ b/docs/USAGE.md @@ -4989,12 +4989,20 @@ the machine's memory, loads and answers on one DGX Spark. This section is the recipe. The mechanism it drives is the previous section, [Streaming routed experts from disk](#streaming-routed-experts-from-disk-capacity-mode), which owns the config schema, the precedence rule, the statistics line, the slot -count warning and what each device can serve. Nothing here repeats them. +count warning and what each device can serve. This section links them rather than +restating them. It repeats three of their facts on purpose: which device to use, +the expert bytes a token reads, and the two streaming decode figures in +[What decode costs](#what-decode-costs-and-why-the-ceiling-is-where-it-is). A +recipe that leaves those out is not a recipe. Each of the three has one record, +so a correction has to change both places. The decode figures are +`ENG-EXPERT-STREAM-DEVICE` W0e in +[`.agents/benchmark-record.md`](../.agents/benchmark-record.md). **Read the speed before you spend the download.** Steady decode on the recipe -below is 11.05 s/token, and the floor under it is storage rather than this -implementation. See [What decode costs](#what-decode-costs-and-why-the-ceiling-is-where-it-is). -This is a capacity result, not an interactive one. +below is measured in seconds per token, and the floor under it is storage rather +than this implementation. This is a capacity result, not an interactive one. +[What decode costs](#what-decode-costs-and-why-the-ceiling-is-where-it-is) gives +the figure and the arithmetic behind it. **Use `--device cpu` for this checkpoint.** `--device cuda` loads and decodes it too, and its token gate against the CPU arm does not pass, so every number below @@ -5038,8 +5046,8 @@ hf download unsloth/Qwen3.8-2.4T-A95B-GGUF \ The files land under a `UD-Q1_0/` subdirectory of `--local-dir`, because that is where they live in the repo. Point `--model` at a copy on **local NVMe**. A -network filesystem puts an uncontrolled variable in front of the 6.95 GB of -expert reads that every token makes. +network filesystem puts an uncontrolled variable in front of the expert reads +that every token makes. **The encoding has no upstream reference.** `UD-Q1_0` stores its expert towers as `IQ1_XXXS`, which upstream llama.cpp does not define. The encoding exists only in @@ -5087,7 +5095,7 @@ Five things in that command are load-bearing. - **`expert_stream` is off by default, and this recipe turns it on at 4000 slots.** That count is the one the published decode figure was measured at, and the previous section explains why 8000 is worse rather than better. -- **`--device cpu`.** It is the arm every number here was measured on. +- **`--device cpu`.** The note at the top of this section says why. `--max-num-seqs 1` and a small `--max-model-len` keep the KV cache out of the way. Nothing is batched at this speed, and the capacity argument itself holds @@ -5099,6 +5107,13 @@ config document: `VT_GGUF_PREFAULT=0`, `VT_MOE_EXPERT_STREAM=1` and `VT_MOE_EXPERT_STREAM_SLOTS=4000`. The two forms are the same switches, and a variable beats a config field wherever both are set. +They also ran a different binary. Every figure below comes from +`benchmarks/expert_stream_device_w0e.cpp`, a purpose-built C ABI client that +reports the token ids, a per-step timestamp and the expert-stream counters +together, which no shipped command does. The command above starts `vllm-server` +over the same engine. At seconds per token, the server's HTTP and SSE framing +sits far below the run-to-run spread recorded below. + ### What the load costs Expect to wait. Two runs of this arm are recorded on `dgx:gpu0`, a GB10 with @@ -5145,7 +5160,7 @@ city in France and is known for its iconic`. ### What decode costs, and why the ceiling is where it is -Every figure here was measured on `--device cpu`, on the box named above. +Every figure here comes from the box named above. | Arm | Steady decode | Where it comes from | |---|---|---| @@ -5153,8 +5168,8 @@ Every figure here was measured on `--device cpu`, on the box named above. | streaming on, 8000 slots | 39.98 and 45.40 s/token | W0e, the medians of two reps | | streaming off | 66.7 s/token | 16 August 2026, streaming not yet enabled | -The 11.05 s/token figure has a min of 9.43 and a max of 13.25 over its window, -and rep 1 of the same arm gives 11.22, which agrees within 1.5%. **The +That 4000-slot figure has a min of 9.43 and a max of 13.25 over its window, and +rep 1 of the same arm gives 11.22, which is 1.54% above it. **The streaming-off row carries no ratio against the other two**, because it was taken on a different source tree on a different date. The two slot counts came from one binary on one lease and are comparable with each other; the previous section @@ -5180,7 +5195,7 @@ first three rows are read from the checkpoint's own metadata: | expert working set per token | **6.95 GB**, that is 6.47 GiB | | slots this recipe reserves | 4000, a 9.28 GiB arena | -That 6.95 GB is a working set and not an I/O rate, because the slot cache serves +That figure is a working set and not an I/O rate, because the slot cache serves part of it from memory. The recorded 32-token run at 4000 slots counted 37 096 hits against 58 538 misses. @@ -5206,9 +5221,9 @@ fewer active parameters, not better software.** the capacity argument stops holding as concurrency rises. - **One box.** Every number was taken on one DGX Spark GB10 with the checkpoint on local NVMe. Different storage or a different host changes them. -- **Nothing here is a `--device cuda` number.** That arm decodes this checkpoint, - its token gate against the CPU arm fails, and `docs/BENCHMARKS.md` therefore - carries its speed axis as `VOID`. +- **Nothing here is a `--device cuda` number.** That arm decodes this checkpoint + and its token gate against the CPU arm fails. The previous section's sixth + limit states what follows for its speed axis. ## Turning CUDA graph capture off, including the break seam