fix(ENG-EXPERT-STREAM): the liveness line could not print the zero the docs told an operator to read, and then its comments outran the code twice (#912, #1091, #1106, #1108)

docs/ENVIRONMENT.md

@@ -79,7 +79,7 @@ allocated up front and never grown — the engine prints the resolved values as
 | `VT_MOE_EXPERT_STREAM` | off | `=1` serves routed expert slices from the bounded host slot cache instead of reading them straight out of the mmap'd tower. Read once per process, and only the FIRST character is examined: a value starting with `0`, and an empty value, are off; anything else is on. Only the CPU path streams — on a device platform the expert slice is already device-resident and is served unchanged. Turning it on also **disables the default-on grouped-MoE path** (`VT_QWEN35_GROUPED_MOE`), which stages the whole tower and therefore cannot stream; the engine says so once on stderr rather than silently doing no streaming. Set `VT_MOE_EXPERT_STREAM=0` to keep grouping |
 | `VT_MOE_EXPERT_STREAM_SLOTS` | `64` | How many expert slices stay resident. Parsed as a decimal integer; unset, empty, zero, negative and unparseable values all keep `64`. Every slot acquired during a step is protected from eviction until the step ends, so a budget smaller than one step's working set exhausts the cache: those slices fall back to reading the tower directly, which is correct but slow, and is counted. Sized against a real model this wants to be large — the measured run used `8000` |
 | `VT_MOE_EXPERT_STREAM_SLOT_BYTES` | the LARGEST of the gate/up/down slices of the first MoE layer reached | Bytes reserved per slot, fixed for the process's life. Parsed as a decimal integer; unset, empty, zero, negative and unparseable values all keep the default. The default is the largest of the three slices rather than the first one taken, because a dynamic (UD) quant keeps `down_proj` at a higher precision than the gate/up pair and sizing from a gate slice then refuses the first down slice mid-decode. A slice that still does not fit is refused BY NAME (`vt: expert stream: a slice of N bytes exceeds the slot budget of M; raise VT_MOE_EXPERT_STREAM_SLOT_BYTES`) rather than truncated or silently routed back to the mmap path, so a streaming benchmark cannot quietly measure the mmap path instead |
-| `VT_MOE_EXPERT_STREAM_STATS_EVERY` | `16` | How many decode steps between the expert-stream statistics line on stderr; `0` silences it. Parsed as a decimal integer; unset, empty, negative and unparseable values all keep `16`. The line is `[expert-stream] steps=N hits=H misses=M evictions=E fills=F bytes=B exhausted=X advised=A`. It exists because the row's first published decode figure was measured on a cache that had switched itself off partway through the third token, and nothing in the run could have said so: the process printed one line at startup and none afterwards. **`steps == 0` or `exhausted > 0` means the lane is not streaming**, whatever the startup line claimed |
+| `VT_MOE_EXPERT_STREAM_STATS_EVERY` | `16` | How many decode steps between the PERIODIC expert-stream statistics line on stderr; `0` silences the periodic line only. Parsed as a decimal integer; unset, empty, negative and unparseable values all keep `16`. The line is `[expert-stream] steps=N hits=H misses=M evictions=E fills=F bytes=B exhausted=X advised=A`. **Exactly one FINAL line is printed when the process ends**, whatever this is set to and whatever the run did, including `steps=0`, for as long as the lane built a store. That is the line to read, and it exists because the row's first published decode figure was measured on a cache that had switched itself off partway through the third token while nothing in the run could say so. **`steps == 0` or `exhausted > 0` means the lane is not streaming**, whatever the startup line claimed. Absence of the final line means either that no store was ever built — in which case the `[expert-stream] ON ...` banner is absent too, and the lane was never reached — or that the process did not run its static destructors (a crash, a signal, `_exit`). A fourth shape exists but no shipped command can produce it: the line is printed once per process, and an internal test seam that flushes it mid-run takes that one print. `docs/USAGE.md` tabulates all four |
 
 ## Rollback and bisect switches
 

docs/FEATURES.md

@@ -61,7 +61,7 @@ are our reading of their documented behavior, not measurements.
 | Scratch allocator keyed by device (two backends, one process) | ✅ since [#516](https://github.com/mudler/vllm.cpp/issues/516); a pool is bound to one backend and refuses any other, and a backend with no registered platform is refused rather than given another's residency cap | ✅ device is field 0 of the allocation handle | ✅ | ✅ |
 | Automatic memory sizing (no hand-tuned budget) | ☐ hand-typed block count | ☐ percent, hand-tuned | ☐ | ◐ |
 | Memory cap with a pre-flight error instead of an OOM | ☐ | ◐ KV pool only | ◐ | ☐ |
-| Routed-expert weight streaming from disk | ◐ default OFF (`VT_MOE_EXPERT_STREAM=1`), CPU keep-quant towers only; bounded host slot cache, hotness-decayed LFU + LRU tiebreak; refuses an unfittable slice by name. Serves c1-c4 capacity, not throughput | ☐ blanket `cpu_offload_gb`, not expert-granular | ☐ | ◐ mmap only |
+| Routed-expert weight streaming from disk | ◐ default OFF (`VT_MOE_EXPERT_STREAM=1`), CPU keep-quant towers only; bounded slot cache; refuses an unfittable slice by name. c1-c4 capacity, not throughput. One `[expert-stream]` line on a clean exit IF a store existed | ☐ blanket `cpu_offload_gb`, not expert-granular | ☐ | ◐ mmap only |
 
 ## Quantization and weight formats
 

docs/USAGE.md

@@ -3206,23 +3206,51 @@ and therefore cannot stream. The engine says that once on stderr rather than
 silently doing no streaming.
 
 **Read the statistics line before you believe any number you measure with it.**
-Every `VT_MOE_EXPERT_STREAM_STATS_EVERY` steps (default 16, `0` silences it) the
-engine prints:
+The engine prints one every `VT_MOE_EXPERT_STREAM_STATS_EVERY` steps (default
+16, `0` silences the periodic line), and **exactly one more when the process
+ends**, whatever the run did:
 
 ```text
 [expert-stream] steps=64 hits=141230 misses=37312 evictions=29312 fills=37312 bytes=92876505088 exhausted=0 advised=37312
 ```
 
-Two of those fields decide whether the run is measuring anything at all:
+**The final line is the one to read**, because it is the only one you are
+guaranteed to get. The periodic line is skipped whenever the step count is not a
+multiple of the interval, so a healthy five-token run prints none of them at the
+default 16; and it used to be skipped on `steps == 0` as well, which meant the
+one run that most needed reporting — the one where the step boundary is never
+reached — printed nothing at all. Treating absence as failure therefore reported
+VOID on a working lane. The final line crosses both of those skips, so it is
+printed even on a run of zero steps.
 
-- `steps` must advance. If it stays at 0 the decode step boundary is not being
-  reached and the cache will stop serving as soon as it fills.
+Two of the fields decide whether the run is measuring anything at all:
+
+- `steps` must advance. If the final line says `steps=0` the decode step
+  boundary is not being reached, and the cache stops serving as soon as it
+  fills — it will fall back to the memory mapping for the rest of the run.
 - `exhausted` must stay 0. Anything above 0 means slices were refused and read
   from the memory mapping instead, which is the slow path streaming exists to
   replace. The usual cause is a budget smaller than one step's working set:
   raise `VT_MOE_EXPERT_STREAM_SLOTS`.
 
-A run whose `steps` is 0 or whose `exhausted` is large is not a measurement of
+Read it together with the `[expert-stream] ON slots=...` banner, which is printed
+once when the lane builds its store. The four shapes are:
+
+| Banner | Final line | What happened |
+|---|---|---|
+| absent | absent | Nothing reached the streamed seam. A CUDA run (a device-resident expert is served unchanged), a checkpoint whose experts are not keep-quant towers, or a prompt that never reached an MoE layer |
+| present | present | The lane ran. Read `steps` and `exhausted` |
+| present | absent, and nothing called `ExpertStreamFlushStats` | The process did not reach its static destructors: a crash, a signal, or `_exit` |
+| present | absent, because `ExpertStreamFlushStats` was called | The internal gate seam took the process's single print, so teardown had none left to make. No shipped command or server path calls it, so an operator never reaches this shape |
+
+The last two shapes are keyed on the CALL and not on what stderr looks like,
+because stderr cannot separate them. `ExpertStreamFlushStats` prints the same
+line in the same shape as the periodic report, so "a statistics line already
+appeared mid-run" is also what a healthy run of 16 steps that then crashes
+produces. What distinguishes the two is whether the seam was called, and only a
+gate calls it.
+
+A run whose `steps` is 0, or whose `exhausted` is large, is not a measurement of
 streaming, whatever the startup line said. See
 [`docs/ENVIRONMENT.md`](ENVIRONMENT.md) for every knob and its parsing rules.
 

include/vllm/model_executor/models/qwen3_5_weights.h

@@ -104,8 +104,19 @@ struct OwnedTensor {
   mutable int mmap_fd = -1;
   mutable size_t mmap_file_offset = 0;
 
-  // A process-unique identity for this tensor's CURRENT bytes, for a cache that
-  // outlives the model.
+  // A process-unique identity for the BUFFER this tensor currently points at,
+  // for a cache that outlives the model.
+  //
+  // READ THAT LITERALLY: the identity is keyed on `bytes.data()`, so it is an
+  // identity for the address, not for the contents. Replacing a buffer's bytes
+  // IN PLACE — same address, different weights — keeps the old uid, and the
+  // cache would then serve the old entries for the new contents. Nothing does
+  // that today: a tower's `bytes` is assigned once when the model loads and is
+  // only ever replaced wholesale, which moves the address. This comment says
+  // where the guarantee stops rather than rounding it up, because #1066 was
+  // caused by a comment on this exact field that rounded it up (it claimed a
+  // base pointer was a stable identity, which is true for one model's life and
+  // false for the cache's). `test_qwen36_weights` pins both halves.
   //
   // The expert slot cache is a process-lifetime singleton keyed by (tower,
   // expert), and it used to derive the tower half from the buffer's ADDRESS.