expert-stream: the liveness instrument cannot report the failure it was built to detect

[localai-bot]

A fresh review of #1076 (row ENG-EXPERT-STREAM, spec .agents/specs/expert-streaming.md) returned FAIL. The TowerUid() fix for #1066 is correct and proven — these are defects in the wiring and observability that shipped beside it.

1. steps=0 is unreachable, so the F1 signature cannot be read off the stats line

src/vllm/model_executor/models/qwen3_5.cpp:5298 calls ReportStats only from EndStep(), and :5321 early-returns on steps == 0. When EndStep is never called — precisely defect F1, which is why the line exists — the line does not print at all.

Measured, same binary, VT_MOE_EXPERT_STREAM_STATS_EVERY=1:

healthy:        8 lines, "[expert-stream] steps=1 hits=0 misses=48 ... exhausted=0 advised=48"
F1 reinjected:  0 lines, only the startup "[expert-stream] ON slots=64 ..."

docs/ENVIRONMENT.md:82 and docs/USAGE.md:3105 both instruct the operator to read steps == 0 off that line. The procedure is unsatisfiable. The absence of the line is the signature, not a zero in it.

A second consequence, found while acting on this: stats_every_ defaults to 16 (:5411), so a short run that reaches ~5 steps prints nothing even when healthy. A benchmark that treats absence as failure therefore reports VOID on a working lane. That was live in the streaming benchmark and is why it had to be restarted.

Remediation: wire the already-present-but-uncalled ReportStats(/*final=*/true) to a teardown flush so exactly one line always prints, and correct both docs to describe absence.

2. CHECK(s.advised > 0) is not load-bearing

tests/vllm/model_executor/test_expert_stream_wiring.cpp:421. Reinjecting the pre-fix unaligned madvise address (page_begin = begin) exits 0 in 40/40 runs under the environment ctest uses. The mutation is live — the identical binary with one extra env var reports advised=0 and fails 20/20 — but > 0 over 48 calls is satisfied whenever heap layout happens to page-align a single slice. #1076's mutation table records this row as CAUGHT; it is not reproducibly so.

Remediation: assert advised == fills (48/48 in the healthy build).

3. "every MoE entry point funnels through here exactly once per forward" is false

qwen3_5.cpp:7361. These reach ExpertMlpKq -> KqExpertSlice with no step guard: Qwen3_5MTPModel::Forward (:7973), ::ForwardPaged (:8021), Qwen3_5Model::ForwardDense (:7859), Qwen3_5ReplayLayer (:8969). MTP is the production spec-decode draft path and runs on CPU in tests/vllm/v1/spec_decode/test_mtp_speculator.cpp:382. Effect: draft acquisitions stay protected_this_step across two forwards and never advance the hotness clock. Same class as F1, smaller blast radius.

4. The pread path is reached by no test

qwen3_5.cpp:5309 takes EnsureFile whenever w.mmap_fd >= 0, i.e. on every real GGUF-mmap checkpoint. Both new tests build owned host vectors (mmap_fd == -1) and only exercise EnsureSpan, so the file_offset + offset composition is unverified end to end. The spec's ## Owed frames this as unmeasured (blocked on hardware); it is also unreached, and a CPU-local test needs neither the box nor the checkpoint.

5. TowerUid() promises more than it delivers

qwen3_5_weights.h:106 claims identity for "this tensor's CURRENT bytes"; qwen3_5_weights.cpp:38 keys on bytes.data() and restamps only when that address changes. Probe: same-address buffer replacement yields a stale uid, 5/5. Not live today (a keep-quant tower's bytes is assigned once), but #1066 was exactly a comment asserting a guarantee the code did not provide.

6. Minor

• ReportStats(bool final) — no caller passes true; the branch is unreachable. Fixing (1) is its natural caller.
• SetForceFallback (:5223) has no production caller and its branch increments the operator-facing exhausted_, conflating "budget refused" with "a test asked for the fallback". Not listed under ## Owed.
• fix(ENG-EXPERT-STREAM): the step clock had no caller, so the decode number measured a dead cache (#912, #1066) #1076's landed body is spliced mid-sentence around the gate evidence. The body is the squash commit message, so this is permanent history; recording it here since it cannot be repaired in place.

Owning row: ENG-EXPERT-STREAM. Full gate on the reviewed commit was clean (498/498, rc 0); every finding above is a gap in what the gate can see, not a red test.

[localai-bot]

Fixed and landed in 268da6b27 (PR #1100).

The teardown flush now prints exactly one [expert-stream] line per process from ~Qwen35ExpertStream, crossing both early returns, so the F1 signature is a line reading steps=0 rather than no line at all — and docs/ENVIRONMENT.md / docs/USAGE.md were corrected to describe what the code does. CHECK(s.advised > 0) was tightened to advised == fills after the loose form was measured passing 40/40 against the reinjected defect. The four MoE entry points are guarded and the nesting refusal is now mutation-pinned. forced_ was split from the operator-facing exhausted_. The pread path gained a CPU-local reachability case driving EnsureFile through a production entry point.

Two follow-ups were split out rather than folded in: #1107 (check-windows-portability.py scans no test TU and lists neither setenv nor unsetenv, so it structurally cannot catch the class it was pointed at) and #1108 (three of the four guards reach nothing outside tests, now recorded under the spec's ## Owed).

Also worth recording here, since it is what this issue was really about: while acting on finding 1 I found that stats_every_ defaults to 16, so a short healthy run printed nothing at all — identical output to a dead lane. The streaming benchmark was gating on that line and could therefore only ever report VOID. It had been running for half an hour on the locked box before that was noticed.