From 0acf2ca1d0b1e6dd2e7792d2664dd7fed5d6fab1 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Tue, 18 Aug 2026 07:55:53 +0000 Subject: [PATCH 1/8] record(ENG-EXPERT-STREAM): the CUDA `DeviceMemoryInfo` override is not a one-line port, and #1126's own step 3 would be a defect (#1126, #1197) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `CudaBackend` still does not override `vt::Backend::DeviceMemoryInfo`, and this change deliberately does not add it. What it fixes is the record that says how to close #1126, because as filed that record would produce a defect. ## What was already true, and what was not #1126 was filed against a seam comment reading "ROCm/CUDA override with hipMemGetInfo/cudaMemGetInfo". That comment no longer exists. #1132 corrected it in both places that carried it, `include/vt/backend.h:78-93` and `gemma4_moe.cpp:440-448`, and recorded the gap under `## Owed` in `expert-streaming.md`. Every anchor either of them cites was re-verified exact against `fd64c76ee`: `rocm_backend.hip:338-345`, `gemma4_moe.cpp:440-448`, `gemma4_moe.cpp:506`, `backend.h:78-93`, `platforms/cuda.cpp:71` and `platforms/rocm.cpp:74`. Six examined, six correct. So the comment half of this issue is done, and the honest question left is whether to build the capability. The answer here is no, and the reason is not the one the issue gives. ## What Gemma4 does today on CUDA `Gemma4MoE` is the seam's only consumer. `FreeBytes` returns false on an absent probe, `MakeRoom` refuses on unknown (`gemma4_moe.cpp:506`), so the device-expert LRU admits nothing on every CUDA device and every routed expert takes `ExpertGeGLUHost` instead (`gemma4_moe.cpp:49-74`, reached at `:1493-1499`), for the life of the process, and nothing reports it. That function's name misleads and it is worth being exact, because the first reading here was wrong: it does not compute on the host. It re-copies the expert's BF16 bytes host-to-device into a scratch buffer (`:59-60`), runs the same `vt::MatmulBT` / `GeluAndMul` pair on the device, and drains the queue once per expert (`:73`). So what the dead LRU costs is bandwidth and a per-expert `Synchronize`, not a wrong answer — the issue's "falls back to host H2D on every expert" is the accurate description. The refuse-on-unknown polarity at that call site is right, so the defect is the missing probe rather than the consumer. ## Why the override is not built here Four things came out of re-reading it, and each raises the price. 1. **There is nothing to run it on.** The LRU needs `ex.is_fp8`, which only `LoadMoeFp8PerExpert` sets (`gemma4_weights.cpp:210-215`) from a per-expert `F8_E4M3` export. No such Gemma-4 checkpoint is pinned anywhere in `docs/USAGE.md`: all 32 lines there matching `gemma` case-insensitively were swept, and every checkpoint among them is the LTX-2.5 text tower (`gemma4-12b-with-proj*.safetensors`), not a per-expert FP8 MoE decoder. The woken path cannot be exercised, here or elsewhere, until one is. 2. **The CUDA consumers are not the ones the ROCm arm measured.** `ExpertGeGLUDeviceAccum` (`gemma4_moe.cpp:76-93`) is generic `vt::MatmulBT` / `GeluAndMul` / `MatmulBTAlphaBeta` and would run. But `RunGemma4FusedTopkExpertGeGLU` (`:1599`), `ExpertGeGLUDeviceBatched` (`:240`), `PeerCopyGemma4Fp8ExpertSlice` and `RunGemma4Fp8TopKOnExpertDevice` are HIP-only, and a non-HIP build links stubs that return `false`. The override would admit experts to VRAM and then route them through the serial per-expert fallback: a third path, measured by nobody. Nor is the swap token-neutral by construction — the resident arm accumulates with `vt::MatmulBTAlphaBeta` (`:90`) where the fallback writes its output outright (`:70`), so the two sum the top-k experts in a different order. 3. **The headroom test does not mean the same thing on the CUDA device this project gates on.** `MakeRoom` admits iff `free_b >= need + 1.5 GiB` (`gemma4_moe.cpp:514`), a constant tuned on discrete dual R9700s where free VRAM is distinct from host RAM. On a GB10 it is not. #1123 measured that `cudaMemGetInfo`'s `total` there is exactly `/proc/meminfo MemTotal` times 1024, which is why it reached for that instrument where `nvidia-smi` answers `[N/A]`; `free` is therefore reported over the same unified pool, and the host BF16 expert cache the device upload exists to relieve is drawn from that pool too. The admission test would double-count, on a box whose unified-memory OOM takes the host down with it. The `free` half of that is an inference from the measured `total` rather than a second measurement, and it wants confirming on the device before any override lands. 4. **The gap is isolated, not a pattern.** Comparing overrides one by one across `src/vt/cuda/cuda_backend.cu` and `src/vt/rocm/rocm_backend.hip`, `DeviceMemoryInfo` is the only optional `vt::Backend` seam ROCm answers and CUDA does not. A dead path with a truthful comment beats a live path nobody has exercised, and points 2 and 3 say the live path would not even be the one the lab throughput note was taken on. ## Which answer to "how much device memory" is authoritative The tree holds two, and they are not duplication. - `vt::Backend::DeviceMemoryInfo(free, total)` is a **live** probe that moves with contention. Authoritative for a **runtime admission** decision — can this allocation succeed right now — and for nothing else. - `vllm::platforms::ResidencyPolicy::device_memory_total_bytes` is a **total** probed once at platform registration, `0 == UNKNOWN`. Authoritative for a **load-time budget** verdict, which has to be reproducible and independent of whatever else the box is doing. This makes #1126's own closing plan wrong at step 3, which proposes that the #1123 fit check "can then read the budget from the backend seam". Collapsing the second onto the first would make the same checkpoint load or be refused depending on the page cache, which is the property `total`-not-`free` was chosen to avoid. What #1126 owes is its steps 1 and 2 **together**, the override and the Gemma4 measurement; never step 1 alone, and never step 3. Both seams already carry the division in prose (`backend.h:90-93`, `platforms/interface.h:61-69`); it is restated in `## Owed` because that is the record a reader of the issue acts on. ## A second defect, filed in flow Reading `MakeRoom` line by line turned up #1197. The slot cap is tested before the eviction loop (`gemma4_moe.cpp:498` against `:499-500`) and `EvictOne` (`:457`) is the only thing that shrinks `slots`, so after 24 admissions `VT_GEMMA4_EXPERT_EVICT=1` is a no-op for the life of the process and the cache degrades permanently to fill-only. It binds only when the slot cap is reached before the byte budget. Filed and not fixed, for the same reason as above rather than for effort: the one-line repair wakes more `hipFree` under load, which the surrounding comments call a permanent `kfd_wait` hang, so the current ordering may be deliberate. It is owned by `ENG-EXPERT-STREAM` and listed under `## Owed`. ## Why this is a record-only change The record is the work here: a filed gap whose closing plan is wrong in two ways, and a newly filed defect. It does not restate what #1132 landed — points 1 through 4, the authority split and #1197 are all new. Leaving it out means the next agent implements step 3 as written. No code changed, so there is nothing to mutate and no behaviour to gate. The gate run is the checker suite. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code] --- .agents/issue-index.md | 1 + .agents/specs/expert-streaming.md | 4 +++- 2 files changed, 4 insertions(+), 1 deletion(-) diff --git a/.agents/issue-index.md b/.agents/issue-index.md index 83876d612..8c9411588 100644 --- a/.agents/issue-index.md +++ b/.agents/issue-index.md @@ -366,3 +366,4 @@ rather than merged. `scripts/check-agent-record.py` gates both. | [#1179](https://github.com/mudler/vllm.cpp/issues/1179) | `ENG-CUDAGRAPH-BREAK` | The hand-rolled decode-graph driver count recorded in `9bc4d7f44` is **eight** and is actually **nine**, and the row it feeds was framed as coverage-only when it is also correctness. The ninth is the DFlash draft graph, file-local with no header declaration, at `src/vllm/model_executor/models/qwen3_dflash.cpp:771,870,1038,1091,1095,1106` — its own `int g_state = 0` three-state machine (`:771`), its own `VT_DFLASH_GRAPH` kill switch (`:870`) instead of the `VLLM_CPP_CUDAGRAPH` the six batched drivers read, its own invalidate-on-block-width-change (`:1038-1047`) and its own `try { EndCaptureGraph(); } catch (...) {}` drain (`:1106`). The eight-count is stated in four places, all corrected here: [`sglang-breakable-cuda-graph.md`](specs/sglang-breakable-cuda-graph.md) §4 and `## Owed`, [`.agents/engine-matrix.md`](engine-matrix.md) rows `ENG-CUDAGRAPH-BREAK` and `ENG-CUDAGRAPH-DEDUP` ("times eight drivers", which sizes #1162's signature table), and [`.agents/roadmap_v1.md`](roadmap_v1.md) track `C12`. The reframing is the substantive half: `ENG-CUDAGRAPH-BREAK` was recorded as a COVERAGE row, and the duplication has already cost a SHIPPED model its decode graph. `src/vllm/model_executor/models/qwen3.cpp:961-986` declines the decode graph outright whenever the asynchronous device-token mirror is live, on its own measured battery — `depth-1, graph ON PASS 78/78`; `depth-2, graph OFF PASS 82/82`; `depth-2, graph ON FAIL, slots 1-3 degenerate` — because `Step()` replays against the HOST `input.token_ids` and the combine has patched the DEVICE ids. The comment names the real fix as reading the identifiers at replay time from a stable device buffer, and that fix exists, in exactly one sibling driver, as `StepDevInputs` (`src/vllm/model_executor/models/qwen3_5.cpp:3894`): `grep -c StepDevInputs` returns 41 lines there and 0 in each of `qwen3_moe.cpp`, `qwen3.cpp`, `deepseek_v2.cpp` and `voxtral.cpp`. One capability, written once, unavailable to four models, with a live mitigation standing in its place. This does NOT weaken the framing rule that `ENG-CUDAGRAPH` established: the row still makes no throughput claim, and the prefill refutation (GB10 3.8% host-idle between launches, GPU-busy >96%, 27B prefill gap 92.5% non-GEMM glue) stands unchanged. Coverage AND correctness, never speed. Fixed in flow with the [`eng-cudagraph-break.md`](specs/eng-cudagraph-break.md) review repair ([#1163](https://github.com/mudler/vllm.cpp/issues/1163)) | record | | [#1181](https://github.com/mudler/vllm.cpp/issues/1181) | `FIX-READ-F32-SCALAR-GUARD` | `ReadF32Scalar` (`src/vllm/model_executor/models/qwen3_5_weights.cpp:312-318` @ `ab6e65216`) bounds its input with `t.data != nullptr && t.nbytes >= sizeof(float)`, a LOWER bound, and then `memcpy`s four bytes into a `float`. Two silent wrong-value paths follow and neither fails: an ARRAY is reduced to element 0, so a block-wise FP8 scale grid of shape `[ceil(N/128), ceil(K/128)]` passes and stands in for the whole weight (measured under [#1166](https://github.com/mudler/vllm.cpp/issues/1166) on `Qwen/Qwen3.8-27B-FP8` @ `017b9c7af6b5689d5dd426a76e0bc077eb5ca20a`, `q_proj.weight_scale_inv` is `[96, 40]`), and ANY dtype is reinterpreted, since that same tensor is `BF16` and its four bytes are two bf16 values read as one float. Both return a finite plausible float, so the output is fluent, plausible and wrong, which is what a token gate cannot see. Upstream makes both facts structural rather than optional: a per-tensor scale is a distinct parameter TYPE that asserts `loaded_weight.shape[0] == 1` (`vllm/model_executor/parameter.py:260-272,304-309` @ `555967922`, plus the `_assert_and_load` shape assert at `:93-96`), the slot is allocated `torch.float32` so a narrow on-disk dtype is VALUE-converted rather than reinterpreted (`utils/fp8_utils.py:1276`), and the declared strategy TENSOR/CHANNEL/BLOCK picks the parameter type before a byte is read (`compressed_tensors/schemes/compressed_tensors_w8a8_fp8.py:63,128`). The AUDIT corrects the issue's own framing twice. The 27 grep hits across five files are 5 definitions, 20 call sites and 2 comment references, and both counts are short: `ReadCtF32Scalar` (`include/vllm/model_executor/models/dense_weight_loaders.h:376`) is a SIXTH copy of the same defect under another name, reached from a SIXTH model file (`src/vllm/model_executor/models/qwen3_weights.cpp:100,126-128` through `LoadCtNvfp4W4A16`). Of the six, three check nothing, `LnReadF32Scalar`/`ShReadF32Scalar` check dtype but not count, and only `nemotron_h_weights.cpp:557-573` is correct, which makes it the model the shared guard generalizes. No call site legitimately passes a multi-element or non-F32 tensor, and every existing fixture emits rank-0 or `{1}` `F32`, so nothing in the tree needed the leniency. It is NOT merely latent: `dense_weight_loaders.h:73-74` and `docs/BENCHMARKS.md:52` both record `unsloth/Qwen3.6-27B-NVFP4` @ `ccdaab7e` as FP8 W8A8 throughout with BF16 PER-OUTPUT-CHANNEL scales, and `LoadAttnDense` branches on the weight dtype alone (`qwen3_5_dense_weights.cpp:478-480`), so those projections enter the per-tensor arm and hit both defects at once under the tensor name the loader actually asked for, with no misspelling to stop them. Fixed in flow by one `dense_loaders::ReadF32Scalar(get, name)` that refuses `numel != 1` naming the shape, refuses a non-`F32` dtype naming the dtype, and requires exactly four readable bytes, with the other five copies deleted onto it and `nemotron_h`'s `Loader`-based twin kept as the one tracked exception. A narrow dtype is refused rather than converted, because a one-element BF16 scale has never been read correctly here and the BF16 layout that IS shipped is per-channel, which the count check refuses first. Per-channel FP8, block-wise FP8 and any explicit narrow-dtype conversion stay owed. Spec [`read-f32-scalar-guard.md`](specs/read-f32-scalar-guard.md) | bug | | [#1185](https://github.com/mudler/vllm.cpp/issues/1185) | `ENV-ORACLE-WHEEL-IN-LEASE` | The pinned vLLM oracle BUILDS, installs, imports and sees the GPU inside an `rc` lease on `dgx:gpu0`, measured 2026-08-18, which falsifies the `nvcc` clause four records carried. [`lease-runtime-staging.md`](specs/lease-runtime-staging.md) said the oracle "needs `nvcc`, which the worker still lacks", and `.agents/environment.md`, [`mtp-k-gt-1.md`](specs/mtp-k-gt-1.md) and [`gpu-lease-methodology.md`](specs/gpu-lease-methodology.md) each derived a blocker from it. The build job (`buildvllm.sh`, staged sha256 prefix `15e140d41f44e7c2`) asserted the checkout against the pin BEFORE compiling, printing `PIN CONFIRMED` at `5559679229bc961848b121ccdeaa8fa5d79bec98` and aborting otherwise, took `nvcc` from the toolkit row `MODEL-NEMOTRON-H-ABI-A3-E2E` staged (`NVCC_RC=0`, CUDA `release 13.3, V13.3.73`) and produced `WHEEL_RC=0`, `PERSIST_RC=0` and a 434 MiB `vllm-0.1.dev1+g555967922.cu133-cp312-cp312-linux_aarch64.whl`, sha256 `7c58b339741a288fbb313f4f5196c9c92a9e3b3c3ebe2ea970b0ff50bb9bcba4`. The identity job (`oracleenv.sh`, prefix `6119f5223f5d818c`) asserted from `cd /`, outside any source tree: `vllm.__version__ = 0.1.dev1+g555967922`, `IDENTITY_RC=0`, `cuda True NVIDIA GB10`, `CUDA_RC=0`. SCOPE, and it carries the same weight as the result: RUNNING A MODEL IS UNTESTED. Only build, install, import and `torch.cuda.is_available()` are measured, and [`mtp-k-gt-1.md`](specs/mtp-k-gt-1.md) records that the last time an oracle reached this far it consumed the host in the step AFTER `torch.compile` and REBOOTED the box, at `gpu_memory_utilization` 0.75 and again at 0.30, so the fraction is not the lever. The version string is an OPEN discrepancy: `.agents/upstream-sync.md` records `vllm_runtime_version = 0.23.1rc1.dev1511+g555967922`, the commit segment matches and satisfies the pin's binding `+g` rule, and the prefix differs because a shallow fetch stops `setuptools_scm` counting commits since the last tag, so a full-string gate needs a deeper fetch or a recorded pretend-version. The venv is NOT staged, because that job was killed at a 90-minute ceiling mid-copy and its partial tree was removed, so only the WHEEL is durable. Four staging walls, all artifacts of the NAS rather than of CUDA: `cp -a` preserves `file_mode=0664` so `nvcc` exited 126. CIFS `nounix` stores no symlink so `include` and `lib64` vanished and CMake reported `Could NOT find CUDA (missing: CUDA_INCLUDE_DIRS CUDA_CUDART_LIBRARY) (found version "13.3")`, naming the version and denying the toolkit in one line. 32 library links `libfoo.so` and `libfoo.so.MAJOR` had to be rebuilt because only the `libfoo.so.X.Y.Z` real files survived. And `markupsafe` existed as a dist-info with NO package files from a `pip --target` killed at a 35-minute ceiling, so Marlin codegen died on `ModuleNotFoundError`. The `rc` worker container is REUSED between jobs, so a repair inside a staging branch is skipped on the next run (`nvcc already in place`) and an environment repair must be unconditional and assert its postcondition. CONSEQUENCE for the rows #1129 blocked, [#1003](https://github.com/mudler/vllm.cpp/issues/1003), [#915](https://github.com/mudler/vllm.cpp/issues/915), [#821](https://github.com/mudler/vllm.cpp/issues/821) and [#81](https://github.com/mudler/vllm.cpp/issues/81): UNBLOCKED FOR THE BUILD STEP and STILL BLOCKED FOR A MODEL RUN. None can take a measurement until a model run is demonstrated. Job details, walls and non-claims in [`oracle-wheel-in-lease.md`](specs/oracle-wheel-in-lease.md) | verification | +| [#1197](https://github.com/mudler/vllm.cpp/issues/1197) | `ENG-EXPERT-STREAM` | `Gemma4MoE`'s device-expert LRU tests its slot cap BEFORE its eviction loop, so the eviction opt-in goes inert once the cap is reached. `DevExpertLru::MakeRoom` runs `if (slots.size() >= kMaxSlots) return false;` at `src/vllm/model_executor/models/gemma4_moe.cpp:498` @ `fd64c76ee`, two lines ahead of the `if (allow_evict) { while (used + need > bud && !slots.empty()) EvictOne(d); }` at `:499-500`, and `EvictOne` (`:457`, the DEVICE LRU's — the file carries a host-cache namesake at `:275`) is the only thing that shrinks `slots`. So after 24 admissions every later `MakeRoom` returns false at that first line, the eviction loop is never reached again, and `VT_GEMMA4_EXPERT_EVICT=1` becomes a no-op for the life of the process — the cache degrades permanently to the fill-only mode the opt-in exists to leave. It binds only when the slot cap is reached before the byte budget, i.e. when `24 * expert_bytes < BudgetBytes()` (below ~85.3 MiB per expert at the 2048 MiB default from `BudgetBytes`, `:416-436`, again the device one and not the host cache's at `:262`); above that the byte budget binds first and eviction behaves. Nothing reports which one happened. FILED, NOT FIXED, and not for effort: the one-line repair (move the cap test after the eviction loop, so it caps RESIDENT slots instead of stopping admission forever) wakes more `hipFree` under load, which the code's own comments call a permanent `kfd_wait` hang with the GPU idle, prefill done and no decode tokens (`:459-461` and `:486-488`), so the current ordering may be deliberate. Deciding it needs the dual-RDNA4 lab box of [`gemma4-rocm-fp8-moe.md`](specs/gemma4-rocm-fp8-moe.md); the host that found it has neither a ROCm nor a CUDA device. Found while establishing the facts for [#1126](https://github.com/mudler/vllm.cpp/issues/1126), which required reading `MakeRoom` line by line. Listed under `## Owed` in [`expert-streaming.md`](specs/expert-streaming.md) | bug | diff --git a/.agents/specs/expert-streaming.md b/.agents/specs/expert-streaming.md index f3d74317b..3e28c08f1 100644 --- a/.agents/specs/expert-streaming.md +++ b/.agents/specs/expert-streaming.md @@ -1606,7 +1606,9 @@ Carried debt for this row. Each item names why it is not closed here. | **`--device cuda` still cannot SERVE a larger-than-pool GGUF; it only refuses by name now.** The device-slot arm is the missing capability: a `DeviceExpertSlotStore` behind `ExpertSlotStore`, a read accessor on that interface (`KqExpertSlice` reads `HostExpertSlotStore::Slot()`, the CONCRETE class, so the seam cannot be swapped today), a device filler that is not `pread`-into-host (`ExpertSlotStore::SlotForWrite` is handed straight to `::pread`, `expert_streamer.cpp:76-94`), and lifting the `is_cpu()` guard at `qwen3_5.cpp:5578`. Sized by the measurement above: 2790 slices per token at 2,490,368 bytes is 6.95 GB per token against a 119.631 GiB pool that already holds the dense remainder. Tracked as [#1124](https://github.com/mudler/vllm.cpp/issues/1124). | It is a campaign, not a fix: W7 (the pluggable backing store) is its declared owner in the work breakdown, and the CPU arm's own I/O rate is still unmeasured on a live cache two rows above. Building a device lane on top of a host lane whose bandwidth number is void would be optimising against a number nobody has. | | **The fit bound omits everything that is not a weight.** KV cache, activations, the scratch pools and the CUDA context are not counted, so a checkpoint at 0.95x of the pool passes the refusal and still dies on the first forward. | A headroom fraction invented here would be exactly the guess the per-tensor bound exists to avoid. The number wants the startup memory profile that `KV-WARMUP-PROFILE` owns (`INVENTORIED`; upstream's is `GPUWorker.determine_available_memory`, `vllm/v1/worker/gpu_worker.py:451-495`, around `profile_run`, `vllm/v1/worker/gpu/model_runner.py:682`), which is a different row. Those two anchors are stated here from the pinned tree because that row's own three anchors are stale at the current pin, and `gguf_device_fit.h` had copied two of them — filed as [#1139](https://github.com/mudler/vllm.cpp/issues/1139), owned by `KV-WARMUP-PROFILE`, blocked here only by the `engine-matrix.md` record lock #1119 holds. | | **The fit bound also counts too MUCH, and that direction can refuse a load that fits.** A tensor present in the file and not staged by THIS load is a positive over-count. On a default load that is the MTP / `nextn` block: 8,940,488,704 bytes, 8.33 GiB, 2.2506 % of the target checkpoint. A budget in that window refuses a weight set that would have fitted. | Not closed here. Closing it means the bound taking a per-tensor staging POLICY as input, which is the caller's knowledge and not the file's, and the exclusion's own failure mode is an under-count to nothing — which restores the 26-minute-then-OOM this row exists to remove, on a device nobody here has to measure it on. So the direction is stated in `gguf_device_fit.h`, pinned executably by `test_gguf_device_fit`, exposed to operators in `docs/USAGE.md`, and tracked as [#1136](https://github.com/mudler/vllm.cpp/issues/1136). `VT_DEVICE_WEIGHT_BUDGET_BYTES` is the way out of the window in the meantime. | -| **`Backend::DeviceMemoryInfo` has no CUDA override.** Only ROCm implements it (`src/vt/rocm/rocm_backend.hip:338-345`), so `Gemma4MoE`'s device-expert LRU refuses on every CUDA device and falls back to host H2D silently. | The CAPABILITY is [#1126](https://github.com/mudler/vllm.cpp/issues/1126) and is not built here, because adding the override wakes a landed residency policy that needs its own measurement. What IS done here is the false COMMENT, corrected in both places that carried it: `include/vt/backend.h:78-93` and `gemma4_moe.cpp:440-448`. The second copy was found by this round's own audit; correcting only the seam would have left the claim in the tree, which is the shape this row keeps hitting. | +| **`Backend::DeviceMemoryInfo` has no CUDA override, and waking it is not the one-line port of the ROCm one that #1126 describes.** Only ROCm implements it (`src/vt/rocm/rocm_backend.hip:338-345`), so `Gemma4MoE`'s device-expert LRU refuses on every CUDA device (`gemma4_moe.cpp:506`) and the expert's BF16 bytes are re-copied HOST->DEVICE on every use instead (`ExpertGeGLUHost`, `gemma4_moe.cpp:49-74`, reached at `:1493-1499`; the H2D is `:59-60` and it drains the queue per expert at `:73`), silently and for the life of the process. The name misleads: it computes on the DEVICE from host-resident weights, so what the dead LRU costs is bandwidth and a per-expert `Synchronize`, not a wrong answer. | The CAPABILITY is [#1126](https://github.com/mudler/vllm.cpp/issues/1126) and is still not built. The false COMMENT was corrected by this row in both places that carried it, `include/vt/backend.h:78-93` and `gemma4_moe.cpp:440-448`; both anchors were re-verified exact against `fd64c76ee`, as were `rocm_backend.hip:338-345`, `gemma4_moe.cpp:506`, `platforms/cuda.cpp:71` and `platforms/rocm.cpp:74`. **Four things were established while re-reading it for #1126, and each one raises the price of the override.** (1) *There is nothing to run it on.* The LRU needs `ex.is_fp8`, which only `LoadMoeFp8PerExpert` sets (`gemma4_weights.cpp:210-215`) from a per-expert `F8_E4M3` export; no such Gemma-4 checkpoint is pinned anywhere in `docs/USAGE.md` — all 32 lines there matching `gemma` case-insensitively were swept, and every checkpoint among them is the LTX-2.5 text tower (`gemma4-12b-with-proj*.safetensors`), not a per-expert FP8 MoE decoder, so the woken path cannot be exercised, here or elsewhere, until one is. (2) *The CUDA consumers are not the ones the ROCm arm measured.* `ExpertGeGLUDeviceAccum` (`gemma4_moe.cpp:76-93`) is generic `vt::MatmulBT`/`GeluAndMul`/`MatmulBTAlphaBeta` and would run, but `RunGemma4FusedTopkExpertGeGLU` (`:1599`), `ExpertGeGLUDeviceBatched` (`:240`), `PeerCopyGemma4Fp8ExpertSlice` and `RunGemma4Fp8TopKOnExpertDevice` are HIP-only and a non-HIP build links stubs that return `false`. So the override would admit experts to VRAM and then route them through the SERIAL per-expert fallback, not the fused kernel the lab note's throughput was taken on — a third path, measured by nobody. Nor is the swap token-neutral by construction: the resident arm accumulates with `vt::MatmulBTAlphaBeta` (`:90`) where the fallback writes its output outright (`:70`), so the two arms sum the top-k experts in a different order. (3) *The headroom test does not mean the same thing on the CUDA device this project gates on.* `MakeRoom` admits iff `free_b >= need + 1.5 GiB` (`:514`), a constant tuned on discrete dual R9700s where free VRAM is a quantity distinct from host RAM. On a GB10 it is not. This row already measured that `cudaMemGetInfo`'s `total` there is EXACTLY `/proc/meminfo MemTotal` times 1024, which is why it reached for that instrument where `nvidia-smi` answers `[N/A]`; its `free` is therefore reported over the same unified pool, and the host BF16 expert cache the device upload exists to relieve (`ex.cached_gu`/`cached_dn` via `EnsureGemma4Fp8ExpertCached`, bounded by the host LRU at `gemma4_moe.cpp:352`) is drawn from that pool too. So the admission test would double-count, on a box whose unified-memory OOM takes the host down with it. The `free` half is an inference from the measured `total`, not a second measurement, and it wants confirming on the device before any override lands. (4) *The gap is isolated, not a pattern.* Comparing overrides one by one across `src/vt/cuda/cuda_backend.cu` and `src/vt/rocm/rocm_backend.hip` at `fd64c76ee`, `DeviceMemoryInfo` is the ONLY optional `vt::Backend` seam ROCm answers and CUDA does not. | +| **#1126's own closing plan, step 3, must NOT be done: the load-time fit check may not read the live seam.** The issue proposes that "the #1123 fit check can then read the budget from the backend seam on every platform that reports one". That would be a defect, and the duplication it means to remove is not duplication. | The tree holds two answers to "how much device memory", and they answer different questions. `vt::Backend::DeviceMemoryInfo(free, total)` is a LIVE probe that moves with contention; it is authoritative for a RUNTIME admission decision — can this allocation succeed right now — and for nothing else. `vllm::platforms::ResidencyPolicy::device_memory_total_bytes` is a TOTAL probed once at platform registration with `0 == UNKNOWN`; it is authoritative for a LOAD-TIME budget verdict, which has to be reproducible and independent of whatever else the box is doing. Collapsing the second onto the first would make the same checkpoint load or be refused depending on the page cache, which is precisely the property **`total`, not `free`** above was chosen to avoid. Both seams already carry this division in prose (`include/vt/backend.h:90-93` and `include/vllm/platforms/interface.h:61-69`); it is restated here because #1126 is the record a reader of that issue will act on, and as filed it points the other way. What #1126 actually owes is its steps 1 and 2 together — the override AND the Gemma4 measurement — never step 1 alone, and never step 3. | +| **The device-expert LRU's slot cap makes its own eviction opt-in inert.** `MakeRoom` tests `slots.size() >= kMaxSlots` (`gemma4_moe.cpp:498`) BEFORE the eviction loop (`:499-500`), and `EvictOne` (`:457`, the device LRU's — a host-cache namesake sits at `:275`) is the only thing that shrinks `slots`, so once 24 slots are resident `VT_GEMMA4_EXPERT_EVICT=1` never runs again and the cache degrades permanently to fill-only. It binds only when `24 * expert_bytes < BudgetBytes()`, so it is condition-dependent and silent either way. Tracked as [#1197](https://github.com/mudler/vllm.cpp/issues/1197). | Filed, not fixed, and for the same reason as the row above rather than for effort: the one-line repair wakes more `hipFree` under load, which the surrounding comments say has been observed as a permanent `kfd_wait` hang with the GPU idle and no decode tokens. The current ordering may well be deliberate belt-and-braces. Deciding that needs the dual-RDNA4 box `.agents/specs/gemma4-rocm-fp8-moe.md` describes; this host has neither a ROCm nor a CUDA device. It closes when the cap moves after the eviction loop and a run stays hang-free, or when the comment says the cap is by design — one of the two, not silence. | | **`model_loader.cpp` is cited by absolute line number from 109 sites in 45 files, and this row's change moved them.** Measured between `e7d0a1f7c` and the repaired head: 203 moved line references over 109 citing sites, 10 unmoved. The file is ~1640 lines and almost every engine and model row edits it, so any edit near its top invalidates citations in files the editing change never opens. | Not swept here, deliberately, and the reason is not effort: several of the 109 were ALREADY stale (`model-matrix.md:197` cites `:184-223` as the "live loader"; line 184 at `e7d0a1f7c` is `static const bool once = [] {`), and rewriting all of them from the current tree would launder pre-existing debt into a clean-looking record. What IS fixed here is the two anchors this change authored itself, checked against the final tree. Tracked as [#1143](https://github.com/mudler/vllm.cpp/issues/1143), which lists the three candidate fixes; it needs a row of its own and is parked here because this row is what measured it. | | **The budget knob is an environment variable, not a config key.** `VT_DEVICE_WEIGHT_BUDGET_BYTES`. | `ENG-RESIDENCY-CONFIG` ([#1110](https://github.com/mudler/vllm.cpp/issues/1110), PR #1119) is in flight and adds exactly the `vllm_cpp` namespace inside `--offload-config` this key belongs in. Landing a second, competing config surface while that one is unmerged would create the conflict both changes then have to resolve. Migrate once #1119 lands; tracked as [#1127](https://github.com/mudler/vllm.cpp/issues/1127). | From 5031bd10c2203f944b7b80881c4b308d38bf1a83 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Tue, 18 Aug 2026 08:35:43 +0000 Subject: [PATCH 2/8] fix(ENG-EXPERT-STREAM): the CUDA blocker is a missing `vt::MatmulBTAlphaBeta`, not a slower third path, and the arm now refuses by name (#1126, #1205) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A fresh review returned FAIL on the record this repairs. The decision it argued for was right and five of its six substantive claims survive. Its central technical paragraph was wrong, and on a record-only change a wrong record is the defect. ## The finding that inverts it `vt::MatmulBTAlphaBeta` has NO CUDA implementation. `src/vt/fused_ops.cpp:102` dispatches to `rocm::MatmulBTAlphaBetaRocm` (`rocm_matmul_hipblaslt.hip:516`) under `#if defined(VLLM_CPP_HIP)` AND `q.device.type == kROCM`; every other device falls through to a throw. There is no CUDA, Vulkan, Metal or CPU arm. So the record's claim that `ExpertGeGLUDeviceAccum` "would run" on CUDA is false. What the override wakes is: `EnsureGemma4Fp8ExpertOnDevice` (`gemma4_moe.cpp:548-586`, no HIP guard) -> `lru.MakeRoom` at `:565` succeeding once `FreeBytes` can answer -> `true` at `:575` -> the call site at `:1486` -> `ExpertGeGLUDeviceAccum` at `:1487` -> `vt::MatmulBTAlphaBeta` at `:90` -> throw. The `try`/`catch (...)` at `:563-585` wraps only the upload; the compute at `:1487` is outside it, so the exception leaves the decode step rather than degrading to the host fallback. It is latent today only because the other route into that function, `same_dev` (`:730-731`), needs `ex.gate_up_dev`, assigned nowhere but `rocm_gemma4_experts.hip:207,226`. The resident arm is unreachable off ROCm, not safe. That makes the missing CUDA arm the actual blocker for #1126 step 1, recorded nowhere, and a stronger argument than any of the record's four. Filed as #1205. ## Three further corrections `ExpertGeGLUDeviceBatched` (`:240`) was named beside three genuine HIP-only stubs. It is not one: it sits outside the file's only `#ifndef VLLM_CPP_HIP` block (`:1574-1628`), has no HIP implementation and no header declaration, and is unconditionally `return false` under its own lab note at `:237-239`. It is disabled everywhere, ROCm included, so listing it inflated the ROCm/CUDA asymmetry. The other three are correct and stay. "The two arms sum the top-k experts in a different order" is false. Both run inside the same `for (int i = 0; i < top_k; ++i)` at `:1431`. The real difference is rounding and where the routing weight is applied: the resident arm folds `ww` into the GEMM `alpha` and accumulates in the epilogue with `beta` (`:1434`, `:1442`, `:90`), while the fallback writes the unweighted product to a BF16 buffer (`:67`, `:70`) and applies `ww` afterwards with separate BF16 kernels (`MulScalar` at `:1524`, `MulScalar` plus `Add` at `:1526-1527`). Two extra BF16 roundings per expert, weight applied post-rounding. Right conclusion, wrong mechanism. "Never step 3" overstated the case, in the spec and in the previous title. #1126 step 3 proposes the fit check read the budget from the backend seam, and that seam returns BOTH `free` and `total` (`include/vt/backend.h:94`). Reading `total` is contention-independent and legitimate. The defensible statement, and the one the row now asserts: step 3 may read only `total`, never `free`, and must keep the value probed once at platform registration (`include/vllm/platforms/interface.h:70-72`). Two smaller ones. `DevExpertLru::Note` also calls `slots.clear()` at `:522`; it is a device-index reset rather than an eviction, frees nothing, and is unreachable in a single-device process, so the #1197 row now names it and the next reader does not conclude the sweep missed it. And the H2D cost the row states in the present tense is conditional: the device LRU is `ex.is_fp8`-gated (`:969`, `:1484`), and by the row's own point (1) no per-expert FP8 Gemma-4 checkpoint is pinned, so on CUDA the fallback is never even asked for today. ## The code this uncovered `fused_ops.cpp` threw a bare "ROCm-only in this build" mid-decode for an unimplemented arm. AGENTS.md requires an unimplemented arm to refuse with a message that names the missing part, and that one named neither the device that asked nor where an implementation would go, so a caller could not tell a missing kernel from a missing build flag. It now names all three plus the issue. `tests/vt/test_gemma4_rocm_fp8_seams.cpp` gates it on a posed CUDA queue and on `kCPU`, `kVULKAN` and `kMETAL` — the refusal reads only `q.device.type`, so a CPU build can pose as any device and no ROCm hardware is needed. This is an in-flow gateable defect independent of #1126, and when someone writes the CUDA implementation the same test is what it has to satisfy. It pins the precondition without any checker that says do not fix it. Red before green: the three message assertions and the three device-name assertions failed against the old message. Mutation-proven twice, both compiled, both applied, three cases each, tree restored byte-exact: restoring the old message goes RED, and deleting the refusal outright goes RED. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code] --- .agents/issue-index.md | 1 + .agents/specs/expert-streaming.md | 7 ++-- src/vt/fused_ops.cpp | 15 ++++++- tests/vt/test_gemma4_rocm_fp8_seams.cpp | 52 +++++++++++++++++++++++++ 4 files changed, 70 insertions(+), 5 deletions(-) diff --git a/.agents/issue-index.md b/.agents/issue-index.md index 8c9411588..a4ef1121d 100644 --- a/.agents/issue-index.md +++ b/.agents/issue-index.md @@ -367,3 +367,4 @@ rather than merged. `scripts/check-agent-record.py` gates both. | [#1181](https://github.com/mudler/vllm.cpp/issues/1181) | `FIX-READ-F32-SCALAR-GUARD` | `ReadF32Scalar` (`src/vllm/model_executor/models/qwen3_5_weights.cpp:312-318` @ `ab6e65216`) bounds its input with `t.data != nullptr && t.nbytes >= sizeof(float)`, a LOWER bound, and then `memcpy`s four bytes into a `float`. Two silent wrong-value paths follow and neither fails: an ARRAY is reduced to element 0, so a block-wise FP8 scale grid of shape `[ceil(N/128), ceil(K/128)]` passes and stands in for the whole weight (measured under [#1166](https://github.com/mudler/vllm.cpp/issues/1166) on `Qwen/Qwen3.8-27B-FP8` @ `017b9c7af6b5689d5dd426a76e0bc077eb5ca20a`, `q_proj.weight_scale_inv` is `[96, 40]`), and ANY dtype is reinterpreted, since that same tensor is `BF16` and its four bytes are two bf16 values read as one float. Both return a finite plausible float, so the output is fluent, plausible and wrong, which is what a token gate cannot see. Upstream makes both facts structural rather than optional: a per-tensor scale is a distinct parameter TYPE that asserts `loaded_weight.shape[0] == 1` (`vllm/model_executor/parameter.py:260-272,304-309` @ `555967922`, plus the `_assert_and_load` shape assert at `:93-96`), the slot is allocated `torch.float32` so a narrow on-disk dtype is VALUE-converted rather than reinterpreted (`utils/fp8_utils.py:1276`), and the declared strategy TENSOR/CHANNEL/BLOCK picks the parameter type before a byte is read (`compressed_tensors/schemes/compressed_tensors_w8a8_fp8.py:63,128`). The AUDIT corrects the issue's own framing twice. The 27 grep hits across five files are 5 definitions, 20 call sites and 2 comment references, and both counts are short: `ReadCtF32Scalar` (`include/vllm/model_executor/models/dense_weight_loaders.h:376`) is a SIXTH copy of the same defect under another name, reached from a SIXTH model file (`src/vllm/model_executor/models/qwen3_weights.cpp:100,126-128` through `LoadCtNvfp4W4A16`). Of the six, three check nothing, `LnReadF32Scalar`/`ShReadF32Scalar` check dtype but not count, and only `nemotron_h_weights.cpp:557-573` is correct, which makes it the model the shared guard generalizes. No call site legitimately passes a multi-element or non-F32 tensor, and every existing fixture emits rank-0 or `{1}` `F32`, so nothing in the tree needed the leniency. It is NOT merely latent: `dense_weight_loaders.h:73-74` and `docs/BENCHMARKS.md:52` both record `unsloth/Qwen3.6-27B-NVFP4` @ `ccdaab7e` as FP8 W8A8 throughout with BF16 PER-OUTPUT-CHANNEL scales, and `LoadAttnDense` branches on the weight dtype alone (`qwen3_5_dense_weights.cpp:478-480`), so those projections enter the per-tensor arm and hit both defects at once under the tensor name the loader actually asked for, with no misspelling to stop them. Fixed in flow by one `dense_loaders::ReadF32Scalar(get, name)` that refuses `numel != 1` naming the shape, refuses a non-`F32` dtype naming the dtype, and requires exactly four readable bytes, with the other five copies deleted onto it and `nemotron_h`'s `Loader`-based twin kept as the one tracked exception. A narrow dtype is refused rather than converted, because a one-element BF16 scale has never been read correctly here and the BF16 layout that IS shipped is per-channel, which the count check refuses first. Per-channel FP8, block-wise FP8 and any explicit narrow-dtype conversion stay owed. Spec [`read-f32-scalar-guard.md`](specs/read-f32-scalar-guard.md) | bug | | [#1185](https://github.com/mudler/vllm.cpp/issues/1185) | `ENV-ORACLE-WHEEL-IN-LEASE` | The pinned vLLM oracle BUILDS, installs, imports and sees the GPU inside an `rc` lease on `dgx:gpu0`, measured 2026-08-18, which falsifies the `nvcc` clause four records carried. [`lease-runtime-staging.md`](specs/lease-runtime-staging.md) said the oracle "needs `nvcc`, which the worker still lacks", and `.agents/environment.md`, [`mtp-k-gt-1.md`](specs/mtp-k-gt-1.md) and [`gpu-lease-methodology.md`](specs/gpu-lease-methodology.md) each derived a blocker from it. The build job (`buildvllm.sh`, staged sha256 prefix `15e140d41f44e7c2`) asserted the checkout against the pin BEFORE compiling, printing `PIN CONFIRMED` at `5559679229bc961848b121ccdeaa8fa5d79bec98` and aborting otherwise, took `nvcc` from the toolkit row `MODEL-NEMOTRON-H-ABI-A3-E2E` staged (`NVCC_RC=0`, CUDA `release 13.3, V13.3.73`) and produced `WHEEL_RC=0`, `PERSIST_RC=0` and a 434 MiB `vllm-0.1.dev1+g555967922.cu133-cp312-cp312-linux_aarch64.whl`, sha256 `7c58b339741a288fbb313f4f5196c9c92a9e3b3c3ebe2ea970b0ff50bb9bcba4`. The identity job (`oracleenv.sh`, prefix `6119f5223f5d818c`) asserted from `cd /`, outside any source tree: `vllm.__version__ = 0.1.dev1+g555967922`, `IDENTITY_RC=0`, `cuda True NVIDIA GB10`, `CUDA_RC=0`. SCOPE, and it carries the same weight as the result: RUNNING A MODEL IS UNTESTED. Only build, install, import and `torch.cuda.is_available()` are measured, and [`mtp-k-gt-1.md`](specs/mtp-k-gt-1.md) records that the last time an oracle reached this far it consumed the host in the step AFTER `torch.compile` and REBOOTED the box, at `gpu_memory_utilization` 0.75 and again at 0.30, so the fraction is not the lever. The version string is an OPEN discrepancy: `.agents/upstream-sync.md` records `vllm_runtime_version = 0.23.1rc1.dev1511+g555967922`, the commit segment matches and satisfies the pin's binding `+g` rule, and the prefix differs because a shallow fetch stops `setuptools_scm` counting commits since the last tag, so a full-string gate needs a deeper fetch or a recorded pretend-version. The venv is NOT staged, because that job was killed at a 90-minute ceiling mid-copy and its partial tree was removed, so only the WHEEL is durable. Four staging walls, all artifacts of the NAS rather than of CUDA: `cp -a` preserves `file_mode=0664` so `nvcc` exited 126. CIFS `nounix` stores no symlink so `include` and `lib64` vanished and CMake reported `Could NOT find CUDA (missing: CUDA_INCLUDE_DIRS CUDA_CUDART_LIBRARY) (found version "13.3")`, naming the version and denying the toolkit in one line. 32 library links `libfoo.so` and `libfoo.so.MAJOR` had to be rebuilt because only the `libfoo.so.X.Y.Z` real files survived. And `markupsafe` existed as a dist-info with NO package files from a `pip --target` killed at a 35-minute ceiling, so Marlin codegen died on `ModuleNotFoundError`. The `rc` worker container is REUSED between jobs, so a repair inside a staging branch is skipped on the next run (`nvcc already in place`) and an environment repair must be unconditional and assert its postcondition. CONSEQUENCE for the rows #1129 blocked, [#1003](https://github.com/mudler/vllm.cpp/issues/1003), [#915](https://github.com/mudler/vllm.cpp/issues/915), [#821](https://github.com/mudler/vllm.cpp/issues/821) and [#81](https://github.com/mudler/vllm.cpp/issues/81): UNBLOCKED FOR THE BUILD STEP and STILL BLOCKED FOR A MODEL RUN. None can take a measurement until a model run is demonstrated. Job details, walls and non-claims in [`oracle-wheel-in-lease.md`](specs/oracle-wheel-in-lease.md) | verification | | [#1197](https://github.com/mudler/vllm.cpp/issues/1197) | `ENG-EXPERT-STREAM` | `Gemma4MoE`'s device-expert LRU tests its slot cap BEFORE its eviction loop, so the eviction opt-in goes inert once the cap is reached. `DevExpertLru::MakeRoom` runs `if (slots.size() >= kMaxSlots) return false;` at `src/vllm/model_executor/models/gemma4_moe.cpp:498` @ `fd64c76ee`, two lines ahead of the `if (allow_evict) { while (used + need > bud && !slots.empty()) EvictOne(d); }` at `:499-500`, and `EvictOne` (`:457`, the DEVICE LRU's — the file carries a host-cache namesake at `:275`) is the only thing that shrinks `slots`. So after 24 admissions every later `MakeRoom` returns false at that first line, the eviction loop is never reached again, and `VT_GEMMA4_EXPERT_EVICT=1` becomes a no-op for the life of the process — the cache degrades permanently to the fill-only mode the opt-in exists to leave. It binds only when the slot cap is reached before the byte budget, i.e. when `24 * expert_bytes < BudgetBytes()` (below ~85.3 MiB per expert at the 2048 MiB default from `BudgetBytes`, `:416-436`, again the device one and not the host cache's at `:262`); above that the byte budget binds first and eviction behaves. Nothing reports which one happened. FILED, NOT FIXED, and not for effort: the one-line repair (move the cap test after the eviction loop, so it caps RESIDENT slots instead of stopping admission forever) wakes more `hipFree` under load, which the code's own comments call a permanent `kfd_wait` hang with the GPU idle, prefill done and no decode tokens (`:459-461` and `:486-488`), so the current ordering may be deliberate. Deciding it needs the dual-RDNA4 lab box of [`gemma4-rocm-fp8-moe.md`](specs/gemma4-rocm-fp8-moe.md); the host that found it has neither a ROCm nor a CUDA device. Found while establishing the facts for [#1126](https://github.com/mudler/vllm.cpp/issues/1126), which required reading `MakeRoom` line by line. Listed under `## Owed` in [`expert-streaming.md`](specs/expert-streaming.md) | bug | +| [#1205](https://github.com/mudler/vllm.cpp/issues/1205) | `ENG-EXPERT-STREAM` | `vt::MatmulBTAlphaBeta` has NO CUDA implementation, and that — not any of the four reasons [#1200](https://github.com/mudler/vllm.cpp/pull/1200) first recorded — is what [#1126](https://github.com/mudler/vllm.cpp/issues/1126) step 1 is blocked on. `src/vt/fused_ops.cpp:102` dispatches to `rocm::MatmulBTAlphaBetaRocm` (`src/vt/rocm/rocm_matmul_hipblaslt.hip:516`, declared `include/vt/rocm/rocm_matmul_batch.h:28`) under `#if defined(VLLM_CPP_HIP)` AND `q.device.type == kROCM`, and every other device falls through to a throw. There is no CUDA, Vulkan, Metal or CPU arm: this is a missing kernel, not a missing build flag. #1126 proposes adding `CudaBackend::DeviceMemoryInfo` via `cudaMemGetInfo` so `Gemma4MoE`'s device-expert LRU stops refusing on CUDA; the record said that would wake a slower third path, and it would in fact wake a THROW, mid-decode. The chain: `EnsureGemma4Fp8ExpertOnDevice` (`src/vllm/model_executor/models/gemma4_moe.cpp:548-586`, no HIP guard) -> `lru.MakeRoom` at `:565` succeeding once `FreeBytes` can answer -> `true` at `:575` -> the call site at `:1486` -> `ExpertGeGLUDeviceAccum` at `:1487` -> `vt::MatmulBTAlphaBeta` at `:90` -> throw. The `try`/`catch (...)` at `:563-585` wraps only the UPLOAD; the compute at `:1487` is outside it, so the exception leaves the decode step rather than degrading to the host fallback. Latent today only because the other route in, `same_dev` (`:730-731`), needs `ex.gate_up_dev`, assigned nowhere but `src/vt/rocm/rocm_gemma4_experts.hip:207,226` — so the resident arm is UNREACHABLE off ROCm rather than safe. FIXED IN FLOW, PARTLY: the REFUSAL now satisfies the rule that an unimplemented arm names the missing part. "ROCm-only in this build" named neither the device that asked nor where an implementation would go, so a caller could not tell a missing kernel from a missing build flag; it now names all three plus this issue, and `tests/vt/test_gemma4_rocm_fp8_seams.cpp` gates it on a posed CUDA queue and on `kCPU`, `kVULKAN` and `kMETAL`, mutation-proven RED by restoring the old message and RED again by deleting the refusal outright. The IMPLEMENTATION stays open and is what this issue tracks: a `beta`-accumulating BT GEMM on cuBLASLt is a kernel with its own correctness gate, no per-expert FP8 Gemma-4 checkpoint is pinned to exercise it on, and the host that found this has neither a ROCm nor a CUDA device. The same file refuses six further arms the same way (`MatmulBTFp8Channel`, `DequantFp8ChannelBf16`, `MoeGatherRows`, `MoeWeightedScatterAdd`, `MoeZeroBf16`, `DualRmsNormPlusRes`); only `MatmulBTAlphaBeta` is on the #1126 path, and the others are named so the next reader need not re-derive the list. Listed under `## Owed` in [`expert-streaming.md`](specs/expert-streaming.md) | bug | diff --git a/.agents/specs/expert-streaming.md b/.agents/specs/expert-streaming.md index 3e28c08f1..9e1af3754 100644 --- a/.agents/specs/expert-streaming.md +++ b/.agents/specs/expert-streaming.md @@ -1606,9 +1606,10 @@ Carried debt for this row. Each item names why it is not closed here. | **`--device cuda` still cannot SERVE a larger-than-pool GGUF; it only refuses by name now.** The device-slot arm is the missing capability: a `DeviceExpertSlotStore` behind `ExpertSlotStore`, a read accessor on that interface (`KqExpertSlice` reads `HostExpertSlotStore::Slot()`, the CONCRETE class, so the seam cannot be swapped today), a device filler that is not `pread`-into-host (`ExpertSlotStore::SlotForWrite` is handed straight to `::pread`, `expert_streamer.cpp:76-94`), and lifting the `is_cpu()` guard at `qwen3_5.cpp:5578`. Sized by the measurement above: 2790 slices per token at 2,490,368 bytes is 6.95 GB per token against a 119.631 GiB pool that already holds the dense remainder. Tracked as [#1124](https://github.com/mudler/vllm.cpp/issues/1124). | It is a campaign, not a fix: W7 (the pluggable backing store) is its declared owner in the work breakdown, and the CPU arm's own I/O rate is still unmeasured on a live cache two rows above. Building a device lane on top of a host lane whose bandwidth number is void would be optimising against a number nobody has. | | **The fit bound omits everything that is not a weight.** KV cache, activations, the scratch pools and the CUDA context are not counted, so a checkpoint at 0.95x of the pool passes the refusal and still dies on the first forward. | A headroom fraction invented here would be exactly the guess the per-tensor bound exists to avoid. The number wants the startup memory profile that `KV-WARMUP-PROFILE` owns (`INVENTORIED`; upstream's is `GPUWorker.determine_available_memory`, `vllm/v1/worker/gpu_worker.py:451-495`, around `profile_run`, `vllm/v1/worker/gpu/model_runner.py:682`), which is a different row. Those two anchors are stated here from the pinned tree because that row's own three anchors are stale at the current pin, and `gguf_device_fit.h` had copied two of them — filed as [#1139](https://github.com/mudler/vllm.cpp/issues/1139), owned by `KV-WARMUP-PROFILE`, blocked here only by the `engine-matrix.md` record lock #1119 holds. | | **The fit bound also counts too MUCH, and that direction can refuse a load that fits.** A tensor present in the file and not staged by THIS load is a positive over-count. On a default load that is the MTP / `nextn` block: 8,940,488,704 bytes, 8.33 GiB, 2.2506 % of the target checkpoint. A budget in that window refuses a weight set that would have fitted. | Not closed here. Closing it means the bound taking a per-tensor staging POLICY as input, which is the caller's knowledge and not the file's, and the exclusion's own failure mode is an under-count to nothing — which restores the 26-minute-then-OOM this row exists to remove, on a device nobody here has to measure it on. So the direction is stated in `gguf_device_fit.h`, pinned executably by `test_gguf_device_fit`, exposed to operators in `docs/USAGE.md`, and tracked as [#1136](https://github.com/mudler/vllm.cpp/issues/1136). `VT_DEVICE_WEIGHT_BUDGET_BYTES` is the way out of the window in the meantime. | -| **`Backend::DeviceMemoryInfo` has no CUDA override, and waking it is not the one-line port of the ROCm one that #1126 describes.** Only ROCm implements it (`src/vt/rocm/rocm_backend.hip:338-345`), so `Gemma4MoE`'s device-expert LRU refuses on every CUDA device (`gemma4_moe.cpp:506`) and the expert's BF16 bytes are re-copied HOST->DEVICE on every use instead (`ExpertGeGLUHost`, `gemma4_moe.cpp:49-74`, reached at `:1493-1499`; the H2D is `:59-60` and it drains the queue per expert at `:73`), silently and for the life of the process. The name misleads: it computes on the DEVICE from host-resident weights, so what the dead LRU costs is bandwidth and a per-expert `Synchronize`, not a wrong answer. | The CAPABILITY is [#1126](https://github.com/mudler/vllm.cpp/issues/1126) and is still not built. The false COMMENT was corrected by this row in both places that carried it, `include/vt/backend.h:78-93` and `gemma4_moe.cpp:440-448`; both anchors were re-verified exact against `fd64c76ee`, as were `rocm_backend.hip:338-345`, `gemma4_moe.cpp:506`, `platforms/cuda.cpp:71` and `platforms/rocm.cpp:74`. **Four things were established while re-reading it for #1126, and each one raises the price of the override.** (1) *There is nothing to run it on.* The LRU needs `ex.is_fp8`, which only `LoadMoeFp8PerExpert` sets (`gemma4_weights.cpp:210-215`) from a per-expert `F8_E4M3` export; no such Gemma-4 checkpoint is pinned anywhere in `docs/USAGE.md` — all 32 lines there matching `gemma` case-insensitively were swept, and every checkpoint among them is the LTX-2.5 text tower (`gemma4-12b-with-proj*.safetensors`), not a per-expert FP8 MoE decoder, so the woken path cannot be exercised, here or elsewhere, until one is. (2) *The CUDA consumers are not the ones the ROCm arm measured.* `ExpertGeGLUDeviceAccum` (`gemma4_moe.cpp:76-93`) is generic `vt::MatmulBT`/`GeluAndMul`/`MatmulBTAlphaBeta` and would run, but `RunGemma4FusedTopkExpertGeGLU` (`:1599`), `ExpertGeGLUDeviceBatched` (`:240`), `PeerCopyGemma4Fp8ExpertSlice` and `RunGemma4Fp8TopKOnExpertDevice` are HIP-only and a non-HIP build links stubs that return `false`. So the override would admit experts to VRAM and then route them through the SERIAL per-expert fallback, not the fused kernel the lab note's throughput was taken on — a third path, measured by nobody. Nor is the swap token-neutral by construction: the resident arm accumulates with `vt::MatmulBTAlphaBeta` (`:90`) where the fallback writes its output outright (`:70`), so the two arms sum the top-k experts in a different order. (3) *The headroom test does not mean the same thing on the CUDA device this project gates on.* `MakeRoom` admits iff `free_b >= need + 1.5 GiB` (`:514`), a constant tuned on discrete dual R9700s where free VRAM is a quantity distinct from host RAM. On a GB10 it is not. This row already measured that `cudaMemGetInfo`'s `total` there is EXACTLY `/proc/meminfo MemTotal` times 1024, which is why it reached for that instrument where `nvidia-smi` answers `[N/A]`; its `free` is therefore reported over the same unified pool, and the host BF16 expert cache the device upload exists to relieve (`ex.cached_gu`/`cached_dn` via `EnsureGemma4Fp8ExpertCached`, bounded by the host LRU at `gemma4_moe.cpp:352`) is drawn from that pool too. So the admission test would double-count, on a box whose unified-memory OOM takes the host down with it. The `free` half is an inference from the measured `total`, not a second measurement, and it wants confirming on the device before any override lands. (4) *The gap is isolated, not a pattern.* Comparing overrides one by one across `src/vt/cuda/cuda_backend.cu` and `src/vt/rocm/rocm_backend.hip` at `fd64c76ee`, `DeviceMemoryInfo` is the ONLY optional `vt::Backend` seam ROCm answers and CUDA does not. | -| **#1126's own closing plan, step 3, must NOT be done: the load-time fit check may not read the live seam.** The issue proposes that "the #1123 fit check can then read the budget from the backend seam on every platform that reports one". That would be a defect, and the duplication it means to remove is not duplication. | The tree holds two answers to "how much device memory", and they answer different questions. `vt::Backend::DeviceMemoryInfo(free, total)` is a LIVE probe that moves with contention; it is authoritative for a RUNTIME admission decision — can this allocation succeed right now — and for nothing else. `vllm::platforms::ResidencyPolicy::device_memory_total_bytes` is a TOTAL probed once at platform registration with `0 == UNKNOWN`; it is authoritative for a LOAD-TIME budget verdict, which has to be reproducible and independent of whatever else the box is doing. Collapsing the second onto the first would make the same checkpoint load or be refused depending on the page cache, which is precisely the property **`total`, not `free`** above was chosen to avoid. Both seams already carry this division in prose (`include/vt/backend.h:90-93` and `include/vllm/platforms/interface.h:61-69`); it is restated here because #1126 is the record a reader of that issue will act on, and as filed it points the other way. What #1126 actually owes is its steps 1 and 2 together — the override AND the Gemma4 measurement — never step 1 alone, and never step 3. | -| **The device-expert LRU's slot cap makes its own eviction opt-in inert.** `MakeRoom` tests `slots.size() >= kMaxSlots` (`gemma4_moe.cpp:498`) BEFORE the eviction loop (`:499-500`), and `EvictOne` (`:457`, the device LRU's — a host-cache namesake sits at `:275`) is the only thing that shrinks `slots`, so once 24 slots are resident `VT_GEMMA4_EXPERT_EVICT=1` never runs again and the cache degrades permanently to fill-only. It binds only when `24 * expert_bytes < BudgetBytes()`, so it is condition-dependent and silent either way. Tracked as [#1197](https://github.com/mudler/vllm.cpp/issues/1197). | Filed, not fixed, and for the same reason as the row above rather than for effort: the one-line repair wakes more `hipFree` under load, which the surrounding comments say has been observed as a permanent `kfd_wait` hang with the GPU idle and no decode tokens. The current ordering may well be deliberate belt-and-braces. Deciding that needs the dual-RDNA4 box `.agents/specs/gemma4-rocm-fp8-moe.md` describes; this host has neither a ROCm nor a CUDA device. It closes when the cap moves after the eviction loop and a run stays hang-free, or when the comment says the cap is by design — one of the two, not silence. | +| **`Backend::DeviceMemoryInfo` has no CUDA override, and waking it is not the one-line port of the ROCm one that #1126 describes: on CUDA it would wake a THROW.** Only ROCm implements it (`src/vt/rocm/rocm_backend.hip:338-345`), so `Gemma4MoE`'s device-expert LRU refuses on every CUDA device (`gemma4_moe.cpp:506`). Where a per-expert FP8 checkpoint is present, the expert's BF16 bytes are re-copied HOST->DEVICE on every use instead (`ExpertGeGLUHost`, `gemma4_moe.cpp:49-74`, reached at `:1493-1499`; the H2D is `:59-60` and it drains the queue per expert at `:73`), silently and for the life of the process. That H2D cost is CONDITIONAL, not present-tense: the whole device LRU is `ex.is_fp8`-gated (`:969`, `:1484`), and by point (1) below no such Gemma-4 checkpoint is pinned anywhere, so on CUDA today the fallback is never even asked for. The name misleads: it computes on the DEVICE from host-resident weights, so what the dead LRU would cost is bandwidth and a per-expert `Synchronize`, not a wrong answer. | The CAPABILITY is [#1126](https://github.com/mudler/vllm.cpp/issues/1126) and is still not built. The false COMMENT was corrected by this row in both places that carried it, `include/vt/backend.h:78-93` and `gemma4_moe.cpp:440-448`; both anchors were re-verified exact against `fd64c76ee`, as were `rocm_backend.hip:338-345`, `gemma4_moe.cpp:506`, `platforms/cuda.cpp:71` and `platforms/rocm.cpp:74`. **Four things were established while re-reading it for #1126, and each one raises the price of the override.** (1) *There is nothing to run it on.* The LRU needs `ex.is_fp8`, which only `LoadMoeFp8PerExpert` sets (`gemma4_weights.cpp:210-215`) from a per-expert `F8_E4M3` export; no such Gemma-4 checkpoint is pinned anywhere in `docs/USAGE.md` — all 32 lines there matching `gemma` case-insensitively were swept, and every checkpoint among them is the LTX-2.5 text tower (`gemma4-12b-with-proj*.safetensors`), not a per-expert FP8 MoE decoder, so the woken path cannot be exercised, here or elsewhere, until one is. (2) *The device-resident arm has no CUDA implementation to route into. It has a throw.* `ExpertGeGLUDeviceAccum` (`gemma4_moe.cpp:76-93`) READS as generic — `vt::MatmulBT`, `GeluAndMul`, `vt::MatmulBTAlphaBeta` — and an earlier draft of this row concluded from that reading that it "would run". It does not. `vt::MatmulBTAlphaBeta` (`src/vt/fused_ops.cpp:102`) is guarded on `#if defined(VLLM_CPP_HIP)` AND `q.device.type == kROCM`; its only implementation in the tree is `rocm::MatmulBTAlphaBetaRocm` (`src/vt/rocm/rocm_matmul_hipblaslt.hip:516`, declared `include/vt/rocm/rocm_matmul_batch.h:28`), and every other device falls through to a refusal. So the chain the override wakes is: `EnsureGemma4Fp8ExpertOnDevice` (`gemma4_moe.cpp:548-586`, carrying NO HIP guard) -> `lru.MakeRoom` (`:565`) succeeding as soon as `FreeBytes` can answer -> `true` at `:575` -> the call site at `:1486` -> `ExpertGeGLUDeviceAccum` at `:1487` -> `vt::MatmulBTAlphaBeta` at `:90` -> THROW, mid-decode. The `try`/`catch (...)` at `:563-585` wraps only the UPLOAD; the compute at `:1487` sits outside it, so the exception propagates out of the decode step rather than degrading to the host fallback. It is latent today only because the other route into that function, `same_dev` (`:730-731`), needs `ex.gate_up_dev`, which is assigned nowhere but `src/vt/rocm/rocm_gemma4_experts.hip:207,226` — so the resident arm is UNREACHABLE off ROCm rather than safe. This is the actual blocker under #1126, it was recorded nowhere, and it is a stronger argument than the other three: filed as [#1205](https://github.com/mudler/vllm.cpp/issues/1205), and the refusal itself is now gated by `tests/vt/test_gemma4_rocm_fp8_seams.cpp`, which is what a CUDA implementation will have to satisfy. **Two corrections to the earlier draft's supporting claims, both of which overstated the case.** *The HIP-only list was one symbol too long.* Three of the four are genuinely HIP-only stubs inside this file's ONLY `#ifndef VLLM_CPP_HIP` block (`:1574-1628`): `RunGemma4FusedTopkExpertGeGLU` (`:1599`), `PeerCopyGemma4Fp8ExpertSlice` (`:1607`) and `RunGemma4Fp8TopKOnExpertDevice` (`:1611`). `ExpertGeGLUDeviceBatched` (`:240`) is NOT: it sits OUTSIDE that block, in an anonymous namespace, with no HIP implementation and no header declaration, and is unconditionally `return false` under its own lab note (`:237-239` — gather+strided produced wrong tokens at ~23 t/s, pointer-batch ~0.8 t/s, serial/fused-gelu kept at ~34 t/s). It is disabled EVERYWHERE, ROCm included, so naming it beside the three inflated the ROCm/CUDA asymmetry. *The token-neutrality argument had the wrong mechanism.* The conclusion stands — the swap would not be token-neutral — but not because "the two arms sum the top-k experts in a different order". They do not: both run inside the SAME `for (int i = 0; i < top_k; ++i)` at `:1431`, so the summation order is identical. The real difference is ROUNDING and where the routing weight is applied. The resident arm folds `ww` into the GEMM `alpha` and accumulates in the epilogue with `beta` (`:1434`, `:1442`, `:90`), so the weight multiplies in the GEMM's own accumulator. The fallback writes the UNWEIGHTED product to a BF16 buffer (`:67`, `:70`) and applies `ww` afterwards with separate BF16 kernels — `vt::MulScalar` at `:1524` on the first expert, `MulScalar` plus `vt::Add` at `:1526-1527` on the rest. Two extra BF16 roundings per expert, with the weight applied post-rounding. (3) *The headroom test does not mean the same thing on the CUDA device this project gates on.* `MakeRoom` admits iff `free_b >= need + 1.5 GiB` (`:514`), a constant tuned on discrete dual R9700s where free VRAM is a quantity distinct from host RAM. On a GB10 it is not. This row already measured that `cudaMemGetInfo`'s `total` there is EXACTLY `/proc/meminfo MemTotal` times 1024, which is why it reached for that instrument where `nvidia-smi` answers `[N/A]`; its `free` is therefore reported over the same unified pool, and the host BF16 expert cache the device upload exists to relieve (`ex.cached_gu`/`cached_dn` via `EnsureGemma4Fp8ExpertCached`, bounded by the host LRU at `gemma4_moe.cpp:352`) is drawn from that pool too. So the admission test would double-count, on a box whose unified-memory OOM takes the host down with it. The `free` half is an inference from the measured `total`, not a second measurement, and it wants confirming on the device before any override lands. (4) *The gap is isolated, not a pattern.* Comparing overrides one by one across `src/vt/cuda/cuda_backend.cu` and `src/vt/rocm/rocm_backend.hip` at `fd64c76ee`, `DeviceMemoryInfo` is the ONLY optional `vt::Backend` seam ROCm answers and CUDA does not. | +| **#1126's own closing plan, step 3, must be NARROWED before it is done: the load-time fit check may read the seam's `total`, and never its `free`.** The issue proposes that "the #1123 fit check can then read the budget from the backend seam on every platform that reports one". As written that invites the live half, which would be a defect — but the seam returns BOTH halves (`bool DeviceMemoryInfo(size_t* free_bytes, size_t* total_bytes)`, `include/vt/backend.h:94`), and only one of them is illegitimate here. | The tree holds two answers to "how much device memory", and they answer different questions. `vt::Backend::DeviceMemoryInfo(free, total)` is a LIVE probe that moves with contention; it is authoritative for a RUNTIME admission decision — can this allocation succeed right now — and for nothing else. `vllm::platforms::ResidencyPolicy::device_memory_total_bytes` is a TOTAL probed once at platform registration with `0 == UNKNOWN`; it is authoritative for a LOAD-TIME budget verdict, which has to be reproducible and independent of whatever else the box is doing. Sourcing the load-time verdict from `free` would make the same checkpoint load or be refused depending on the page cache, which is precisely the property **`total`, not `free`** above was chosen to avoid. Reading `total` through the seam is not that. It is contention-independent, it is the same quantity `ResidencyPolicy` already carries, and a seam that answers it on every platform is a defensible place to source it from. What step 3 must additionally preserve is the PROBE-ONCE semantics: `device_memory_total_bytes` is probed at platform registration (`include/vllm/platforms/interface.h:70-72`), and a per-load live call would reintroduce the contention dependence by the back door even reading only `total`, because a load-time verdict has to be reproducible from the record rather than from the moment. So the defensible statement, and the one this row asserts: **step 3 may read only `total`, never `free`, and must keep the value probed once at platform registration.** An earlier draft of this row said "never step 3" outright; that overstated it and would have blocked a legitimate simplification, so it is corrected here rather than quietly narrowed. Both seams already carry the division in prose (`include/vt/backend.h:90-93` and `include/vllm/platforms/interface.h:61-69`); it is restated here because #1126 is the record a reader of that issue will act on, and as filed it points the other way. What #1126 owes is its steps 1 and 2 together — the override AND the Gemma4 measurement, with [#1205](https://github.com/mudler/vllm.cpp/issues/1205) ahead of both — never step 1 alone, and step 3 only in the narrowed form above. | +| **The device-expert LRU's slot cap makes its own eviction opt-in inert.** `MakeRoom` tests `slots.size() >= kMaxSlots` (`gemma4_moe.cpp:498`) BEFORE the eviction loop (`:499-500`), and `EvictOne` (`:457`, the device LRU's — a host-cache namesake sits at `:275`) is the only thing that SHRINKS `slots`. The one other statement that touches its size, `slots.clear()` in `DevExpertLru::Note` (`:522`), is a device-index RESET rather than an eviction: it drops bookkeeping when `dev != d.q.device.index` and frees nothing, and it is unreachable in a single-device process. It is named here so the next reader does not conclude the #1197 sweep missed it. So once 24 slots are resident `VT_GEMMA4_EXPERT_EVICT=1` never runs again and the cache degrades permanently to fill-only. It binds only when `24 * expert_bytes < BudgetBytes()`, so it is condition-dependent and silent either way. Tracked as [#1197](https://github.com/mudler/vllm.cpp/issues/1197). | Filed, not fixed, and for the same reason as the row above rather than for effort: the one-line repair wakes more `hipFree` under load, which the surrounding comments say has been observed as a permanent `kfd_wait` hang with the GPU idle and no decode tokens. The current ordering may well be deliberate belt-and-braces. Deciding that needs the dual-RDNA4 box `.agents/specs/gemma4-rocm-fp8-moe.md` describes; this host has neither a ROCm nor a CUDA device. It closes when the cap moves after the eviction loop and a run stays hang-free, or when the comment says the cap is by design — one of the two, not silence. | +| **`vt::MatmulBTAlphaBeta` is ROCm-only and has no CUDA implementation at all, which is what #1126 step 1 is actually blocked on.** `src/vt/fused_ops.cpp:102` dispatches to `rocm::MatmulBTAlphaBetaRocm` (`src/vt/rocm/rocm_matmul_hipblaslt.hip:516`) under `#if defined(VLLM_CPP_HIP)` and `q.device.type == kROCM`, and refuses on every other device. There is no CUDA, Vulkan, Metal or CPU arm. The full chain from the missing `DeviceMemoryInfo` override to that refusal is traced in the first row above. Tracked as [#1205](https://github.com/mudler/vllm.cpp/issues/1205). | The REFUSAL is fixed in flow, because a bare `std::runtime_error` reading "ROCm-only in this build" does not satisfy the standing rule that an unimplemented arm refuses with a message NAMING the missing part: a caller who hits it on CUDA cannot tell a missing kernel from a missing build flag. It now names the device that asked, names where the implementation lives, and names the issue, and `tests/vt/test_gemma4_rocm_fp8_seams.cpp` gates it on a posed CUDA queue plus three other non-ROCm devices — mutation-proven by restoring the old message (RED) and by deleting the refusal outright (RED). Pinning the precondition executably is also the answer to "should the absence be recorded": the gate says what is missing without any checker that says do not fix it, and when someone writes the CUDA arm the same test is what it has to satisfy. The IMPLEMENTATION stays owed and is what [#1205](https://github.com/mudler/vllm.cpp/issues/1205) tracks. It is not written here: a `beta`-accumulating BT GEMM on cuBLASLt is a kernel with its own correctness gate, this row's point (1) says there is no checkpoint to exercise it on, and this host has neither a ROCm nor a CUDA device to measure either arm. | | **`model_loader.cpp` is cited by absolute line number from 109 sites in 45 files, and this row's change moved them.** Measured between `e7d0a1f7c` and the repaired head: 203 moved line references over 109 citing sites, 10 unmoved. The file is ~1640 lines and almost every engine and model row edits it, so any edit near its top invalidates citations in files the editing change never opens. | Not swept here, deliberately, and the reason is not effort: several of the 109 were ALREADY stale (`model-matrix.md:197` cites `:184-223` as the "live loader"; line 184 at `e7d0a1f7c` is `static const bool once = [] {`), and rewriting all of them from the current tree would launder pre-existing debt into a clean-looking record. What IS fixed here is the two anchors this change authored itself, checked against the final tree. Tracked as [#1143](https://github.com/mudler/vllm.cpp/issues/1143), which lists the three candidate fixes; it needs a row of its own and is parked here because this row is what measured it. | | **The budget knob is an environment variable, not a config key.** `VT_DEVICE_WEIGHT_BUDGET_BYTES`. | `ENG-RESIDENCY-CONFIG` ([#1110](https://github.com/mudler/vllm.cpp/issues/1110), PR #1119) is in flight and adds exactly the `vllm_cpp` namespace inside `--offload-config` this key belongs in. Landing a second, competing config surface while that one is unmerged would create the conflict both changes then have to resolve. Migrate once #1119 lands; tracked as [#1127](https://github.com/mudler/vllm.cpp/issues/1127). | diff --git a/src/vt/fused_ops.cpp b/src/vt/fused_ops.cpp index 236c97e40..937e986cf 100644 --- a/src/vt/fused_ops.cpp +++ b/src/vt/fused_ops.cpp @@ -1,6 +1,7 @@ #include "vt/fused_ops.h" #include +#include #include "vt/backend.h" #include "vt/dtype.h" @@ -106,7 +107,6 @@ void MatmulBTAlphaBeta(Queue& q, void* out, const void* a, const void* b, int M, return; } #endif - (void)q; (void)out; (void)a; (void)b; @@ -116,7 +116,18 @@ void MatmulBTAlphaBeta(Queue& q, void* out, const void* a, const void* b, int M, (void)alpha; (void)beta; (void)dtype; - throw std::runtime_error("vt::MatmulBTAlphaBeta: ROCm-only in this build"); + // No CUDA arm exists: the only implementation in the tree is + // rocm::MatmulBTAlphaBetaRocm (src/vt/rocm/rocm_matmul_hipblaslt.hip), so this + // is a missing kernel and not a missing build flag. Name the device that asked + // and where the implementation lives, because the caller cannot tell those two + // apart from "ROCm-only" alone. Reaching this on CUDA is issue #1205 and blocks + // #1126 step 1: waking Gemma4's device-expert LRU routes decode into + // ExpertGeGLUDeviceAccum, which lands here outside the upload's try/catch. + throw std::runtime_error( + std::string("vt::MatmulBTAlphaBeta: no implementation for device '") + + DeviceTypeName(q.device.type) + + "'; this arm is implemented for 'rocm' only (src/vt/rocm/rocm_matmul_hipblaslt.hip). " + "A CUDA implementation is missing; see issue #1205."); } void MatmulBTFp8Channel(Queue& q, void* out, const void* a, const void* b_fp8, diff --git a/tests/vt/test_gemma4_rocm_fp8_seams.cpp b/tests/vt/test_gemma4_rocm_fp8_seams.cpp index 12abaf6d2..105ebbfdf 100644 --- a/tests/vt/test_gemma4_rocm_fp8_seams.cpp +++ b/tests/vt/test_gemma4_rocm_fp8_seams.cpp @@ -4,8 +4,11 @@ #include #include +#include #include +#include "vt/device.h" +#include "vt/dtype.h" #include "vt/fused_ops.h" namespace { @@ -76,3 +79,52 @@ TEST_CASE("gemma4 rocm fp8 seams: recipe env knobs parse inert defaults") { CHECK(EnvInt("VT_GEMMA4_FP8_HW_CVT", 1) == 0); } } + +// #1205: `vt::MatmulBTAlphaBeta` has NO CUDA implementation. The only one in the +// tree is `rocm::MatmulBTAlphaBetaRocm` (rocm_matmul_hipblaslt.hip:516), reached +// through a `#if defined(VLLM_CPP_HIP)` + `kROCM` guard in fused_ops.cpp; every +// other device falls through to a throw. That is the actual blocker under #1126: +// adding `CudaBackend::DeviceMemoryInfo` alone would let the Gemma4 expert LRU +// admit (gemma4_moe.cpp:565,575), take ExpertGeGLUDeviceAccum at :1487, and reach +// this call at :90 — outside the upload's try/catch (:563-585), so mid-decode. +// +// AGENTS.md requires an unimplemented arm to refuse with a message that NAMES the +// missing part. "ROCm-only in this build" names neither the device that asked nor +// where the implementation would go, so a caller who hits it on CUDA cannot tell a +// missing kernel from a missing build flag. This pins the contract; when the CUDA +// arm is written, this is what it has to satisfy. +TEST_CASE("gemma4 rocm fp8 seams: MatmulBTAlphaBeta refuses a non-ROCm queue by name") { + // The refusal reads `q.device.type` and nothing else, so a CPU build can pose + // as any device here. Buffers are never dereferenced on the refusing path. + auto refusal_for = [](vt::DeviceType type) -> std::string { + vt::Queue q; + q.device.type = type; + q.device.index = 0; + try { + vt::MatmulBTAlphaBeta(q, nullptr, nullptr, nullptr, /*M=*/1, /*N=*/1, /*K=*/1, + /*alpha=*/1.f, /*beta=*/0.f, vt::DType::kBF16); + } catch (const std::runtime_error& e) { + return e.what(); + } + return std::string(); + }; + + // CUDA is the device #1126 would wake, so it is the one the message must name. + const std::string cuda_msg = refusal_for(vt::DeviceType::kCUDA); + REQUIRE_FALSE(cuda_msg.empty()); + INFO("cuda refusal: " << cuda_msg); + CHECK(cuda_msg.find("MatmulBTAlphaBeta") != std::string::npos); + CHECK(cuda_msg.find("cuda") != std::string::npos); + CHECK(cuda_msg.find("rocm") != std::string::npos); + CHECK(cuda_msg.find("1205") != std::string::npos); + + // Not a CUDA special case: every non-ROCm device refuses, naming itself. + for (const auto type : {vt::DeviceType::kCPU, vt::DeviceType::kVULKAN, + vt::DeviceType::kMETAL}) { + const std::string msg = refusal_for(type); + // std::string, not the char* — doctest stringifies a bare char* as a bool. + INFO("refusal for " << std::string(vt::DeviceTypeName(type)) << ": " << msg); + REQUIRE_FALSE(msg.empty()); + CHECK(msg.find(vt::DeviceTypeName(type)) != std::string::npos); + } +} From 7beada17c39e97690b9e7455c74434e337b4cb0a Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Tue, 18 Aug 2026 08:40:26 +0000 Subject: [PATCH 3/8] record(ENG-EXPERT-STREAM): name the anchor re-audit and point the #1205 row at the right numbered point (#1205) The `## Owed` rows now say the review repair re-audited every anchor the three rows cite against the repaired tree, 53 examined and 53 exact, so a reader can tell a re-verified anchor from a carried-over one. The #1205 row said "this row's point (1)" for a point that lives in the `DeviceMemoryInfo` row above it. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code] --- .agents/specs/expert-streaming.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/.agents/specs/expert-streaming.md b/.agents/specs/expert-streaming.md index 9e1af3754..75d5959d8 100644 --- a/.agents/specs/expert-streaming.md +++ b/.agents/specs/expert-streaming.md @@ -1606,10 +1606,10 @@ Carried debt for this row. Each item names why it is not closed here. | **`--device cuda` still cannot SERVE a larger-than-pool GGUF; it only refuses by name now.** The device-slot arm is the missing capability: a `DeviceExpertSlotStore` behind `ExpertSlotStore`, a read accessor on that interface (`KqExpertSlice` reads `HostExpertSlotStore::Slot()`, the CONCRETE class, so the seam cannot be swapped today), a device filler that is not `pread`-into-host (`ExpertSlotStore::SlotForWrite` is handed straight to `::pread`, `expert_streamer.cpp:76-94`), and lifting the `is_cpu()` guard at `qwen3_5.cpp:5578`. Sized by the measurement above: 2790 slices per token at 2,490,368 bytes is 6.95 GB per token against a 119.631 GiB pool that already holds the dense remainder. Tracked as [#1124](https://github.com/mudler/vllm.cpp/issues/1124). | It is a campaign, not a fix: W7 (the pluggable backing store) is its declared owner in the work breakdown, and the CPU arm's own I/O rate is still unmeasured on a live cache two rows above. Building a device lane on top of a host lane whose bandwidth number is void would be optimising against a number nobody has. | | **The fit bound omits everything that is not a weight.** KV cache, activations, the scratch pools and the CUDA context are not counted, so a checkpoint at 0.95x of the pool passes the refusal and still dies on the first forward. | A headroom fraction invented here would be exactly the guess the per-tensor bound exists to avoid. The number wants the startup memory profile that `KV-WARMUP-PROFILE` owns (`INVENTORIED`; upstream's is `GPUWorker.determine_available_memory`, `vllm/v1/worker/gpu_worker.py:451-495`, around `profile_run`, `vllm/v1/worker/gpu/model_runner.py:682`), which is a different row. Those two anchors are stated here from the pinned tree because that row's own three anchors are stale at the current pin, and `gguf_device_fit.h` had copied two of them — filed as [#1139](https://github.com/mudler/vllm.cpp/issues/1139), owned by `KV-WARMUP-PROFILE`, blocked here only by the `engine-matrix.md` record lock #1119 holds. | | **The fit bound also counts too MUCH, and that direction can refuse a load that fits.** A tensor present in the file and not staged by THIS load is a positive over-count. On a default load that is the MTP / `nextn` block: 8,940,488,704 bytes, 8.33 GiB, 2.2506 % of the target checkpoint. A budget in that window refuses a weight set that would have fitted. | Not closed here. Closing it means the bound taking a per-tensor staging POLICY as input, which is the caller's knowledge and not the file's, and the exclusion's own failure mode is an under-count to nothing — which restores the 26-minute-then-OOM this row exists to remove, on a device nobody here has to measure it on. So the direction is stated in `gguf_device_fit.h`, pinned executably by `test_gguf_device_fit`, exposed to operators in `docs/USAGE.md`, and tracked as [#1136](https://github.com/mudler/vllm.cpp/issues/1136). `VT_DEVICE_WEIGHT_BUDGET_BYTES` is the way out of the window in the meantime. | -| **`Backend::DeviceMemoryInfo` has no CUDA override, and waking it is not the one-line port of the ROCm one that #1126 describes: on CUDA it would wake a THROW.** Only ROCm implements it (`src/vt/rocm/rocm_backend.hip:338-345`), so `Gemma4MoE`'s device-expert LRU refuses on every CUDA device (`gemma4_moe.cpp:506`). Where a per-expert FP8 checkpoint is present, the expert's BF16 bytes are re-copied HOST->DEVICE on every use instead (`ExpertGeGLUHost`, `gemma4_moe.cpp:49-74`, reached at `:1493-1499`; the H2D is `:59-60` and it drains the queue per expert at `:73`), silently and for the life of the process. That H2D cost is CONDITIONAL, not present-tense: the whole device LRU is `ex.is_fp8`-gated (`:969`, `:1484`), and by point (1) below no such Gemma-4 checkpoint is pinned anywhere, so on CUDA today the fallback is never even asked for. The name misleads: it computes on the DEVICE from host-resident weights, so what the dead LRU would cost is bandwidth and a per-expert `Synchronize`, not a wrong answer. | The CAPABILITY is [#1126](https://github.com/mudler/vllm.cpp/issues/1126) and is still not built. The false COMMENT was corrected by this row in both places that carried it, `include/vt/backend.h:78-93` and `gemma4_moe.cpp:440-448`; both anchors were re-verified exact against `fd64c76ee`, as were `rocm_backend.hip:338-345`, `gemma4_moe.cpp:506`, `platforms/cuda.cpp:71` and `platforms/rocm.cpp:74`. **Four things were established while re-reading it for #1126, and each one raises the price of the override.** (1) *There is nothing to run it on.* The LRU needs `ex.is_fp8`, which only `LoadMoeFp8PerExpert` sets (`gemma4_weights.cpp:210-215`) from a per-expert `F8_E4M3` export; no such Gemma-4 checkpoint is pinned anywhere in `docs/USAGE.md` — all 32 lines there matching `gemma` case-insensitively were swept, and every checkpoint among them is the LTX-2.5 text tower (`gemma4-12b-with-proj*.safetensors`), not a per-expert FP8 MoE decoder, so the woken path cannot be exercised, here or elsewhere, until one is. (2) *The device-resident arm has no CUDA implementation to route into. It has a throw.* `ExpertGeGLUDeviceAccum` (`gemma4_moe.cpp:76-93`) READS as generic — `vt::MatmulBT`, `GeluAndMul`, `vt::MatmulBTAlphaBeta` — and an earlier draft of this row concluded from that reading that it "would run". It does not. `vt::MatmulBTAlphaBeta` (`src/vt/fused_ops.cpp:102`) is guarded on `#if defined(VLLM_CPP_HIP)` AND `q.device.type == kROCM`; its only implementation in the tree is `rocm::MatmulBTAlphaBetaRocm` (`src/vt/rocm/rocm_matmul_hipblaslt.hip:516`, declared `include/vt/rocm/rocm_matmul_batch.h:28`), and every other device falls through to a refusal. So the chain the override wakes is: `EnsureGemma4Fp8ExpertOnDevice` (`gemma4_moe.cpp:548-586`, carrying NO HIP guard) -> `lru.MakeRoom` (`:565`) succeeding as soon as `FreeBytes` can answer -> `true` at `:575` -> the call site at `:1486` -> `ExpertGeGLUDeviceAccum` at `:1487` -> `vt::MatmulBTAlphaBeta` at `:90` -> THROW, mid-decode. The `try`/`catch (...)` at `:563-585` wraps only the UPLOAD; the compute at `:1487` sits outside it, so the exception propagates out of the decode step rather than degrading to the host fallback. It is latent today only because the other route into that function, `same_dev` (`:730-731`), needs `ex.gate_up_dev`, which is assigned nowhere but `src/vt/rocm/rocm_gemma4_experts.hip:207,226` — so the resident arm is UNREACHABLE off ROCm rather than safe. This is the actual blocker under #1126, it was recorded nowhere, and it is a stronger argument than the other three: filed as [#1205](https://github.com/mudler/vllm.cpp/issues/1205), and the refusal itself is now gated by `tests/vt/test_gemma4_rocm_fp8_seams.cpp`, which is what a CUDA implementation will have to satisfy. **Two corrections to the earlier draft's supporting claims, both of which overstated the case.** *The HIP-only list was one symbol too long.* Three of the four are genuinely HIP-only stubs inside this file's ONLY `#ifndef VLLM_CPP_HIP` block (`:1574-1628`): `RunGemma4FusedTopkExpertGeGLU` (`:1599`), `PeerCopyGemma4Fp8ExpertSlice` (`:1607`) and `RunGemma4Fp8TopKOnExpertDevice` (`:1611`). `ExpertGeGLUDeviceBatched` (`:240`) is NOT: it sits OUTSIDE that block, in an anonymous namespace, with no HIP implementation and no header declaration, and is unconditionally `return false` under its own lab note (`:237-239` — gather+strided produced wrong tokens at ~23 t/s, pointer-batch ~0.8 t/s, serial/fused-gelu kept at ~34 t/s). It is disabled EVERYWHERE, ROCm included, so naming it beside the three inflated the ROCm/CUDA asymmetry. *The token-neutrality argument had the wrong mechanism.* The conclusion stands — the swap would not be token-neutral — but not because "the two arms sum the top-k experts in a different order". They do not: both run inside the SAME `for (int i = 0; i < top_k; ++i)` at `:1431`, so the summation order is identical. The real difference is ROUNDING and where the routing weight is applied. The resident arm folds `ww` into the GEMM `alpha` and accumulates in the epilogue with `beta` (`:1434`, `:1442`, `:90`), so the weight multiplies in the GEMM's own accumulator. The fallback writes the UNWEIGHTED product to a BF16 buffer (`:67`, `:70`) and applies `ww` afterwards with separate BF16 kernels — `vt::MulScalar` at `:1524` on the first expert, `MulScalar` plus `vt::Add` at `:1526-1527` on the rest. Two extra BF16 roundings per expert, with the weight applied post-rounding. (3) *The headroom test does not mean the same thing on the CUDA device this project gates on.* `MakeRoom` admits iff `free_b >= need + 1.5 GiB` (`:514`), a constant tuned on discrete dual R9700s where free VRAM is a quantity distinct from host RAM. On a GB10 it is not. This row already measured that `cudaMemGetInfo`'s `total` there is EXACTLY `/proc/meminfo MemTotal` times 1024, which is why it reached for that instrument where `nvidia-smi` answers `[N/A]`; its `free` is therefore reported over the same unified pool, and the host BF16 expert cache the device upload exists to relieve (`ex.cached_gu`/`cached_dn` via `EnsureGemma4Fp8ExpertCached`, bounded by the host LRU at `gemma4_moe.cpp:352`) is drawn from that pool too. So the admission test would double-count, on a box whose unified-memory OOM takes the host down with it. The `free` half is an inference from the measured `total`, not a second measurement, and it wants confirming on the device before any override lands. (4) *The gap is isolated, not a pattern.* Comparing overrides one by one across `src/vt/cuda/cuda_backend.cu` and `src/vt/rocm/rocm_backend.hip` at `fd64c76ee`, `DeviceMemoryInfo` is the ONLY optional `vt::Backend` seam ROCm answers and CUDA does not. | +| **`Backend::DeviceMemoryInfo` has no CUDA override, and waking it is not the one-line port of the ROCm one that #1126 describes: on CUDA it would wake a THROW.** Only ROCm implements it (`src/vt/rocm/rocm_backend.hip:338-345`), so `Gemma4MoE`'s device-expert LRU refuses on every CUDA device (`gemma4_moe.cpp:506`). Where a per-expert FP8 checkpoint is present, the expert's BF16 bytes are re-copied HOST->DEVICE on every use instead (`ExpertGeGLUHost`, `gemma4_moe.cpp:49-74`, reached at `:1493-1499`; the H2D is `:59-60` and it drains the queue per expert at `:73`), silently and for the life of the process. That H2D cost is CONDITIONAL, not present-tense: the whole device LRU is `ex.is_fp8`-gated (`:969`, `:1484`), and by point (1) below no such Gemma-4 checkpoint is pinned anywhere, so on CUDA today the LRU-MISS fallback is never even asked for. (A BF16 Gemma-4 checkpoint reaches `ExpertGeGLUHost` at `:1503` too, but through the host-weight branch the LRU never governs, so it is not a cost of the missing probe.) The name misleads: it computes on the DEVICE from host-resident weights, so what the dead LRU would cost is bandwidth and a per-expert `Synchronize`, not a wrong answer. | The CAPABILITY is [#1126](https://github.com/mudler/vllm.cpp/issues/1126) and is still not built. The false COMMENT was corrected by this row in both places that carried it, `include/vt/backend.h:78-93` and `gemma4_moe.cpp:440-448`; both anchors were re-verified exact against `fd64c76ee`, as were `rocm_backend.hip:338-345`, `gemma4_moe.cpp:506`, `platforms/cuda.cpp:71` and `platforms/rocm.cpp:74`. The review repair below re-audited every anchor these three rows cite against the repaired tree: 53 examined, 53 exact, 0 stale. **Four things were established while re-reading it for #1126, and each one raises the price of the override.** (1) *There is nothing to run it on.* The LRU needs `ex.is_fp8`, which only `LoadMoeFp8PerExpert` sets (`gemma4_weights.cpp:210-215`) from a per-expert `F8_E4M3` export; no such Gemma-4 checkpoint is pinned anywhere in `docs/USAGE.md` — all 32 lines there matching `gemma` case-insensitively were swept, and every checkpoint among them is the LTX-2.5 text tower (`gemma4-12b-with-proj*.safetensors`), not a per-expert FP8 MoE decoder, so the woken path cannot be exercised, here or elsewhere, until one is. (2) *The device-resident arm has no CUDA implementation to route into. It has a throw.* `ExpertGeGLUDeviceAccum` (`gemma4_moe.cpp:76-93`) READS as generic — `vt::MatmulBT`, `GeluAndMul`, `vt::MatmulBTAlphaBeta` — and an earlier draft of this row concluded from that reading that it "would run". It does not. `vt::MatmulBTAlphaBeta` (`src/vt/fused_ops.cpp:102`) is guarded on `#if defined(VLLM_CPP_HIP)` AND `q.device.type == kROCM`; its only implementation in the tree is `rocm::MatmulBTAlphaBetaRocm` (`src/vt/rocm/rocm_matmul_hipblaslt.hip:516`, declared `include/vt/rocm/rocm_matmul_batch.h:28`), and every other device falls through to a refusal. So the chain the override wakes is: `EnsureGemma4Fp8ExpertOnDevice` (`gemma4_moe.cpp:548-586`, carrying NO HIP guard) -> `lru.MakeRoom` (`:565`) succeeding as soon as `FreeBytes` can answer -> `true` at `:575` -> the call site at `:1486` -> `ExpertGeGLUDeviceAccum` at `:1487` -> `vt::MatmulBTAlphaBeta` at `:90` -> THROW, mid-decode. The `try`/`catch (...)` at `:563-585` wraps only the UPLOAD; the compute at `:1487` sits outside it, so the exception propagates out of the decode step rather than degrading to the host fallback. It is latent today only because the other route into that function, `same_dev` (`:730-731`), needs `ex.gate_up_dev`, which is assigned nowhere but `src/vt/rocm/rocm_gemma4_experts.hip:207,226` — so the resident arm is UNREACHABLE off ROCm rather than safe. This is the actual blocker under #1126, it was recorded nowhere, and it is a stronger argument than the other three: filed as [#1205](https://github.com/mudler/vllm.cpp/issues/1205), and the refusal itself is now gated by `tests/vt/test_gemma4_rocm_fp8_seams.cpp`, which is what a CUDA implementation will have to satisfy. **Two corrections to the earlier draft's supporting claims, both of which overstated the case.** *The HIP-only list was one symbol too long.* Three of the four are genuinely HIP-only stubs inside this file's ONLY `#ifndef VLLM_CPP_HIP` block (`:1574-1628`): `RunGemma4FusedTopkExpertGeGLU` (`:1599`), `PeerCopyGemma4Fp8ExpertSlice` (`:1607`) and `RunGemma4Fp8TopKOnExpertDevice` (`:1611`). `ExpertGeGLUDeviceBatched` (`:240`) is NOT: it sits OUTSIDE that block, in an anonymous namespace, with no HIP implementation and no header declaration, and is unconditionally `return false` under its own lab note (`:237-239` — gather+strided produced wrong tokens at ~23 t/s, pointer-batch ~0.8 t/s, serial/fused-gelu kept at ~34 t/s). It is disabled EVERYWHERE, ROCm included, so naming it beside the three inflated the ROCm/CUDA asymmetry. *The token-neutrality argument had the wrong mechanism.* The conclusion stands — the swap would not be token-neutral — but not because "the two arms sum the top-k experts in a different order". They do not: both run inside the SAME `for (int i = 0; i < top_k; ++i)` at `:1431`, so the summation order is identical. The real difference is ROUNDING and where the routing weight is applied. The resident arm folds `ww` into the GEMM `alpha` and accumulates in the epilogue with `beta` (`:1434`, `:1442`, `:90`), so the weight multiplies in the GEMM's own accumulator. The fallback writes the UNWEIGHTED product to a BF16 buffer (`:67`, `:70`) and applies `ww` afterwards with separate BF16 kernels — `vt::MulScalar` at `:1524` on the first expert, `MulScalar` plus `vt::Add` at `:1526-1527` on the rest. Two extra BF16 roundings per expert, with the weight applied post-rounding. (3) *The headroom test does not mean the same thing on the CUDA device this project gates on.* `MakeRoom` admits iff `free_b >= need + 1.5 GiB` (`:514`), a constant tuned on discrete dual R9700s where free VRAM is a quantity distinct from host RAM. On a GB10 it is not. This row already measured that `cudaMemGetInfo`'s `total` there is EXACTLY `/proc/meminfo MemTotal` times 1024, which is why it reached for that instrument where `nvidia-smi` answers `[N/A]`; its `free` is therefore reported over the same unified pool, and the host BF16 expert cache the device upload exists to relieve (`ex.cached_gu`/`cached_dn` via `EnsureGemma4Fp8ExpertCached`, bounded by the host LRU at `gemma4_moe.cpp:352`) is drawn from that pool too. So the admission test would double-count, on a box whose unified-memory OOM takes the host down with it. The `free` half is an inference from the measured `total`, not a second measurement, and it wants confirming on the device before any override lands. (4) *The gap is isolated, not a pattern.* Comparing overrides one by one across `src/vt/cuda/cuda_backend.cu` and `src/vt/rocm/rocm_backend.hip` at `fd64c76ee`, `DeviceMemoryInfo` is the ONLY optional `vt::Backend` seam ROCm answers and CUDA does not. | | **#1126's own closing plan, step 3, must be NARROWED before it is done: the load-time fit check may read the seam's `total`, and never its `free`.** The issue proposes that "the #1123 fit check can then read the budget from the backend seam on every platform that reports one". As written that invites the live half, which would be a defect — but the seam returns BOTH halves (`bool DeviceMemoryInfo(size_t* free_bytes, size_t* total_bytes)`, `include/vt/backend.h:94`), and only one of them is illegitimate here. | The tree holds two answers to "how much device memory", and they answer different questions. `vt::Backend::DeviceMemoryInfo(free, total)` is a LIVE probe that moves with contention; it is authoritative for a RUNTIME admission decision — can this allocation succeed right now — and for nothing else. `vllm::platforms::ResidencyPolicy::device_memory_total_bytes` is a TOTAL probed once at platform registration with `0 == UNKNOWN`; it is authoritative for a LOAD-TIME budget verdict, which has to be reproducible and independent of whatever else the box is doing. Sourcing the load-time verdict from `free` would make the same checkpoint load or be refused depending on the page cache, which is precisely the property **`total`, not `free`** above was chosen to avoid. Reading `total` through the seam is not that. It is contention-independent, it is the same quantity `ResidencyPolicy` already carries, and a seam that answers it on every platform is a defensible place to source it from. What step 3 must additionally preserve is the PROBE-ONCE semantics: `device_memory_total_bytes` is probed at platform registration (`include/vllm/platforms/interface.h:70-72`), and a per-load live call would reintroduce the contention dependence by the back door even reading only `total`, because a load-time verdict has to be reproducible from the record rather than from the moment. So the defensible statement, and the one this row asserts: **step 3 may read only `total`, never `free`, and must keep the value probed once at platform registration.** An earlier draft of this row said "never step 3" outright; that overstated it and would have blocked a legitimate simplification, so it is corrected here rather than quietly narrowed. Both seams already carry the division in prose (`include/vt/backend.h:90-93` and `include/vllm/platforms/interface.h:61-69`); it is restated here because #1126 is the record a reader of that issue will act on, and as filed it points the other way. What #1126 owes is its steps 1 and 2 together — the override AND the Gemma4 measurement, with [#1205](https://github.com/mudler/vllm.cpp/issues/1205) ahead of both — never step 1 alone, and step 3 only in the narrowed form above. | | **The device-expert LRU's slot cap makes its own eviction opt-in inert.** `MakeRoom` tests `slots.size() >= kMaxSlots` (`gemma4_moe.cpp:498`) BEFORE the eviction loop (`:499-500`), and `EvictOne` (`:457`, the device LRU's — a host-cache namesake sits at `:275`) is the only thing that SHRINKS `slots`. The one other statement that touches its size, `slots.clear()` in `DevExpertLru::Note` (`:522`), is a device-index RESET rather than an eviction: it drops bookkeeping when `dev != d.q.device.index` and frees nothing, and it is unreachable in a single-device process. It is named here so the next reader does not conclude the #1197 sweep missed it. So once 24 slots are resident `VT_GEMMA4_EXPERT_EVICT=1` never runs again and the cache degrades permanently to fill-only. It binds only when `24 * expert_bytes < BudgetBytes()`, so it is condition-dependent and silent either way. Tracked as [#1197](https://github.com/mudler/vllm.cpp/issues/1197). | Filed, not fixed, and for the same reason as the row above rather than for effort: the one-line repair wakes more `hipFree` under load, which the surrounding comments say has been observed as a permanent `kfd_wait` hang with the GPU idle and no decode tokens. The current ordering may well be deliberate belt-and-braces. Deciding that needs the dual-RDNA4 box `.agents/specs/gemma4-rocm-fp8-moe.md` describes; this host has neither a ROCm nor a CUDA device. It closes when the cap moves after the eviction loop and a run stays hang-free, or when the comment says the cap is by design — one of the two, not silence. | -| **`vt::MatmulBTAlphaBeta` is ROCm-only and has no CUDA implementation at all, which is what #1126 step 1 is actually blocked on.** `src/vt/fused_ops.cpp:102` dispatches to `rocm::MatmulBTAlphaBetaRocm` (`src/vt/rocm/rocm_matmul_hipblaslt.hip:516`) under `#if defined(VLLM_CPP_HIP)` and `q.device.type == kROCM`, and refuses on every other device. There is no CUDA, Vulkan, Metal or CPU arm. The full chain from the missing `DeviceMemoryInfo` override to that refusal is traced in the first row above. Tracked as [#1205](https://github.com/mudler/vllm.cpp/issues/1205). | The REFUSAL is fixed in flow, because a bare `std::runtime_error` reading "ROCm-only in this build" does not satisfy the standing rule that an unimplemented arm refuses with a message NAMING the missing part: a caller who hits it on CUDA cannot tell a missing kernel from a missing build flag. It now names the device that asked, names where the implementation lives, and names the issue, and `tests/vt/test_gemma4_rocm_fp8_seams.cpp` gates it on a posed CUDA queue plus three other non-ROCm devices — mutation-proven by restoring the old message (RED) and by deleting the refusal outright (RED). Pinning the precondition executably is also the answer to "should the absence be recorded": the gate says what is missing without any checker that says do not fix it, and when someone writes the CUDA arm the same test is what it has to satisfy. The IMPLEMENTATION stays owed and is what [#1205](https://github.com/mudler/vllm.cpp/issues/1205) tracks. It is not written here: a `beta`-accumulating BT GEMM on cuBLASLt is a kernel with its own correctness gate, this row's point (1) says there is no checkpoint to exercise it on, and this host has neither a ROCm nor a CUDA device to measure either arm. | +| **`vt::MatmulBTAlphaBeta` is ROCm-only and has no CUDA implementation at all, which is what #1126 step 1 is actually blocked on.** `src/vt/fused_ops.cpp:102` dispatches to `rocm::MatmulBTAlphaBetaRocm` (`src/vt/rocm/rocm_matmul_hipblaslt.hip:516`) under `#if defined(VLLM_CPP_HIP)` and `q.device.type == kROCM`, and refuses on every other device. There is no CUDA, Vulkan, Metal or CPU arm. The full chain from the missing `DeviceMemoryInfo` override to that refusal is traced in the first row above. Tracked as [#1205](https://github.com/mudler/vllm.cpp/issues/1205). | The REFUSAL is fixed in flow, because a bare `std::runtime_error` reading "ROCm-only in this build" does not satisfy the standing rule that an unimplemented arm refuses with a message NAMING the missing part: a caller who hits it on CUDA cannot tell a missing kernel from a missing build flag. It now names the device that asked, names where the implementation lives, and names the issue, and `tests/vt/test_gemma4_rocm_fp8_seams.cpp` gates it on a posed CUDA queue plus three other non-ROCm devices — mutation-proven by restoring the old message (RED) and by deleting the refusal outright (RED). Pinning the precondition executably is also the answer to "should the absence be recorded": the gate says what is missing without any checker that says do not fix it, and when someone writes the CUDA arm the same test is what it has to satisfy. The IMPLEMENTATION stays owed and is what [#1205](https://github.com/mudler/vllm.cpp/issues/1205) tracks. It is not written here: a `beta`-accumulating BT GEMM on cuBLASLt is a kernel with its own correctness gate, the `DeviceMemoryInfo` row's point (1) above says there is no checkpoint to exercise it on, and this host has neither a ROCm nor a CUDA device to measure either arm. | | **`model_loader.cpp` is cited by absolute line number from 109 sites in 45 files, and this row's change moved them.** Measured between `e7d0a1f7c` and the repaired head: 203 moved line references over 109 citing sites, 10 unmoved. The file is ~1640 lines and almost every engine and model row edits it, so any edit near its top invalidates citations in files the editing change never opens. | Not swept here, deliberately, and the reason is not effort: several of the 109 were ALREADY stale (`model-matrix.md:197` cites `:184-223` as the "live loader"; line 184 at `e7d0a1f7c` is `static const bool once = [] {`), and rewriting all of them from the current tree would launder pre-existing debt into a clean-looking record. What IS fixed here is the two anchors this change authored itself, checked against the final tree. Tracked as [#1143](https://github.com/mudler/vllm.cpp/issues/1143), which lists the three candidate fixes; it needs a row of its own and is parked here because this row is what measured it. | | **The budget knob is an environment variable, not a config key.** `VT_DEVICE_WEIGHT_BUDGET_BYTES`. | `ENG-RESIDENCY-CONFIG` ([#1110](https://github.com/mudler/vllm.cpp/issues/1110), PR #1119) is in flight and adds exactly the `vllm_cpp` namespace inside `--offload-config` this key belongs in. Landing a second, competing config surface while that one is unmerged would create the conflict both changes then have to resolve. Migrate once #1119 lands; tracked as [#1127](https://github.com/mudler/vllm.cpp/issues/1127). | From 85a9c6e5fece6a9dba9ef5f95b7178df1e1831cc Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Tue, 18 Aug 2026 09:51:19 +0000 Subject: [PATCH 4/8] fix(ENG-EXPERT-STREAM): refuse the Gemma-4 device-expert upload before it happens, and stop telling a ROCm caller the kernel is missing (#1126, #1205, #1197, #1218) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit FOLLOWING_AGENTS_PROTOCOL The review of #1200 returned FAIL on three findings and recommended a fourth change. This is the repair, and the recommended change is the substance of it. THE GUARD. #1205 named a cheaper and reachable fix that the first pass did not take: refuse at the `gemma4_moe.cpp` call site BEFORE the upload rather than throwing after it. `EnsureGemma4Fp8ExpertOnDevice` now returns false at `gemma4_moe.cpp:571` when `vt::HasMatmulBTAlphaBeta(d.q)` is false, so the caller takes the host fallback already sitting in the `else` at `:1515-1521` and the decode step answers instead of throwing mid-decode. That matters more than a better message: the reviewer's M3 deleted the production call site at `gemma4_moe.cpp:90` and the old focused gate stayed fully green, because it constructed a `vt::Queue` by hand and so measured the class rather than the capability. The predicate is not a device-name test and not a build-macro test. `vt::MatmulBTAlphaBeta` dispatches on `HasMatmulBTAlphaBeta` itself (`src/vt/fused_ops.cpp:102-109`, dispatch at `:117`), so the two cannot disagree, and writing a CUDA arm wakes the device path with no edit at the call site. `tests/vllm/models/test_gemma4_moe_device_arm_guard.cpp` gates it through `vllm::RunGemma4Moe` — the layer entry `gemma4.cpp:634` calls — with a real `Gemma4MoeLayerWeights`, the real router, the real LRU and the real fallback. It decorates the registered CPU backend so `vt::Backend::DeviceMemoryInfo` ANSWERS, because that is the post-#1126 state and the only state in which the guard binds at all: on a stock build `MakeRoom` refuses on the absent probe, and deleting the guard would change nothing observable. Both directions of the probe are asserted before anything is concluded, so an instrument that failed to arm cannot report the guard proven by a run that never reached it. Deleting the guard turns that test RED with the exact `no implementation for device 'cpu'` throw; forcing the predicate to `true` turns it RED too. F1. The new refusal was wrong for the one case the old one got right. A kROCM queue in a build without `-DVLLM_CPP_HIP` falls through the same `#if`, and the message told it a kernel was missing and pointed it at a CUDA issue. It now gets its own message naming the absent BUILD FLAG, and the seams test poses kROCM alongside the four devices that made the old message look right. Say plainly what that test pins: a contract in a unit test, not observable behaviour. With the guard in place no production run off ROCm reaches either string. F2, F4, F6. #1205's issue body anchors matched `0acf2ca1d` and were eleven lines stale at `7beada17c`; this change moves them again by twenty-two. Every anchor in the cluster was re-derived against the final tree — 168 endpoints across nine rows, none out of range. `fused_ops.cpp:102` was the signature and the dispatch is at `:117`. The audit headline now states its SCOPE, because "53 examined, 53 exact, 0 stale" covered four spec rows and never included an issue body, which is exactly how the body went stale unnoticed. Several bare `:NNN` anchors also inherited the wrong file after a mid-sentence mention of a different one; those are spelled out. #1218 is filed and deliberately NOT fixed here. `EnsureGemma4Fp8NativeOnDevice` is the DEFAULT arm and carries the same shape, but it depends on three different ops and needs a predicate per op; reusing `HasMatmulBTAlphaBeta` there would be a guard naming the wrong arm, which is the exact defect this review just corrected in a refusal message. It is owned by `ENG-EXPERT-STREAM` and listed under `## Owed`. Two `.agents/issue-index.md` rows are EDITED rather than appended. Both were added by this branch and have not merged, so no other branch can hold a different version of them, and the append-only rule protects merged rows against a duplicating union merge. Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5-1m [Claude Code] --- .agents/issue-index.md | 3 +- .agents/specs/expert-streaming.md | 6 +- include/vt/fused_ops.h | 13 + src/vllm/model_executor/models/gemma4_moe.cpp | 22 ++ src/vt/fused_ops.cpp | 48 ++- tests/CMakeLists.txt | 7 + .../test_gemma4_moe_device_arm_guard.cpp | 301 ++++++++++++++++++ tests/vt/test_gemma4_rocm_fp8_seams.cpp | 75 ++++- 8 files changed, 458 insertions(+), 17 deletions(-) create mode 100644 tests/vllm/models/test_gemma4_moe_device_arm_guard.cpp diff --git a/.agents/issue-index.md b/.agents/issue-index.md index 6b2e74f65..8baf5befc 100644 --- a/.agents/issue-index.md +++ b/.agents/issue-index.md @@ -367,5 +367,6 @@ rather than merged. `scripts/check-agent-record.py` gates both. | [#1181](https://github.com/mudler/vllm.cpp/issues/1181) | `FIX-READ-F32-SCALAR-GUARD` | `ReadF32Scalar` (`src/vllm/model_executor/models/qwen3_5_weights.cpp:312-318` @ `ab6e65216`) bounds its input with `t.data != nullptr && t.nbytes >= sizeof(float)`, a LOWER bound, and then `memcpy`s four bytes into a `float`. Two silent wrong-value paths follow and neither fails: an ARRAY is reduced to element 0, so a block-wise FP8 scale grid of shape `[ceil(N/128), ceil(K/128)]` passes and stands in for the whole weight (measured under [#1166](https://github.com/mudler/vllm.cpp/issues/1166) on `Qwen/Qwen3.8-27B-FP8` @ `017b9c7af6b5689d5dd426a76e0bc077eb5ca20a`, `q_proj.weight_scale_inv` is `[96, 40]`), and ANY dtype is reinterpreted, since that same tensor is `BF16` and its four bytes are two bf16 values read as one float. Both return a finite plausible float, so the output is fluent, plausible and wrong, which is what a token gate cannot see. Upstream makes both facts structural rather than optional: a per-tensor scale is a distinct parameter TYPE that asserts `loaded_weight.shape[0] == 1` (`vllm/model_executor/parameter.py:260-272,304-309` @ `555967922`, plus the `_assert_and_load` shape assert at `:93-96`), the slot is allocated `torch.float32` so a narrow on-disk dtype is VALUE-converted rather than reinterpreted (`utils/fp8_utils.py:1276`), and the declared strategy TENSOR/CHANNEL/BLOCK picks the parameter type before a byte is read (`compressed_tensors/schemes/compressed_tensors_w8a8_fp8.py:63,128`). The AUDIT corrects the issue's own framing twice. The 27 grep hits across five files are 5 definitions, 20 call sites and 2 comment references, and both counts are short: `ReadCtF32Scalar` (`include/vllm/model_executor/models/dense_weight_loaders.h:376`) is a SIXTH copy of the same defect under another name, reached from a SIXTH model file (`src/vllm/model_executor/models/qwen3_weights.cpp:100,126-128` through `LoadCtNvfp4W4A16`). Of the six, three check nothing, `LnReadF32Scalar`/`ShReadF32Scalar` check dtype but not count, and only `nemotron_h_weights.cpp:557-573` is correct, which makes it the model the shared guard generalizes. No call site legitimately passes a multi-element or non-F32 tensor, and every existing fixture emits rank-0 or `{1}` `F32`, so nothing in the tree needed the leniency. It is NOT merely latent: `dense_weight_loaders.h:73-74` and `docs/BENCHMARKS.md:52` both record `unsloth/Qwen3.6-27B-NVFP4` @ `ccdaab7e` as FP8 W8A8 throughout with BF16 PER-OUTPUT-CHANNEL scales, and `LoadAttnDense` branches on the weight dtype alone (`qwen3_5_dense_weights.cpp:478-480`), so those projections enter the per-tensor arm and hit both defects at once under the tensor name the loader actually asked for, with no misspelling to stop them. Fixed in flow by one `dense_loaders::ReadF32Scalar(get, name)` that refuses `numel != 1` naming the shape, refuses a non-`F32` dtype naming the dtype, and requires exactly four readable bytes, with the other five copies deleted onto it and `nemotron_h`'s `Loader`-based twin kept as the one tracked exception. A narrow dtype is refused rather than converted, because a one-element BF16 scale has never been read correctly here and the BF16 layout that IS shipped is per-channel, which the count check refuses first. Per-channel FP8, block-wise FP8 and any explicit narrow-dtype conversion stay owed. Spec [`read-f32-scalar-guard.md`](specs/read-f32-scalar-guard.md) | bug | | [#1185](https://github.com/mudler/vllm.cpp/issues/1185) | `ENV-ORACLE-WHEEL-IN-LEASE` | The pinned vLLM oracle BUILDS, installs, imports and sees the GPU inside an `rc` lease on `dgx:gpu0`, measured 2026-08-18, which falsifies the `nvcc` clause four records carried. [`lease-runtime-staging.md`](specs/lease-runtime-staging.md) said the oracle "needs `nvcc`, which the worker still lacks", and `.agents/environment.md`, [`mtp-k-gt-1.md`](specs/mtp-k-gt-1.md) and [`gpu-lease-methodology.md`](specs/gpu-lease-methodology.md) each derived a blocker from it. The build job (`buildvllm.sh`, staged sha256 prefix `15e140d41f44e7c2`) asserted the checkout against the pin BEFORE compiling, printing `PIN CONFIRMED` at `5559679229bc961848b121ccdeaa8fa5d79bec98` and aborting otherwise, took `nvcc` from the toolkit row `MODEL-NEMOTRON-H-ABI-A3-E2E` staged (`NVCC_RC=0`, CUDA `release 13.3, V13.3.73`) and produced `WHEEL_RC=0`, `PERSIST_RC=0` and a 434 MiB `vllm-0.1.dev1+g555967922.cu133-cp312-cp312-linux_aarch64.whl`, sha256 `7c58b339741a288fbb313f4f5196c9c92a9e3b3c3ebe2ea970b0ff50bb9bcba4`. The identity job (`oracleenv.sh`, prefix `6119f5223f5d818c`) asserted from `cd /`, outside any source tree: `vllm.__version__ = 0.1.dev1+g555967922`, `IDENTITY_RC=0`, `cuda True NVIDIA GB10`, `CUDA_RC=0`. SCOPE, and it carries the same weight as the result: RUNNING A MODEL IS UNTESTED. Only build, install, import and `torch.cuda.is_available()` are measured, and [`mtp-k-gt-1.md`](specs/mtp-k-gt-1.md) records that the last time an oracle reached this far it consumed the host in the step AFTER `torch.compile` and REBOOTED the box, at `gpu_memory_utilization` 0.75 and again at 0.30, so the fraction is not the lever. The version string is an OPEN discrepancy: `.agents/upstream-sync.md` records `vllm_runtime_version = 0.23.1rc1.dev1511+g555967922`, the commit segment matches and satisfies the pin's binding `+g` rule, and the prefix differs because a shallow fetch stops `setuptools_scm` counting commits since the last tag, so a full-string gate needs a deeper fetch or a recorded pretend-version. The venv is NOT staged, because that job was killed at a 90-minute ceiling mid-copy and its partial tree was removed, so only the WHEEL is durable. Four staging walls, all artifacts of the NAS rather than of CUDA: `cp -a` preserves `file_mode=0664` so `nvcc` exited 126. CIFS `nounix` stores no symlink so `include` and `lib64` vanished and CMake reported `Could NOT find CUDA (missing: CUDA_INCLUDE_DIRS CUDA_CUDART_LIBRARY) (found version "13.3")`, naming the version and denying the toolkit in one line. 32 library links `libfoo.so` and `libfoo.so.MAJOR` had to be rebuilt because only the `libfoo.so.X.Y.Z` real files survived. And `markupsafe` existed as a dist-info with NO package files from a `pip --target` killed at a 35-minute ceiling, so Marlin codegen died on `ModuleNotFoundError`. The `rc` worker container is REUSED between jobs, so a repair inside a staging branch is skipped on the next run (`nvcc already in place`) and an environment repair must be unconditional and assert its postcondition. CONSEQUENCE for the rows #1129 blocked, [#1003](https://github.com/mudler/vllm.cpp/issues/1003), [#915](https://github.com/mudler/vllm.cpp/issues/915), [#821](https://github.com/mudler/vllm.cpp/issues/821) and [#81](https://github.com/mudler/vllm.cpp/issues/81): UNBLOCKED FOR THE BUILD STEP and STILL BLOCKED FOR A MODEL RUN. None can take a measurement until a model run is demonstrated. Job details, walls and non-claims in [`oracle-wheel-in-lease.md`](specs/oracle-wheel-in-lease.md) | verification | | [#1197](https://github.com/mudler/vllm.cpp/issues/1197) | `ENG-EXPERT-STREAM` | `Gemma4MoE`'s device-expert LRU tests its slot cap BEFORE its eviction loop, so the eviction opt-in goes inert once the cap is reached. `DevExpertLru::MakeRoom` runs `if (slots.size() >= kMaxSlots) return false;` at `src/vllm/model_executor/models/gemma4_moe.cpp:498` @ `fd64c76ee`, two lines ahead of the `if (allow_evict) { while (used + need > bud && !slots.empty()) EvictOne(d); }` at `:499-500`, and `EvictOne` (`:457`, the DEVICE LRU's — the file carries a host-cache namesake at `:275`) is the only thing that shrinks `slots`. So after 24 admissions every later `MakeRoom` returns false at that first line, the eviction loop is never reached again, and `VT_GEMMA4_EXPERT_EVICT=1` becomes a no-op for the life of the process — the cache degrades permanently to the fill-only mode the opt-in exists to leave. It binds only when the slot cap is reached before the byte budget, i.e. when `24 * expert_bytes < BudgetBytes()` (below ~85.3 MiB per expert at the 2048 MiB default from `BudgetBytes`, `:416-436`, again the device one and not the host cache's at `:262`); above that the byte budget binds first and eviction behaves. Nothing reports which one happened. FILED, NOT FIXED, and not for effort: the one-line repair (move the cap test after the eviction loop, so it caps RESIDENT slots instead of stopping admission forever) wakes more `hipFree` under load, which the code's own comments call a permanent `kfd_wait` hang with the GPU idle, prefill done and no decode tokens (`:459-461` and `:486-488`), so the current ordering may be deliberate. Deciding it needs the dual-RDNA4 lab box of [`gemma4-rocm-fp8-moe.md`](specs/gemma4-rocm-fp8-moe.md); the host that found it has neither a ROCm nor a CUDA device. Found while establishing the facts for [#1126](https://github.com/mudler/vllm.cpp/issues/1126), which required reading `MakeRoom` line by line. Listed under `## Owed` in [`expert-streaming.md`](specs/expert-streaming.md) | bug | -| [#1205](https://github.com/mudler/vllm.cpp/issues/1205) | `ENG-EXPERT-STREAM` | `vt::MatmulBTAlphaBeta` has NO CUDA implementation, and that — not any of the four reasons [#1200](https://github.com/mudler/vllm.cpp/pull/1200) first recorded — is what [#1126](https://github.com/mudler/vllm.cpp/issues/1126) step 1 is blocked on. `src/vt/fused_ops.cpp:102` dispatches to `rocm::MatmulBTAlphaBetaRocm` (`src/vt/rocm/rocm_matmul_hipblaslt.hip:516`, declared `include/vt/rocm/rocm_matmul_batch.h:28`) under `#if defined(VLLM_CPP_HIP)` AND `q.device.type == kROCM`, and every other device falls through to a throw. There is no CUDA, Vulkan, Metal or CPU arm: this is a missing kernel, not a missing build flag. #1126 proposes adding `CudaBackend::DeviceMemoryInfo` via `cudaMemGetInfo` so `Gemma4MoE`'s device-expert LRU stops refusing on CUDA; the record said that would wake a slower third path, and it would in fact wake a THROW, mid-decode. The chain: `EnsureGemma4Fp8ExpertOnDevice` (`src/vllm/model_executor/models/gemma4_moe.cpp:548-586`, no HIP guard) -> `lru.MakeRoom` at `:565` succeeding once `FreeBytes` can answer -> `true` at `:575` -> the call site at `:1486` -> `ExpertGeGLUDeviceAccum` at `:1487` -> `vt::MatmulBTAlphaBeta` at `:90` -> throw. The `try`/`catch (...)` at `:563-585` wraps only the UPLOAD; the compute at `:1487` is outside it, so the exception leaves the decode step rather than degrading to the host fallback. Latent today only because the other route in, `same_dev` (`:730-731`), needs `ex.gate_up_dev`, assigned nowhere but `src/vt/rocm/rocm_gemma4_experts.hip:207,226` — so the resident arm is UNREACHABLE off ROCm rather than safe. FIXED IN FLOW, PARTLY: the REFUSAL now satisfies the rule that an unimplemented arm names the missing part. "ROCm-only in this build" named neither the device that asked nor where an implementation would go, so a caller could not tell a missing kernel from a missing build flag; it now names all three plus this issue, and `tests/vt/test_gemma4_rocm_fp8_seams.cpp` gates it on a posed CUDA queue and on `kCPU`, `kVULKAN` and `kMETAL`, mutation-proven RED by restoring the old message and RED again by deleting the refusal outright. The IMPLEMENTATION stays open and is what this issue tracks: a `beta`-accumulating BT GEMM on cuBLASLt is a kernel with its own correctness gate, no per-expert FP8 Gemma-4 checkpoint is pinned to exercise it on, and the host that found this has neither a ROCm nor a CUDA device. The same file refuses six further arms the same way (`MatmulBTFp8Channel`, `DequantFp8ChannelBf16`, `MoeGatherRows`, `MoeWeightedScatterAdd`, `MoeZeroBf16`, `DualRmsNormPlusRes`); only `MatmulBTAlphaBeta` is on the #1126 path, and the others are named so the next reader need not re-derive the list. Listed under `## Owed` in [`expert-streaming.md`](specs/expert-streaming.md) | bug | +| [#1205](https://github.com/mudler/vllm.cpp/issues/1205) | `ENG-EXPERT-STREAM` | `vt::MatmulBTAlphaBeta` has NO CUDA implementation, and that — not any of the four reasons [#1200](https://github.com/mudler/vllm.cpp/pull/1200) first recorded — is what [#1126](https://github.com/mudler/vllm.cpp/issues/1126) step 1 is blocked on. `src/vt/fused_ops.cpp:117` dispatches to `rocm::MatmulBTAlphaBetaRocm` (`src/vt/rocm/rocm_matmul_hipblaslt.hip:516`, declared `include/vt/rocm/rocm_matmul_batch.h:28`) under `#if defined(VLLM_CPP_HIP)` AND `q.device.type == kROCM` — `src/vt/fused_ops.cpp:111-112` is the signature, not the dispatch — and every other device falls through to a throw (`src/vt/fused_ops.cpp:152`). There is no CUDA, Vulkan, Metal or CPU arm: this is a missing kernel, not a missing build flag. #1126 proposes adding `CudaBackend::DeviceMemoryInfo` via `cudaMemGetInfo` so `Gemma4MoE`'s device-expert LRU stops refusing on CUDA; the record said that would wake a slower third path, and it would in fact wake a THROW, mid-decode. The chain: `EnsureGemma4Fp8ExpertOnDevice` (`src/vllm/model_executor/models/gemma4_moe.cpp:548-608`) -> `lru.MakeRoom` at `:587` succeeding once `FreeBytes` can answer -> `true` at `:597` -> the call site at `:1508` -> `ExpertGeGLUDeviceAccum` at `:1509` -> `vt::MatmulBTAlphaBeta` at `:90` -> throw. The `try`/`catch (...)` at `:585-607` wraps only the UPLOAD; the compute at `:1509` is outside it, so the exception leaves the decode step rather than degrading to the host fallback. Latent today only because the other route in, `same_dev` (`:752-753`), needs `ex.gate_up_dev`, assigned nowhere but `src/vt/rocm/rocm_gemma4_experts.hip:207,226` — so the resident arm is UNREACHABLE off ROCm rather than safe. FIXED IN FLOW, PARTLY. (a) The HAZARD is guarded: `EnsureGemma4Fp8ExpertOnDevice` refuses at `src/vllm/model_executor/models/gemma4_moe.cpp:571` when `vt::HasMatmulBTAlphaBeta(d.q)` is false, BEFORE the upload, so the caller takes the host fallback already sitting in the `else` at `:1515-1521` and the decode step answers instead of throwing. The predicate (`src/vt/fused_ops.cpp:102-109`) is the same condition the dispatch at `:117` uses, so the two cannot drift. Gated by `tests/vllm/models/test_gemma4_moe_device_arm_guard.cpp`, which enters through `vllm::RunGemma4Moe` and decorates the registered CPU backend so `DeviceMemoryInfo` answers — the post-#1126 state, and the only one in which the guard binds; deleting the guard turns it RED. (b) The REFUSAL now satisfies the rule that an unimplemented arm names the missing part. "ROCm-only in this build" named neither the device that asked nor where an implementation would go; a non-ROCm queue now gets all three plus this issue (`src/vt/fused_ops.cpp:152`), and a kROCM queue in a build without `-DVLLM_CPP_HIP` gets a different message naming the absent FLAG (`src/vt/fused_ops.cpp:138`), because for that caller the kernel exists. `tests/vt/test_gemma4_rocm_fp8_seams.cpp` gates both on posed queues — CUDA, `kCPU`, `kVULKAN`, `kMETAL` and kROCM — mutation-proven RED by restoring the old message, RED by deleting the refusal outright, and RED by deleting the kROCM branch. That message gate is a UNIT CONTRACT, not observable behaviour: with the guard in place the throw is unreachable off ROCm, which is the point of the guard. The IMPLEMENTATION stays open and is what this issue tracks: a `beta`-accumulating BT GEMM on cuBLASLt is a kernel with its own correctness gate, no per-expert FP8 Gemma-4 checkpoint is pinned to exercise it on, and the host that found this has neither a ROCm nor a CUDA device. The same file refuses six further arms the same way (`MatmulBTFp8Channel`, `DequantFp8ChannelBf16`, `MoeGatherRows`, `MoeWeightedScatterAdd`, `MoeZeroBf16`, `DualRmsNormPlusRes`); only `MatmulBTAlphaBeta` is on the #1126 path, and the others are named so the next reader need not re-derive the list. Listed under `## Owed` in [`expert-streaming.md`](specs/expert-streaming.md) | bug | | [#1193](https://github.com/mudler/vllm.cpp/issues/1193) | `SPEC-DSPARK-QWEN3-ROUTING` | A Qwen3 DSpark draft declaring `architectures=["DSparkDraftModel"]` with `model_type` `qwen3` has no route. The pin forces every DSpark draft that is not `Qwen3DSparkModel` or `Gemma4DSparkModel` onto `model_type` `deepseek_v4` (`vllm/config/speculative.py:934-944` @ `555967922`), and vLLM PR 52197 (merged 2026-08-17 at `7075ddac`) replaced that with a leading branch normalizing the pair to `Qwen3DSparkModel`. We diverge from BOTH: the forced rewrite was never ported, so nothing in `src/vllm/entrypoints/model_loader.cpp` reads a draft config's `architectures` key at all, and `SpeculativeConfig::IsDsparkDraft` (`include/vllm/config/speculative.h:120-136`) has no production caller — every reference outside its header is in `tests/vllm/config/test_speculative_dspark.cpp:132-140`, and `ResolveSpecConfig` branches on `cli.method` alone. The checkpoint is real and gateable here: `RadixArk/Qwen3.8-27B-DSpark` at revision `85ef153be924f17ce4bf62726954eeaa4a73e854` carries exactly that config shape in one 2718576122-byte shard, drafting five layers for a 64-layer Qwen3.8-27B target | bug | +| [#1218](https://github.com/mudler/vllm.cpp/issues/1218) | `ENG-EXPERT-STREAM` | `EnsureGemma4Fp8NativeOnDevice` has no arm-existence guard, and it is the DEFAULT Gemma-4 expert arm, so [#1126](https://github.com/mudler/vllm.cpp/issues/1126) step 1 would still throw mid-decode after [#1205](https://github.com/mudler/vllm.cpp/issues/1205)'s guard lands. That guard covers the BF16 device-expert arm (`src/vllm/model_executor/models/gemma4_moe.cpp:571`, inside `:548-608`). The FP8-native twin at `:611` has none, and `VT_GEMMA4_FP8_NATIVE` returns true when unset (`:969-974`), so on a per-expert FP8 checkpoint the expert loop reaches the twin at `:1359` and `:1484` BEFORE the guarded arm. A `true` from it routes into `ExpertGeGLUFp8Native` (`:95-130`), which needs `vt::ExpertGeGLUFp8TopKM1` (`:105`, which merely returns false off ROCm), `vt::DequantFp8ChannelBf16` (`:117`, `:119`, refusing at `src/vt/fused_ops.cpp:194`) and `vt::MatmulBTAlphaBeta` (`gemma4_moe.cpp:128`, refusing at `src/vt/fused_ops.cpp:152`) — so T>1 throws at the first dequant and T==1 falls through the fused kernel and throws too, in both cases outside the upload's own `try`/`catch`. Latent for the same reason and for exactly as long: the twin's `MakeRoom` also needs `vt::Backend::DeviceMemoryInfo`, which only ROCm overrides (`src/vt/rocm/rocm_backend.hip:338-345`). FILED, NOT FIXED, and not for effort: the BF16 guard keys on ONE predicate that is the same condition its own dispatch uses, which is what makes it honest, whereas the twin depends on three different ops and needs a predicate per op — reusing `HasMatmulBTAlphaBeta` there would be a guard naming the wrong arm, the exact defect this row's review had just corrected in a refusal message. Found while repairing [#1200](https://github.com/mudler/vllm.cpp/pull/1200). Listed under `## Owed` in [`expert-streaming.md`](specs/expert-streaming.md) | bug | diff --git a/.agents/specs/expert-streaming.md b/.agents/specs/expert-streaming.md index 75d5959d8..cb2b2636c 100644 --- a/.agents/specs/expert-streaming.md +++ b/.agents/specs/expert-streaming.md @@ -1606,12 +1606,14 @@ Carried debt for this row. Each item names why it is not closed here. | **`--device cuda` still cannot SERVE a larger-than-pool GGUF; it only refuses by name now.** The device-slot arm is the missing capability: a `DeviceExpertSlotStore` behind `ExpertSlotStore`, a read accessor on that interface (`KqExpertSlice` reads `HostExpertSlotStore::Slot()`, the CONCRETE class, so the seam cannot be swapped today), a device filler that is not `pread`-into-host (`ExpertSlotStore::SlotForWrite` is handed straight to `::pread`, `expert_streamer.cpp:76-94`), and lifting the `is_cpu()` guard at `qwen3_5.cpp:5578`. Sized by the measurement above: 2790 slices per token at 2,490,368 bytes is 6.95 GB per token against a 119.631 GiB pool that already holds the dense remainder. Tracked as [#1124](https://github.com/mudler/vllm.cpp/issues/1124). | It is a campaign, not a fix: W7 (the pluggable backing store) is its declared owner in the work breakdown, and the CPU arm's own I/O rate is still unmeasured on a live cache two rows above. Building a device lane on top of a host lane whose bandwidth number is void would be optimising against a number nobody has. | | **The fit bound omits everything that is not a weight.** KV cache, activations, the scratch pools and the CUDA context are not counted, so a checkpoint at 0.95x of the pool passes the refusal and still dies on the first forward. | A headroom fraction invented here would be exactly the guess the per-tensor bound exists to avoid. The number wants the startup memory profile that `KV-WARMUP-PROFILE` owns (`INVENTORIED`; upstream's is `GPUWorker.determine_available_memory`, `vllm/v1/worker/gpu_worker.py:451-495`, around `profile_run`, `vllm/v1/worker/gpu/model_runner.py:682`), which is a different row. Those two anchors are stated here from the pinned tree because that row's own three anchors are stale at the current pin, and `gguf_device_fit.h` had copied two of them — filed as [#1139](https://github.com/mudler/vllm.cpp/issues/1139), owned by `KV-WARMUP-PROFILE`, blocked here only by the `engine-matrix.md` record lock #1119 holds. | | **The fit bound also counts too MUCH, and that direction can refuse a load that fits.** A tensor present in the file and not staged by THIS load is a positive over-count. On a default load that is the MTP / `nextn` block: 8,940,488,704 bytes, 8.33 GiB, 2.2506 % of the target checkpoint. A budget in that window refuses a weight set that would have fitted. | Not closed here. Closing it means the bound taking a per-tensor staging POLICY as input, which is the caller's knowledge and not the file's, and the exclusion's own failure mode is an under-count to nothing — which restores the 26-minute-then-OOM this row exists to remove, on a device nobody here has to measure it on. So the direction is stated in `gguf_device_fit.h`, pinned executably by `test_gguf_device_fit`, exposed to operators in `docs/USAGE.md`, and tracked as [#1136](https://github.com/mudler/vllm.cpp/issues/1136). `VT_DEVICE_WEIGHT_BUDGET_BYTES` is the way out of the window in the meantime. | -| **`Backend::DeviceMemoryInfo` has no CUDA override, and waking it is not the one-line port of the ROCm one that #1126 describes: on CUDA it would wake a THROW.** Only ROCm implements it (`src/vt/rocm/rocm_backend.hip:338-345`), so `Gemma4MoE`'s device-expert LRU refuses on every CUDA device (`gemma4_moe.cpp:506`). Where a per-expert FP8 checkpoint is present, the expert's BF16 bytes are re-copied HOST->DEVICE on every use instead (`ExpertGeGLUHost`, `gemma4_moe.cpp:49-74`, reached at `:1493-1499`; the H2D is `:59-60` and it drains the queue per expert at `:73`), silently and for the life of the process. That H2D cost is CONDITIONAL, not present-tense: the whole device LRU is `ex.is_fp8`-gated (`:969`, `:1484`), and by point (1) below no such Gemma-4 checkpoint is pinned anywhere, so on CUDA today the LRU-MISS fallback is never even asked for. (A BF16 Gemma-4 checkpoint reaches `ExpertGeGLUHost` at `:1503` too, but through the host-weight branch the LRU never governs, so it is not a cost of the missing probe.) The name misleads: it computes on the DEVICE from host-resident weights, so what the dead LRU would cost is bandwidth and a per-expert `Synchronize`, not a wrong answer. | The CAPABILITY is [#1126](https://github.com/mudler/vllm.cpp/issues/1126) and is still not built. The false COMMENT was corrected by this row in both places that carried it, `include/vt/backend.h:78-93` and `gemma4_moe.cpp:440-448`; both anchors were re-verified exact against `fd64c76ee`, as were `rocm_backend.hip:338-345`, `gemma4_moe.cpp:506`, `platforms/cuda.cpp:71` and `platforms/rocm.cpp:74`. The review repair below re-audited every anchor these three rows cite against the repaired tree: 53 examined, 53 exact, 0 stale. **Four things were established while re-reading it for #1126, and each one raises the price of the override.** (1) *There is nothing to run it on.* The LRU needs `ex.is_fp8`, which only `LoadMoeFp8PerExpert` sets (`gemma4_weights.cpp:210-215`) from a per-expert `F8_E4M3` export; no such Gemma-4 checkpoint is pinned anywhere in `docs/USAGE.md` — all 32 lines there matching `gemma` case-insensitively were swept, and every checkpoint among them is the LTX-2.5 text tower (`gemma4-12b-with-proj*.safetensors`), not a per-expert FP8 MoE decoder, so the woken path cannot be exercised, here or elsewhere, until one is. (2) *The device-resident arm has no CUDA implementation to route into. It has a throw.* `ExpertGeGLUDeviceAccum` (`gemma4_moe.cpp:76-93`) READS as generic — `vt::MatmulBT`, `GeluAndMul`, `vt::MatmulBTAlphaBeta` — and an earlier draft of this row concluded from that reading that it "would run". It does not. `vt::MatmulBTAlphaBeta` (`src/vt/fused_ops.cpp:102`) is guarded on `#if defined(VLLM_CPP_HIP)` AND `q.device.type == kROCM`; its only implementation in the tree is `rocm::MatmulBTAlphaBetaRocm` (`src/vt/rocm/rocm_matmul_hipblaslt.hip:516`, declared `include/vt/rocm/rocm_matmul_batch.h:28`), and every other device falls through to a refusal. So the chain the override wakes is: `EnsureGemma4Fp8ExpertOnDevice` (`gemma4_moe.cpp:548-586`, carrying NO HIP guard) -> `lru.MakeRoom` (`:565`) succeeding as soon as `FreeBytes` can answer -> `true` at `:575` -> the call site at `:1486` -> `ExpertGeGLUDeviceAccum` at `:1487` -> `vt::MatmulBTAlphaBeta` at `:90` -> THROW, mid-decode. The `try`/`catch (...)` at `:563-585` wraps only the UPLOAD; the compute at `:1487` sits outside it, so the exception propagates out of the decode step rather than degrading to the host fallback. It is latent today only because the other route into that function, `same_dev` (`:730-731`), needs `ex.gate_up_dev`, which is assigned nowhere but `src/vt/rocm/rocm_gemma4_experts.hip:207,226` — so the resident arm is UNREACHABLE off ROCm rather than safe. This is the actual blocker under #1126, it was recorded nowhere, and it is a stronger argument than the other three: filed as [#1205](https://github.com/mudler/vllm.cpp/issues/1205), and the refusal itself is now gated by `tests/vt/test_gemma4_rocm_fp8_seams.cpp`, which is what a CUDA implementation will have to satisfy. **Two corrections to the earlier draft's supporting claims, both of which overstated the case.** *The HIP-only list was one symbol too long.* Three of the four are genuinely HIP-only stubs inside this file's ONLY `#ifndef VLLM_CPP_HIP` block (`:1574-1628`): `RunGemma4FusedTopkExpertGeGLU` (`:1599`), `PeerCopyGemma4Fp8ExpertSlice` (`:1607`) and `RunGemma4Fp8TopKOnExpertDevice` (`:1611`). `ExpertGeGLUDeviceBatched` (`:240`) is NOT: it sits OUTSIDE that block, in an anonymous namespace, with no HIP implementation and no header declaration, and is unconditionally `return false` under its own lab note (`:237-239` — gather+strided produced wrong tokens at ~23 t/s, pointer-batch ~0.8 t/s, serial/fused-gelu kept at ~34 t/s). It is disabled EVERYWHERE, ROCm included, so naming it beside the three inflated the ROCm/CUDA asymmetry. *The token-neutrality argument had the wrong mechanism.* The conclusion stands — the swap would not be token-neutral — but not because "the two arms sum the top-k experts in a different order". They do not: both run inside the SAME `for (int i = 0; i < top_k; ++i)` at `:1431`, so the summation order is identical. The real difference is ROUNDING and where the routing weight is applied. The resident arm folds `ww` into the GEMM `alpha` and accumulates in the epilogue with `beta` (`:1434`, `:1442`, `:90`), so the weight multiplies in the GEMM's own accumulator. The fallback writes the UNWEIGHTED product to a BF16 buffer (`:67`, `:70`) and applies `ww` afterwards with separate BF16 kernels — `vt::MulScalar` at `:1524` on the first expert, `MulScalar` plus `vt::Add` at `:1526-1527` on the rest. Two extra BF16 roundings per expert, with the weight applied post-rounding. (3) *The headroom test does not mean the same thing on the CUDA device this project gates on.* `MakeRoom` admits iff `free_b >= need + 1.5 GiB` (`:514`), a constant tuned on discrete dual R9700s where free VRAM is a quantity distinct from host RAM. On a GB10 it is not. This row already measured that `cudaMemGetInfo`'s `total` there is EXACTLY `/proc/meminfo MemTotal` times 1024, which is why it reached for that instrument where `nvidia-smi` answers `[N/A]`; its `free` is therefore reported over the same unified pool, and the host BF16 expert cache the device upload exists to relieve (`ex.cached_gu`/`cached_dn` via `EnsureGemma4Fp8ExpertCached`, bounded by the host LRU at `gemma4_moe.cpp:352`) is drawn from that pool too. So the admission test would double-count, on a box whose unified-memory OOM takes the host down with it. The `free` half is an inference from the measured `total`, not a second measurement, and it wants confirming on the device before any override lands. (4) *The gap is isolated, not a pattern.* Comparing overrides one by one across `src/vt/cuda/cuda_backend.cu` and `src/vt/rocm/rocm_backend.hip` at `fd64c76ee`, `DeviceMemoryInfo` is the ONLY optional `vt::Backend` seam ROCm answers and CUDA does not. | +| **`Backend::DeviceMemoryInfo` has no CUDA override, and waking it is not the one-line port of the ROCm one that #1126 describes: on CUDA it would wake a THROW.** Only ROCm implements it (`src/vt/rocm/rocm_backend.hip:338-345`), so `Gemma4MoE`'s device-expert LRU refuses on every CUDA device (`gemma4_moe.cpp:506`). Where a per-expert FP8 checkpoint is present, the expert's BF16 bytes are re-copied HOST->DEVICE on every use instead (`ExpertGeGLUHost`, `gemma4_moe.cpp:49-74`, reached at `:1515-1521`; the H2D is `:59-60` and it drains the queue per expert at `:73`), silently and for the life of the process. That H2D cost is CONDITIONAL, not present-tense: the whole device LRU is `ex.is_fp8`-gated (`:991`, `:1506`), and by point (1) below no such Gemma-4 checkpoint is pinned anywhere, so on CUDA today the LRU-MISS fallback is never even asked for. (A BF16 Gemma-4 checkpoint reaches `ExpertGeGLUHost` at `:1525` too, but through the host-weight branch the LRU never governs, so it is not a cost of the missing probe.) The name misleads: it computes on the DEVICE from host-resident weights, so what the dead LRU would cost is bandwidth and a per-expert `Synchronize`, not a wrong answer. | The CAPABILITY is [#1126](https://github.com/mudler/vllm.cpp/issues/1126) and is still not built. The false COMMENT was corrected by this row in both places that carried it, `include/vt/backend.h:78-93` and `gemma4_moe.cpp:440-448`; both anchors were re-verified exact against `fd64c76ee`, as were `rocm_backend.hip:338-345`, `gemma4_moe.cpp:506`, `platforms/cuda.cpp:71` and `platforms/rocm.cpp:74`. SCOPE, because the previous headline read as coverage it did not have. The first review repair re-audited every anchor cited by the FOUR `## Owed` rows in this cluster — this one, the #1126 step-3 row, the #1197 row and the #1205 row — against the repaired tree: 53 examined, 53 exact, 0 stale. It audited those four rows and nothing else. The GitHub ISSUE BODIES were never in the audited set, and #1205's body was in fact 11 lines stale at `7beada17c` for exactly that reason. The second review repair widened the set: it re-derived every anchor in those four rows AND in the #1197 and #1205 issue bodies AND in the #1205 index row against the final tree — 71 examined, 71 exact, 0 stale after repair. Anchors from `gemma4_moe.cpp:549` onward moved by 22 lines in that repair, because the arm-existence guard it added sits at `:571`. **Four things were established while re-reading it for #1126, and each one raises the price of the override.** (1) *There is nothing to run it on.* The LRU needs `ex.is_fp8`, which only `LoadMoeFp8PerExpert` sets (`gemma4_weights.cpp:210-215`) from a per-expert `F8_E4M3` export; no such Gemma-4 checkpoint is pinned anywhere in `docs/USAGE.md` — all 32 lines there matching `gemma` case-insensitively were swept, and every checkpoint among them is the LTX-2.5 text tower (`gemma4-12b-with-proj*.safetensors`), not a per-expert FP8 MoE decoder, so the woken path cannot be exercised, here or elsewhere, until one is. (2) *The device-resident arm has no CUDA implementation to route into. It has a throw.* `ExpertGeGLUDeviceAccum` (`gemma4_moe.cpp:76-93`) READS as generic — `vt::MatmulBT`, `GeluAndMul`, `vt::MatmulBTAlphaBeta` — and an earlier draft of this row concluded from that reading that it "would run". It does not. `vt::MatmulBTAlphaBeta` (`src/vt/fused_ops.cpp:111-157`, dispatching at `:117`) is guarded on `#if defined(VLLM_CPP_HIP)` AND `q.device.type == kROCM`; its only implementation in the tree is `rocm::MatmulBTAlphaBetaRocm` (`src/vt/rocm/rocm_matmul_hipblaslt.hip:516`, declared `include/vt/rocm/rocm_matmul_batch.h:28`), and every other device falls through to a refusal. So the chain the override WOULD wake is: `EnsureGemma4Fp8ExpertOnDevice` (`gemma4_moe.cpp:548-608`) -> `lru.MakeRoom` (`:587`) succeeding as soon as `FreeBytes` can answer -> `true` at `:597` -> the call site at `:1508` -> `ExpertGeGLUDeviceAccum` at `:1509` -> `vt::MatmulBTAlphaBeta` at `:90` -> THROW, mid-decode. The `try`/`catch (...)` at `:585-607` wraps only the UPLOAD; the compute at `:1509` sits outside it, so the exception would propagate out of the decode step rather than degrading to the host fallback. **That chain is now cut at its first link.** `EnsureGemma4Fp8ExpertOnDevice` refuses at `:571` when `vt::HasMatmulBTAlphaBeta(d.q)` is false, BEFORE the upload, so the caller takes the `else` at `:1515-1521` — `EnsureGemma4Fp8ExpertCached` plus `ExpertGeGLUHost` — and the step answers instead of throwing. The refusal at `:90` stays as the backstop. It is latent today only because the other route into that function, `same_dev` (`:752-753`), needs `ex.gate_up_dev`, which is assigned nowhere but `src/vt/rocm/rocm_gemma4_experts.hip:207,226` — so the resident arm is UNREACHABLE off ROCm rather than safe. This is the actual blocker under #1126, it was recorded nowhere, and it is a stronger argument than the other three: filed as [#1205](https://github.com/mudler/vllm.cpp/issues/1205), and the refusal itself is now gated by `tests/vt/test_gemma4_rocm_fp8_seams.cpp`, which is what a CUDA implementation will have to satisfy. **Two corrections to the earlier draft's supporting claims, both of which overstated the case.** *The HIP-only list was one symbol too long.* Three of the four are genuinely HIP-only stubs inside `gemma4_moe.cpp`'s ONLY `#ifndef VLLM_CPP_HIP` block (`gemma4_moe.cpp:1596-1650`): `RunGemma4FusedTopkExpertGeGLU` (`:1621`), `PeerCopyGemma4Fp8ExpertSlice` (`:1629`) and `RunGemma4Fp8TopKOnExpertDevice` (`:1633`). `ExpertGeGLUDeviceBatched` (`:240`) is NOT: it sits OUTSIDE that block, in an anonymous namespace, with no HIP implementation and no header declaration, and is unconditionally `return false` under its own lab note (`:237-239` — gather+strided produced wrong tokens at ~23 t/s, pointer-batch ~0.8 t/s, serial/fused-gelu kept at ~34 t/s). It is disabled EVERYWHERE, ROCm included, so naming it beside the three inflated the ROCm/CUDA asymmetry. *The token-neutrality argument had the wrong mechanism.* The conclusion stands — the swap would not be token-neutral — but not because "the two arms sum the top-k experts in a different order". They do not: both run inside the SAME `for (int i = 0; i < top_k; ++i)` at `:1453`, so the summation order is identical. The real difference is ROUNDING and where the routing weight is applied. The resident arm folds `ww` into the GEMM `alpha` and accumulates in the epilogue with `beta` (`:1456`, `:1464`, `:90`), so the weight multiplies in the GEMM's own accumulator. The fallback writes the UNWEIGHTED product to a BF16 buffer (`:67`, `:70`) and applies `ww` afterwards with separate BF16 kernels — `vt::MulScalar` at `:1546` on the first expert, `MulScalar` plus `vt::Add` at `:1548-1549` on the rest. Two extra BF16 roundings per expert, with the weight applied post-rounding. (3) *The headroom test does not mean the same thing on the CUDA device this project gates on.* `MakeRoom` admits iff `free_b >= need + 1.5 GiB` (`:514`), a constant tuned on discrete dual R9700s where free VRAM is a quantity distinct from host RAM. On a GB10 it is not. This row already measured that `cudaMemGetInfo`'s `total` there is EXACTLY `/proc/meminfo MemTotal` times 1024, which is why it reached for that instrument where `nvidia-smi` answers `[N/A]`; its `free` is therefore reported over the same unified pool, and the host BF16 expert cache the device upload exists to relieve (`ex.cached_gu`/`cached_dn` via `EnsureGemma4Fp8ExpertCached`, bounded by the host LRU at `gemma4_moe.cpp:352`) is drawn from that pool too. So the admission test would double-count, on a box whose unified-memory OOM takes the host down with it. The `free` half is an inference from the measured `total`, not a second measurement, and it wants confirming on the device before any override lands. (4) *The gap is isolated, not a pattern.* Comparing overrides one by one across `src/vt/cuda/cuda_backend.cu` and `src/vt/rocm/rocm_backend.hip` at `fd64c76ee`, `DeviceMemoryInfo` is the ONLY optional `vt::Backend` seam ROCm answers and CUDA does not. | | **#1126's own closing plan, step 3, must be NARROWED before it is done: the load-time fit check may read the seam's `total`, and never its `free`.** The issue proposes that "the #1123 fit check can then read the budget from the backend seam on every platform that reports one". As written that invites the live half, which would be a defect — but the seam returns BOTH halves (`bool DeviceMemoryInfo(size_t* free_bytes, size_t* total_bytes)`, `include/vt/backend.h:94`), and only one of them is illegitimate here. | The tree holds two answers to "how much device memory", and they answer different questions. `vt::Backend::DeviceMemoryInfo(free, total)` is a LIVE probe that moves with contention; it is authoritative for a RUNTIME admission decision — can this allocation succeed right now — and for nothing else. `vllm::platforms::ResidencyPolicy::device_memory_total_bytes` is a TOTAL probed once at platform registration with `0 == UNKNOWN`; it is authoritative for a LOAD-TIME budget verdict, which has to be reproducible and independent of whatever else the box is doing. Sourcing the load-time verdict from `free` would make the same checkpoint load or be refused depending on the page cache, which is precisely the property **`total`, not `free`** above was chosen to avoid. Reading `total` through the seam is not that. It is contention-independent, it is the same quantity `ResidencyPolicy` already carries, and a seam that answers it on every platform is a defensible place to source it from. What step 3 must additionally preserve is the PROBE-ONCE semantics: `device_memory_total_bytes` is probed at platform registration (`include/vllm/platforms/interface.h:70-72`), and a per-load live call would reintroduce the contention dependence by the back door even reading only `total`, because a load-time verdict has to be reproducible from the record rather than from the moment. So the defensible statement, and the one this row asserts: **step 3 may read only `total`, never `free`, and must keep the value probed once at platform registration.** An earlier draft of this row said "never step 3" outright; that overstated it and would have blocked a legitimate simplification, so it is corrected here rather than quietly narrowed. Both seams already carry the division in prose (`include/vt/backend.h:90-93` and `include/vllm/platforms/interface.h:61-69`); it is restated here because #1126 is the record a reader of that issue will act on, and as filed it points the other way. What #1126 owes is its steps 1 and 2 together — the override AND the Gemma4 measurement, with [#1205](https://github.com/mudler/vllm.cpp/issues/1205) ahead of both — never step 1 alone, and step 3 only in the narrowed form above. | | **The device-expert LRU's slot cap makes its own eviction opt-in inert.** `MakeRoom` tests `slots.size() >= kMaxSlots` (`gemma4_moe.cpp:498`) BEFORE the eviction loop (`:499-500`), and `EvictOne` (`:457`, the device LRU's — a host-cache namesake sits at `:275`) is the only thing that SHRINKS `slots`. The one other statement that touches its size, `slots.clear()` in `DevExpertLru::Note` (`:522`), is a device-index RESET rather than an eviction: it drops bookkeeping when `dev != d.q.device.index` and frees nothing, and it is unreachable in a single-device process. It is named here so the next reader does not conclude the #1197 sweep missed it. So once 24 slots are resident `VT_GEMMA4_EXPERT_EVICT=1` never runs again and the cache degrades permanently to fill-only. It binds only when `24 * expert_bytes < BudgetBytes()`, so it is condition-dependent and silent either way. Tracked as [#1197](https://github.com/mudler/vllm.cpp/issues/1197). | Filed, not fixed, and for the same reason as the row above rather than for effort: the one-line repair wakes more `hipFree` under load, which the surrounding comments say has been observed as a permanent `kfd_wait` hang with the GPU idle and no decode tokens. The current ordering may well be deliberate belt-and-braces. Deciding that needs the dual-RDNA4 box `.agents/specs/gemma4-rocm-fp8-moe.md` describes; this host has neither a ROCm nor a CUDA device. It closes when the cap moves after the eviction loop and a run stays hang-free, or when the comment says the cap is by design — one of the two, not silence. | -| **`vt::MatmulBTAlphaBeta` is ROCm-only and has no CUDA implementation at all, which is what #1126 step 1 is actually blocked on.** `src/vt/fused_ops.cpp:102` dispatches to `rocm::MatmulBTAlphaBetaRocm` (`src/vt/rocm/rocm_matmul_hipblaslt.hip:516`) under `#if defined(VLLM_CPP_HIP)` and `q.device.type == kROCM`, and refuses on every other device. There is no CUDA, Vulkan, Metal or CPU arm. The full chain from the missing `DeviceMemoryInfo` override to that refusal is traced in the first row above. Tracked as [#1205](https://github.com/mudler/vllm.cpp/issues/1205). | The REFUSAL is fixed in flow, because a bare `std::runtime_error` reading "ROCm-only in this build" does not satisfy the standing rule that an unimplemented arm refuses with a message NAMING the missing part: a caller who hits it on CUDA cannot tell a missing kernel from a missing build flag. It now names the device that asked, names where the implementation lives, and names the issue, and `tests/vt/test_gemma4_rocm_fp8_seams.cpp` gates it on a posed CUDA queue plus three other non-ROCm devices — mutation-proven by restoring the old message (RED) and by deleting the refusal outright (RED). Pinning the precondition executably is also the answer to "should the absence be recorded": the gate says what is missing without any checker that says do not fix it, and when someone writes the CUDA arm the same test is what it has to satisfy. The IMPLEMENTATION stays owed and is what [#1205](https://github.com/mudler/vllm.cpp/issues/1205) tracks. It is not written here: a `beta`-accumulating BT GEMM on cuBLASLt is a kernel with its own correctness gate, the `DeviceMemoryInfo` row's point (1) above says there is no checkpoint to exercise it on, and this host has neither a ROCm nor a CUDA device to measure either arm. | +| **`vt::MatmulBTAlphaBeta` is ROCm-only and has no CUDA implementation at all, which is what #1126 step 1 is actually blocked on.** `src/vt/fused_ops.cpp:117` dispatches to `rocm::MatmulBTAlphaBetaRocm` (`src/vt/rocm/rocm_matmul_hipblaslt.hip:516`) under `#if defined(VLLM_CPP_HIP)` and `q.device.type == kROCM` — `src/vt/fused_ops.cpp:111-112` is the signature, not the dispatch — and every other device falls through to the refusal at `src/vt/fused_ops.cpp:152`. There is no CUDA, Vulkan, Metal or CPU arm. The full chain from the missing `DeviceMemoryInfo` override to that refusal is traced in the first row above. Tracked as [#1205](https://github.com/mudler/vllm.cpp/issues/1205). | The REFUSAL is fixed in flow, because a bare `std::runtime_error` reading "ROCm-only in this build" does not satisfy the standing rule that an unimplemented arm refuses with a message NAMING the missing part: a caller who hits it on CUDA cannot tell a missing kernel from a missing build flag. It now names the device that asked, names the one arm that exists, and names the issue (`:152`), and a kROCM queue — which reaches the same line in a build configured without `-DVLLM_CPP_HIP` — gets a DIFFERENT message naming the absent build flag (`:138`), because for that caller the kernel exists and telling them to write one would send them to fix the wrong thing. `tests/vt/test_gemma4_rocm_fp8_seams.cpp` gates both messages on a posed CUDA queue, on `kCPU`/`kVULKAN`/`kMETAL`, and on kROCM — mutation-proven by restoring the old message (RED), by deleting the refusal outright (RED), and by deleting the kROCM branch so that case falls to the generic message (RED). **Say plainly what that message change does and does not pin: a contract in a unit test, not observable behaviour.** The throw is unreachable off ROCm in any shipped configuration, so no production run can print either string today; what the test fixes is what a CUDA implementation has to satisfy when someone writes one. **The reachable half of this row is the GUARD.** `EnsureGemma4Fp8ExpertOnDevice` refuses at `gemma4_moe.cpp:571` when `vt::HasMatmulBTAlphaBeta(d.q)` is false, BEFORE the upload rather than after it, which converts the mid-decode exception traced above into the host fallback that was already sitting in the `else` at `:1515-1521`: slower, two extra BF16 roundings per expert, and correct. The predicate (`include/vt/fused_ops.h`, defined `src/vt/fused_ops.cpp:102-109`) is the same condition the dispatch at `:117` uses rather than a second copy of it, so the two cannot drift and writing the CUDA kernel wakes the device arm with no edit at the call site. It is gated by `tests/vllm/models/test_gemma4_moe_device_arm_guard.cpp`, which enters through `vllm::RunGemma4Moe` — the production layer entry `src/vllm/model_executor/models/gemma4.cpp:634` calls — and decorates the registered CPU backend so `DeviceMemoryInfo` ANSWERS, which is the post-#1126 state and the only state in which the guard binds at all. Deleting the guard makes that test RED with the exact `no implementation for device 'cpu'` throw; forcing `HasMatmulBTAlphaBeta` to `true` makes it RED too. A test that constructed the `Dev` or the LRU by hand would have stayed green under both. The IMPLEMENTATION stays owed and is what [#1205](https://github.com/mudler/vllm.cpp/issues/1205) tracks. It is not written here: a `beta`-accumulating BT GEMM on cuBLASLt is a kernel with its own correctness gate, the `DeviceMemoryInfo` row's point (1) above says there is no checkpoint to exercise it on, and this host has neither a ROCm nor a CUDA device to measure either arm. | | **`model_loader.cpp` is cited by absolute line number from 109 sites in 45 files, and this row's change moved them.** Measured between `e7d0a1f7c` and the repaired head: 203 moved line references over 109 citing sites, 10 unmoved. The file is ~1640 lines and almost every engine and model row edits it, so any edit near its top invalidates citations in files the editing change never opens. | Not swept here, deliberately, and the reason is not effort: several of the 109 were ALREADY stale (`model-matrix.md:197` cites `:184-223` as the "live loader"; line 184 at `e7d0a1f7c` is `static const bool once = [] {`), and rewriting all of them from the current tree would launder pre-existing debt into a clean-looking record. What IS fixed here is the two anchors this change authored itself, checked against the final tree. Tracked as [#1143](https://github.com/mudler/vllm.cpp/issues/1143), which lists the three candidate fixes; it needs a row of its own and is parked here because this row is what measured it. | | **The budget knob is an environment variable, not a config key.** `VT_DEVICE_WEIGHT_BUDGET_BYTES`. | `ENG-RESIDENCY-CONFIG` ([#1110](https://github.com/mudler/vllm.cpp/issues/1110), PR #1119) is in flight and adds exactly the `vllm_cpp` namespace inside `--offload-config` this key belongs in. Landing a second, competing config surface while that one is unmerged would create the conflict both changes then have to resolve. Migrate once #1119 lands; tracked as [#1127](https://github.com/mudler/vllm.cpp/issues/1127). | +| **`EnsureGemma4Fp8NativeOnDevice` has the same missing-arm shape and no guard, and it is the DEFAULT arm.** The guard this row added covers the BF16 device-expert arm (`gemma4_moe.cpp:571`). Its FP8-native twin at `:611` does not have one, and `VT_GEMMA4_FP8_NATIVE` defaults to TRUE (`:969-974`), so on a per-expert FP8 checkpoint the expert loop reaches the twin at `:1359` and `:1484` FIRST. A `true` from it routes into `ExpertGeGLUFp8Native` (`:95-130`), which needs `vt::DequantFp8ChannelBf16` (`:117`, `:119`; refuses at `src/vt/fused_ops.cpp:194`) and `vt::MatmulBTAlphaBeta` (`gemma4_moe.cpp:128`; refuses at `src/vt/fused_ops.cpp:152`). Latent for the same reason and for exactly as long: its `MakeRoom` also needs `Backend::DeviceMemoryInfo`, so #1126 step 1 wakes this arm BEFORE it wakes the guarded one. Tracked as [#1218](https://github.com/mudler/vllm.cpp/issues/1218). | Not fixed in flow, and not for effort. The BF16 guard keys on ONE predicate that is the same condition its dispatch uses, which is what makes it honest. The twin depends on three different ops, so an honest guard for it needs a predicate per op; reusing `HasMatmulBTAlphaBeta` there would be a guard naming the wrong arm, which is the defect this row's own review just corrected in a refusal message. That is a distinct change with its own gate. Recording it is what stops the default arm being discovered by whoever lands #1126. | +| **A production-entered gate for the guard exists; a production-entered gate for the REFUSAL MESSAGE does not, and cannot be built here.** `test_gemma4_moe_device_arm_guard.cpp` drives `vllm::RunGemma4Moe`, so the guard is measured as a capability. The message itself is only reachable when the guard is absent, which is precisely what that test forbids, so the message's own gate is a unit contract on a posed `vt::Queue`. | This is a property of the fix, not a gap in the test. A refusal that a correct program never reaches has no production path by construction; the alternative would be to leave the hazard unguarded so the string could be observed. Naming it here so no later reader reads the seams suite as a reachability proof. Closed when a CUDA `MatmulBTAlphaBeta` lands under [#1205](https://github.com/mudler/vllm.cpp/issues/1205) and the message stops being the answer at all. | ## Risks/decisions diff --git a/include/vt/fused_ops.h b/include/vt/fused_ops.h index ceb31e8d0..c051b70c3 100644 --- a/include/vt/fused_ops.h +++ b/include/vt/fused_ops.h @@ -21,6 +21,19 @@ void DualRmsNormPlusRes(Queue& q, Tensor& out, const Tensor& x1, const Tensor& w void GeluMulSeparate(Queue& q, void* out, const void* gate, const void* up, int64_t n, DType dtype); +// Does `MatmulBTAlphaBeta` have an arm for this queue's device in THIS build? +// It answers the question a caller has to ask BEFORE committing to a device +// path, because the alternative is finding out from a throw: the only +// implementation in the tree is `rocm::MatmulBTAlphaBetaRocm`, so on every other +// device — and on ROCm in a build configured without `-DVLLM_CPP_HIP` — the call +// below refuses instead of computing (issue #1205). +// +// It is not a device-name test that a reader has to keep in sync by hand. +// `MatmulBTAlphaBeta` itself dispatches on this predicate, so the two cannot +// disagree: false here means the very next line throws, and a future CUDA arm +// makes both true in one edit. +bool HasMatmulBTAlphaBeta(const Queue& q); + void MatmulBTAlphaBeta(Queue& q, void* out, const void* a, const void* b, int M, int N, int K, float alpha, float beta, DType dtype); diff --git a/src/vllm/model_executor/models/gemma4_moe.cpp b/src/vllm/model_executor/models/gemma4_moe.cpp index ede647835..31c711cb9 100644 --- a/src/vllm/model_executor/models/gemma4_moe.cpp +++ b/src/vllm/model_executor/models/gemma4_moe.cpp @@ -547,6 +547,28 @@ DevExpertLru& ExpertLru() { bool EnsureGemma4Fp8ExpertOnDevice(Dev d, const Gemma4Fp8ExpertMats& ex, int64_t I, int64_t H) { + // Refuse BEFORE the upload on a device whose down-projection GEMM does not + // exist. Returning true here is a PROMISE that the caller may run the + // device-resident arm, and every caller that takes that promise ends in + // `vt::MatmulBTAlphaBeta` — `ExpertGeGLUDeviceAccum` (:76-93) and + // `ExpertGeGLUTopKFusedGelu` (:181-234) both do. That call has exactly one + // implementation in the tree, `rocm::MatmulBTAlphaBetaRocm`, so off ROCm it + // throws (issue #1205). The upload's own `try`/`catch (...)` below does NOT + // cover the compute, so without this line the exception leaves the decode step + // instead of degrading: the `else` arms at the call sites already fall back to + // `EnsureGemma4Fp8ExpertCached` + `ExpertGeGLUHost`, which is slower and + // rounds twice more, but answers. + // + // It is latent rather than live today only because `MakeRoom` needs + // `Backend::DeviceMemoryInfo`, which only ROCm overrides. #1126 step 1 is + // exactly the change that adds the CUDA override, which is why the refusal has + // to be here before it lands and not after. + // + // Keyed on whether the arm EXISTS, not on a device name or a build macro: + // `vt::HasMatmulBTAlphaBeta` is the same predicate the dispatch itself uses, so + // writing the CUDA kernel wakes this path with no edit here, and on ROCm the + // answer is true and nothing about this function changes. + if (!vt::HasMatmulBTAlphaBeta(d.q)) return false; // When device LRU disabled, do NOT host-cache-dequant here — that path was // unbounded (every expert forever) and OOM'd the 30G host (~27G RSS) under pollution. if (!ExpertLru().Enabled()) return false; diff --git a/src/vt/fused_ops.cpp b/src/vt/fused_ops.cpp index 937e986cf..209cef360 100644 --- a/src/vt/fused_ops.cpp +++ b/src/vt/fused_ops.cpp @@ -99,10 +99,21 @@ void GeluMulSeparate(Queue& q, void* out, const void* gate, const void* up, int6 b.Free(tmp); } +bool HasMatmulBTAlphaBeta(const Queue& q) { +#if defined(VLLM_CPP_HIP) + return q.device.type == DeviceType::kROCM; +#else + (void)q; + return false; +#endif +} + void MatmulBTAlphaBeta(Queue& q, void* out, const void* a, const void* b, int M, int N, int K, float alpha, float beta, DType dtype) { #if defined(VLLM_CPP_HIP) - if (q.device.type == DeviceType::kROCM) { + // Dispatch on the predicate rather than on a second copy of its condition, so + // `HasMatmulBTAlphaBeta` cannot drift from what this function actually does. + if (HasMatmulBTAlphaBeta(q)) { rocm::MatmulBTAlphaBetaRocm(q, out, a, b, M, N, K, alpha, beta, dtype); return; } @@ -116,18 +127,33 @@ void MatmulBTAlphaBeta(Queue& q, void* out, const void* a, const void* b, int M, (void)alpha; (void)beta; (void)dtype; - // No CUDA arm exists: the only implementation in the tree is - // rocm::MatmulBTAlphaBetaRocm (src/vt/rocm/rocm_matmul_hipblaslt.hip), so this - // is a missing kernel and not a missing build flag. Name the device that asked - // and where the implementation lives, because the caller cannot tell those two - // apart from "ROCm-only" alone. Reaching this on CUDA is issue #1205 and blocks - // #1126 step 1: waking Gemma4's device-expert LRU routes decode into - // ExpertGeGLUDeviceAccum, which lands here outside the upload's try/catch. + // Two different absences reach this line, and telling a caller the wrong one + // sends them to fix the wrong thing. + // + // A kROCM queue arrives here only when the build was configured without + // `-DVLLM_CPP_HIP`, so the 'rocm' arm exists in the tree and is compiled out. + // That is a build-configuration problem, not a missing kernel, and it is not + // #1205 — the previous "ROCm-only in this build" got exactly this case right. + if (q.device.type == DeviceType::kROCM) { + throw std::runtime_error( + "vt::MatmulBTAlphaBeta: the 'rocm' arm " + "(src/vt/rocm/rocm_matmul_hipblaslt.hip) is compiled out of this build; " + "reconfigure with -DVLLM_CPP_HIP to enable it."); + } + // Every other device arrives here because no such kernel was ever written: + // the only implementation in the tree is rocm::MatmulBTAlphaBetaRocm. Name the + // device that asked, name the one arm that exists, and name the issue that + // owes the rest — "ROCm-only" alone left the caller unable to tell a missing + // kernel from a missing build flag. Reaching this on CUDA is issue #1205 and + // blocks #1126 step 1: waking Gemma4's device-expert LRU would route decode + // into ExpertGeGLUDeviceAccum, which lands here outside the upload's + // try/catch. `EnsureGemma4Fp8ExpertOnDevice` now refuses before that upload + // (gemma4_moe.cpp), so this throw is the backstop rather than the guard. throw std::runtime_error( std::string("vt::MatmulBTAlphaBeta: no implementation for device '") + - DeviceTypeName(q.device.type) + - "'; this arm is implemented for 'rocm' only (src/vt/rocm/rocm_matmul_hipblaslt.hip). " - "A CUDA implementation is missing; see issue #1205."); + DeviceTypeName(q.device.type) + "'; no '" + DeviceTypeName(q.device.type) + + "' kernel has been written and the only arm in the tree is 'rocm' " + "(src/vt/rocm/rocm_matmul_hipblaslt.hip); see issue #1205."); } void MatmulBTFp8Channel(Queue& q, void* out, const void* a, const void* b_fp8, diff --git a/tests/CMakeLists.txt b/tests/CMakeLists.txt index 82c362fbe..20cfa4099 100644 --- a/tests/CMakeLists.txt +++ b/tests/CMakeLists.txt @@ -537,6 +537,13 @@ vllm_cpp_add_test(test_gemma_load vllm/models/test_gemma_load.cpp) vllm_cpp_add_test(test_gemma_forward vllm/models/test_gemma_forward.cpp) target_include_directories(test_gemma_forward PRIVATE ${CMAKE_SOURCE_DIR}/src) vllm_cpp_add_test(test_gemma4_honesty vllm/models/test_gemma4_honesty.cpp) +# #1205: the Gemma-4 device-expert upload must refuse where `vt::MatmulBTAlphaBeta` +# has no arm, entered through `vllm::RunGemma4Moe`. Its own binary: the layer's +# env knobs are frozen into function-local statics on the first call, so the arm +# it measures has to be decided before main() and cannot share a process with a +# case that wants a different one. +vllm_cpp_add_test(test_gemma4_moe_device_arm_guard + vllm/models/test_gemma4_moe_device_arm_guard.cpp) vllm_cpp_add_test(test_qwen3_moe_load vllm/models/test_qwen3_moe_load.cpp) vllm_cpp_add_test(test_qwen3_moe_forward vllm/models/test_qwen3_moe_forward.cpp) target_include_directories(test_qwen3_moe_forward PRIVATE ${CMAKE_SOURCE_DIR}/src) diff --git a/tests/vllm/models/test_gemma4_moe_device_arm_guard.cpp b/tests/vllm/models/test_gemma4_moe_device_arm_guard.cpp new file mode 100644 index 000000000..5de60da6d --- /dev/null +++ b/tests/vllm/models/test_gemma4_moe_device_arm_guard.cpp @@ -0,0 +1,301 @@ +// #1205 / #1126 step 1: `EnsureGemma4Fp8ExpertOnDevice` must REFUSE the device +// upload on a device whose down-projection GEMM does not exist, instead of +// admitting and then throwing out of the middle of a decode step. +// +// WHAT MAKES THIS A REACHABILITY GATE AND NOT A UNIT TEST. The guard is entered +// through `vllm::RunGemma4Moe` — the Gemma-4 MoE layer entry point that +// `gemma4.cpp:634` calls — with a real `Gemma4MoeLayerWeights`, the real router, +// the real device-expert LRU and the real fallback. Nothing here constructs the +// LRU, the `Dev`, or `vt::MatmulBTAlphaBeta` by hand. Delete the `if +// (!vt::HasMatmulBTAlphaBeta(d.q)) return false;` line from +// `EnsureGemma4Fp8ExpertOnDevice` and this file goes red, because the layer then +// throws `vt::MatmulBTAlphaBeta: no implementation for device 'cpu'` out of +// `ExpertGeGLUTopKFusedGelu`. +// +// WHY THE BACKEND IS DECORATED. The hazard is latent on every machine that can +// run this suite, and for exactly one reason: the LRU's `MakeRoom` refuses +// unless `vt::Backend::DeviceMemoryInfo` answers, and only ROCm overrides it +// (`src/vt/rocm/rocm_backend.hip`). So on a stock CPU or CUDA build the upload +// is refused by the memory probe and the guard never binds — an +// unmoved-mutation trap, not a proof. #1126 step 1 IS the change that makes +// `DeviceMemoryInfo` answer on a device with no `MatmulBTAlphaBeta` arm, so this +// test puts the tree in exactly that state: the registered CPU backend, wrapped, +// with `DeviceMemoryInfo` answering and nothing else altered. Every op still +// runs its real CPU kernel; `vt::MatmulBTAlphaBeta` still has no CPU arm. That +// combination is the one the guard exists for, and it is the only combination +// under which deleting the guard is observable. +// +// The decoration is a real `vt::Backend` registered through the public +// `vt::RegisterBackend`, restored byte-for-byte in a destructor, and the test +// asserts BOTH directions of the probe (real backend: false; wrapped: true) +// before it concludes anything — an instrument that silently failed to arm would +// otherwise report the guard as proven when the run never reached it. +#include "vllm/model_executor/models/gemma4_moe.h" + +#include + +#include +#include +#include +#include +#include +#include + +#include "vt/backend.h" +#include "vt/device.h" +#include "vt/dtype.h" +#include "vt/fused_ops.h" +#include "vt/tensor.h" + +namespace { + +using vt::DType; + +// Every knob `RunGemma4Moe` reads lands in a function-local `static const`, so +// the FIRST call in the process freezes it. Set them during static +// initialization rather than inside a TEST_CASE, so no ordering between cases +// can decide which arm the suite measured. +struct EnvOnce { + EnvOnce() { + // The BF16 device-expert LRU arm is the one #1205 traces and the one this + // guard covers. Its FP8-native twin (`EnsureGemma4Fp8NativeOnDevice`, the + // default) carries the same latent shape through `vt::DequantFp8ChannelBf16` + // and is NOT guarded here — filed as its own issue and recorded under + // `## Owed` in .agents/specs/expert-streaming.md. + ::setenv("VT_GEMMA4_FP8_NATIVE", "0", 1); + // A definite budget: the default is 2048 MiB, but reading it from a default + // makes the precondition invisible in the log. + ::setenv("VT_GEMMA4_EXPERT_VRAM_MB", "256", 1); + // Everything else must stay off — each of these reroutes the expert loop. + ::unsetenv("VT_GEMMA4_HOST_AXPY"); + ::unsetenv("VT_GEMMA4_BATCH_EXPERTS"); + ::unsetenv("VT_GEMMA4_CUSTOM_EXPERT"); + ::unsetenv("VT_GEMMA4_EXPERT_EVICT"); + ::unsetenv("VT_GEMMA4_PREFILL_BATCH_MOE"); + ::unsetenv("VT_GEMMA4_PROFILE"); + } +}; +const EnvOnce g_env_once; + +// The registered CPU backend with ONE behaviour changed: the optional device +// memory probe answers. Everything else forwards, so the allocations, copies and +// kernels under test are the production CPU ones. +class ProbingBackend final : public vt::Backend { + public: + explicit ProbingBackend(vt::Backend& inner) : inner_(inner) {} + + void* Alloc(size_t bytes) override { return inner_.Alloc(bytes); } + void Free(void* p) override { inner_.Free(p); } + void Memset(vt::Queue& q, void* p, int value, size_t bytes) override { + inner_.Memset(q, p, value, bytes); + } + void Copy(vt::Queue& q, void* dst, const void* src, size_t bytes) override { + inner_.Copy(q, dst, src, bytes); + } + vt::Queue CreateQueue() override { return inner_.CreateQueue(); } + void DestroyQueue(vt::Queue& q) override { inner_.DestroyQueue(q); } + void Synchronize(vt::Queue& q) override { inner_.Synchronize(q); } + void FlushPending() override { inner_.FlushPending(); } + bool UnifiedMemory() const override { return inner_.UnifiedMemory(); } + bool DeviceMemoryIsHostAddressable() const override { + return inner_.DeviceMemoryIsHostAddressable(); + } + bool SupportsAuxStream() const override { return inner_.SupportsAuxStream(); } + bool SupportsAsyncSampledTokenReadback() const override { + return inner_.SupportsAsyncSampledTokenReadback(); + } + bool SupportsCompressedConvState() const override { + return inner_.SupportsCompressedConvState(); + } + bool SupportsCompressedGdnState() const override { + return inner_.SupportsCompressedGdnState(); + } + bool SupportsGraphCapture() const override { return inner_.SupportsGraphCapture(); } + int DeviceCapabilityMajor() const override { return inner_.DeviceCapabilityMajor(); } + int DeviceCapabilityMinor() const override { return inner_.DeviceCapabilityMinor(); } + + // The one override. Numbers chosen well above `MakeRoom`'s 1.5 GiB headroom so + // the admission decision turns on the guard and not on arithmetic. + bool DeviceMemoryInfo(size_t* free_bytes, size_t* total_bytes) const override { + if (free_bytes != nullptr) *free_bytes = 8ull << 30; + if (total_bytes != nullptr) *total_bytes = 16ull << 30; + return true; + } + + private: + vt::Backend& inner_; +}; + +struct BackendSwap { + vt::Backend* prev; + explicit BackendSwap(vt::Backend& replacement) + : prev(&vt::GetBackend(vt::DeviceType::kCPU)) { + vt::RegisterBackend(vt::DeviceType::kCPU, &replacement); + } + ~BackendSwap() { vt::RegisterBackend(vt::DeviceType::kCPU, prev); } + BackendSwap(const BackendSwap&) = delete; + BackendSwap& operator=(const BackendSwap&) = delete; +}; + +constexpr int64_t kE = 4; // experts +constexpr int64_t kI = 16; // expert intermediate +constexpr int64_t kH = 32; // hidden +constexpr int kTopK = 2; + +// Finite, well-formed F8_E4M3 codes. Random bytes would put the e4m3 NaN +// encodings (0x7F/0xFF) into the weights, which propagate through the GEMM and +// make every later comparison vacuous. +constexpr uint8_t kF8Codes[] = {0x38, 0x3C, 0x30, 0xB8, 0x34, 0x40, 0xB0, 0xBC}; + +uint8_t F8Code(uint64_t i) { return kF8Codes[i % (sizeof(kF8Codes) / sizeof(kF8Codes[0]))]; } + +vllm::OwnedTensor MakeF8(int64_t n_rows, int64_t n_cols, uint64_t seed) { + vllm::OwnedTensor t; + t.dtype = DType::kI8; // F8_E4M3 carried as I8, as the loader does + t.rank = 2; + t.shape[0] = n_rows; + t.shape[1] = n_cols; + std::vector b(static_cast(n_rows * n_cols)); + for (size_t i = 0; i < b.size(); ++i) b[i] = F8Code(seed + i); + t.bytes = vllm::OwnedBytes(std::move(b)); + return t; +} + +vllm::OwnedTensor MakeBf16(const std::vector& shape, float base, float step) { + vllm::OwnedTensor t; + t.dtype = DType::kBF16; + t.rank = static_cast(shape.size()); + int64_t n = 1; + for (int i = 0; i < t.rank; ++i) { + t.shape[i] = shape[static_cast(i)]; + n *= t.shape[i]; + } + std::vector b(static_cast(n) * 2); + auto* p = reinterpret_cast(b.data()); + for (int64_t i = 0; i < n; ++i) + p[i] = vt::F32ToBF16(base + step * static_cast(i % 7)); + t.bytes = vllm::OwnedBytes(std::move(b)); + return t; +} + +vllm::Gemma4MoeLayerWeights MakeMoeWeights() { + vllm::Gemma4MoeLayerWeights moe; + moe.enabled = true; + moe.top_k = kTopK; + moe.moe_intermediate = kI; + // `nk = true` [E, H]: the orientation RunGemma4Moe's VT_CHECK requires. + moe.router_proj = MakeBf16({kE, kH}, -0.25f, 0.1f); + moe.router_proj.nk = true; + + auto& ex = moe.experts; + ex.is_fp8 = true; + ex.num_experts = kE; + ex.intermediate = kI; + ex.hidden = kH; + ex.fp8.resize(static_cast(kE)); + for (int64_t e = 0; e < kE; ++e) { + auto& fex = ex.fp8[static_cast(e)]; + const uint64_t s = static_cast(e) * 131u; + fex.gate_w = MakeF8(kI, kH, s + 1); + fex.up_w = MakeF8(kI, kH, s + 2); + fex.down_w = MakeF8(kH, kI, s + 3); + fex.gate_s = MakeBf16({kI}, 0.05f, 0.01f); + fex.up_s = MakeBf16({kI}, 0.04f, 0.01f); + fex.down_s = MakeBf16({kH}, 0.03f, 0.01f); + } + return moe; +} + +// One decode token through the real layer entry point. Returns the [1, H] bf16 +// result copied out immediately: for T == 1 the layer hands back a view of a +// thread_local buffer that the NEXT call overwrites. +std::vector RunOneToken(vt::Queue& q, const vllm::Gemma4MoeLayerWeights& moe, + const std::vector& x) { + vt::Tensor xin = vt::Tensor::Contiguous(const_cast(x.data()), DType::kBF16, + q.device, {1, kH}); + vllm::Gemma4MoeScratch out = + vllm::RunGemma4Moe(q, moe, /*router_in=*/xin, /*expert_in=*/xin, /*T=*/1, kH, + /*rms_eps=*/1e-6f); + std::vector got(static_cast(kH)); + std::memcpy(got.data(), out.tensor.data, got.size() * sizeof(uint16_t)); + return got; +} + +bool AnyNonZero(const std::vector& v) { + for (const uint16_t h : v) + if (h != 0) return true; + return false; +} + +} // namespace + +TEST_CASE("gemma4 moe: the device-expert upload refuses where MatmulBTAlphaBeta has no arm") { + vt::Queue q; + q.device.type = vt::DeviceType::kCPU; + q.device.index = 0; + + // Precondition 1 — the arm really is absent for this queue. True on every + // build, HIP included: the only arm is kROCM's. + REQUIRE_FALSE(vt::HasMatmulBTAlphaBeta(q)); + + // Precondition 2 — the STOCK backend does not answer the memory probe, which + // is why the hazard is latent and why the decoration below is needed at all. + { + size_t free_b = 1, total_b = 1; + REQUIRE_FALSE(vt::GetBackend(vt::DeviceType::kCPU).DeviceMemoryInfo(&free_b, &total_b)); + } + + const vllm::Gemma4MoeLayerWeights moe = MakeMoeWeights(); + std::vector x(static_cast(kH)); + for (size_t i = 0; i < x.size(); ++i) + x[i] = vt::F32ToBF16(0.1f + 0.05f * static_cast(i % 5)); + + // Baseline: the layer as it runs today, host-fallback expert path. + const std::vector want = RunOneToken(q, moe, x); + REQUIRE(want.size() == static_cast(kH)); + // A layer that produced all zeros would make the equality below vacuous. + REQUIRE(AnyNonZero(want)); + + // Now put the tree in the post-#1126 state: the memory probe answers, so the + // LRU's `MakeRoom` will admit, and only the guard stands between the upload + // and a `vt::MatmulBTAlphaBeta` that has no arm for this device. + std::vector got; + { + vt::Backend& real = vt::GetBackend(vt::DeviceType::kCPU); + ProbingBackend probing(real); + const BackendSwap swap(probing); + + // Precondition 3 — the instrument is ARMED. Without this, a decoration that + // failed to take would leave the run on the same refuse-on-unknown path as + // the baseline and report the guard proven by a run that never reached it. + size_t free_b = 0, total_b = 0; + REQUIRE(vt::GetBackend(vt::DeviceType::kCPU).DeviceMemoryInfo(&free_b, &total_b)); + REQUIRE(free_b > (2ull << 30)); + + // THE ASSERTION. With the guard: the upload is refused, the expert loop + // takes `EnsureGemma4Fp8ExpertCached` + `ExpertGeGLUHost`, and the layer + // answers. Without it: `ExpertGeGLUTopKFusedGelu` reaches + // `vt::MatmulBTAlphaBeta` and this throws out of the decode step. + REQUIRE_NOTHROW(got = RunOneToken(q, moe, x)); + } + + // Degraded, but CORRECT: refusing the device arm must not change the answer. + REQUIRE(got.size() == want.size()); + int same = 0; + for (size_t i = 0; i < want.size(); ++i) { + if (got[i] == want[i]) ++same; + } + CHECK(same == static_cast(want.size())); + + // The refusal must not have leaked device pointers into the expert records: + // a `true` return is what assigns them, so they stay null. + int null_dev = 0; + for (const auto& fex : moe.experts.fp8) { + if (fex.dev_gu == nullptr && fex.dev_dn == nullptr) ++null_dev; + } + CHECK(null_dev == static_cast(kE)); + + // The backend registry is back to what it was. + size_t free_b = 1, total_b = 1; + CHECK_FALSE(vt::GetBackend(vt::DeviceType::kCPU).DeviceMemoryInfo(&free_b, &total_b)); +} diff --git a/tests/vt/test_gemma4_rocm_fp8_seams.cpp b/tests/vt/test_gemma4_rocm_fp8_seams.cpp index 105ebbfdf..dc698877c 100644 --- a/tests/vt/test_gemma4_rocm_fp8_seams.cpp +++ b/tests/vt/test_gemma4_rocm_fp8_seams.cpp @@ -84,9 +84,16 @@ TEST_CASE("gemma4 rocm fp8 seams: recipe env knobs parse inert defaults") { // tree is `rocm::MatmulBTAlphaBetaRocm` (rocm_matmul_hipblaslt.hip:516), reached // through a `#if defined(VLLM_CPP_HIP)` + `kROCM` guard in fused_ops.cpp; every // other device falls through to a throw. That is the actual blocker under #1126: -// adding `CudaBackend::DeviceMemoryInfo` alone would let the Gemma4 expert LRU -// admit (gemma4_moe.cpp:565,575), take ExpertGeGLUDeviceAccum at :1487, and reach -// this call at :90 — outside the upload's try/catch (:563-585), so mid-decode. +// adding `CudaBackend::DeviceMemoryInfo` alone would have let the Gemma4 expert +// LRU admit (gemma4_moe.cpp:587,597), take ExpertGeGLUDeviceAccum at :1509, and +// reach this call at :90 — outside the upload's try/catch (:585-607), so +// mid-decode. `EnsureGemma4Fp8ExpertOnDevice` now refuses that upload at :571 +// when `vt::HasMatmulBTAlphaBeta` is false, so the throw below is the backstop +// and not the guard; the capability gate for the refusal-before-upload is +// tests/vllm/models/test_gemma4_moe_device_arm_guard.cpp, which enters through +// `vllm::RunGemma4Moe`. What THIS file pins is the message contract a CUDA +// implementation has to satisfy — a unit contract, since with the guard in place +// no production run off ROCm reaches the throw. // // AGENTS.md requires an unimplemented arm to refuse with a message that NAMES the // missing part. "ROCm-only in this build" names neither the device that asked nor @@ -126,5 +133,67 @@ TEST_CASE("gemma4 rocm fp8 seams: MatmulBTAlphaBeta refuses a non-ROCm queue by INFO("refusal for " << std::string(vt::DeviceTypeName(type)) << ": " << msg); REQUIRE_FALSE(msg.empty()); CHECK(msg.find(vt::DeviceTypeName(type)) != std::string::npos); + CHECK(msg.find("1205") != std::string::npos); } + + // kROCM is the one device the four above make the message look right FOR and + // wrong ABOUT. A kROCM queue in a build without -DVLLM_CPP_HIP falls through + // the same `#if` and refuses too — but the 'rocm' kernel EXISTS and was + // compiled out, so telling that caller "no implementation ... see #1205" would + // send them to write a kernel the tree already has. The two absences get two + // messages, and this pins the difference in both directions. + { + vt::Queue rocm_q; + rocm_q.device.type = vt::DeviceType::kROCM; + rocm_q.device.index = 0; + // Build-agnostic: on a HIP build the arm is present and the call would run a + // real GEMM over these null pointers, so ask the same predicate the dispatch + // asks rather than repeating the `#if` here. + if (!vt::HasMatmulBTAlphaBeta(rocm_q)) { + const std::string rocm_msg = refusal_for(vt::DeviceType::kROCM); + REQUIRE_FALSE(rocm_msg.empty()); + INFO("rocm refusal: " << rocm_msg); + CHECK(rocm_msg.find("MatmulBTAlphaBeta") != std::string::npos); + CHECK(rocm_msg.find("rocm") != std::string::npos); + // Names the ABSENT BUILD FLAG, which is what is actually missing here. + CHECK(rocm_msg.find("VLLM_CPP_HIP") != std::string::npos); + // And does NOT claim the kernel is unwritten, nor point at the CUDA issue. + CHECK(rocm_msg.find("1205") == std::string::npos); + CHECK(rocm_msg.find("no implementation for device") == std::string::npos); + } + } +} + +// The predicate the guard in `EnsureGemma4Fp8ExpertOnDevice` keys on. It has to +// agree with the dispatch EXACTLY — a caller that trusts `true` and then throws +// is worse than no predicate — so this asserts the agreement over every device +// rather than reading the two conditions and calling them equal. +TEST_CASE("gemma4 rocm fp8 seams: HasMatmulBTAlphaBeta agrees with the dispatch") { + int checked = 0; + for (const auto type : {vt::DeviceType::kCPU, vt::DeviceType::kCUDA, vt::DeviceType::kROCM, + vt::DeviceType::kVULKAN, vt::DeviceType::kMETAL, + vt::DeviceType::kXPU}) { + vt::Queue q; + q.device.type = type; + q.device.index = 0; + const bool has = vt::HasMatmulBTAlphaBeta(q); + INFO("device " << std::string(vt::DeviceTypeName(type)) << " has=" << has); + if (has) { + // Do NOT call through: a present arm would run a real GEMM over null + // pointers. The absence direction is the one this guard depends on. + ++checked; + continue; + } + bool threw = false; + try { + vt::MatmulBTAlphaBeta(q, nullptr, nullptr, nullptr, /*M=*/1, /*N=*/1, /*K=*/1, + /*alpha=*/1.f, /*beta=*/0.f, vt::DType::kBF16); + } catch (const std::runtime_error&) { + threw = true; + } + CHECK(threw); + ++checked; + } + // A loop that ran zero times would report SUCCESS with nothing examined. + CHECK(checked == 6); } From ee2df78a78809019ed8c114309dd89993b83525b Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Tue, 18 Aug 2026 11:21:11 +0000 Subject: [PATCH 5/8] docs(ENG-EXPERT-STREAM): say why the guard test may re-register a backend after main, and name gemma4.cpp in full (#1205) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit FOLLOWING_AGENTS_PROTOCOL `vt::RegisterBackend` documents that all registration completes before `main()`, and the guard test calls it after. The contract buys the absence of synchronization on the read path, and this binary is single-threaded across the swap, so there is no concurrent reader to race — but a reviewer should not have to derive that, and an undiscussed contract violation reads as one that was not noticed. `gemma4.cpp` is also spelled in full. Two files in this tree carry that name and the shorter form resolved to the 123-line tool parser, where `:634` is out of range — the same class of defect as a stale line number. Comment-only: the diff contains no non-comment line. Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5-1m [Claude Code] --- .../test_gemma4_moe_device_arm_guard.cpp | 21 ++++++++++++++----- 1 file changed, 16 insertions(+), 5 deletions(-) diff --git a/tests/vllm/models/test_gemma4_moe_device_arm_guard.cpp b/tests/vllm/models/test_gemma4_moe_device_arm_guard.cpp index 5de60da6d..1c6a92f27 100644 --- a/tests/vllm/models/test_gemma4_moe_device_arm_guard.cpp +++ b/tests/vllm/models/test_gemma4_moe_device_arm_guard.cpp @@ -4,7 +4,8 @@ // // WHAT MAKES THIS A REACHABILITY GATE AND NOT A UNIT TEST. The guard is entered // through `vllm::RunGemma4Moe` — the Gemma-4 MoE layer entry point that -// `gemma4.cpp:634` calls — with a real `Gemma4MoeLayerWeights`, the real router, +// `src/vllm/model_executor/models/gemma4.cpp:634` calls — with a real +// `Gemma4MoeLayerWeights`, the real router, // the real device-expert LRU and the real fallback. Nothing here constructs the // LRU, the `Dev`, or `vt::MatmulBTAlphaBeta` by hand. Delete the `if // (!vt::HasMatmulBTAlphaBeta(d.q)) return false;` line from @@ -26,10 +27,20 @@ // under which deleting the guard is observable. // // The decoration is a real `vt::Backend` registered through the public -// `vt::RegisterBackend`, restored byte-for-byte in a destructor, and the test -// asserts BOTH directions of the probe (real backend: false; wrapped: true) -// before it concludes anything — an instrument that silently failed to arm would -// otherwise report the guard as proven when the run never reached it. +// `vt::RegisterBackend`, restored in a destructor, and the test asserts BOTH +// directions of the probe (real backend: false; wrapped: true) before it +// concludes anything — an instrument that silently failed to arm would otherwise +// report the guard as proven when the run never reached it. +// +// `vt::RegisterBackend` documents that all registration completes before +// `main()`, and this call is after it. What that contract buys is the absence of +// synchronization on the read path: the registry is a plain pointer store and +// lock-free reads thereafter. This binary is single-threaded across the swap — +// one doctest case, no queue with a worker, no other case touching the CPU +// backend — so there is no concurrent reader to race, and the pointer is put +// back before the case returns. It is the narrowest way to reach the state the +// guard exists for; the alternative is a ROCm device, which no host running this +// suite has. #include "vllm/model_executor/models/gemma4_moe.h" #include From e85eee14a15ad19bfbc68d3b7e909789dd4d36f7 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Tue, 18 Aug 2026 11:36:07 +0000 Subject: [PATCH 6/8] record(ENG-EXPERT-STREAM): re-derive the rocm_backend.hip anchor the merge moved, and check anchors semantically rather than only for range (#1205) FOLLOWING_AGENTS_PROTOCOL Merging origin/main moved `RocmBackend::DeviceMemoryInfo` from `:338-345` to `:358-365`, and five citations across the spec and the issue index still pointed at the old lines. That is this row's third stale-anchor defect, and the first one caused by somebody else's diff rather than its own. The range-only audit did not catch it. An anchor that is still IN RANGE can point at the wrong line, and a checker that only asks "does this line exist" reports OK for exactly the failure it was written to find. The audit is now two instruments: 168 endpoints resolved for range, and 69 load-bearing anchors asserted against the substring the prose claims lives there. The second one is what found this. Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5-1m [Claude Code] --- .agents/issue-index.md | 4 ++-- .agents/specs/expert-streaming.md | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/.agents/issue-index.md b/.agents/issue-index.md index fa79f6802..154243e2d 100644 --- a/.agents/issue-index.md +++ b/.agents/issue-index.md @@ -338,7 +338,7 @@ rather than merged. `scripts/check-agent-record.py` gates both. | [#1135](https://github.com/mudler/vllm.cpp/issues/1135) | `ENG-RESIDENCY-CONFIG` | `--offload-config` does not reach three entry points, and two of them are server-side. It is parsed once in `server_main.cpp`, AFTER the architecture resolution, so the server's POOLING/embedding path (the `if (pooling_model)` block) and its transcription-only path build their `EngineParams` without it, and `vllm-cli` has no such flag at all. Not specific to the `vllm_cpp` residency extension: the MIRRORED `uva`/`prefetch` half is dropped on the same two paths and has been since before that key existed, so the new key inherits a pre-existing gap rather than introducing one. An embedding or transcription server started with `--offload-config` therefore places weights as though the flag were absent and says nothing; for the residency half that is the 370 GiB case, where the difference is whether the process fits in host RAM at all. Fixing the server half means moving the offload parse ahead of the architecture branch, which is `ENG-WEIGHT-OFFLOAD`'s surface as much as this row's; `vllm-cli` having no flag is a deliberate scope line for [#1110](https://github.com/mudler/vllm.cpp/issues/1110) rather than a defect, recorded together so a reader need not rediscover which of the three is which. FILED because the gap was listed under `## Owed` against [#1122](https://github.com/mudler/vllm.cpp/issues/1122), the review issue [#1119](https://github.com/mudler/vllm.cpp/pull/1119) closes, so on landing it would have had no open issue (#1133 L8). Documented in `docs/USAGE.md` beside the config form. Listed under `## Owed` in [`weight-residency-config.md`](specs/weight-residency-config.md) | bug | | [#1123](https://github.com/mudler/vllm.cpp/issues/1123) | `ENG-EXPERT-STREAM` | `Qwen3.8-2.4T-A95B UD-Q1_0` (369.96 GiB) reached a serving state on `--device cuda` on a 119.631 GiB GB10 after 26 minutes and then died on the FIRST forward with `vt cuda: cudaMalloc: out of memory`, in the EngineCore busy loop rather than the loader, while the same checkpoint and binary serve on `--device cpu` (TTFT 667.0 s, 44.2 s/token, coherent). The log line could not name the allocation because `CudaBackend::Alloc` throws `"vt cuda: " + "cudaMalloc" + ": " + cudaGetErrorString(err)` and DISCARDS `bytes` (`src/vt/cuda/cuda_backend.cu:48-52,75-81`). Named by reading: `ResidentWeight` uploads the WHOLE stacked `[E*N,K]` keep-quant expert tower, `d.b.Alloc(w.bytes.size())` at `src/vllm/model_executor/models/qwen3_5.cpp:1011`, and BOTH switch positions reach it — the default grouped path (`MoeBlock:6615,6616,6620` -> `KqGrouped:5694`) and `VT_MOE_EXPERT_STREAM=1`, which disables grouping and then takes `KqExpertSlice:5595` -> `KqResidentSlice:5112` because the slot arm is guarded by `is_cpu()` (`:5578`). `BuildMoeMarlinResident` is NOT on this path (`MoeBlock:6555` needs `expert_*_fp4`, a GGUF populates `expert_*_kq`). Sized by re-censusing both GGUF tensor tables at revision `567d3e6ac26c5474b18311e619c04350fb9a5556` over all ten shards by HTTP range request, 1702 records parsed against 1702 declared in `split.tensors.count`: one IQ1_XXXS tower is **1,275,068,416 B (1.1875 GiB)**, the three Q2_K MTP-block towers are 2,818,572,288 B each, all 279 total **360,374,599,680 B = 335.62 GiB**, and `1,275,068,416 / 512 = 2,490,368` matches the W4 banner's `slot_bytes` exactly. Budget measured with the instrument that works where `nvidia-smi` answers `[N/A], [N/A], [N/A]`: `cudaMemGetInfo` on `dgx:gpu0` reports total `128,452,956,160` (119.631 GiB), free 113.677 GiB, `Integrated=1`. The load survives because a borrowed tower costs ZERO anonymous bytes; staging converts each into a real allocation, exhausting the pool after roughly 48 towers, partway through layer 16 of 93. FIXED here by a load-time refusal keyed on the measured condition (`needs_weight_staging` AND a known budget AND a LOWER-bound footprint above it), never on "CUDA + GGUF" and never on an architecture name, so a GGUF that fits still loads. The device-slot arm is NOT built and is [#1124](https://github.com/mudler/vllm.cpp/issues/1124) | bug | | [#1124](https://github.com/mudler/vllm.cpp/issues/1124) | `ENG-EXPERT-STREAM` | `--device cuda` still cannot SERVE a larger-than-pool GGUF after [#1123](https://github.com/mudler/vllm.cpp/issues/1123); it refuses by name instead of dying mid-stream. The missing capability is a DEVICE expert slot store, and it is four pieces: `HostExpertSlotStore` is the only production `ExpertSlotStore` (`include/vllm/model_executor/host_expert_slot_store.h:28`, the only other subclass being a test double) while `include/vllm/model_executor/expert_streamer.h:8-9,30-31` claims "the production destination is a contiguous device-side slot array" and is FALSE today; the interface has no device-capable read, because `KqExpertSlice` reads back through `HostExpertSlotStore::Slot()`, the CONCRETE class (`qwen3_5.cpp:5258,5314`); the filler is `pread`-into-host, since `SlotForWrite` is handed straight to `::pread` (`expert_streamer.cpp:76-94`); and the consumer is device-gated by `is_cpu()` at `qwen3_5.cpp:5578`. Sized: 2790 slices per token at 2,490,368 B is 6.95 GB per token against a 119.631 GiB pool already holding the dense remainder. Deferred because W7 owns the pluggable backing store in the row's work breakdown, and the CPU arm's own decode bandwidth is still VOID (#912 F1 measured it with the step clock dead from token 3), so a device lane would be optimised against a number nobody has. Listed under `## Owed` in [`expert-streaming.md`](specs/expert-streaming.md) | gap | -| [#1126](https://github.com/mudler/vllm.cpp/issues/1126) | `ENG-EXPERT-STREAM` | `CudaBackend` never overrides `Backend::DeviceMemoryInfo`, and the seam's own comment says it does: `include/vt/backend.h:79-83` reads "ROCm/CUDA override with hipMemGetInfo/cudaMemGetInfo" while only `src/vt/rocm/rocm_backend.hip:338-345` does, and `cudaMemGetInfo` is called NOWHERE in the repository. The comment is corrected in prose by [#1123](https://github.com/mudler/vllm.cpp/issues/1123); the capability is this issue. The consequence is not only a comment: `Gemma4MoE` is the seam's only consumer, `FreeBytes` returns false on an absent probe (`src/vllm/model_executor/models/gemma4_moe.cpp:439-447`) and `MakeRoom` refuses on unknown by design (`:494-506`), so on EVERY CUDA device the device-expert LRU (`kMaxSlots = 24`, `kHeadroom = 1.5 GiB`) admits nothing and falls back to host H2D permanently, silently. That polarity is right for that call site; the defect is the missing probe. #1123 therefore probed `cudaMemGetInfo` in `CudaPlatform` (which already includes `` and already probes attributes at registration) and carried the total on `ResidencyPolicy`, touching nothing Gemma4 reads, because adding the override wakes another model's residency policy and that needs its own measurement. Measured on `dgx:gpu0`: total 128,452,956,160 (119.631 GiB), free 113.677 GiB, against `nvidia-smi` answering `[N/A], [N/A], [N/A]`. Listed under `## Owed` in [`expert-streaming.md`](specs/expert-streaming.md) | bug | +| [#1126](https://github.com/mudler/vllm.cpp/issues/1126) | `ENG-EXPERT-STREAM` | `CudaBackend` never overrides `Backend::DeviceMemoryInfo`, and the seam's own comment says it does: `include/vt/backend.h:79-83` reads "ROCm/CUDA override with hipMemGetInfo/cudaMemGetInfo" while only `src/vt/rocm/rocm_backend.hip:358-365` does, and `cudaMemGetInfo` is called NOWHERE in the repository. The comment is corrected in prose by [#1123](https://github.com/mudler/vllm.cpp/issues/1123); the capability is this issue. The consequence is not only a comment: `Gemma4MoE` is the seam's only consumer, `FreeBytes` returns false on an absent probe (`src/vllm/model_executor/models/gemma4_moe.cpp:439-447`) and `MakeRoom` refuses on unknown by design (`:494-506`), so on EVERY CUDA device the device-expert LRU (`kMaxSlots = 24`, `kHeadroom = 1.5 GiB`) admits nothing and falls back to host H2D permanently, silently. That polarity is right for that call site; the defect is the missing probe. #1123 therefore probed `cudaMemGetInfo` in `CudaPlatform` (which already includes `` and already probes attributes at registration) and carried the total on `ResidencyPolicy`, touching nothing Gemma4 reads, because adding the override wakes another model's residency policy and that needs its own measurement. Measured on `dgx:gpu0`: total 128,452,956,160 (119.631 GiB), free 113.677 GiB, against `nvidia-smi` answering `[N/A], [N/A], [N/A]`. Listed under `## Owed` in [`expert-streaming.md`](specs/expert-streaming.md) | bug | | [#1127](https://github.com/mudler/vllm.cpp/issues/1127) | `ENG-EXPERT-STREAM` | `VT_DEVICE_WEIGHT_BUDGET_BYTES`, added by [#1123](https://github.com/mudler/vllm.cpp/issues/1123) to override the probed device memory pool for the load-time fit refusal, should be a weight-residency CONFIG key rather than an environment variable, for the reason `ENG-RESIDENCY-CONFIG` gives for the five `VT_GGUF_*` / `VT_MOE_EXPERT_STREAM*` knobs it is converting. It was left as an environment variable ON PURPOSE: [#1110](https://github.com/mudler/vllm.cpp/issues/1110) / PR #1119 is in flight, adds exactly the `vllm_cpp` namespace inside `--offload-config` this key belongs in plus `include/vllm/config/weight_residency.h`, and touches the same `src/vllm/entrypoints/model_loader.cpp`, so landing a competing config surface first would create the conflict both changes then resolve. Closing it means a `device_weight_budget_bytes` key under that object, the loader reading `EngineParams::weight_residency` instead of `std::getenv`, and `docs/ENVIRONMENT.md` plus `docs/USAGE.md` following. Listed under `## Owed` in [`expert-streaming.md`](specs/expert-streaming.md) | gap | | [#1136](https://github.com/mudler/vllm.cpp/issues/1136) | `ENG-EXPERT-STREAM` | The load-time GGUF fit bound of [#1123](https://github.com/mudler/vllm.cpp/issues/1123) can OVER-count, and therefore over-refuse. Per tensor it is `min(gguf_bytes, elems * model_dtype_bytes)`, a true lower bound on THAT tensor's staged size; the sum is not a lower bound on the load, because a tensor present in the file and never staged is a positive error. One such class exists on every default load: the MTP / `nextn` head is attached only under `params.speculative_config.has_value() && method == "mtp"` (`src/vllm/entrypoints/model_loader.cpp`, the `maybe_attach_mtp` GGUF arm), and the main model reads `block_count - nextn_predict_layers` blocks (`qwen3_5_gguf_weights.cpp:877-878`), so the head's blocks are outside its range. Measured on `unsloth/Qwen3.8-2.4T-A95B-GGUF` at `567d3e6ac26c5474b18311e619c04350fb9a5556`: block 92 is 20 tensors, **8,940,488,704 bytes (8.33 GiB) of 397,245,341,184**, so **2.2506 %**. A budget in `[what a default load stages, what the bound counts)` refuses a weight set that fits. The "under-count dominates" argument does NOT close this: the two errors are on different quantities and never cancel. NOT fixed, and the reason is the fix's own failure mode. Excluding those tensors means the bound taking a per-tensor staging POLICY as input, which is the caller's knowledge and not the file's, and an exclusion that is wrong under-counts toward zero — which restores exactly the 26-minute-load-then-`cudaMalloc`-OOM this row removed, on a device nobody on this fleet has to measure the change against. What IS done: `gguf_device_fit.h` states the direction, `tests/vllm/model_executor/test_gguf_device_fit.cpp` pins it executably with a counted-but-unstaged fixture and asserts both ends of the over-refusal window, `docs/USAGE.md` tells an operator that `VT_DEVICE_WEIGHT_BUDGET_BYTES` is the way out of it, and the spec's risk table records it. Also filed here rather than lost: the same review found `ResolveModelDeviceType` and `SelectQueueForModel` diverging on the AUTO arm (fixed in flow — the resolver now resolves through an attempted queue, pinned by two cases in `test_gguf_device_fit_reach.cpp`), the false "ROCm/CUDA override" comment in TWO places (both corrected in flow), and `CudaPlatform`'s policy assembly being reachable only in a CUDA build (extracted to `CudaResidencyPolicy` in `vllm/platforms/interface.h` and unit-tested on every host). Listed under `## Owed` in [`expert-streaming.md`](specs/expert-streaming.md) | bug | | [#1139](https://github.com/mudler/vllm.cpp/issues/1139) | `KV-WARMUP-PROFILE` | All three upstream anchors on that row (`.agents/engine-matrix.md:112`) point at unrelated code at the current parity pin `555967922`, verified by reading the pinned tree: `vllm/v1/worker/gpu/model_runner.py:504` is inside a `DraftModelSpeculator.set_attn(...)` call, `:647` is a `torch.zeros(...)` argument in a `dummy_run=True` construction, and `vllm/v1/worker/gpu_worker.py:430` is a comment about `max_split_size_mb` inside `load_model`. The startup memory profile the row describes is `GPUWorker.determine_available_memory` (`gpu_worker.py:451-495`, `memory_profiling` at `:491-494` around `profile_run()` at `:495`) and `GPUModelRunner.profile_run` (`gpu/model_runner.py:682`); `model_memory_usage` is recorded AFTER the load at `gpu/model_runner.py:315`, which is why upstream never asks whether the weights will fit and why [#1123](https://github.com/mudler/vllm.cpp/issues/1123) has no upstream counterpart to mirror. Found while repairing [#1136](https://github.com/mudler/vllm.cpp/issues/1136): `gguf_device_fit.h` and `expert-streaming.md` had both COPIED the `:504,647` pair from this row, and both are corrected there, so this row is the surviving source. Filed and not fixed in flow because the fix is one cell in `.agents/engine-matrix.md`, which PR #1119 ([#1110](https://github.com/mudler/vllm.cpp/issues/1110)) is concurrently bumping alongside the hardcoded `ENGINE` count in `scripts/check-agent-record.py` — the record-lock hazard AGENTS.md names, and the reason the repairing session was told to leave both files alone. Most likely cause: correct at the previous `e24d1b24` pin and not reconciled when the pin advanced; whether other `INVENTORIED` rows citing `vllm/v1/worker/gpu/**` share the defect is a wider sweep than one cell | bug | @@ -369,7 +369,7 @@ rather than merged. `scripts/check-agent-record.py` gates both. | [#1197](https://github.com/mudler/vllm.cpp/issues/1197) | `ENG-EXPERT-STREAM` | `Gemma4MoE`'s device-expert LRU tests its slot cap BEFORE its eviction loop, so the eviction opt-in goes inert once the cap is reached. `DevExpertLru::MakeRoom` runs `if (slots.size() >= kMaxSlots) return false;` at `src/vllm/model_executor/models/gemma4_moe.cpp:498` @ `fd64c76ee`, two lines ahead of the `if (allow_evict) { while (used + need > bud && !slots.empty()) EvictOne(d); }` at `:499-500`, and `EvictOne` (`:457`, the DEVICE LRU's — the file carries a host-cache namesake at `:275`) is the only thing that shrinks `slots`. So after 24 admissions every later `MakeRoom` returns false at that first line, the eviction loop is never reached again, and `VT_GEMMA4_EXPERT_EVICT=1` becomes a no-op for the life of the process — the cache degrades permanently to the fill-only mode the opt-in exists to leave. It binds only when the slot cap is reached before the byte budget, i.e. when `24 * expert_bytes < BudgetBytes()` (below ~85.3 MiB per expert at the 2048 MiB default from `BudgetBytes`, `:416-436`, again the device one and not the host cache's at `:262`); above that the byte budget binds first and eviction behaves. Nothing reports which one happened. FILED, NOT FIXED, and not for effort: the one-line repair (move the cap test after the eviction loop, so it caps RESIDENT slots instead of stopping admission forever) wakes more `hipFree` under load, which the code's own comments call a permanent `kfd_wait` hang with the GPU idle, prefill done and no decode tokens (`:459-461` and `:486-488`), so the current ordering may be deliberate. Deciding it needs the dual-RDNA4 lab box of [`gemma4-rocm-fp8-moe.md`](specs/gemma4-rocm-fp8-moe.md); the host that found it has neither a ROCm nor a CUDA device. Found while establishing the facts for [#1126](https://github.com/mudler/vllm.cpp/issues/1126), which required reading `MakeRoom` line by line. Listed under `## Owed` in [`expert-streaming.md`](specs/expert-streaming.md) | bug | | [#1205](https://github.com/mudler/vllm.cpp/issues/1205) | `ENG-EXPERT-STREAM` | `vt::MatmulBTAlphaBeta` has NO CUDA implementation, and that — not any of the four reasons [#1200](https://github.com/mudler/vllm.cpp/pull/1200) first recorded — is what [#1126](https://github.com/mudler/vllm.cpp/issues/1126) step 1 is blocked on. `src/vt/fused_ops.cpp:117` dispatches to `rocm::MatmulBTAlphaBetaRocm` (`src/vt/rocm/rocm_matmul_hipblaslt.hip:516`, declared `include/vt/rocm/rocm_matmul_batch.h:28`) under `#if defined(VLLM_CPP_HIP)` AND `q.device.type == kROCM` — `src/vt/fused_ops.cpp:111-112` is the signature, not the dispatch — and every other device falls through to a throw (`src/vt/fused_ops.cpp:152`). There is no CUDA, Vulkan, Metal or CPU arm: this is a missing kernel, not a missing build flag. #1126 proposes adding `CudaBackend::DeviceMemoryInfo` via `cudaMemGetInfo` so `Gemma4MoE`'s device-expert LRU stops refusing on CUDA; the record said that would wake a slower third path, and it would in fact wake a THROW, mid-decode. The chain: `EnsureGemma4Fp8ExpertOnDevice` (`src/vllm/model_executor/models/gemma4_moe.cpp:548-608`) -> `lru.MakeRoom` at `:587` succeeding once `FreeBytes` can answer -> `true` at `:597` -> the call site at `:1508` -> `ExpertGeGLUDeviceAccum` at `:1509` -> `vt::MatmulBTAlphaBeta` at `:90` -> throw. The `try`/`catch (...)` at `:585-607` wraps only the UPLOAD; the compute at `:1509` is outside it, so the exception leaves the decode step rather than degrading to the host fallback. Latent today only because the other route in, `same_dev` (`:752-753`), needs `ex.gate_up_dev`, assigned nowhere but `src/vt/rocm/rocm_gemma4_experts.hip:207,226` — so the resident arm is UNREACHABLE off ROCm rather than safe. FIXED IN FLOW, PARTLY. (a) The HAZARD is guarded: `EnsureGemma4Fp8ExpertOnDevice` refuses at `src/vllm/model_executor/models/gemma4_moe.cpp:571` when `vt::HasMatmulBTAlphaBeta(d.q)` is false, BEFORE the upload, so the caller takes the host fallback already sitting in the `else` at `:1515-1521` and the decode step answers instead of throwing. The predicate (`src/vt/fused_ops.cpp:102-109`) is the same condition the dispatch at `:117` uses, so the two cannot drift. Gated by `tests/vllm/models/test_gemma4_moe_device_arm_guard.cpp`, which enters through `vllm::RunGemma4Moe` and decorates the registered CPU backend so `DeviceMemoryInfo` answers — the post-#1126 state, and the only one in which the guard binds; deleting the guard turns it RED. (b) The REFUSAL now satisfies the rule that an unimplemented arm names the missing part. "ROCm-only in this build" named neither the device that asked nor where an implementation would go; a non-ROCm queue now gets all three plus this issue (`src/vt/fused_ops.cpp:152`), and a kROCM queue in a build without `-DVLLM_CPP_HIP` gets a different message naming the absent FLAG (`src/vt/fused_ops.cpp:138`), because for that caller the kernel exists. `tests/vt/test_gemma4_rocm_fp8_seams.cpp` gates both on posed queues — CUDA, `kCPU`, `kVULKAN`, `kMETAL` and kROCM — mutation-proven RED by restoring the old message, RED by deleting the refusal outright, and RED by deleting the kROCM branch. That message gate is a UNIT CONTRACT, not observable behaviour: with the guard in place the throw is unreachable off ROCm, which is the point of the guard. The IMPLEMENTATION stays open and is what this issue tracks: a `beta`-accumulating BT GEMM on cuBLASLt is a kernel with its own correctness gate, no per-expert FP8 Gemma-4 checkpoint is pinned to exercise it on, and the host that found this has neither a ROCm nor a CUDA device. The same file refuses six further arms the same way (`MatmulBTFp8Channel`, `DequantFp8ChannelBf16`, `MoeGatherRows`, `MoeWeightedScatterAdd`, `MoeZeroBf16`, `DualRmsNormPlusRes`); only `MatmulBTAlphaBeta` is on the #1126 path, and the others are named so the next reader need not re-derive the list. Listed under `## Owed` in [`expert-streaming.md`](specs/expert-streaming.md) | bug | | [#1193](https://github.com/mudler/vllm.cpp/issues/1193) | `SPEC-DSPARK-QWEN3-ROUTING` | A Qwen3 DSpark draft declaring `architectures=["DSparkDraftModel"]` with `model_type` `qwen3` has no route. The pin forces every DSpark draft that is not `Qwen3DSparkModel` or `Gemma4DSparkModel` onto `model_type` `deepseek_v4` (`vllm/config/speculative.py:934-944` @ `555967922`), and vLLM PR 52197 (merged 2026-08-17 at `7075ddac`) replaced that with a leading branch normalizing the pair to `Qwen3DSparkModel`. We diverge from BOTH: the forced rewrite was never ported, so nothing in `src/vllm/entrypoints/model_loader.cpp` reads a draft config's `architectures` key at all, and `SpeculativeConfig::IsDsparkDraft` (`include/vllm/config/speculative.h:120-136`) has no production caller — every reference outside its header is in `tests/vllm/config/test_speculative_dspark.cpp:132-140`, and `ResolveSpecConfig` branches on `cli.method` alone. The checkpoint is real and gateable here: `RadixArk/Qwen3.8-27B-DSpark` at revision `85ef153be924f17ce4bf62726954eeaa4a73e854` carries exactly that config shape in one 2718576122-byte shard, drafting five layers for a 64-layer Qwen3.8-27B target | bug | -| [#1218](https://github.com/mudler/vllm.cpp/issues/1218) | `ENG-EXPERT-STREAM` | `EnsureGemma4Fp8NativeOnDevice` has no arm-existence guard, and it is the DEFAULT Gemma-4 expert arm, so [#1126](https://github.com/mudler/vllm.cpp/issues/1126) step 1 would still throw mid-decode after [#1205](https://github.com/mudler/vllm.cpp/issues/1205)'s guard lands. That guard covers the BF16 device-expert arm (`src/vllm/model_executor/models/gemma4_moe.cpp:571`, inside `:548-608`). The FP8-native twin at `:611` has none, and `VT_GEMMA4_FP8_NATIVE` returns true when unset (`:969-974`), so on a per-expert FP8 checkpoint the expert loop reaches the twin at `:1359` and `:1484` BEFORE the guarded arm. A `true` from it routes into `ExpertGeGLUFp8Native` (`:95-130`), which needs `vt::ExpertGeGLUFp8TopKM1` (`:105`, which merely returns false off ROCm), `vt::DequantFp8ChannelBf16` (`:117`, `:119`, refusing at `src/vt/fused_ops.cpp:194`) and `vt::MatmulBTAlphaBeta` (`gemma4_moe.cpp:128`, refusing at `src/vt/fused_ops.cpp:152`) — so T>1 throws at the first dequant and T==1 falls through the fused kernel and throws too, in both cases outside the upload's own `try`/`catch`. Latent for the same reason and for exactly as long: the twin's `MakeRoom` also needs `vt::Backend::DeviceMemoryInfo`, which only ROCm overrides (`src/vt/rocm/rocm_backend.hip:338-345`). FILED, NOT FIXED, and not for effort: the BF16 guard keys on ONE predicate that is the same condition its own dispatch uses, which is what makes it honest, whereas the twin depends on three different ops and needs a predicate per op — reusing `HasMatmulBTAlphaBeta` there would be a guard naming the wrong arm, the exact defect this row's review had just corrected in a refusal message. Found while repairing [#1200](https://github.com/mudler/vllm.cpp/pull/1200). Listed under `## Owed` in [`expert-streaming.md`](specs/expert-streaming.md) | bug | +| [#1218](https://github.com/mudler/vllm.cpp/issues/1218) | `ENG-EXPERT-STREAM` | `EnsureGemma4Fp8NativeOnDevice` has no arm-existence guard, and it is the DEFAULT Gemma-4 expert arm, so [#1126](https://github.com/mudler/vllm.cpp/issues/1126) step 1 would still throw mid-decode after [#1205](https://github.com/mudler/vllm.cpp/issues/1205)'s guard lands. That guard covers the BF16 device-expert arm (`src/vllm/model_executor/models/gemma4_moe.cpp:571`, inside `:548-608`). The FP8-native twin at `:611` has none, and `VT_GEMMA4_FP8_NATIVE` returns true when unset (`:969-974`), so on a per-expert FP8 checkpoint the expert loop reaches the twin at `:1359` and `:1484` BEFORE the guarded arm. A `true` from it routes into `ExpertGeGLUFp8Native` (`:95-130`), which needs `vt::ExpertGeGLUFp8TopKM1` (`:105`, which merely returns false off ROCm), `vt::DequantFp8ChannelBf16` (`:117`, `:119`, refusing at `src/vt/fused_ops.cpp:194`) and `vt::MatmulBTAlphaBeta` (`gemma4_moe.cpp:128`, refusing at `src/vt/fused_ops.cpp:152`) — so T>1 throws at the first dequant and T==1 falls through the fused kernel and throws too, in both cases outside the upload's own `try`/`catch`. Latent for the same reason and for exactly as long: the twin's `MakeRoom` also needs `vt::Backend::DeviceMemoryInfo`, which only ROCm overrides (`src/vt/rocm/rocm_backend.hip:358-365`). FILED, NOT FIXED, and not for effort: the BF16 guard keys on ONE predicate that is the same condition its own dispatch uses, which is what makes it honest, whereas the twin depends on three different ops and needs a predicate per op — reusing `HasMatmulBTAlphaBeta` there would be a guard naming the wrong arm, the exact defect this row's review had just corrected in a refusal message. Found while repairing [#1200](https://github.com/mudler/vllm.cpp/pull/1200). Listed under `## Owed` in [`expert-streaming.md`](specs/expert-streaming.md) | bug | | [#1184](https://github.com/mudler/vllm.cpp/issues/1184) | `ENG-CUDAGRAPH-DEDUP` | `VT_CUDA_GRAPH_DEDUP=1` cannot complete a single decode step once a CUDA graph is actually captured. Measured 6/6 deterministic on GB10: `vt graph dedup: captured 1 graphs, deduped to 1 execs`, then `[Qwen3DenseDecodeGraph] captured dense decode graph for padded size S=8`, then `engine-fatal: EngineCore busy loop threw: vt cuda: greedy_argmax launch: invalid device function` -- from a `greedy_argmax` launch that had SUCCEEDED. The OFF and `=0` arms are clean and byte-identical across 7 runs on the identical binary and libraries, which is what makes the asymmetry arm-attributable. `greedy_argmax` is not at fault. `src/vt/graph_dedup_runtime.h` @ `3ed6253c6` walks the retained raw graph with CUDA runtime APIs and is DESIGNED to see several of them fail -- the `cudaGraphExecUpdate` probe refusing a fold is the feature working, not an exception, and the topology walk has five more escapes that degrade the key rather than abort inside a capture -- but it never consumed the runtime's sticky per-thread error: `grep -n 'cudaGetLastError\|hipGetLastError' src/vt/graph_dedup_runtime.h` returned nothing. Twelve sites swallowed a non-success return without clearing (`:98,100,115,130,137,146,154,173,179,281,307,317`); the two that mattered are the probe (`:307,:317`), whose refusal is routine, and `cudaGraphInstantiate` (`:281`). The next unrelated kernel, launched with the ordinary `kernel<<<>>>(); Check(cudaGetLastError())` pattern, then reported our refusal as its own failure. Every symptom follows: it needs BOTH `dedup=1` and a real capture, `CUDA_LAUNCH_BLOCKING=1` does not move it because the latch is host-side and synchronous rather than a deferred async error, and `cudaGraphLaunch` returns success because a return value does not consume the latch. Fixed in flow, structurally rather than site-by-site: twelve hand-placed clears are a fix the thirteenth fallible call misses, so the clear lives in `ScopedLatchClear`'s destructor (`src/vt/graph_dedup_latch.h`, new) installed at the binding's entry points, which are exactly the six `GraphDedupOps` members; `MakeLatchGuardedOps` is the table's only constructor and takes the raw functions as template arguments, so no raw address reaches a field and a seventh operation wired elsewhere leaves its field null, which the registry constructor refuses. One line covers CUDA and HIP because there is one source. The coverage gap that hid it is closed in the same change: the device-free half of the signature walk moves to `src/vt/graph_dedup_signature.h` and is gated by the new `tests/vt/test_graph_dedup_runtime.cpp` (13 cases, 51 assertions, 7/7 negative mutations detected) -- Kahn ordering, topological re-index, sorted edge emission, the depth-4 child bound and the four graph-level escapes, none of which was reached by any test on any tier before. STATED LIMIT: a CPU test drives a fake runtime and cannot observe the real latched error, so it proves the guard's structure and not the device outcome; the device A/B re-run is owed under `## Owed` in [`eng-cudagraph-dedup.md`](specs/eng-cudagraph-dedup.md), which also records that the shipped async serving path never reaches this feature at all | bug | | [#1192](https://github.com/mudler/vllm.cpp/issues/1192) | `ENG-CUDAGRAPH-BREAK` | W1 of the break-point capture seam: `vt::BreakableGraph`, `vt::GraphCaptureScope` and `vt::GraphBreak` land in `include/vt/breakable_graph.h` + `src/vt/breakable_graph.cpp`, with the SGLang unit suite ported case for case and ONE break point registered on ONE model. **The stage's exit criterion was answered first, on a leased GPU, because W0 deliberately did not answer it**: `cudaStreamEndCapture` followed by `cudaStreamBeginCapture` on the SAME stream mid-forward with EAGER work between them is LEGAL under `cudaStreamCaptureModeThreadLocal`, our backend's mode (`src/vt/cuda/cuda_backend.cu:204-206`) — measured on `orin:gpu0` through an `rc` lease, driver 12060, three replays with fresh inputs and 0 mismatches, plus the bare zero-work re-begin (`breakable_cuda_graph.py:370-374`). The first probe REFUSED and that refusal was the probe's, not CUDA's: `dlsym` on the bare symbol name binds the LEGACY v1 entry points, which are not capture-aware, so `cuMemcpyDtoDAsync` returned `CUDA_ERROR_STREAM_CAPTURE_UNSUPPORTED` and `cuStreamBeginCapture` (v1) silently took no capture-mode argument — a broken instrument failing toward a verdict about the code. Bound by exact versioned name, the criterion holds. The break point is the DENSE ATTENTION ENTRY of `Qwen3ForCausalLM` (`src/vllm/model_executor/models/qwen3.cpp`, inside `RunLayer`), which is vLLM's boundary and not one this row invents (`splitting_ops` defaulted to the attention family, `vllm/config/compilation.py:517,764-772,1145` @ `5559679229`) registered in SGLang's form because we have no compiler to match operation names against (`layers/radix_attention.py:256` @ `f63458b5be`). The G2 reachability gate is `tests/vllm/models/test_qwen3_break_point.cpp`: it drives the production `Qwen3DenseModel::Forward` with a scope open and counts `num_hidden_layers + 1` segments, so deleting the one call site takes it to 1 and turns the gate RED — a unit test that constructs the container by hand could not have asked that. It also holds G4 in the same case: the logits are compared BIT FOR BIT against the unscoped forward, 500 values, 0 differing. **STAGED SLICE, named rather than implied:** `GraphCaptureScope` and `BreakableGraph` are not yet ENTERED from a production step, because no driver opens a scope until W2 migrates `Qwen3DenseDecodeGraph`; the break point itself is on the production path on every forward and takes the pass-through arm. Listed under `## Owed` in [`eng-cudagraph-break.md`](specs/eng-cudagraph-break.md). The capture-failure drain is NOT owed: it landed here, as behaviour and as three gated arms. The destructor `catch` alone covered a throwing `EndCaptureGraph` and nothing else, so a break function or ordinary model code throwing mid-capture left a container reporting `captured() == true` over a PARTIAL forward; comparing `std::uncaught_exceptions()` against the depth recorded at scope entry is what tells those apart, and tests 13a, 13b and 13c gate the three arms. Still owed by later stages and NOT claimed here: the auxiliary-stream auto-join before every segment close (spec D10, live at `qwen3_5.cpp:6254-6255,6384` and `laguna.cpp:2572-2576,2612`, which W4 and W5 migrate), the non-capturing-backend arm on ROCm and Tenstorrent (G5), G1 bit-exactness on a real GPU over more than one replay, and one residual the fresh review surfaced: an exception CAUGHT INSIDE the capture scope leaves the segment closed and the rest of the forward uncaptured while `captured()` stays true, which the drain cannot see because nothing is unwinding at scope exit (spec `## Owed`, W2). No throughput claim is made anywhere in this change: the prefill refutation stands (GB10 3.8% host-idle between launches, GPU-busy above 96%, 27B prefill gap 92.5% non-GEMM glue) | feature | | [#1189](https://github.com/mudler/vllm.cpp/issues/1189) | `VT-QUANT-FP8-GROUP` | Block-wise (128x128) FP8 so `Qwen/Qwen3.8-27B-FP8` runs instead of being refused. `weight_block_size` appears nowhere in `src/` or `include/`, so `469f38395` refuses the arm by name (#1166). Six independently landable milestones; M1 lands here. M1 is `vt::QuantFp8Group`, the dynamic per-token per-group activation quant, CPU and CUDA. The numerics mirror the kernel that ACTUALLY EXECUTES on a CUDA-alike platform with a contiguous input, which is the C++ custom op at `csrc/libtorch_stable/quantization/w8a8/fp8/per_token_group_quant.cu:42-96` and NOT the Triton kernel at `fp8_utils.py:95-150`: `fp8_utils.py:635-650` calls the former and returns before the latter. The two arms differ, and the difference is measurable rather than cosmetic. The CUDA kernel divides twice, `local_absmax / max_8bit` at `:68` and `static_cast(src) / y_s` at `:85`; the Triton kernel multiplies by `(1.0 / fp8_max)` at `fp8_utils.py:145` under a comment that names the 1-ULP gap. Upstream's own test tolerates the gap with `rtol=0.15` (`test_block_fp8.py:112-114`), so a value comparison cannot tell the two apart and only a byte comparison against a spelled-out reference can. `eps` is the reduction's INITIAL value (`:47`), not a post-clamp, which is what keeps an all-zero group from dividing by zero. Scope refused here and owed to later milestones: the block-scaled GEMM (M2), `Fp8BlockWeight` and the loader (M3), `Fp8BlockLinearMethod` and the Qwen3.5 wiring (M4), the mainloop-scaled CUTLASS kernel and the column-major/TMA-aligned scale layouts (M5), merged `gate_up`/QKV (M6). M1 lands UNREACHED: no production entry point dispatches `vt::QuantFp8Group` at its merge commit, M4 owns the wiring, and `.agents/specs/vt-quant-fp8-group.md` lists it under `## Owed`. The CUDA arm compiles and its on-hardware leg is owed too, because the row took no GPU lease by design: the CPU arm is the gateable one | feature | diff --git a/.agents/specs/expert-streaming.md b/.agents/specs/expert-streaming.md index cb2b2636c..705e021b7 100644 --- a/.agents/specs/expert-streaming.md +++ b/.agents/specs/expert-streaming.md @@ -1556,7 +1556,7 @@ The probe is added to `CudaPlatform`, which already includes `` and already probes device attributes at registration, and NOT to `Backend::DeviceMemoryInfo`. That seam's comment claimed "ROCm/CUDA override with hipMemGetInfo/cudaMemGetInfo" and only ROCm does -(`src/vt/rocm/rocm_backend.hip:338-345`). The comment is **corrected in this +(`src/vt/rocm/rocm_backend.hip:358-365`). The comment is **corrected in this change**, in the two places that carried it: `include/vt/backend.h:78-93` on the seam, and `gemma4_moe.cpp:440-448` on the only call site — the second copy was found by this round's audit and is why the first correction alone would have left @@ -1606,7 +1606,7 @@ Carried debt for this row. Each item names why it is not closed here. | **`--device cuda` still cannot SERVE a larger-than-pool GGUF; it only refuses by name now.** The device-slot arm is the missing capability: a `DeviceExpertSlotStore` behind `ExpertSlotStore`, a read accessor on that interface (`KqExpertSlice` reads `HostExpertSlotStore::Slot()`, the CONCRETE class, so the seam cannot be swapped today), a device filler that is not `pread`-into-host (`ExpertSlotStore::SlotForWrite` is handed straight to `::pread`, `expert_streamer.cpp:76-94`), and lifting the `is_cpu()` guard at `qwen3_5.cpp:5578`. Sized by the measurement above: 2790 slices per token at 2,490,368 bytes is 6.95 GB per token against a 119.631 GiB pool that already holds the dense remainder. Tracked as [#1124](https://github.com/mudler/vllm.cpp/issues/1124). | It is a campaign, not a fix: W7 (the pluggable backing store) is its declared owner in the work breakdown, and the CPU arm's own I/O rate is still unmeasured on a live cache two rows above. Building a device lane on top of a host lane whose bandwidth number is void would be optimising against a number nobody has. | | **The fit bound omits everything that is not a weight.** KV cache, activations, the scratch pools and the CUDA context are not counted, so a checkpoint at 0.95x of the pool passes the refusal and still dies on the first forward. | A headroom fraction invented here would be exactly the guess the per-tensor bound exists to avoid. The number wants the startup memory profile that `KV-WARMUP-PROFILE` owns (`INVENTORIED`; upstream's is `GPUWorker.determine_available_memory`, `vllm/v1/worker/gpu_worker.py:451-495`, around `profile_run`, `vllm/v1/worker/gpu/model_runner.py:682`), which is a different row. Those two anchors are stated here from the pinned tree because that row's own three anchors are stale at the current pin, and `gguf_device_fit.h` had copied two of them — filed as [#1139](https://github.com/mudler/vllm.cpp/issues/1139), owned by `KV-WARMUP-PROFILE`, blocked here only by the `engine-matrix.md` record lock #1119 holds. | | **The fit bound also counts too MUCH, and that direction can refuse a load that fits.** A tensor present in the file and not staged by THIS load is a positive over-count. On a default load that is the MTP / `nextn` block: 8,940,488,704 bytes, 8.33 GiB, 2.2506 % of the target checkpoint. A budget in that window refuses a weight set that would have fitted. | Not closed here. Closing it means the bound taking a per-tensor staging POLICY as input, which is the caller's knowledge and not the file's, and the exclusion's own failure mode is an under-count to nothing — which restores the 26-minute-then-OOM this row exists to remove, on a device nobody here has to measure it on. So the direction is stated in `gguf_device_fit.h`, pinned executably by `test_gguf_device_fit`, exposed to operators in `docs/USAGE.md`, and tracked as [#1136](https://github.com/mudler/vllm.cpp/issues/1136). `VT_DEVICE_WEIGHT_BUDGET_BYTES` is the way out of the window in the meantime. | -| **`Backend::DeviceMemoryInfo` has no CUDA override, and waking it is not the one-line port of the ROCm one that #1126 describes: on CUDA it would wake a THROW.** Only ROCm implements it (`src/vt/rocm/rocm_backend.hip:338-345`), so `Gemma4MoE`'s device-expert LRU refuses on every CUDA device (`gemma4_moe.cpp:506`). Where a per-expert FP8 checkpoint is present, the expert's BF16 bytes are re-copied HOST->DEVICE on every use instead (`ExpertGeGLUHost`, `gemma4_moe.cpp:49-74`, reached at `:1515-1521`; the H2D is `:59-60` and it drains the queue per expert at `:73`), silently and for the life of the process. That H2D cost is CONDITIONAL, not present-tense: the whole device LRU is `ex.is_fp8`-gated (`:991`, `:1506`), and by point (1) below no such Gemma-4 checkpoint is pinned anywhere, so on CUDA today the LRU-MISS fallback is never even asked for. (A BF16 Gemma-4 checkpoint reaches `ExpertGeGLUHost` at `:1525` too, but through the host-weight branch the LRU never governs, so it is not a cost of the missing probe.) The name misleads: it computes on the DEVICE from host-resident weights, so what the dead LRU would cost is bandwidth and a per-expert `Synchronize`, not a wrong answer. | The CAPABILITY is [#1126](https://github.com/mudler/vllm.cpp/issues/1126) and is still not built. The false COMMENT was corrected by this row in both places that carried it, `include/vt/backend.h:78-93` and `gemma4_moe.cpp:440-448`; both anchors were re-verified exact against `fd64c76ee`, as were `rocm_backend.hip:338-345`, `gemma4_moe.cpp:506`, `platforms/cuda.cpp:71` and `platforms/rocm.cpp:74`. SCOPE, because the previous headline read as coverage it did not have. The first review repair re-audited every anchor cited by the FOUR `## Owed` rows in this cluster — this one, the #1126 step-3 row, the #1197 row and the #1205 row — against the repaired tree: 53 examined, 53 exact, 0 stale. It audited those four rows and nothing else. The GitHub ISSUE BODIES were never in the audited set, and #1205's body was in fact 11 lines stale at `7beada17c` for exactly that reason. The second review repair widened the set: it re-derived every anchor in those four rows AND in the #1197 and #1205 issue bodies AND in the #1205 index row against the final tree — 71 examined, 71 exact, 0 stale after repair. Anchors from `gemma4_moe.cpp:549` onward moved by 22 lines in that repair, because the arm-existence guard it added sits at `:571`. **Four things were established while re-reading it for #1126, and each one raises the price of the override.** (1) *There is nothing to run it on.* The LRU needs `ex.is_fp8`, which only `LoadMoeFp8PerExpert` sets (`gemma4_weights.cpp:210-215`) from a per-expert `F8_E4M3` export; no such Gemma-4 checkpoint is pinned anywhere in `docs/USAGE.md` — all 32 lines there matching `gemma` case-insensitively were swept, and every checkpoint among them is the LTX-2.5 text tower (`gemma4-12b-with-proj*.safetensors`), not a per-expert FP8 MoE decoder, so the woken path cannot be exercised, here or elsewhere, until one is. (2) *The device-resident arm has no CUDA implementation to route into. It has a throw.* `ExpertGeGLUDeviceAccum` (`gemma4_moe.cpp:76-93`) READS as generic — `vt::MatmulBT`, `GeluAndMul`, `vt::MatmulBTAlphaBeta` — and an earlier draft of this row concluded from that reading that it "would run". It does not. `vt::MatmulBTAlphaBeta` (`src/vt/fused_ops.cpp:111-157`, dispatching at `:117`) is guarded on `#if defined(VLLM_CPP_HIP)` AND `q.device.type == kROCM`; its only implementation in the tree is `rocm::MatmulBTAlphaBetaRocm` (`src/vt/rocm/rocm_matmul_hipblaslt.hip:516`, declared `include/vt/rocm/rocm_matmul_batch.h:28`), and every other device falls through to a refusal. So the chain the override WOULD wake is: `EnsureGemma4Fp8ExpertOnDevice` (`gemma4_moe.cpp:548-608`) -> `lru.MakeRoom` (`:587`) succeeding as soon as `FreeBytes` can answer -> `true` at `:597` -> the call site at `:1508` -> `ExpertGeGLUDeviceAccum` at `:1509` -> `vt::MatmulBTAlphaBeta` at `:90` -> THROW, mid-decode. The `try`/`catch (...)` at `:585-607` wraps only the UPLOAD; the compute at `:1509` sits outside it, so the exception would propagate out of the decode step rather than degrading to the host fallback. **That chain is now cut at its first link.** `EnsureGemma4Fp8ExpertOnDevice` refuses at `:571` when `vt::HasMatmulBTAlphaBeta(d.q)` is false, BEFORE the upload, so the caller takes the `else` at `:1515-1521` — `EnsureGemma4Fp8ExpertCached` plus `ExpertGeGLUHost` — and the step answers instead of throwing. The refusal at `:90` stays as the backstop. It is latent today only because the other route into that function, `same_dev` (`:752-753`), needs `ex.gate_up_dev`, which is assigned nowhere but `src/vt/rocm/rocm_gemma4_experts.hip:207,226` — so the resident arm is UNREACHABLE off ROCm rather than safe. This is the actual blocker under #1126, it was recorded nowhere, and it is a stronger argument than the other three: filed as [#1205](https://github.com/mudler/vllm.cpp/issues/1205), and the refusal itself is now gated by `tests/vt/test_gemma4_rocm_fp8_seams.cpp`, which is what a CUDA implementation will have to satisfy. **Two corrections to the earlier draft's supporting claims, both of which overstated the case.** *The HIP-only list was one symbol too long.* Three of the four are genuinely HIP-only stubs inside `gemma4_moe.cpp`'s ONLY `#ifndef VLLM_CPP_HIP` block (`gemma4_moe.cpp:1596-1650`): `RunGemma4FusedTopkExpertGeGLU` (`:1621`), `PeerCopyGemma4Fp8ExpertSlice` (`:1629`) and `RunGemma4Fp8TopKOnExpertDevice` (`:1633`). `ExpertGeGLUDeviceBatched` (`:240`) is NOT: it sits OUTSIDE that block, in an anonymous namespace, with no HIP implementation and no header declaration, and is unconditionally `return false` under its own lab note (`:237-239` — gather+strided produced wrong tokens at ~23 t/s, pointer-batch ~0.8 t/s, serial/fused-gelu kept at ~34 t/s). It is disabled EVERYWHERE, ROCm included, so naming it beside the three inflated the ROCm/CUDA asymmetry. *The token-neutrality argument had the wrong mechanism.* The conclusion stands — the swap would not be token-neutral — but not because "the two arms sum the top-k experts in a different order". They do not: both run inside the SAME `for (int i = 0; i < top_k; ++i)` at `:1453`, so the summation order is identical. The real difference is ROUNDING and where the routing weight is applied. The resident arm folds `ww` into the GEMM `alpha` and accumulates in the epilogue with `beta` (`:1456`, `:1464`, `:90`), so the weight multiplies in the GEMM's own accumulator. The fallback writes the UNWEIGHTED product to a BF16 buffer (`:67`, `:70`) and applies `ww` afterwards with separate BF16 kernels — `vt::MulScalar` at `:1546` on the first expert, `MulScalar` plus `vt::Add` at `:1548-1549` on the rest. Two extra BF16 roundings per expert, with the weight applied post-rounding. (3) *The headroom test does not mean the same thing on the CUDA device this project gates on.* `MakeRoom` admits iff `free_b >= need + 1.5 GiB` (`:514`), a constant tuned on discrete dual R9700s where free VRAM is a quantity distinct from host RAM. On a GB10 it is not. This row already measured that `cudaMemGetInfo`'s `total` there is EXACTLY `/proc/meminfo MemTotal` times 1024, which is why it reached for that instrument where `nvidia-smi` answers `[N/A]`; its `free` is therefore reported over the same unified pool, and the host BF16 expert cache the device upload exists to relieve (`ex.cached_gu`/`cached_dn` via `EnsureGemma4Fp8ExpertCached`, bounded by the host LRU at `gemma4_moe.cpp:352`) is drawn from that pool too. So the admission test would double-count, on a box whose unified-memory OOM takes the host down with it. The `free` half is an inference from the measured `total`, not a second measurement, and it wants confirming on the device before any override lands. (4) *The gap is isolated, not a pattern.* Comparing overrides one by one across `src/vt/cuda/cuda_backend.cu` and `src/vt/rocm/rocm_backend.hip` at `fd64c76ee`, `DeviceMemoryInfo` is the ONLY optional `vt::Backend` seam ROCm answers and CUDA does not. | +| **`Backend::DeviceMemoryInfo` has no CUDA override, and waking it is not the one-line port of the ROCm one that #1126 describes: on CUDA it would wake a THROW.** Only ROCm implements it (`src/vt/rocm/rocm_backend.hip:358-365`), so `Gemma4MoE`'s device-expert LRU refuses on every CUDA device (`gemma4_moe.cpp:506`). Where a per-expert FP8 checkpoint is present, the expert's BF16 bytes are re-copied HOST->DEVICE on every use instead (`ExpertGeGLUHost`, `gemma4_moe.cpp:49-74`, reached at `:1515-1521`; the H2D is `:59-60` and it drains the queue per expert at `:73`), silently and for the life of the process. That H2D cost is CONDITIONAL, not present-tense: the whole device LRU is `ex.is_fp8`-gated (`:991`, `:1506`), and by point (1) below no such Gemma-4 checkpoint is pinned anywhere, so on CUDA today the LRU-MISS fallback is never even asked for. (A BF16 Gemma-4 checkpoint reaches `ExpertGeGLUHost` at `:1525` too, but through the host-weight branch the LRU never governs, so it is not a cost of the missing probe.) The name misleads: it computes on the DEVICE from host-resident weights, so what the dead LRU would cost is bandwidth and a per-expert `Synchronize`, not a wrong answer. | The CAPABILITY is [#1126](https://github.com/mudler/vllm.cpp/issues/1126) and is still not built. The false COMMENT was corrected by this row in both places that carried it, `include/vt/backend.h:78-93` and `gemma4_moe.cpp:440-448`; both anchors were re-verified exact against `fd64c76ee`, as were `rocm_backend.hip:358-365`, `gemma4_moe.cpp:506`, `platforms/cuda.cpp:71` and `platforms/rocm.cpp:74`. SCOPE, because the previous headline read as coverage it did not have. The first review repair re-audited every anchor cited by the FOUR `## Owed` rows in this cluster — this one, the #1126 step-3 row, the #1197 row and the #1205 row — against the repaired tree: 53 examined, 53 exact, 0 stale. It audited those four rows and nothing else. The GitHub ISSUE BODIES were never in the audited set, and #1205's body was in fact 11 lines stale at `7beada17c` for exactly that reason. The second review repair widened the set: it re-derived every anchor in those four rows AND in the #1197 and #1205 issue bodies AND in the #1205 index row against the final tree — 71 examined, 71 exact, 0 stale after repair. Anchors from `gemma4_moe.cpp:549` onward moved by 22 lines in that repair, because the arm-existence guard it added sits at `:571`. **Four things were established while re-reading it for #1126, and each one raises the price of the override.** (1) *There is nothing to run it on.* The LRU needs `ex.is_fp8`, which only `LoadMoeFp8PerExpert` sets (`gemma4_weights.cpp:210-215`) from a per-expert `F8_E4M3` export; no such Gemma-4 checkpoint is pinned anywhere in `docs/USAGE.md` — all 32 lines there matching `gemma` case-insensitively were swept, and every checkpoint among them is the LTX-2.5 text tower (`gemma4-12b-with-proj*.safetensors`), not a per-expert FP8 MoE decoder, so the woken path cannot be exercised, here or elsewhere, until one is. (2) *The device-resident arm has no CUDA implementation to route into. It has a throw.* `ExpertGeGLUDeviceAccum` (`gemma4_moe.cpp:76-93`) READS as generic — `vt::MatmulBT`, `GeluAndMul`, `vt::MatmulBTAlphaBeta` — and an earlier draft of this row concluded from that reading that it "would run". It does not. `vt::MatmulBTAlphaBeta` (`src/vt/fused_ops.cpp:111-157`, dispatching at `:117`) is guarded on `#if defined(VLLM_CPP_HIP)` AND `q.device.type == kROCM`; its only implementation in the tree is `rocm::MatmulBTAlphaBetaRocm` (`src/vt/rocm/rocm_matmul_hipblaslt.hip:516`, declared `include/vt/rocm/rocm_matmul_batch.h:28`), and every other device falls through to a refusal. So the chain the override WOULD wake is: `EnsureGemma4Fp8ExpertOnDevice` (`gemma4_moe.cpp:548-608`) -> `lru.MakeRoom` (`:587`) succeeding as soon as `FreeBytes` can answer -> `true` at `:597` -> the call site at `:1508` -> `ExpertGeGLUDeviceAccum` at `:1509` -> `vt::MatmulBTAlphaBeta` at `:90` -> THROW, mid-decode. The `try`/`catch (...)` at `:585-607` wraps only the UPLOAD; the compute at `:1509` sits outside it, so the exception would propagate out of the decode step rather than degrading to the host fallback. **That chain is now cut at its first link.** `EnsureGemma4Fp8ExpertOnDevice` refuses at `:571` when `vt::HasMatmulBTAlphaBeta(d.q)` is false, BEFORE the upload, so the caller takes the `else` at `:1515-1521` — `EnsureGemma4Fp8ExpertCached` plus `ExpertGeGLUHost` — and the step answers instead of throwing. The refusal at `:90` stays as the backstop. It is latent today only because the other route into that function, `same_dev` (`:752-753`), needs `ex.gate_up_dev`, which is assigned nowhere but `src/vt/rocm/rocm_gemma4_experts.hip:207,226` — so the resident arm is UNREACHABLE off ROCm rather than safe. This is the actual blocker under #1126, it was recorded nowhere, and it is a stronger argument than the other three: filed as [#1205](https://github.com/mudler/vllm.cpp/issues/1205), and the refusal itself is now gated by `tests/vt/test_gemma4_rocm_fp8_seams.cpp`, which is what a CUDA implementation will have to satisfy. **Two corrections to the earlier draft's supporting claims, both of which overstated the case.** *The HIP-only list was one symbol too long.* Three of the four are genuinely HIP-only stubs inside `gemma4_moe.cpp`'s ONLY `#ifndef VLLM_CPP_HIP` block (`gemma4_moe.cpp:1596-1650`): `RunGemma4FusedTopkExpertGeGLU` (`:1621`), `PeerCopyGemma4Fp8ExpertSlice` (`:1629`) and `RunGemma4Fp8TopKOnExpertDevice` (`:1633`). `ExpertGeGLUDeviceBatched` (`:240`) is NOT: it sits OUTSIDE that block, in an anonymous namespace, with no HIP implementation and no header declaration, and is unconditionally `return false` under its own lab note (`:237-239` — gather+strided produced wrong tokens at ~23 t/s, pointer-batch ~0.8 t/s, serial/fused-gelu kept at ~34 t/s). It is disabled EVERYWHERE, ROCm included, so naming it beside the three inflated the ROCm/CUDA asymmetry. *The token-neutrality argument had the wrong mechanism.* The conclusion stands — the swap would not be token-neutral — but not because "the two arms sum the top-k experts in a different order". They do not: both run inside the SAME `for (int i = 0; i < top_k; ++i)` at `:1453`, so the summation order is identical. The real difference is ROUNDING and where the routing weight is applied. The resident arm folds `ww` into the GEMM `alpha` and accumulates in the epilogue with `beta` (`:1456`, `:1464`, `:90`), so the weight multiplies in the GEMM's own accumulator. The fallback writes the UNWEIGHTED product to a BF16 buffer (`:67`, `:70`) and applies `ww` afterwards with separate BF16 kernels — `vt::MulScalar` at `:1546` on the first expert, `MulScalar` plus `vt::Add` at `:1548-1549` on the rest. Two extra BF16 roundings per expert, with the weight applied post-rounding. (3) *The headroom test does not mean the same thing on the CUDA device this project gates on.* `MakeRoom` admits iff `free_b >= need + 1.5 GiB` (`:514`), a constant tuned on discrete dual R9700s where free VRAM is a quantity distinct from host RAM. On a GB10 it is not. This row already measured that `cudaMemGetInfo`'s `total` there is EXACTLY `/proc/meminfo MemTotal` times 1024, which is why it reached for that instrument where `nvidia-smi` answers `[N/A]`; its `free` is therefore reported over the same unified pool, and the host BF16 expert cache the device upload exists to relieve (`ex.cached_gu`/`cached_dn` via `EnsureGemma4Fp8ExpertCached`, bounded by the host LRU at `gemma4_moe.cpp:352`) is drawn from that pool too. So the admission test would double-count, on a box whose unified-memory OOM takes the host down with it. The `free` half is an inference from the measured `total`, not a second measurement, and it wants confirming on the device before any override lands. (4) *The gap is isolated, not a pattern.* Comparing overrides one by one across `src/vt/cuda/cuda_backend.cu` and `src/vt/rocm/rocm_backend.hip` at `fd64c76ee`, `DeviceMemoryInfo` is the ONLY optional `vt::Backend` seam ROCm answers and CUDA does not. | | **#1126's own closing plan, step 3, must be NARROWED before it is done: the load-time fit check may read the seam's `total`, and never its `free`.** The issue proposes that "the #1123 fit check can then read the budget from the backend seam on every platform that reports one". As written that invites the live half, which would be a defect — but the seam returns BOTH halves (`bool DeviceMemoryInfo(size_t* free_bytes, size_t* total_bytes)`, `include/vt/backend.h:94`), and only one of them is illegitimate here. | The tree holds two answers to "how much device memory", and they answer different questions. `vt::Backend::DeviceMemoryInfo(free, total)` is a LIVE probe that moves with contention; it is authoritative for a RUNTIME admission decision — can this allocation succeed right now — and for nothing else. `vllm::platforms::ResidencyPolicy::device_memory_total_bytes` is a TOTAL probed once at platform registration with `0 == UNKNOWN`; it is authoritative for a LOAD-TIME budget verdict, which has to be reproducible and independent of whatever else the box is doing. Sourcing the load-time verdict from `free` would make the same checkpoint load or be refused depending on the page cache, which is precisely the property **`total`, not `free`** above was chosen to avoid. Reading `total` through the seam is not that. It is contention-independent, it is the same quantity `ResidencyPolicy` already carries, and a seam that answers it on every platform is a defensible place to source it from. What step 3 must additionally preserve is the PROBE-ONCE semantics: `device_memory_total_bytes` is probed at platform registration (`include/vllm/platforms/interface.h:70-72`), and a per-load live call would reintroduce the contention dependence by the back door even reading only `total`, because a load-time verdict has to be reproducible from the record rather than from the moment. So the defensible statement, and the one this row asserts: **step 3 may read only `total`, never `free`, and must keep the value probed once at platform registration.** An earlier draft of this row said "never step 3" outright; that overstated it and would have blocked a legitimate simplification, so it is corrected here rather than quietly narrowed. Both seams already carry the division in prose (`include/vt/backend.h:90-93` and `include/vllm/platforms/interface.h:61-69`); it is restated here because #1126 is the record a reader of that issue will act on, and as filed it points the other way. What #1126 owes is its steps 1 and 2 together — the override AND the Gemma4 measurement, with [#1205](https://github.com/mudler/vllm.cpp/issues/1205) ahead of both — never step 1 alone, and step 3 only in the narrowed form above. | | **The device-expert LRU's slot cap makes its own eviction opt-in inert.** `MakeRoom` tests `slots.size() >= kMaxSlots` (`gemma4_moe.cpp:498`) BEFORE the eviction loop (`:499-500`), and `EvictOne` (`:457`, the device LRU's — a host-cache namesake sits at `:275`) is the only thing that SHRINKS `slots`. The one other statement that touches its size, `slots.clear()` in `DevExpertLru::Note` (`:522`), is a device-index RESET rather than an eviction: it drops bookkeeping when `dev != d.q.device.index` and frees nothing, and it is unreachable in a single-device process. It is named here so the next reader does not conclude the #1197 sweep missed it. So once 24 slots are resident `VT_GEMMA4_EXPERT_EVICT=1` never runs again and the cache degrades permanently to fill-only. It binds only when `24 * expert_bytes < BudgetBytes()`, so it is condition-dependent and silent either way. Tracked as [#1197](https://github.com/mudler/vllm.cpp/issues/1197). | Filed, not fixed, and for the same reason as the row above rather than for effort: the one-line repair wakes more `hipFree` under load, which the surrounding comments say has been observed as a permanent `kfd_wait` hang with the GPU idle and no decode tokens. The current ordering may well be deliberate belt-and-braces. Deciding that needs the dual-RDNA4 box `.agents/specs/gemma4-rocm-fp8-moe.md` describes; this host has neither a ROCm nor a CUDA device. It closes when the cap moves after the eviction loop and a run stays hang-free, or when the comment says the cap is by design — one of the two, not silence. | | **`vt::MatmulBTAlphaBeta` is ROCm-only and has no CUDA implementation at all, which is what #1126 step 1 is actually blocked on.** `src/vt/fused_ops.cpp:117` dispatches to `rocm::MatmulBTAlphaBetaRocm` (`src/vt/rocm/rocm_matmul_hipblaslt.hip:516`) under `#if defined(VLLM_CPP_HIP)` and `q.device.type == kROCM` — `src/vt/fused_ops.cpp:111-112` is the signature, not the dispatch — and every other device falls through to the refusal at `src/vt/fused_ops.cpp:152`. There is no CUDA, Vulkan, Metal or CPU arm. The full chain from the missing `DeviceMemoryInfo` override to that refusal is traced in the first row above. Tracked as [#1205](https://github.com/mudler/vllm.cpp/issues/1205). | The REFUSAL is fixed in flow, because a bare `std::runtime_error` reading "ROCm-only in this build" does not satisfy the standing rule that an unimplemented arm refuses with a message NAMING the missing part: a caller who hits it on CUDA cannot tell a missing kernel from a missing build flag. It now names the device that asked, names the one arm that exists, and names the issue (`:152`), and a kROCM queue — which reaches the same line in a build configured without `-DVLLM_CPP_HIP` — gets a DIFFERENT message naming the absent build flag (`:138`), because for that caller the kernel exists and telling them to write one would send them to fix the wrong thing. `tests/vt/test_gemma4_rocm_fp8_seams.cpp` gates both messages on a posed CUDA queue, on `kCPU`/`kVULKAN`/`kMETAL`, and on kROCM — mutation-proven by restoring the old message (RED), by deleting the refusal outright (RED), and by deleting the kROCM branch so that case falls to the generic message (RED). **Say plainly what that message change does and does not pin: a contract in a unit test, not observable behaviour.** The throw is unreachable off ROCm in any shipped configuration, so no production run can print either string today; what the test fixes is what a CUDA implementation has to satisfy when someone writes one. **The reachable half of this row is the GUARD.** `EnsureGemma4Fp8ExpertOnDevice` refuses at `gemma4_moe.cpp:571` when `vt::HasMatmulBTAlphaBeta(d.q)` is false, BEFORE the upload rather than after it, which converts the mid-decode exception traced above into the host fallback that was already sitting in the `else` at `:1515-1521`: slower, two extra BF16 roundings per expert, and correct. The predicate (`include/vt/fused_ops.h`, defined `src/vt/fused_ops.cpp:102-109`) is the same condition the dispatch at `:117` uses rather than a second copy of it, so the two cannot drift and writing the CUDA kernel wakes the device arm with no edit at the call site. It is gated by `tests/vllm/models/test_gemma4_moe_device_arm_guard.cpp`, which enters through `vllm::RunGemma4Moe` — the production layer entry `src/vllm/model_executor/models/gemma4.cpp:634` calls — and decorates the registered CPU backend so `DeviceMemoryInfo` ANSWERS, which is the post-#1126 state and the only state in which the guard binds at all. Deleting the guard makes that test RED with the exact `no implementation for device 'cpu'` throw; forcing `HasMatmulBTAlphaBeta` to `true` makes it RED too. A test that constructed the `Dev` or the LRU by hand would have stayed green under both. The IMPLEMENTATION stays owed and is what [#1205](https://github.com/mudler/vllm.cpp/issues/1205) tracks. It is not written here: a `beta`-accumulating BT GEMM on cuBLASLt is a kernel with its own correctness gate, the `DeviceMemoryInfo` row's point (1) above says there is no checkpoint to exercise it on, and this host has neither a ROCm nor a CUDA device to measure either arm. | From 2188c88daec70c7b1f464830c46583fd8da5b20e Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Tue, 18 Aug 2026 12:08:23 +0000 Subject: [PATCH 7/8] fix(ENG-EXPERT-STREAM): put the merged #1126 index row back, byte-for-byte (#1205) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit FOLLOWING_AGENTS_PROTOCOL Re-deriving the moved `rocm_backend.hip` anchor was done with a blanket string replace over `.agents/issue-index.md`, and it hit the `#1126` row as well as this branch's own. That row is already on `main`. Editing a merged row is exactly what the append-only rule forbids, because the union driver DUPLICATES an edited line instead of merging it — the gate caught it, and it was right to. The row is restored from `origin/main`, and the file now removes or edits nothing that main has: it adds three rows, #1197, #1205 and #1218, all appended by this branch. Those three are still editable because they have not merged, so no other branch can hold a different version of them. The merged `#1126` row keeps its pre-move anchor deliberately. One stale number in an append-only log is a smaller problem than a duplicated row, and the spec's `## Owed` entry carries the current anchor and says so. Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5-1m [Claude Code] --- .agents/issue-index.md | 2 +- .agents/specs/expert-streaming.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/.agents/issue-index.md b/.agents/issue-index.md index 154243e2d..a2b50dd32 100644 --- a/.agents/issue-index.md +++ b/.agents/issue-index.md @@ -338,7 +338,7 @@ rather than merged. `scripts/check-agent-record.py` gates both. | [#1135](https://github.com/mudler/vllm.cpp/issues/1135) | `ENG-RESIDENCY-CONFIG` | `--offload-config` does not reach three entry points, and two of them are server-side. It is parsed once in `server_main.cpp`, AFTER the architecture resolution, so the server's POOLING/embedding path (the `if (pooling_model)` block) and its transcription-only path build their `EngineParams` without it, and `vllm-cli` has no such flag at all. Not specific to the `vllm_cpp` residency extension: the MIRRORED `uva`/`prefetch` half is dropped on the same two paths and has been since before that key existed, so the new key inherits a pre-existing gap rather than introducing one. An embedding or transcription server started with `--offload-config` therefore places weights as though the flag were absent and says nothing; for the residency half that is the 370 GiB case, where the difference is whether the process fits in host RAM at all. Fixing the server half means moving the offload parse ahead of the architecture branch, which is `ENG-WEIGHT-OFFLOAD`'s surface as much as this row's; `vllm-cli` having no flag is a deliberate scope line for [#1110](https://github.com/mudler/vllm.cpp/issues/1110) rather than a defect, recorded together so a reader need not rediscover which of the three is which. FILED because the gap was listed under `## Owed` against [#1122](https://github.com/mudler/vllm.cpp/issues/1122), the review issue [#1119](https://github.com/mudler/vllm.cpp/pull/1119) closes, so on landing it would have had no open issue (#1133 L8). Documented in `docs/USAGE.md` beside the config form. Listed under `## Owed` in [`weight-residency-config.md`](specs/weight-residency-config.md) | bug | | [#1123](https://github.com/mudler/vllm.cpp/issues/1123) | `ENG-EXPERT-STREAM` | `Qwen3.8-2.4T-A95B UD-Q1_0` (369.96 GiB) reached a serving state on `--device cuda` on a 119.631 GiB GB10 after 26 minutes and then died on the FIRST forward with `vt cuda: cudaMalloc: out of memory`, in the EngineCore busy loop rather than the loader, while the same checkpoint and binary serve on `--device cpu` (TTFT 667.0 s, 44.2 s/token, coherent). The log line could not name the allocation because `CudaBackend::Alloc` throws `"vt cuda: " + "cudaMalloc" + ": " + cudaGetErrorString(err)` and DISCARDS `bytes` (`src/vt/cuda/cuda_backend.cu:48-52,75-81`). Named by reading: `ResidentWeight` uploads the WHOLE stacked `[E*N,K]` keep-quant expert tower, `d.b.Alloc(w.bytes.size())` at `src/vllm/model_executor/models/qwen3_5.cpp:1011`, and BOTH switch positions reach it — the default grouped path (`MoeBlock:6615,6616,6620` -> `KqGrouped:5694`) and `VT_MOE_EXPERT_STREAM=1`, which disables grouping and then takes `KqExpertSlice:5595` -> `KqResidentSlice:5112` because the slot arm is guarded by `is_cpu()` (`:5578`). `BuildMoeMarlinResident` is NOT on this path (`MoeBlock:6555` needs `expert_*_fp4`, a GGUF populates `expert_*_kq`). Sized by re-censusing both GGUF tensor tables at revision `567d3e6ac26c5474b18311e619c04350fb9a5556` over all ten shards by HTTP range request, 1702 records parsed against 1702 declared in `split.tensors.count`: one IQ1_XXXS tower is **1,275,068,416 B (1.1875 GiB)**, the three Q2_K MTP-block towers are 2,818,572,288 B each, all 279 total **360,374,599,680 B = 335.62 GiB**, and `1,275,068,416 / 512 = 2,490,368` matches the W4 banner's `slot_bytes` exactly. Budget measured with the instrument that works where `nvidia-smi` answers `[N/A], [N/A], [N/A]`: `cudaMemGetInfo` on `dgx:gpu0` reports total `128,452,956,160` (119.631 GiB), free 113.677 GiB, `Integrated=1`. The load survives because a borrowed tower costs ZERO anonymous bytes; staging converts each into a real allocation, exhausting the pool after roughly 48 towers, partway through layer 16 of 93. FIXED here by a load-time refusal keyed on the measured condition (`needs_weight_staging` AND a known budget AND a LOWER-bound footprint above it), never on "CUDA + GGUF" and never on an architecture name, so a GGUF that fits still loads. The device-slot arm is NOT built and is [#1124](https://github.com/mudler/vllm.cpp/issues/1124) | bug | | [#1124](https://github.com/mudler/vllm.cpp/issues/1124) | `ENG-EXPERT-STREAM` | `--device cuda` still cannot SERVE a larger-than-pool GGUF after [#1123](https://github.com/mudler/vllm.cpp/issues/1123); it refuses by name instead of dying mid-stream. The missing capability is a DEVICE expert slot store, and it is four pieces: `HostExpertSlotStore` is the only production `ExpertSlotStore` (`include/vllm/model_executor/host_expert_slot_store.h:28`, the only other subclass being a test double) while `include/vllm/model_executor/expert_streamer.h:8-9,30-31` claims "the production destination is a contiguous device-side slot array" and is FALSE today; the interface has no device-capable read, because `KqExpertSlice` reads back through `HostExpertSlotStore::Slot()`, the CONCRETE class (`qwen3_5.cpp:5258,5314`); the filler is `pread`-into-host, since `SlotForWrite` is handed straight to `::pread` (`expert_streamer.cpp:76-94`); and the consumer is device-gated by `is_cpu()` at `qwen3_5.cpp:5578`. Sized: 2790 slices per token at 2,490,368 B is 6.95 GB per token against a 119.631 GiB pool already holding the dense remainder. Deferred because W7 owns the pluggable backing store in the row's work breakdown, and the CPU arm's own decode bandwidth is still VOID (#912 F1 measured it with the step clock dead from token 3), so a device lane would be optimised against a number nobody has. Listed under `## Owed` in [`expert-streaming.md`](specs/expert-streaming.md) | gap | -| [#1126](https://github.com/mudler/vllm.cpp/issues/1126) | `ENG-EXPERT-STREAM` | `CudaBackend` never overrides `Backend::DeviceMemoryInfo`, and the seam's own comment says it does: `include/vt/backend.h:79-83` reads "ROCm/CUDA override with hipMemGetInfo/cudaMemGetInfo" while only `src/vt/rocm/rocm_backend.hip:358-365` does, and `cudaMemGetInfo` is called NOWHERE in the repository. The comment is corrected in prose by [#1123](https://github.com/mudler/vllm.cpp/issues/1123); the capability is this issue. The consequence is not only a comment: `Gemma4MoE` is the seam's only consumer, `FreeBytes` returns false on an absent probe (`src/vllm/model_executor/models/gemma4_moe.cpp:439-447`) and `MakeRoom` refuses on unknown by design (`:494-506`), so on EVERY CUDA device the device-expert LRU (`kMaxSlots = 24`, `kHeadroom = 1.5 GiB`) admits nothing and falls back to host H2D permanently, silently. That polarity is right for that call site; the defect is the missing probe. #1123 therefore probed `cudaMemGetInfo` in `CudaPlatform` (which already includes `` and already probes attributes at registration) and carried the total on `ResidencyPolicy`, touching nothing Gemma4 reads, because adding the override wakes another model's residency policy and that needs its own measurement. Measured on `dgx:gpu0`: total 128,452,956,160 (119.631 GiB), free 113.677 GiB, against `nvidia-smi` answering `[N/A], [N/A], [N/A]`. Listed under `## Owed` in [`expert-streaming.md`](specs/expert-streaming.md) | bug | +| [#1126](https://github.com/mudler/vllm.cpp/issues/1126) | `ENG-EXPERT-STREAM` | `CudaBackend` never overrides `Backend::DeviceMemoryInfo`, and the seam's own comment says it does: `include/vt/backend.h:79-83` reads "ROCm/CUDA override with hipMemGetInfo/cudaMemGetInfo" while only `src/vt/rocm/rocm_backend.hip:338-345` does, and `cudaMemGetInfo` is called NOWHERE in the repository. The comment is corrected in prose by [#1123](https://github.com/mudler/vllm.cpp/issues/1123); the capability is this issue. The consequence is not only a comment: `Gemma4MoE` is the seam's only consumer, `FreeBytes` returns false on an absent probe (`src/vllm/model_executor/models/gemma4_moe.cpp:439-447`) and `MakeRoom` refuses on unknown by design (`:494-506`), so on EVERY CUDA device the device-expert LRU (`kMaxSlots = 24`, `kHeadroom = 1.5 GiB`) admits nothing and falls back to host H2D permanently, silently. That polarity is right for that call site; the defect is the missing probe. #1123 therefore probed `cudaMemGetInfo` in `CudaPlatform` (which already includes `` and already probes attributes at registration) and carried the total on `ResidencyPolicy`, touching nothing Gemma4 reads, because adding the override wakes another model's residency policy and that needs its own measurement. Measured on `dgx:gpu0`: total 128,452,956,160 (119.631 GiB), free 113.677 GiB, against `nvidia-smi` answering `[N/A], [N/A], [N/A]`. Listed under `## Owed` in [`expert-streaming.md`](specs/expert-streaming.md) | bug | | [#1127](https://github.com/mudler/vllm.cpp/issues/1127) | `ENG-EXPERT-STREAM` | `VT_DEVICE_WEIGHT_BUDGET_BYTES`, added by [#1123](https://github.com/mudler/vllm.cpp/issues/1123) to override the probed device memory pool for the load-time fit refusal, should be a weight-residency CONFIG key rather than an environment variable, for the reason `ENG-RESIDENCY-CONFIG` gives for the five `VT_GGUF_*` / `VT_MOE_EXPERT_STREAM*` knobs it is converting. It was left as an environment variable ON PURPOSE: [#1110](https://github.com/mudler/vllm.cpp/issues/1110) / PR #1119 is in flight, adds exactly the `vllm_cpp` namespace inside `--offload-config` this key belongs in plus `include/vllm/config/weight_residency.h`, and touches the same `src/vllm/entrypoints/model_loader.cpp`, so landing a competing config surface first would create the conflict both changes then resolve. Closing it means a `device_weight_budget_bytes` key under that object, the loader reading `EngineParams::weight_residency` instead of `std::getenv`, and `docs/ENVIRONMENT.md` plus `docs/USAGE.md` following. Listed under `## Owed` in [`expert-streaming.md`](specs/expert-streaming.md) | gap | | [#1136](https://github.com/mudler/vllm.cpp/issues/1136) | `ENG-EXPERT-STREAM` | The load-time GGUF fit bound of [#1123](https://github.com/mudler/vllm.cpp/issues/1123) can OVER-count, and therefore over-refuse. Per tensor it is `min(gguf_bytes, elems * model_dtype_bytes)`, a true lower bound on THAT tensor's staged size; the sum is not a lower bound on the load, because a tensor present in the file and never staged is a positive error. One such class exists on every default load: the MTP / `nextn` head is attached only under `params.speculative_config.has_value() && method == "mtp"` (`src/vllm/entrypoints/model_loader.cpp`, the `maybe_attach_mtp` GGUF arm), and the main model reads `block_count - nextn_predict_layers` blocks (`qwen3_5_gguf_weights.cpp:877-878`), so the head's blocks are outside its range. Measured on `unsloth/Qwen3.8-2.4T-A95B-GGUF` at `567d3e6ac26c5474b18311e619c04350fb9a5556`: block 92 is 20 tensors, **8,940,488,704 bytes (8.33 GiB) of 397,245,341,184**, so **2.2506 %**. A budget in `[what a default load stages, what the bound counts)` refuses a weight set that fits. The "under-count dominates" argument does NOT close this: the two errors are on different quantities and never cancel. NOT fixed, and the reason is the fix's own failure mode. Excluding those tensors means the bound taking a per-tensor staging POLICY as input, which is the caller's knowledge and not the file's, and an exclusion that is wrong under-counts toward zero — which restores exactly the 26-minute-load-then-`cudaMalloc`-OOM this row removed, on a device nobody on this fleet has to measure the change against. What IS done: `gguf_device_fit.h` states the direction, `tests/vllm/model_executor/test_gguf_device_fit.cpp` pins it executably with a counted-but-unstaged fixture and asserts both ends of the over-refusal window, `docs/USAGE.md` tells an operator that `VT_DEVICE_WEIGHT_BUDGET_BYTES` is the way out of it, and the spec's risk table records it. Also filed here rather than lost: the same review found `ResolveModelDeviceType` and `SelectQueueForModel` diverging on the AUTO arm (fixed in flow — the resolver now resolves through an attempted queue, pinned by two cases in `test_gguf_device_fit_reach.cpp`), the false "ROCm/CUDA override" comment in TWO places (both corrected in flow), and `CudaPlatform`'s policy assembly being reachable only in a CUDA build (extracted to `CudaResidencyPolicy` in `vllm/platforms/interface.h` and unit-tested on every host). Listed under `## Owed` in [`expert-streaming.md`](specs/expert-streaming.md) | bug | | [#1139](https://github.com/mudler/vllm.cpp/issues/1139) | `KV-WARMUP-PROFILE` | All three upstream anchors on that row (`.agents/engine-matrix.md:112`) point at unrelated code at the current parity pin `555967922`, verified by reading the pinned tree: `vllm/v1/worker/gpu/model_runner.py:504` is inside a `DraftModelSpeculator.set_attn(...)` call, `:647` is a `torch.zeros(...)` argument in a `dummy_run=True` construction, and `vllm/v1/worker/gpu_worker.py:430` is a comment about `max_split_size_mb` inside `load_model`. The startup memory profile the row describes is `GPUWorker.determine_available_memory` (`gpu_worker.py:451-495`, `memory_profiling` at `:491-494` around `profile_run()` at `:495`) and `GPUModelRunner.profile_run` (`gpu/model_runner.py:682`); `model_memory_usage` is recorded AFTER the load at `gpu/model_runner.py:315`, which is why upstream never asks whether the weights will fit and why [#1123](https://github.com/mudler/vllm.cpp/issues/1123) has no upstream counterpart to mirror. Found while repairing [#1136](https://github.com/mudler/vllm.cpp/issues/1136): `gguf_device_fit.h` and `expert-streaming.md` had both COPIED the `:504,647` pair from this row, and both are corrected there, so this row is the surviving source. Filed and not fixed in flow because the fix is one cell in `.agents/engine-matrix.md`, which PR #1119 ([#1110](https://github.com/mudler/vllm.cpp/issues/1110)) is concurrently bumping alongside the hardcoded `ENGINE` count in `scripts/check-agent-record.py` — the record-lock hazard AGENTS.md names, and the reason the repairing session was told to leave both files alone. Most likely cause: correct at the previous `e24d1b24` pin and not reconciled when the pin advanced; whether other `INVENTORIED` rows citing `vllm/v1/worker/gpu/**` share the defect is a wider sweep than one cell | bug | diff --git a/.agents/specs/expert-streaming.md b/.agents/specs/expert-streaming.md index 705e021b7..232114652 100644 --- a/.agents/specs/expert-streaming.md +++ b/.agents/specs/expert-streaming.md @@ -1606,7 +1606,7 @@ Carried debt for this row. Each item names why it is not closed here. | **`--device cuda` still cannot SERVE a larger-than-pool GGUF; it only refuses by name now.** The device-slot arm is the missing capability: a `DeviceExpertSlotStore` behind `ExpertSlotStore`, a read accessor on that interface (`KqExpertSlice` reads `HostExpertSlotStore::Slot()`, the CONCRETE class, so the seam cannot be swapped today), a device filler that is not `pread`-into-host (`ExpertSlotStore::SlotForWrite` is handed straight to `::pread`, `expert_streamer.cpp:76-94`), and lifting the `is_cpu()` guard at `qwen3_5.cpp:5578`. Sized by the measurement above: 2790 slices per token at 2,490,368 bytes is 6.95 GB per token against a 119.631 GiB pool that already holds the dense remainder. Tracked as [#1124](https://github.com/mudler/vllm.cpp/issues/1124). | It is a campaign, not a fix: W7 (the pluggable backing store) is its declared owner in the work breakdown, and the CPU arm's own I/O rate is still unmeasured on a live cache two rows above. Building a device lane on top of a host lane whose bandwidth number is void would be optimising against a number nobody has. | | **The fit bound omits everything that is not a weight.** KV cache, activations, the scratch pools and the CUDA context are not counted, so a checkpoint at 0.95x of the pool passes the refusal and still dies on the first forward. | A headroom fraction invented here would be exactly the guess the per-tensor bound exists to avoid. The number wants the startup memory profile that `KV-WARMUP-PROFILE` owns (`INVENTORIED`; upstream's is `GPUWorker.determine_available_memory`, `vllm/v1/worker/gpu_worker.py:451-495`, around `profile_run`, `vllm/v1/worker/gpu/model_runner.py:682`), which is a different row. Those two anchors are stated here from the pinned tree because that row's own three anchors are stale at the current pin, and `gguf_device_fit.h` had copied two of them — filed as [#1139](https://github.com/mudler/vllm.cpp/issues/1139), owned by `KV-WARMUP-PROFILE`, blocked here only by the `engine-matrix.md` record lock #1119 holds. | | **The fit bound also counts too MUCH, and that direction can refuse a load that fits.** A tensor present in the file and not staged by THIS load is a positive over-count. On a default load that is the MTP / `nextn` block: 8,940,488,704 bytes, 8.33 GiB, 2.2506 % of the target checkpoint. A budget in that window refuses a weight set that would have fitted. | Not closed here. Closing it means the bound taking a per-tensor staging POLICY as input, which is the caller's knowledge and not the file's, and the exclusion's own failure mode is an under-count to nothing — which restores the 26-minute-then-OOM this row exists to remove, on a device nobody here has to measure it on. So the direction is stated in `gguf_device_fit.h`, pinned executably by `test_gguf_device_fit`, exposed to operators in `docs/USAGE.md`, and tracked as [#1136](https://github.com/mudler/vllm.cpp/issues/1136). `VT_DEVICE_WEIGHT_BUDGET_BYTES` is the way out of the window in the meantime. | -| **`Backend::DeviceMemoryInfo` has no CUDA override, and waking it is not the one-line port of the ROCm one that #1126 describes: on CUDA it would wake a THROW.** Only ROCm implements it (`src/vt/rocm/rocm_backend.hip:358-365`), so `Gemma4MoE`'s device-expert LRU refuses on every CUDA device (`gemma4_moe.cpp:506`). Where a per-expert FP8 checkpoint is present, the expert's BF16 bytes are re-copied HOST->DEVICE on every use instead (`ExpertGeGLUHost`, `gemma4_moe.cpp:49-74`, reached at `:1515-1521`; the H2D is `:59-60` and it drains the queue per expert at `:73`), silently and for the life of the process. That H2D cost is CONDITIONAL, not present-tense: the whole device LRU is `ex.is_fp8`-gated (`:991`, `:1506`), and by point (1) below no such Gemma-4 checkpoint is pinned anywhere, so on CUDA today the LRU-MISS fallback is never even asked for. (A BF16 Gemma-4 checkpoint reaches `ExpertGeGLUHost` at `:1525` too, but through the host-weight branch the LRU never governs, so it is not a cost of the missing probe.) The name misleads: it computes on the DEVICE from host-resident weights, so what the dead LRU would cost is bandwidth and a per-expert `Synchronize`, not a wrong answer. | The CAPABILITY is [#1126](https://github.com/mudler/vllm.cpp/issues/1126) and is still not built. The false COMMENT was corrected by this row in both places that carried it, `include/vt/backend.h:78-93` and `gemma4_moe.cpp:440-448`; both anchors were re-verified exact against `fd64c76ee`, as were `rocm_backend.hip:358-365`, `gemma4_moe.cpp:506`, `platforms/cuda.cpp:71` and `platforms/rocm.cpp:74`. SCOPE, because the previous headline read as coverage it did not have. The first review repair re-audited every anchor cited by the FOUR `## Owed` rows in this cluster — this one, the #1126 step-3 row, the #1197 row and the #1205 row — against the repaired tree: 53 examined, 53 exact, 0 stale. It audited those four rows and nothing else. The GitHub ISSUE BODIES were never in the audited set, and #1205's body was in fact 11 lines stale at `7beada17c` for exactly that reason. The second review repair widened the set: it re-derived every anchor in those four rows AND in the #1197 and #1205 issue bodies AND in the #1205 index row against the final tree — 71 examined, 71 exact, 0 stale after repair. Anchors from `gemma4_moe.cpp:549` onward moved by 22 lines in that repair, because the arm-existence guard it added sits at `:571`. **Four things were established while re-reading it for #1126, and each one raises the price of the override.** (1) *There is nothing to run it on.* The LRU needs `ex.is_fp8`, which only `LoadMoeFp8PerExpert` sets (`gemma4_weights.cpp:210-215`) from a per-expert `F8_E4M3` export; no such Gemma-4 checkpoint is pinned anywhere in `docs/USAGE.md` — all 32 lines there matching `gemma` case-insensitively were swept, and every checkpoint among them is the LTX-2.5 text tower (`gemma4-12b-with-proj*.safetensors`), not a per-expert FP8 MoE decoder, so the woken path cannot be exercised, here or elsewhere, until one is. (2) *The device-resident arm has no CUDA implementation to route into. It has a throw.* `ExpertGeGLUDeviceAccum` (`gemma4_moe.cpp:76-93`) READS as generic — `vt::MatmulBT`, `GeluAndMul`, `vt::MatmulBTAlphaBeta` — and an earlier draft of this row concluded from that reading that it "would run". It does not. `vt::MatmulBTAlphaBeta` (`src/vt/fused_ops.cpp:111-157`, dispatching at `:117`) is guarded on `#if defined(VLLM_CPP_HIP)` AND `q.device.type == kROCM`; its only implementation in the tree is `rocm::MatmulBTAlphaBetaRocm` (`src/vt/rocm/rocm_matmul_hipblaslt.hip:516`, declared `include/vt/rocm/rocm_matmul_batch.h:28`), and every other device falls through to a refusal. So the chain the override WOULD wake is: `EnsureGemma4Fp8ExpertOnDevice` (`gemma4_moe.cpp:548-608`) -> `lru.MakeRoom` (`:587`) succeeding as soon as `FreeBytes` can answer -> `true` at `:597` -> the call site at `:1508` -> `ExpertGeGLUDeviceAccum` at `:1509` -> `vt::MatmulBTAlphaBeta` at `:90` -> THROW, mid-decode. The `try`/`catch (...)` at `:585-607` wraps only the UPLOAD; the compute at `:1509` sits outside it, so the exception would propagate out of the decode step rather than degrading to the host fallback. **That chain is now cut at its first link.** `EnsureGemma4Fp8ExpertOnDevice` refuses at `:571` when `vt::HasMatmulBTAlphaBeta(d.q)` is false, BEFORE the upload, so the caller takes the `else` at `:1515-1521` — `EnsureGemma4Fp8ExpertCached` plus `ExpertGeGLUHost` — and the step answers instead of throwing. The refusal at `:90` stays as the backstop. It is latent today only because the other route into that function, `same_dev` (`:752-753`), needs `ex.gate_up_dev`, which is assigned nowhere but `src/vt/rocm/rocm_gemma4_experts.hip:207,226` — so the resident arm is UNREACHABLE off ROCm rather than safe. This is the actual blocker under #1126, it was recorded nowhere, and it is a stronger argument than the other three: filed as [#1205](https://github.com/mudler/vllm.cpp/issues/1205), and the refusal itself is now gated by `tests/vt/test_gemma4_rocm_fp8_seams.cpp`, which is what a CUDA implementation will have to satisfy. **Two corrections to the earlier draft's supporting claims, both of which overstated the case.** *The HIP-only list was one symbol too long.* Three of the four are genuinely HIP-only stubs inside `gemma4_moe.cpp`'s ONLY `#ifndef VLLM_CPP_HIP` block (`gemma4_moe.cpp:1596-1650`): `RunGemma4FusedTopkExpertGeGLU` (`:1621`), `PeerCopyGemma4Fp8ExpertSlice` (`:1629`) and `RunGemma4Fp8TopKOnExpertDevice` (`:1633`). `ExpertGeGLUDeviceBatched` (`:240`) is NOT: it sits OUTSIDE that block, in an anonymous namespace, with no HIP implementation and no header declaration, and is unconditionally `return false` under its own lab note (`:237-239` — gather+strided produced wrong tokens at ~23 t/s, pointer-batch ~0.8 t/s, serial/fused-gelu kept at ~34 t/s). It is disabled EVERYWHERE, ROCm included, so naming it beside the three inflated the ROCm/CUDA asymmetry. *The token-neutrality argument had the wrong mechanism.* The conclusion stands — the swap would not be token-neutral — but not because "the two arms sum the top-k experts in a different order". They do not: both run inside the SAME `for (int i = 0; i < top_k; ++i)` at `:1453`, so the summation order is identical. The real difference is ROUNDING and where the routing weight is applied. The resident arm folds `ww` into the GEMM `alpha` and accumulates in the epilogue with `beta` (`:1456`, `:1464`, `:90`), so the weight multiplies in the GEMM's own accumulator. The fallback writes the UNWEIGHTED product to a BF16 buffer (`:67`, `:70`) and applies `ww` afterwards with separate BF16 kernels — `vt::MulScalar` at `:1546` on the first expert, `MulScalar` plus `vt::Add` at `:1548-1549` on the rest. Two extra BF16 roundings per expert, with the weight applied post-rounding. (3) *The headroom test does not mean the same thing on the CUDA device this project gates on.* `MakeRoom` admits iff `free_b >= need + 1.5 GiB` (`:514`), a constant tuned on discrete dual R9700s where free VRAM is a quantity distinct from host RAM. On a GB10 it is not. This row already measured that `cudaMemGetInfo`'s `total` there is EXACTLY `/proc/meminfo MemTotal` times 1024, which is why it reached for that instrument where `nvidia-smi` answers `[N/A]`; its `free` is therefore reported over the same unified pool, and the host BF16 expert cache the device upload exists to relieve (`ex.cached_gu`/`cached_dn` via `EnsureGemma4Fp8ExpertCached`, bounded by the host LRU at `gemma4_moe.cpp:352`) is drawn from that pool too. So the admission test would double-count, on a box whose unified-memory OOM takes the host down with it. The `free` half is an inference from the measured `total`, not a second measurement, and it wants confirming on the device before any override lands. (4) *The gap is isolated, not a pattern.* Comparing overrides one by one across `src/vt/cuda/cuda_backend.cu` and `src/vt/rocm/rocm_backend.hip` at `fd64c76ee`, `DeviceMemoryInfo` is the ONLY optional `vt::Backend` seam ROCm answers and CUDA does not. | +| **`Backend::DeviceMemoryInfo` has no CUDA override, and waking it is not the one-line port of the ROCm one that #1126 describes: on CUDA it would wake a THROW.** Only ROCm implements it (`src/vt/rocm/rocm_backend.hip:358-365`), so `Gemma4MoE`'s device-expert LRU refuses on every CUDA device (`gemma4_moe.cpp:506`). Where a per-expert FP8 checkpoint is present, the expert's BF16 bytes are re-copied HOST->DEVICE on every use instead (`ExpertGeGLUHost`, `gemma4_moe.cpp:49-74`, reached at `:1515-1521`; the H2D is `:59-60` and it drains the queue per expert at `:73`), silently and for the life of the process. That H2D cost is CONDITIONAL, not present-tense: the whole device LRU is `ex.is_fp8`-gated (`:991`, `:1506`), and by point (1) below no such Gemma-4 checkpoint is pinned anywhere, so on CUDA today the LRU-MISS fallback is never even asked for. (A BF16 Gemma-4 checkpoint reaches `ExpertGeGLUHost` at `:1525` too, but through the host-weight branch the LRU never governs, so it is not a cost of the missing probe.) The name misleads: it computes on the DEVICE from host-resident weights, so what the dead LRU would cost is bandwidth and a per-expert `Synchronize`, not a wrong answer. | The CAPABILITY is [#1126](https://github.com/mudler/vllm.cpp/issues/1126) and is still not built. Note that the MERGED `#1126` index row cites `rocm_backend.hip:338-345`, which a later commit moved to `:358-365`. That row is append-only and is deliberately left alone: editing a merged row makes a union merge DUPLICATE it rather than merge it, which is a worse outcome than one stale number. This spec row carries the current anchor and is authoritative for it. The false COMMENT was corrected by this row in both places that carried it, `include/vt/backend.h:78-93` and `gemma4_moe.cpp:440-448`; both anchors were re-verified exact against `fd64c76ee`, as were `rocm_backend.hip:358-365`, `gemma4_moe.cpp:506`, `platforms/cuda.cpp:71` and `platforms/rocm.cpp:74`. SCOPE, because the previous headline read as coverage it did not have. The first review repair re-audited every anchor cited by the FOUR `## Owed` rows in this cluster — this one, the #1126 step-3 row, the #1197 row and the #1205 row — against the repaired tree: 53 examined, 53 exact, 0 stale. It audited those four rows and nothing else. The GitHub ISSUE BODIES were never in the audited set, and #1205's body was in fact 11 lines stale at `7beada17c` for exactly that reason. The second review repair widened the set: it re-derived every anchor in those four rows AND in the #1197 and #1205 issue bodies AND in the #1205 index row against the final tree — 71 examined, 71 exact, 0 stale after repair. Anchors from `gemma4_moe.cpp:549` onward moved by 22 lines in that repair, because the arm-existence guard it added sits at `:571`. **Four things were established while re-reading it for #1126, and each one raises the price of the override.** (1) *There is nothing to run it on.* The LRU needs `ex.is_fp8`, which only `LoadMoeFp8PerExpert` sets (`gemma4_weights.cpp:210-215`) from a per-expert `F8_E4M3` export; no such Gemma-4 checkpoint is pinned anywhere in `docs/USAGE.md` — all 32 lines there matching `gemma` case-insensitively were swept, and every checkpoint among them is the LTX-2.5 text tower (`gemma4-12b-with-proj*.safetensors`), not a per-expert FP8 MoE decoder, so the woken path cannot be exercised, here or elsewhere, until one is. (2) *The device-resident arm has no CUDA implementation to route into. It has a throw.* `ExpertGeGLUDeviceAccum` (`gemma4_moe.cpp:76-93`) READS as generic — `vt::MatmulBT`, `GeluAndMul`, `vt::MatmulBTAlphaBeta` — and an earlier draft of this row concluded from that reading that it "would run". It does not. `vt::MatmulBTAlphaBeta` (`src/vt/fused_ops.cpp:111-157`, dispatching at `:117`) is guarded on `#if defined(VLLM_CPP_HIP)` AND `q.device.type == kROCM`; its only implementation in the tree is `rocm::MatmulBTAlphaBetaRocm` (`src/vt/rocm/rocm_matmul_hipblaslt.hip:516`, declared `include/vt/rocm/rocm_matmul_batch.h:28`), and every other device falls through to a refusal. So the chain the override WOULD wake is: `EnsureGemma4Fp8ExpertOnDevice` (`gemma4_moe.cpp:548-608`) -> `lru.MakeRoom` (`:587`) succeeding as soon as `FreeBytes` can answer -> `true` at `:597` -> the call site at `:1508` -> `ExpertGeGLUDeviceAccum` at `:1509` -> `vt::MatmulBTAlphaBeta` at `:90` -> THROW, mid-decode. The `try`/`catch (...)` at `:585-607` wraps only the UPLOAD; the compute at `:1509` sits outside it, so the exception would propagate out of the decode step rather than degrading to the host fallback. **That chain is now cut at its first link.** `EnsureGemma4Fp8ExpertOnDevice` refuses at `:571` when `vt::HasMatmulBTAlphaBeta(d.q)` is false, BEFORE the upload, so the caller takes the `else` at `:1515-1521` — `EnsureGemma4Fp8ExpertCached` plus `ExpertGeGLUHost` — and the step answers instead of throwing. The refusal at `:90` stays as the backstop. It is latent today only because the other route into that function, `same_dev` (`:752-753`), needs `ex.gate_up_dev`, which is assigned nowhere but `src/vt/rocm/rocm_gemma4_experts.hip:207,226` — so the resident arm is UNREACHABLE off ROCm rather than safe. This is the actual blocker under #1126, it was recorded nowhere, and it is a stronger argument than the other three: filed as [#1205](https://github.com/mudler/vllm.cpp/issues/1205), and the refusal itself is now gated by `tests/vt/test_gemma4_rocm_fp8_seams.cpp`, which is what a CUDA implementation will have to satisfy. **Two corrections to the earlier draft's supporting claims, both of which overstated the case.** *The HIP-only list was one symbol too long.* Three of the four are genuinely HIP-only stubs inside `gemma4_moe.cpp`'s ONLY `#ifndef VLLM_CPP_HIP` block (`gemma4_moe.cpp:1596-1650`): `RunGemma4FusedTopkExpertGeGLU` (`:1621`), `PeerCopyGemma4Fp8ExpertSlice` (`:1629`) and `RunGemma4Fp8TopKOnExpertDevice` (`:1633`). `ExpertGeGLUDeviceBatched` (`:240`) is NOT: it sits OUTSIDE that block, in an anonymous namespace, with no HIP implementation and no header declaration, and is unconditionally `return false` under its own lab note (`:237-239` — gather+strided produced wrong tokens at ~23 t/s, pointer-batch ~0.8 t/s, serial/fused-gelu kept at ~34 t/s). It is disabled EVERYWHERE, ROCm included, so naming it beside the three inflated the ROCm/CUDA asymmetry. *The token-neutrality argument had the wrong mechanism.* The conclusion stands — the swap would not be token-neutral — but not because "the two arms sum the top-k experts in a different order". They do not: both run inside the SAME `for (int i = 0; i < top_k; ++i)` at `:1453`, so the summation order is identical. The real difference is ROUNDING and where the routing weight is applied. The resident arm folds `ww` into the GEMM `alpha` and accumulates in the epilogue with `beta` (`:1456`, `:1464`, `:90`), so the weight multiplies in the GEMM's own accumulator. The fallback writes the UNWEIGHTED product to a BF16 buffer (`:67`, `:70`) and applies `ww` afterwards with separate BF16 kernels — `vt::MulScalar` at `:1546` on the first expert, `MulScalar` plus `vt::Add` at `:1548-1549` on the rest. Two extra BF16 roundings per expert, with the weight applied post-rounding. (3) *The headroom test does not mean the same thing on the CUDA device this project gates on.* `MakeRoom` admits iff `free_b >= need + 1.5 GiB` (`:514`), a constant tuned on discrete dual R9700s where free VRAM is a quantity distinct from host RAM. On a GB10 it is not. This row already measured that `cudaMemGetInfo`'s `total` there is EXACTLY `/proc/meminfo MemTotal` times 1024, which is why it reached for that instrument where `nvidia-smi` answers `[N/A]`; its `free` is therefore reported over the same unified pool, and the host BF16 expert cache the device upload exists to relieve (`ex.cached_gu`/`cached_dn` via `EnsureGemma4Fp8ExpertCached`, bounded by the host LRU at `gemma4_moe.cpp:352`) is drawn from that pool too. So the admission test would double-count, on a box whose unified-memory OOM takes the host down with it. The `free` half is an inference from the measured `total`, not a second measurement, and it wants confirming on the device before any override lands. (4) *The gap is isolated, not a pattern.* Comparing overrides one by one across `src/vt/cuda/cuda_backend.cu` and `src/vt/rocm/rocm_backend.hip` at `fd64c76ee`, `DeviceMemoryInfo` is the ONLY optional `vt::Backend` seam ROCm answers and CUDA does not. | | **#1126's own closing plan, step 3, must be NARROWED before it is done: the load-time fit check may read the seam's `total`, and never its `free`.** The issue proposes that "the #1123 fit check can then read the budget from the backend seam on every platform that reports one". As written that invites the live half, which would be a defect — but the seam returns BOTH halves (`bool DeviceMemoryInfo(size_t* free_bytes, size_t* total_bytes)`, `include/vt/backend.h:94`), and only one of them is illegitimate here. | The tree holds two answers to "how much device memory", and they answer different questions. `vt::Backend::DeviceMemoryInfo(free, total)` is a LIVE probe that moves with contention; it is authoritative for a RUNTIME admission decision — can this allocation succeed right now — and for nothing else. `vllm::platforms::ResidencyPolicy::device_memory_total_bytes` is a TOTAL probed once at platform registration with `0 == UNKNOWN`; it is authoritative for a LOAD-TIME budget verdict, which has to be reproducible and independent of whatever else the box is doing. Sourcing the load-time verdict from `free` would make the same checkpoint load or be refused depending on the page cache, which is precisely the property **`total`, not `free`** above was chosen to avoid. Reading `total` through the seam is not that. It is contention-independent, it is the same quantity `ResidencyPolicy` already carries, and a seam that answers it on every platform is a defensible place to source it from. What step 3 must additionally preserve is the PROBE-ONCE semantics: `device_memory_total_bytes` is probed at platform registration (`include/vllm/platforms/interface.h:70-72`), and a per-load live call would reintroduce the contention dependence by the back door even reading only `total`, because a load-time verdict has to be reproducible from the record rather than from the moment. So the defensible statement, and the one this row asserts: **step 3 may read only `total`, never `free`, and must keep the value probed once at platform registration.** An earlier draft of this row said "never step 3" outright; that overstated it and would have blocked a legitimate simplification, so it is corrected here rather than quietly narrowed. Both seams already carry the division in prose (`include/vt/backend.h:90-93` and `include/vllm/platforms/interface.h:61-69`); it is restated here because #1126 is the record a reader of that issue will act on, and as filed it points the other way. What #1126 owes is its steps 1 and 2 together — the override AND the Gemma4 measurement, with [#1205](https://github.com/mudler/vllm.cpp/issues/1205) ahead of both — never step 1 alone, and step 3 only in the narrowed form above. | | **The device-expert LRU's slot cap makes its own eviction opt-in inert.** `MakeRoom` tests `slots.size() >= kMaxSlots` (`gemma4_moe.cpp:498`) BEFORE the eviction loop (`:499-500`), and `EvictOne` (`:457`, the device LRU's — a host-cache namesake sits at `:275`) is the only thing that SHRINKS `slots`. The one other statement that touches its size, `slots.clear()` in `DevExpertLru::Note` (`:522`), is a device-index RESET rather than an eviction: it drops bookkeeping when `dev != d.q.device.index` and frees nothing, and it is unreachable in a single-device process. It is named here so the next reader does not conclude the #1197 sweep missed it. So once 24 slots are resident `VT_GEMMA4_EXPERT_EVICT=1` never runs again and the cache degrades permanently to fill-only. It binds only when `24 * expert_bytes < BudgetBytes()`, so it is condition-dependent and silent either way. Tracked as [#1197](https://github.com/mudler/vllm.cpp/issues/1197). | Filed, not fixed, and for the same reason as the row above rather than for effort: the one-line repair wakes more `hipFree` under load, which the surrounding comments say has been observed as a permanent `kfd_wait` hang with the GPU idle and no decode tokens. The current ordering may well be deliberate belt-and-braces. Deciding that needs the dual-RDNA4 box `.agents/specs/gemma4-rocm-fp8-moe.md` describes; this host has neither a ROCm nor a CUDA device. It closes when the cap moves after the eviction loop and a run stays hang-free, or when the comment says the cap is by design — one of the two, not silence. | | **`vt::MatmulBTAlphaBeta` is ROCm-only and has no CUDA implementation at all, which is what #1126 step 1 is actually blocked on.** `src/vt/fused_ops.cpp:117` dispatches to `rocm::MatmulBTAlphaBetaRocm` (`src/vt/rocm/rocm_matmul_hipblaslt.hip:516`) under `#if defined(VLLM_CPP_HIP)` and `q.device.type == kROCM` — `src/vt/fused_ops.cpp:111-112` is the signature, not the dispatch — and every other device falls through to the refusal at `src/vt/fused_ops.cpp:152`. There is no CUDA, Vulkan, Metal or CPU arm. The full chain from the missing `DeviceMemoryInfo` override to that refusal is traced in the first row above. Tracked as [#1205](https://github.com/mudler/vllm.cpp/issues/1205). | The REFUSAL is fixed in flow, because a bare `std::runtime_error` reading "ROCm-only in this build" does not satisfy the standing rule that an unimplemented arm refuses with a message NAMING the missing part: a caller who hits it on CUDA cannot tell a missing kernel from a missing build flag. It now names the device that asked, names the one arm that exists, and names the issue (`:152`), and a kROCM queue — which reaches the same line in a build configured without `-DVLLM_CPP_HIP` — gets a DIFFERENT message naming the absent build flag (`:138`), because for that caller the kernel exists and telling them to write one would send them to fix the wrong thing. `tests/vt/test_gemma4_rocm_fp8_seams.cpp` gates both messages on a posed CUDA queue, on `kCPU`/`kVULKAN`/`kMETAL`, and on kROCM — mutation-proven by restoring the old message (RED), by deleting the refusal outright (RED), and by deleting the kROCM branch so that case falls to the generic message (RED). **Say plainly what that message change does and does not pin: a contract in a unit test, not observable behaviour.** The throw is unreachable off ROCm in any shipped configuration, so no production run can print either string today; what the test fixes is what a CUDA implementation has to satisfy when someone writes one. **The reachable half of this row is the GUARD.** `EnsureGemma4Fp8ExpertOnDevice` refuses at `gemma4_moe.cpp:571` when `vt::HasMatmulBTAlphaBeta(d.q)` is false, BEFORE the upload rather than after it, which converts the mid-decode exception traced above into the host fallback that was already sitting in the `else` at `:1515-1521`: slower, two extra BF16 roundings per expert, and correct. The predicate (`include/vt/fused_ops.h`, defined `src/vt/fused_ops.cpp:102-109`) is the same condition the dispatch at `:117` uses rather than a second copy of it, so the two cannot drift and writing the CUDA kernel wakes the device arm with no edit at the call site. It is gated by `tests/vllm/models/test_gemma4_moe_device_arm_guard.cpp`, which enters through `vllm::RunGemma4Moe` — the production layer entry `src/vllm/model_executor/models/gemma4.cpp:634` calls — and decorates the registered CPU backend so `DeviceMemoryInfo` ANSWERS, which is the post-#1126 state and the only state in which the guard binds at all. Deleting the guard makes that test RED with the exact `no implementation for device 'cpu'` throw; forcing `HasMatmulBTAlphaBeta` to `true` makes it RED too. A test that constructed the `Dev` or the LRU by hand would have stayed green under both. The IMPLEMENTATION stays owed and is what [#1205](https://github.com/mudler/vllm.cpp/issues/1205) tracks. It is not written here: a `beta`-accumulating BT GEMM on cuBLASLt is a kernel with its own correctness gate, the `DeviceMemoryInfo` row's point (1) above says there is no checkpoint to exercise it on, and this host has neither a ROCm nor a CUDA device to measure either arm. | From 581942de8f10b6f5445335d2eefc2593b7e85ea5 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Tue, 18 Aug 2026 12:42:13 +0000 Subject: [PATCH 8/8] fix(ENG-EXPERT-STREAM): drop the #1193 row a union merge duplicated (#1205) FOLLOWING_AGENTS_PROTOCOL `.agents/issue-index.md` carried two byte-identical `#1193` rows after merging origin/main, and `check-agent-record.py` refused: under `merge=union` a duplicate is what two branches appending the same issue look like, so the checker cannot tell that case from this one and is right to stop. The two copies were compared before either was touched, and only then was the stray one removed. The surviving row is the one whose position matches origin/main's ordering (after `#1190`); the deleted one sat inside this branch's own appended block, which is where the union driver placed it. The file now removes and edits nothing that main has, adds exactly `#1197`, `#1205` and `#1218`, and contains no duplicate row. Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5-1m [Claude Code] --- .agents/issue-index.md | 1 - 1 file changed, 1 deletion(-) diff --git a/.agents/issue-index.md b/.agents/issue-index.md index a2b50dd32..c654f7fff 100644 --- a/.agents/issue-index.md +++ b/.agents/issue-index.md @@ -368,7 +368,6 @@ rather than merged. `scripts/check-agent-record.py` gates both. | [#1185](https://github.com/mudler/vllm.cpp/issues/1185) | `ENV-ORACLE-WHEEL-IN-LEASE` | The pinned vLLM oracle BUILDS, installs, imports and sees the GPU inside an `rc` lease on `dgx:gpu0`, measured 2026-08-18, which falsifies the `nvcc` clause four records carried. [`lease-runtime-staging.md`](specs/lease-runtime-staging.md) said the oracle "needs `nvcc`, which the worker still lacks", and `.agents/environment.md`, [`mtp-k-gt-1.md`](specs/mtp-k-gt-1.md) and [`gpu-lease-methodology.md`](specs/gpu-lease-methodology.md) each derived a blocker from it. The build job (`buildvllm.sh`, staged sha256 prefix `15e140d41f44e7c2`) asserted the checkout against the pin BEFORE compiling, printing `PIN CONFIRMED` at `5559679229bc961848b121ccdeaa8fa5d79bec98` and aborting otherwise, took `nvcc` from the toolkit row `MODEL-NEMOTRON-H-ABI-A3-E2E` staged (`NVCC_RC=0`, CUDA `release 13.3, V13.3.73`) and produced `WHEEL_RC=0`, `PERSIST_RC=0` and a 434 MiB `vllm-0.1.dev1+g555967922.cu133-cp312-cp312-linux_aarch64.whl`, sha256 `7c58b339741a288fbb313f4f5196c9c92a9e3b3c3ebe2ea970b0ff50bb9bcba4`. The identity job (`oracleenv.sh`, prefix `6119f5223f5d818c`) asserted from `cd /`, outside any source tree: `vllm.__version__ = 0.1.dev1+g555967922`, `IDENTITY_RC=0`, `cuda True NVIDIA GB10`, `CUDA_RC=0`. SCOPE, and it carries the same weight as the result: RUNNING A MODEL IS UNTESTED. Only build, install, import and `torch.cuda.is_available()` are measured, and [`mtp-k-gt-1.md`](specs/mtp-k-gt-1.md) records that the last time an oracle reached this far it consumed the host in the step AFTER `torch.compile` and REBOOTED the box, at `gpu_memory_utilization` 0.75 and again at 0.30, so the fraction is not the lever. The version string is an OPEN discrepancy: `.agents/upstream-sync.md` records `vllm_runtime_version = 0.23.1rc1.dev1511+g555967922`, the commit segment matches and satisfies the pin's binding `+g` rule, and the prefix differs because a shallow fetch stops `setuptools_scm` counting commits since the last tag, so a full-string gate needs a deeper fetch or a recorded pretend-version. The venv is NOT staged, because that job was killed at a 90-minute ceiling mid-copy and its partial tree was removed, so only the WHEEL is durable. Four staging walls, all artifacts of the NAS rather than of CUDA: `cp -a` preserves `file_mode=0664` so `nvcc` exited 126. CIFS `nounix` stores no symlink so `include` and `lib64` vanished and CMake reported `Could NOT find CUDA (missing: CUDA_INCLUDE_DIRS CUDA_CUDART_LIBRARY) (found version "13.3")`, naming the version and denying the toolkit in one line. 32 library links `libfoo.so` and `libfoo.so.MAJOR` had to be rebuilt because only the `libfoo.so.X.Y.Z` real files survived. And `markupsafe` existed as a dist-info with NO package files from a `pip --target` killed at a 35-minute ceiling, so Marlin codegen died on `ModuleNotFoundError`. The `rc` worker container is REUSED between jobs, so a repair inside a staging branch is skipped on the next run (`nvcc already in place`) and an environment repair must be unconditional and assert its postcondition. CONSEQUENCE for the rows #1129 blocked, [#1003](https://github.com/mudler/vllm.cpp/issues/1003), [#915](https://github.com/mudler/vllm.cpp/issues/915), [#821](https://github.com/mudler/vllm.cpp/issues/821) and [#81](https://github.com/mudler/vllm.cpp/issues/81): UNBLOCKED FOR THE BUILD STEP and STILL BLOCKED FOR A MODEL RUN. None can take a measurement until a model run is demonstrated. Job details, walls and non-claims in [`oracle-wheel-in-lease.md`](specs/oracle-wheel-in-lease.md) | verification | | [#1197](https://github.com/mudler/vllm.cpp/issues/1197) | `ENG-EXPERT-STREAM` | `Gemma4MoE`'s device-expert LRU tests its slot cap BEFORE its eviction loop, so the eviction opt-in goes inert once the cap is reached. `DevExpertLru::MakeRoom` runs `if (slots.size() >= kMaxSlots) return false;` at `src/vllm/model_executor/models/gemma4_moe.cpp:498` @ `fd64c76ee`, two lines ahead of the `if (allow_evict) { while (used + need > bud && !slots.empty()) EvictOne(d); }` at `:499-500`, and `EvictOne` (`:457`, the DEVICE LRU's — the file carries a host-cache namesake at `:275`) is the only thing that shrinks `slots`. So after 24 admissions every later `MakeRoom` returns false at that first line, the eviction loop is never reached again, and `VT_GEMMA4_EXPERT_EVICT=1` becomes a no-op for the life of the process — the cache degrades permanently to the fill-only mode the opt-in exists to leave. It binds only when the slot cap is reached before the byte budget, i.e. when `24 * expert_bytes < BudgetBytes()` (below ~85.3 MiB per expert at the 2048 MiB default from `BudgetBytes`, `:416-436`, again the device one and not the host cache's at `:262`); above that the byte budget binds first and eviction behaves. Nothing reports which one happened. FILED, NOT FIXED, and not for effort: the one-line repair (move the cap test after the eviction loop, so it caps RESIDENT slots instead of stopping admission forever) wakes more `hipFree` under load, which the code's own comments call a permanent `kfd_wait` hang with the GPU idle, prefill done and no decode tokens (`:459-461` and `:486-488`), so the current ordering may be deliberate. Deciding it needs the dual-RDNA4 lab box of [`gemma4-rocm-fp8-moe.md`](specs/gemma4-rocm-fp8-moe.md); the host that found it has neither a ROCm nor a CUDA device. Found while establishing the facts for [#1126](https://github.com/mudler/vllm.cpp/issues/1126), which required reading `MakeRoom` line by line. Listed under `## Owed` in [`expert-streaming.md`](specs/expert-streaming.md) | bug | | [#1205](https://github.com/mudler/vllm.cpp/issues/1205) | `ENG-EXPERT-STREAM` | `vt::MatmulBTAlphaBeta` has NO CUDA implementation, and that — not any of the four reasons [#1200](https://github.com/mudler/vllm.cpp/pull/1200) first recorded — is what [#1126](https://github.com/mudler/vllm.cpp/issues/1126) step 1 is blocked on. `src/vt/fused_ops.cpp:117` dispatches to `rocm::MatmulBTAlphaBetaRocm` (`src/vt/rocm/rocm_matmul_hipblaslt.hip:516`, declared `include/vt/rocm/rocm_matmul_batch.h:28`) under `#if defined(VLLM_CPP_HIP)` AND `q.device.type == kROCM` — `src/vt/fused_ops.cpp:111-112` is the signature, not the dispatch — and every other device falls through to a throw (`src/vt/fused_ops.cpp:152`). There is no CUDA, Vulkan, Metal or CPU arm: this is a missing kernel, not a missing build flag. #1126 proposes adding `CudaBackend::DeviceMemoryInfo` via `cudaMemGetInfo` so `Gemma4MoE`'s device-expert LRU stops refusing on CUDA; the record said that would wake a slower third path, and it would in fact wake a THROW, mid-decode. The chain: `EnsureGemma4Fp8ExpertOnDevice` (`src/vllm/model_executor/models/gemma4_moe.cpp:548-608`) -> `lru.MakeRoom` at `:587` succeeding once `FreeBytes` can answer -> `true` at `:597` -> the call site at `:1508` -> `ExpertGeGLUDeviceAccum` at `:1509` -> `vt::MatmulBTAlphaBeta` at `:90` -> throw. The `try`/`catch (...)` at `:585-607` wraps only the UPLOAD; the compute at `:1509` is outside it, so the exception leaves the decode step rather than degrading to the host fallback. Latent today only because the other route in, `same_dev` (`:752-753`), needs `ex.gate_up_dev`, assigned nowhere but `src/vt/rocm/rocm_gemma4_experts.hip:207,226` — so the resident arm is UNREACHABLE off ROCm rather than safe. FIXED IN FLOW, PARTLY. (a) The HAZARD is guarded: `EnsureGemma4Fp8ExpertOnDevice` refuses at `src/vllm/model_executor/models/gemma4_moe.cpp:571` when `vt::HasMatmulBTAlphaBeta(d.q)` is false, BEFORE the upload, so the caller takes the host fallback already sitting in the `else` at `:1515-1521` and the decode step answers instead of throwing. The predicate (`src/vt/fused_ops.cpp:102-109`) is the same condition the dispatch at `:117` uses, so the two cannot drift. Gated by `tests/vllm/models/test_gemma4_moe_device_arm_guard.cpp`, which enters through `vllm::RunGemma4Moe` and decorates the registered CPU backend so `DeviceMemoryInfo` answers — the post-#1126 state, and the only one in which the guard binds; deleting the guard turns it RED. (b) The REFUSAL now satisfies the rule that an unimplemented arm names the missing part. "ROCm-only in this build" named neither the device that asked nor where an implementation would go; a non-ROCm queue now gets all three plus this issue (`src/vt/fused_ops.cpp:152`), and a kROCM queue in a build without `-DVLLM_CPP_HIP` gets a different message naming the absent FLAG (`src/vt/fused_ops.cpp:138`), because for that caller the kernel exists. `tests/vt/test_gemma4_rocm_fp8_seams.cpp` gates both on posed queues — CUDA, `kCPU`, `kVULKAN`, `kMETAL` and kROCM — mutation-proven RED by restoring the old message, RED by deleting the refusal outright, and RED by deleting the kROCM branch. That message gate is a UNIT CONTRACT, not observable behaviour: with the guard in place the throw is unreachable off ROCm, which is the point of the guard. The IMPLEMENTATION stays open and is what this issue tracks: a `beta`-accumulating BT GEMM on cuBLASLt is a kernel with its own correctness gate, no per-expert FP8 Gemma-4 checkpoint is pinned to exercise it on, and the host that found this has neither a ROCm nor a CUDA device. The same file refuses six further arms the same way (`MatmulBTFp8Channel`, `DequantFp8ChannelBf16`, `MoeGatherRows`, `MoeWeightedScatterAdd`, `MoeZeroBf16`, `DualRmsNormPlusRes`); only `MatmulBTAlphaBeta` is on the #1126 path, and the others are named so the next reader need not re-derive the list. Listed under `## Owed` in [`expert-streaming.md`](specs/expert-streaming.md) | bug | -| [#1193](https://github.com/mudler/vllm.cpp/issues/1193) | `SPEC-DSPARK-QWEN3-ROUTING` | A Qwen3 DSpark draft declaring `architectures=["DSparkDraftModel"]` with `model_type` `qwen3` has no route. The pin forces every DSpark draft that is not `Qwen3DSparkModel` or `Gemma4DSparkModel` onto `model_type` `deepseek_v4` (`vllm/config/speculative.py:934-944` @ `555967922`), and vLLM PR 52197 (merged 2026-08-17 at `7075ddac`) replaced that with a leading branch normalizing the pair to `Qwen3DSparkModel`. We diverge from BOTH: the forced rewrite was never ported, so nothing in `src/vllm/entrypoints/model_loader.cpp` reads a draft config's `architectures` key at all, and `SpeculativeConfig::IsDsparkDraft` (`include/vllm/config/speculative.h:120-136`) has no production caller — every reference outside its header is in `tests/vllm/config/test_speculative_dspark.cpp:132-140`, and `ResolveSpecConfig` branches on `cli.method` alone. The checkpoint is real and gateable here: `RadixArk/Qwen3.8-27B-DSpark` at revision `85ef153be924f17ce4bf62726954eeaa4a73e854` carries exactly that config shape in one 2718576122-byte shard, drafting five layers for a 64-layer Qwen3.8-27B target | bug | | [#1218](https://github.com/mudler/vllm.cpp/issues/1218) | `ENG-EXPERT-STREAM` | `EnsureGemma4Fp8NativeOnDevice` has no arm-existence guard, and it is the DEFAULT Gemma-4 expert arm, so [#1126](https://github.com/mudler/vllm.cpp/issues/1126) step 1 would still throw mid-decode after [#1205](https://github.com/mudler/vllm.cpp/issues/1205)'s guard lands. That guard covers the BF16 device-expert arm (`src/vllm/model_executor/models/gemma4_moe.cpp:571`, inside `:548-608`). The FP8-native twin at `:611` has none, and `VT_GEMMA4_FP8_NATIVE` returns true when unset (`:969-974`), so on a per-expert FP8 checkpoint the expert loop reaches the twin at `:1359` and `:1484` BEFORE the guarded arm. A `true` from it routes into `ExpertGeGLUFp8Native` (`:95-130`), which needs `vt::ExpertGeGLUFp8TopKM1` (`:105`, which merely returns false off ROCm), `vt::DequantFp8ChannelBf16` (`:117`, `:119`, refusing at `src/vt/fused_ops.cpp:194`) and `vt::MatmulBTAlphaBeta` (`gemma4_moe.cpp:128`, refusing at `src/vt/fused_ops.cpp:152`) — so T>1 throws at the first dequant and T==1 falls through the fused kernel and throws too, in both cases outside the upload's own `try`/`catch`. Latent for the same reason and for exactly as long: the twin's `MakeRoom` also needs `vt::Backend::DeviceMemoryInfo`, which only ROCm overrides (`src/vt/rocm/rocm_backend.hip:358-365`). FILED, NOT FIXED, and not for effort: the BF16 guard keys on ONE predicate that is the same condition its own dispatch uses, which is what makes it honest, whereas the twin depends on three different ops and needs a predicate per op — reusing `HasMatmulBTAlphaBeta` there would be a guard naming the wrong arm, the exact defect this row's review had just corrected in a refusal message. Found while repairing [#1200](https://github.com/mudler/vllm.cpp/pull/1200). Listed under `## Owed` in [`expert-streaming.md`](specs/expert-streaming.md) | bug | | [#1184](https://github.com/mudler/vllm.cpp/issues/1184) | `ENG-CUDAGRAPH-DEDUP` | `VT_CUDA_GRAPH_DEDUP=1` cannot complete a single decode step once a CUDA graph is actually captured. Measured 6/6 deterministic on GB10: `vt graph dedup: captured 1 graphs, deduped to 1 execs`, then `[Qwen3DenseDecodeGraph] captured dense decode graph for padded size S=8`, then `engine-fatal: EngineCore busy loop threw: vt cuda: greedy_argmax launch: invalid device function` -- from a `greedy_argmax` launch that had SUCCEEDED. The OFF and `=0` arms are clean and byte-identical across 7 runs on the identical binary and libraries, which is what makes the asymmetry arm-attributable. `greedy_argmax` is not at fault. `src/vt/graph_dedup_runtime.h` @ `3ed6253c6` walks the retained raw graph with CUDA runtime APIs and is DESIGNED to see several of them fail -- the `cudaGraphExecUpdate` probe refusing a fold is the feature working, not an exception, and the topology walk has five more escapes that degrade the key rather than abort inside a capture -- but it never consumed the runtime's sticky per-thread error: `grep -n 'cudaGetLastError\|hipGetLastError' src/vt/graph_dedup_runtime.h` returned nothing. Twelve sites swallowed a non-success return without clearing (`:98,100,115,130,137,146,154,173,179,281,307,317`); the two that mattered are the probe (`:307,:317`), whose refusal is routine, and `cudaGraphInstantiate` (`:281`). The next unrelated kernel, launched with the ordinary `kernel<<<>>>(); Check(cudaGetLastError())` pattern, then reported our refusal as its own failure. Every symptom follows: it needs BOTH `dedup=1` and a real capture, `CUDA_LAUNCH_BLOCKING=1` does not move it because the latch is host-side and synchronous rather than a deferred async error, and `cudaGraphLaunch` returns success because a return value does not consume the latch. Fixed in flow, structurally rather than site-by-site: twelve hand-placed clears are a fix the thirteenth fallible call misses, so the clear lives in `ScopedLatchClear`'s destructor (`src/vt/graph_dedup_latch.h`, new) installed at the binding's entry points, which are exactly the six `GraphDedupOps` members; `MakeLatchGuardedOps` is the table's only constructor and takes the raw functions as template arguments, so no raw address reaches a field and a seventh operation wired elsewhere leaves its field null, which the registry constructor refuses. One line covers CUDA and HIP because there is one source. The coverage gap that hid it is closed in the same change: the device-free half of the signature walk moves to `src/vt/graph_dedup_signature.h` and is gated by the new `tests/vt/test_graph_dedup_runtime.cpp` (13 cases, 51 assertions, 7/7 negative mutations detected) -- Kahn ordering, topological re-index, sorted edge emission, the depth-4 child bound and the four graph-level escapes, none of which was reached by any test on any tier before. STATED LIMIT: a CPU test drives a fake runtime and cannot observe the real latched error, so it proves the guard's structure and not the device outcome; the device A/B re-run is owed under `## Owed` in [`eng-cudagraph-dedup.md`](specs/eng-cudagraph-dedup.md), which also records that the shipped async serving path never reaches this feature at all | bug | | [#1192](https://github.com/mudler/vllm.cpp/issues/1192) | `ENG-CUDAGRAPH-BREAK` | W1 of the break-point capture seam: `vt::BreakableGraph`, `vt::GraphCaptureScope` and `vt::GraphBreak` land in `include/vt/breakable_graph.h` + `src/vt/breakable_graph.cpp`, with the SGLang unit suite ported case for case and ONE break point registered on ONE model. **The stage's exit criterion was answered first, on a leased GPU, because W0 deliberately did not answer it**: `cudaStreamEndCapture` followed by `cudaStreamBeginCapture` on the SAME stream mid-forward with EAGER work between them is LEGAL under `cudaStreamCaptureModeThreadLocal`, our backend's mode (`src/vt/cuda/cuda_backend.cu:204-206`) — measured on `orin:gpu0` through an `rc` lease, driver 12060, three replays with fresh inputs and 0 mismatches, plus the bare zero-work re-begin (`breakable_cuda_graph.py:370-374`). The first probe REFUSED and that refusal was the probe's, not CUDA's: `dlsym` on the bare symbol name binds the LEGACY v1 entry points, which are not capture-aware, so `cuMemcpyDtoDAsync` returned `CUDA_ERROR_STREAM_CAPTURE_UNSUPPORTED` and `cuStreamBeginCapture` (v1) silently took no capture-mode argument — a broken instrument failing toward a verdict about the code. Bound by exact versioned name, the criterion holds. The break point is the DENSE ATTENTION ENTRY of `Qwen3ForCausalLM` (`src/vllm/model_executor/models/qwen3.cpp`, inside `RunLayer`), which is vLLM's boundary and not one this row invents (`splitting_ops` defaulted to the attention family, `vllm/config/compilation.py:517,764-772,1145` @ `5559679229`) registered in SGLang's form because we have no compiler to match operation names against (`layers/radix_attention.py:256` @ `f63458b5be`). The G2 reachability gate is `tests/vllm/models/test_qwen3_break_point.cpp`: it drives the production `Qwen3DenseModel::Forward` with a scope open and counts `num_hidden_layers + 1` segments, so deleting the one call site takes it to 1 and turns the gate RED — a unit test that constructs the container by hand could not have asked that. It also holds G4 in the same case: the logits are compared BIT FOR BIT against the unscoped forward, 500 values, 0 differing. **STAGED SLICE, named rather than implied:** `GraphCaptureScope` and `BreakableGraph` are not yet ENTERED from a production step, because no driver opens a scope until W2 migrates `Qwen3DenseDecodeGraph`; the break point itself is on the production path on every forward and takes the pass-through arm. Listed under `## Owed` in [`eng-cudagraph-break.md`](specs/eng-cudagraph-break.md). The capture-failure drain is NOT owed: it landed here, as behaviour and as three gated arms. The destructor `catch` alone covered a throwing `EndCaptureGraph` and nothing else, so a break function or ordinary model code throwing mid-capture left a container reporting `captured() == true` over a PARTIAL forward; comparing `std::uncaught_exceptions()` against the depth recorded at scope entry is what tells those apart, and tests 13a, 13b and 13c gate the three arms. Still owed by later stages and NOT claimed here: the auxiliary-stream auto-join before every segment close (spec D10, live at `qwen3_5.cpp:6254-6255,6384` and `laguna.cpp:2572-2576,2612`, which W4 and W5 migrate), the non-capturing-backend arm on ROCm and Tenstorrent (G5), G1 bit-exactness on a real GPU over more than one replay, and one residual the fresh review surfaced: an exception CAUGHT INSIDE the capture scope leaves the segment closed and the rest of the forward uncaptured while `captured()` stays true, which the drain cannot see because nothing is unwinding at scope exit (spec `## Owed`, W2). No throughput claim is made anywhere in this change: the prefill refutation stands (GB10 3.8% host-idle between launches, GPU-busy above 96%, 27B prefill gap 92.5% non-GEMM glue) | feature |