spec(ENG-CUDAGRAPH-BREAK): the capture seam has no break points, and the nine hand-rolled drivers each re-derive one

.agents/issue-index.md

@@ -358,3 +358,4 @@ rather than merged. `scripts/check-agent-record.py` gates both.
 | [#1169](https://github.com/mudler/vllm.cpp/issues/1169) | — | Packed GDN decode is unreachable on every MoE checkpoint for a second, independent reason: `detail::ShouldUsePackedGdnDecode` (`src/vllm/model_executor/models/qwen3_5.cpp:76-84` @ `dd8a3b0e1`) requires `e.has_packed_ba`, populated at `:4410` as `!w.in_proj_ba.Empty()`, and `in_proj_ba` is written at exactly ONE site in the tree — `src/vllm/model_executor/models/qwen3_5_dense_weights.cpp:436`, the DENSE loader. The MoE loader loads the shards split (`qwen3_5_weights.cpp:563-564`) and so does the GGUF loader (`qwen3_5_gguf_weights.cpp:1067-1070`); `qwen3_5.cpp:3220` states it in the tree ("the only path that populates `in_proj_ba`"). Consequence: removing the `dense_model` term and narrowing `GdnOutDType` (#1168) is NOT sufficient to reach packed decode on a MoE model, which corrects the premise row `GDN-MOE-BF16-OUT` started from. It matters because the 35B's geometry (`Hv=32`, `Hg=16`, `Dk=Dv=128`) has a vendored Triton AOT decode cubin for every supported architecture (`src/vt/cuda/triton_aot_vendored/*/gdn_decode_h32.*`) that is exactly the FLA kernel vLLM runs, and `.agents/kernel-matrix.md` records the contrast the vendoring exists to capture — Triton REG:205 / 0 spill against the hand CUDA port at REG:255 + STACK:48. Upstream has one physical `in_proj_ba` on every Qwen3.5/3.6 GDN layer, dense and MoE alike (`qwen_gdn_linear_attn.py:843`; `packed_modules_mapping` on the shared `Qwen3_5ForCausalLMBase`, `vllm/model_executor/models/qwen3_5.py:287-297` @ `5559679`), so the merged owner is the upstream topology and our split MoE load is the deviation. Same class of gap on `in_proj_qkvz`, which no MoE or GGUF loader builds either. Closing it also owes a repair to the stale `KERNEL-GDN-PACKED-DECODE` sentence in `.agents/kernel-matrix.md` claiming "the launcher guard rejects its `Hv=32` shape anyway": `TryTritonPackedDecode` accepts `Hv=32` and dispatches `gdn_decode_h32_default` (`src/vt/cuda/cuda_gdn.cu:5207`, `:5239`). Filed, not fixed — it is a loader change with its own resident-weight lifetime and byte-exactness argument, and the finding row is spec-only. Listed under `## Owed` in [`gdn-moe-bf16-out.md`](specs/gdn-moe-bf16-out.md) | gap |
 | [#1170](https://github.com/mudler/vllm.cpp/issues/1170) | — | All four GDN Triton AOT fast paths reject any geometry whose linear V-head count is not 48 or 32 — `TryTritonPackedDecode` (`src/vt/cuda/cuda_gdn.cu:5207` @ `dd8a3b0e1`), `TryTritonDeltaH` (`:5264`), `TryTritonChunkO` (`:5298`) and `TryTritonWU` (`:5361`), each reading `if (hv_n != 48 && hv_n != 32) return false;` on top of `dk == 128 && dv == 128 && hk_n == 16`. Those two are the only vendored specializations (`src/vt/cuda/triton_aot_vendored/*/gdn_{decode,deltah,chunko,wu}_h{48,32}.*`): 48 is the dense 27B and 32 is `Qwen3.6-35B-A3B`. `Qwen/Qwen3.8-2.4T-A95B` has 128 linear V-heads ([`qwen38-text-only.md`](specs/qwen38-text-only.md)) and clears every other term, so it is rejected on `hv_n` alone and runs the hand CUDA kernels on all four legs — the ones `.agents/kernel-matrix.md` measured by cuobjdump at REG:255 + STACK:48 (spilling) against the vLLM FLA cubin's REG:205 / 0 spill, which is the whole reason the vendored cubins exist and are default-on. `Qwen3.8-27B` is NOT affected: it is the `Qwen3.6-27B` geometry retrained, 48 V-heads, and hits every AOT arm. Neither reference restricts the head count — SGLang's `TritonGDNKernel` sets `supports_packed_decode` from the platform alone and takes `num_v_heads` as a runtime argument (`python/sglang/srt/layers/attention/linear/kernels/gdn_triton.py:43` @ `f63458b5be`), and `VLLM_ENABLE_FLA_PACKED_RECURRENT_DECODE` has no shape term (`vllm/envs.py:124` @ `5559679`); both JIT-compile per shape, which is the property the AOT vendoring trades away for a Python-free runtime. Closing it needs `h128` specializations vendored across the supported architectures, or a stated rule for which head counts get an AOT arm plus a visible fallback cost at the call site. Filed, not fixed: either close needs the checkpoint that motivates it, and this hardware cannot run the 2.4T (~4.8 TB bf16 against 128 GB unified memory), so the fallback cannot be measured here today. Listed under `## Owed` in [`gdn-moe-bf16-out.md`](specs/gdn-moe-bf16-out.md) | perf |
 | [#1171](https://github.com/mudler/vllm.cpp/issues/1171) | `KERNEL-GDN-REPLAYSSM` | GDN decode rewrites the whole `[HV,V,K]` fp32 state every step (`src/vt/cuda/cuda_gdn.cu:2393` reads the tile, `:2425` writes it back), which at the 27B shape `HV=32, V=128, K=128` is 2,097,152 bytes read plus the same written per layer, per request, per token. ReplaySSM keeps a per-slot ring of the last `L` steps' rank-1 factors `(d, k, g)`, reconstructs the state in registers, and writes it back only every `L` steps. vLLM implements the algorithm at the pin `555967922` for Mamba2 selective-state ONLY (`layers/mamba/ops/selective_state_update_replayssm_output_only.py`, ring shapes/dtypes `mamba_utils.py:84-93,202-221`, `use_replayssm` default `False` at `config/cache.py:152`, introduced `866fea2b` #48018) and it cannot reach GDN: `config/vllm.py:2318-2322` refuses any model not setting `supports_replayssm`, only `NemotronHForCausalLM` sets it (`models/nemotron_h.py:711`), `GDNAttentionMetadataBuilder` does not subclass the Mamba builder that derives the cursor (`v1/attention/backends/gdn_attn.py:82` vs `mamba_attn.py:575-638`), and the kernel hard-requires a scalar-per-head `A` (`:540-542`) with the Mamba2 `(B,C)` group structure (`:529`). Still true 877 commits past the pin. SGLang ported it to GDN at our recorded pin `f63458b5` (`layers/attention/fla/fused_recurrent_linear_replayssm.py`, whose `:50` credits vLLM; `--enable-linear-replayssm` default `False` and `--linear-replayssm-cache-len` default 16 at `server_args.py:1972-1986`; rings `memory_pool.py:465-483`; commit `a10a24e9` #28451), so the algorithm is a vLLM mirror and the GDN application is a secondary-oracle port. PAYOFF UNMEASURED HERE and deliberately not claimed: ReplaySSM removes the state WRITE and not the read, the flush step reads the checkpoint a SECOND time, so the honest state ratio is `(1+2/L)/2` = 0.5625 at `L=16` against SGLang's published 0.53x which models neither the flush re-read nor any ring read; the ring itself adds 395,264 bytes per slot per layer at the 27B shape = **+18.9% KV page**, worse than vLLM's ~7% on Nemotron because GDN's state is `V*K` while the ring is `L*(V+K)`; and SGLang's own end-to-end figure is ~2.3% TPOT at 128 concurrency on an MoE model. Neither upstream is bit-exact against its unbuffered path and neither claims to be. Motivation is the open Qwen3.8-27B bf16 decode gap (c4 total 0.918x, output 0.963x, `docs/BENCHMARKS.md:192-205`). Spec [`gdn-replayssm.md`](specs/gdn-replayssm.md) | perf |
+| [#1179](https://github.com/mudler/vllm.cpp/issues/1179) | `ENG-CUDAGRAPH-BREAK` | The hand-rolled decode-graph driver count recorded in `9bc4d7f44` is **eight** and is actually **nine**, and the row it feeds was framed as coverage-only when it is also correctness. The ninth is the DFlash draft graph, file-local with no header declaration, at `src/vllm/model_executor/models/qwen3_dflash.cpp:771,870,1038,1091,1095,1106` — its own `int g_state = 0` three-state machine (`:771`), its own `VT_DFLASH_GRAPH` kill switch (`:870`) instead of the `VLLM_CPP_CUDAGRAPH` the six batched drivers read, its own invalidate-on-block-width-change (`:1038-1047`) and its own `try { EndCaptureGraph(); } catch (...) {}` drain (`:1106`). The eight-count is stated in four places, all corrected here: [`sglang-breakable-cuda-graph.md`](specs/sglang-breakable-cuda-graph.md) §4 and `## Owed`, [`.agents/engine-matrix.md`](engine-matrix.md) rows `ENG-CUDAGRAPH-BREAK` and `ENG-CUDAGRAPH-DEDUP` ("times eight drivers", which sizes #1162's signature table), and [`.agents/roadmap_v1.md`](roadmap_v1.md) track `C12`. The reframing is the substantive half: `ENG-CUDAGRAPH-BREAK` was recorded as a COVERAGE row, and the duplication has already cost a SHIPPED model its decode graph. `src/vllm/model_executor/models/qwen3.cpp:961-986` declines the decode graph outright whenever the asynchronous device-token mirror is live, on its own measured battery — `depth-1, graph ON PASS 78/78`; `depth-2, graph OFF PASS 82/82`; `depth-2, graph ON FAIL, slots 1-3 degenerate` — because `Step()` replays against the HOST `input.token_ids` and the combine has patched the DEVICE ids. The comment names the real fix as reading the identifiers at replay time from a stable device buffer, and that fix exists, in exactly one sibling driver, as `StepDevInputs` (`src/vllm/model_executor/models/qwen3_5.cpp:3894`): `grep -c StepDevInputs` returns 41 lines there and 0 in each of `qwen3_moe.cpp`, `qwen3.cpp`, `deepseek_v2.cpp` and `voxtral.cpp`. One capability, written once, unavailable to four models, with a live mitigation standing in its place. This does NOT weaken the framing rule that `ENG-CUDAGRAPH` established: the row still makes no throughput claim, and the prefill refutation (GB10 3.8% host-idle between launches, GPU-busy >96%, 27B prefill gap 92.5% non-GEMM glue) stands unchanged. Coverage AND correctness, never speed. Fixed in flow with the [`eng-cudagraph-break.md`](specs/eng-cudagraph-break.md) review repair ([#1163](https://github.com/mudler/vllm.cpp/issues/1163)) | record |

.agents/roadmap_v1.md

@@ -149,7 +149,7 @@ live in [feature-matrix.md](feature-matrix.md).
 | C9 | Recurring upstream sync cycle and P1 sync tooling | 🔁 recurring; [upstream-sync.md](upstream-sync.md) |
 | C10 | **Qwen3.5 high-throughput lever set** (user-directed 2026-08-08, from [vllm.ai 2026-08-06 "25K tok/s/GPU"](https://vllm.ai/blog/2026-08-06-qwen35-25k-tps)): mirror the three levers upstream credits. **(a) Blackwell GDN prefill** replacing the FLA/Triton kernel (FlashInfer PR #3001, vLLM PR #40717, `--gdn-prefill-backend flashinfer`): 1.02x-5.78x on the kernel across sizes, 5.92x microbenchmark, **1.13x e2e prefill throughput, 12% lower mean TTFT**. **(b) Hybrid cache + GDN-state transfer** (vLLM PR #36687 + #37416/#37635/#37310/#41869, `VLLM_SSM_CONV_STATE_LAYOUT=DS`): ~7% intra-node H100 from cutting transferred descriptors 4284 to 1650. **(c) Race-free async scheduling** (vLLM PRs #48481/#45357, `--async-scheduling`), credited as key but not isolated | ☐ SPIKE-first. **PIN MOVE REQUIRED, and it is the first deliverable:** every referenced vLLM PR post-dates our parity pin `555967922` (0.26.0.dev0), so none of this is mirrorable until the pin advances. The spike must (1) find the earliest upstream ref containing #40717 + #36687 + #48481, (2) prove that ref BUILDS AND RUNS a greedy golden on GB10 per [`oracle-gateability`](verification.md), (3) re-capture the SACRED goldens and show ZERO drift before any lever is ported. **Honest scoping:** upstream measured a GB200/NVLink72 DISAGGREGATED cluster (1xDEP8 decode, 4-8 prefill endpoints, 8192/1024, concurrency 64-5120) - one GB10 cannot reproduce it, so **25K is NOT adopted as our bar**. Only (a) is directly portable single-GPU; (c) overlaps the `VT_ASYNC_RUNNER` W3 work already default-OFF under C6; (b) is gated on the scale-out lane |
 | C11 | **Krea 2 image generation** (user-directed 2026-08-08, [krea-ai/krea-2](https://github.com/krea-ai/krea-2)): text to image diffusion transformer. Two community-licensed checkpoints, `krea/krea-2-raw` (base) and `krea/krea-2-turbo` (8-step distilled, up to 2k). Reference impl is PyTorch in-repo (`mmdit.py` MultiModalDiT + `autoencoder.py` + `sampling.py`) | ☐ SPIKE-first: arch inventory + reuse map against the MiniMax-H3 DiT lane, which already owns a joint video+audio diffusion transformer, an NVFP4/GGUF DiT weight-streaming loader and the `vllm_video_*` ABI slice - a text-to-image DiT is the SIMPLER sibling and should reuse the sampler / VAE-decode / DiT-forward seams rather than start new. **No pin move needed** (nothing upstream to track). **Upstream-mirror caveat:** vLLM has no Krea 2 path, so this is a from-scratch port owing a `porting-inventory.md` section 9 entry with the krea-2 repo as reference root and its PyTorch reference as the numerical oracle |
-| C12 | **SGLang Breakable CUDA Graph (BCG), evaluated and mostly REFUTED** (user-directed 2026-08-17, from a public SGLang claim: *"first engine to develop and land BCG, the full CUDA Graph, and graph memory reuse"*). Grounded in the pinned SGLang tree `v0.5.15`/`f63458b5be` rather than in the claim text: BCG captures one forward as a SEQUENCE of `torch.cuda.CUDAGraph` segments split at eager break points on the attention and recurrent calls, with no `torch.compile` (`runner_backend_utils/breakable_cuda_graph/breakable_cuda_graph.py:204-241,244-260`; `runner_backend/breakable_cuda_graph_backend.py:14-17`). **The priority claim is narrower than it reads:** vLLM's v1 default is already `FULL_AND_PIECEWISE`, splitting at `splitting_ops` (`vllm/config/compilation.py:60-63,615,630` @ `555967922`), so the CAPABILITY is in our denominator; what SGLang landed first is the CONSTRUCTION — piecewise coverage with no Dynamo/Inductor/FX. | ☐ T1 (2026-08-17, [#1161](https://github.com/mudler/vllm.cpp/issues/1161)); spec [sglang-breakable-cuda-graph.md](specs/sglang-breakable-cuda-graph.md); row `SGLANG-BCG` in [sglang-matrix.md](sglang-matrix.md). **Four outcomes, three of them not throughput.** (a) *Dropping `torch.compile`* is NO lever here — we never had a compiler; `src/vt/cuda/cuda_backend.cu:203-232` is already raw stream capture. (b) *Prefill capture* is **REFUTED** on our CUDA gate models and recorded against `ENG-CUDAGRAPH` so nobody re-derives it: GB10 2026-07-09 measured prefill GPU-idle-between-launches at **3.8%** with GPU-busy >96% on both arms, and the 27B prefill gap at **92.5% non-GEMM glue GPU work** (dominant GEMM +0.17%, attention AHEAD). No launch bubbles to collapse. (c) *Graph memory reuse* is **real and unported** — `cudaGraphExecUpdate` appears nowhere in `src/` or `include/`, while we instantiate one exec per padded bucket across eight hand-rolled drivers → `ENG-CUDAGRAPH-DEDUP` `INVENTORIED` ([#1162](https://github.com/mudler/vllm.cpp/issues/1162)). (d) The analysis surfaced a STRUCTURAL row the claim did not: our capture is all-or-nothing (`runner.cpp:1338-1341` routes only `pure_decode`) and hand-rolled eight times → `ENG-CUDAGRAPH-BREAK` `INVENTORIED` ([#1163](https://github.com/mudler/vllm.cpp/issues/1163)), a COVERAGE row, not a throughput row. Diffusion looked like the one real lever (SGLang measured LTX-2 H200 10.75s→6.90s post-pin) but is **BLOCKED by our own defects**: an LTX-2.5 render does almost no device compute to capture — GPU util **exactly 0 in 321 of 347 samples** ([#1024](https://github.com/mudler/vllm.cpp/issues/1024)), no device arm for the VAE decode ([#1007](https://github.com/mudler/vllm.cpp/issues/1007)), and **57-66% of wall** in one resolution-CONSTANT serial host phase ([#1087](https://github.com/mudler/vllm.cpp/issues/1087)) → `ENG-CUDAGRAPH-DIFFUSION` `INVENTORIED` ([#1164](https://github.com/mudler/vllm.cpp/issues/1164)) |
+| C12 | **SGLang Breakable CUDA Graph (BCG), evaluated and mostly REFUTED** (user-directed 2026-08-17, from a public SGLang claim: *"first engine to develop and land BCG, the full CUDA Graph, and graph memory reuse"*). Grounded in the pinned SGLang tree `v0.5.15`/`f63458b5be` rather than in the claim text: BCG captures one forward as a SEQUENCE of `torch.cuda.CUDAGraph` segments split at eager break points on the attention and recurrent calls, with no `torch.compile` (`runner_backend_utils/breakable_cuda_graph/breakable_cuda_graph.py:204-241,244-260`; `runner_backend/breakable_cuda_graph_backend.py:14-17`). **The priority claim is narrower than it reads:** vLLM's v1 default is already `FULL_AND_PIECEWISE`, splitting at `splitting_ops` (`vllm/config/compilation.py:60-63,615,630` @ `555967922`), so the CAPABILITY is in our denominator; what SGLang landed first is the CONSTRUCTION — piecewise coverage with no Dynamo/Inductor/FX. | ☐ T1 (2026-08-17, [#1161](https://github.com/mudler/vllm.cpp/issues/1161)); spec [sglang-breakable-cuda-graph.md](specs/sglang-breakable-cuda-graph.md); row `SGLANG-BCG` in [sglang-matrix.md](sglang-matrix.md). **Four outcomes, three of them not throughput.** (a) *Dropping `torch.compile`* is NO lever here — we never had a compiler; `src/vt/cuda/cuda_backend.cu:203-232` is already raw stream capture. (b) *Prefill capture* is **REFUTED** on our CUDA gate models and recorded against `ENG-CUDAGRAPH` so nobody re-derives it: GB10 2026-07-09 measured prefill GPU-idle-between-launches at **3.8%** with GPU-busy >96% on both arms, and the 27B prefill gap at **92.5% non-GEMM glue GPU work** (dominant GEMM +0.17%, attention AHEAD). No launch bubbles to collapse. (c) *Graph memory reuse* is **real and unported** — `cudaGraphExecUpdate` appears nowhere in `src/` or `include/`, while we instantiate one exec per padded bucket across NINE hand-rolled drivers → `ENG-CUDAGRAPH-DEDUP` `INVENTORIED` ([#1162](https://github.com/mudler/vllm.cpp/issues/1162)). (d) The analysis surfaced a STRUCTURAL row the claim did not: our capture is all-or-nothing (`runner.cpp:1338-1341` routes only `pure_decode`) and hand-rolled **nine** times (count corrected 2026-08-18, [#1179](https://github.com/mudler/vllm.cpp/issues/1179): the DFlash draft graph `qwen3_dflash.cpp:771,870,1038,1091,1095,1106` was missed) → `ENG-CUDAGRAPH-BREAK` `READY` ([#1163](https://github.com/mudler/vllm.cpp/issues/1163)), a COVERAGE **and CORRECTNESS** row, not a throughput row — `qwen3.cpp:961-986` already DECLINES its decode graph while the async device-token mirror is live (`depth-2, graph ON FAIL, slots 1-3 degenerate`) because `StepDevInputs` (`qwen3_5.cpp:3894`) exists in one driver only. Diffusion looked like the one real lever (SGLang measured LTX-2 H200 10.75s→6.90s post-pin) but is **BLOCKED by our own defects**: an LTX-2.5 render does almost no device compute to capture — GPU util **exactly 0 in 321 of 347 samples** ([#1024](https://github.com/mudler/vllm.cpp/issues/1024)), no device arm for the VAE decode ([#1007](https://github.com/mudler/vllm.cpp/issues/1007)), and **57-66% of wall** in one resolution-CONSTANT serial host phase ([#1087](https://github.com/mudler/vllm.cpp/issues/1087)) → `ENG-CUDAGRAPH-DIFFUSION` `INVENTORIED` ([#1164](https://github.com/mudler/vllm.cpp/issues/1164)) |
 
 ## D. T2 (after T1, per porting-inventory.md)