diff --git a/.agents/engine-matrix.md b/.agents/engine-matrix.md index 0b049156d..4c42aa9f4 100644 --- a/.agents/engine-matrix.md +++ b/.agents/engine-matrix.md @@ -38,7 +38,7 @@ forensics: roadmap_v1.md and the parity ledger. | Area | Rows | `ANCHOR-BACKFILL` | `PARTIAL` | `SPIKE` | `READY` | `ACTIVE` | `GATING` | `DONE` | `INVENTORIED` | |---|---:|---:|---:|---:|---:|---:|---:|---:|---:| -| Engine and scheduling | 30 | 6 | 3 | 1 | 3 | 6 | 2 | 2 | 7 | +| Engine and scheduling | 30 | 6 | 3 | 1 | 4 | 6 | 2 | 2 | 6 | | KV cache and memory | 23 | 7 | 3 | 2 | 3 | 2 | 2 | 0 | 4 | | Parallelism | 6 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 5 | | Sampling and generation | 15 | 4 | 2 | 0 | 0 | 4 | 0 | 1 | 4 | @@ -48,7 +48,7 @@ forensics: roadmap_v1.md and the parity ledger. | LoRA and adapters | 2 | 0 | 0 | 0 | 0 | 1 | 0 | 0 | 1 | | Long context and attention | 10 | 5 | 0 | 0 | 1 | 0 | 1 | 0 | 3 | | Loading, tokenizer, config | 10 | 3 | 3 | 0 | 0 | 1 | 1 | 1 | 1 | -| **Total** | **160** | **35** | **17** | **5** | **11** | **32** | **8** | **11** | **40** | +| **Total** | **160** | **35** | **17** | **5** | **12** | **32** | **8** | **11** | **39** | ## Engine core and scheduling @@ -60,8 +60,8 @@ forensics: roadmap_v1.md and the parity ledger. | `KV-PREFIX-MATCH-UNIT` | `--prefix-match-unit` (config `prefix_match_unit`): the finest token boundary a prefix-cache hit can land on == the `hash_block_size`/"prefix match unit" the block hasher uses. NEW in 0.26 (absent at the prior `e24d1b24`/0.25.0 pin). For a HYBRID/multi-group model the resolver `resolve_kv_cache_block_sizes` computes `hash_block_size = prefix_match_unit if set else gcd(group_block_sizes)` (scheduler block size = `lcm`), letting matching land FINER than a physical block (e.g. 16/32 tokens inside a 1024-token block) provided every group block size is divisible by it; single-group (dense) models ignore the knob. Backs off to the scheduler block size when no prefix-cache/connector consumer is active or a mamba group diverges from `cache_block_size` (mamba_cache_mode != "align"); throws on a non-divisible unit. **W0 spike + W1 resolver LANDED 2026-07-28 (`CLAIM-PREFIX-MATCH-UNIT`, NOT pushed):** `resolve_kv_cache_block_sizes` ported 1:1 (explicit-parameter signature vs upstream's `VllmConfig`, our config surface is threaded), RED-first unit-gated (default gcd `!=` `=16` override). `PARTIAL`: the config/CLI/ABI field (W2), the scheduler threading of a resolved `hash_block_size != block_size` + mamba partial-tail stop (W3, needs the `KV-BLOCK-POOL` align path that still throws), and the benchmark (W4) are deferred. Default path byte-identical (single-group inert; scheduler still passes `block_size`). | T1 | `vllm/engine/arg_utils.py:696,1222,1940`; `vllm/config/cache.py:56-67`; resolver `vllm/v1/core/kv_cache_utils.py:626-688`; hasher `:691-748`; call site `vllm/v1/engine/core.py:154`; scheduler `vllm/v1/core/sched/scheduler.py:76,268-270,282,312-318`; fine-grained view `vllm/v1/core/single_type_kv_cache_manager.py:683,697` | resolver `src/vllm/v1/core/kv_cache_utils.cpp:638` (`resolve_kv_cache_block_sizes`), decl `include/vllm/v1/core/kv_cache_utils.h`; hash_block_size already plumbed `get_request_block_hasher` `src/vllm/v1/core/kv_cache_utils.cpp:577`; DEFERRED align path throws `src/vllm/v1/core/block_pool.cpp:93,220` (shared with `KV-BLOCK-POOL`) | `tests/vllm/v1/test_prefix_match_unit.cpp:64,88,99,119,129,145,164,186` 8/8 (29 assertions): single-group inert + DCP scale, multi-group default=gcd, `=16` override finer-than-default (RED), finer-than-1024-block, non-divisible throws, no-consumer back-off + connector-alone re-enable, mamba non-align back-off vs align gcd, hasher-granularity RED (coarse 2 vs fine 4 hashes); [parity-ledger.md](parity-ledger.md) | [prefix-match-unit.md](specs/prefix-match-unit.md) | `PARTIAL` | `CLAIM-PREFIX-MATCH-UNIT` | | `ENG-PREEMPT-RECOMPUTE` | FCFS tail preemption with recompute | T0 | `vllm/v1/core/sched/scheduler.py:1142`; `tests/v1/core/test_scheduler.py:930` | `src/vllm/v1/core/sched/scheduler.cpp:102,157`; `src/vllm/v1/core/sched/request_queue.cpp:36` | `tests/vllm/v1/test_scheduler.cpp:247,295`; `tests/vllm/v1/test_request_queue.cpp:91` | `planned: specs/preemption.md` | `ANCHOR-BACKFILL` | - | | `ENG-CUDAGRAPH` | Decode graph capture/replay modes (host-cluster cleanup: capture-size set derived from `max_num_seqs` mirroring vLLM `_set_cudagraph_sizes`; 2026-07-18 graph-baked-scratch use-after-free fix — the 35B c2+ online-serving IMA blocker) | T0 | `vllm/config/compilation.py:53,1319,683-684,1438-1444`; `vllm/config/vllm.py:1667-1770`; `vllm/v1/worker/gpu/cudagraph_utils.py:116`; `tests/compile/test_config.py:122,229` | `src/vt/cuda/cuda_backend.cu:76,97,105`; `include/vllm/model_executor/models/decode_graph_sizes.h`; `src/vllm/model_executor/models/qwen3_5.cpp:3754,3952`; `src/vllm/v1/worker/gpu/runner.cpp:577,597`; graph-safe scratch (retire-on-grow so graph-baked scratch pointers stay valid) `src/vt/cuda/graph_safe_scratch.h`, `src/vt/cuda/cuda_moe_marlin.cu:75`, `src/vt/cuda/cuda_matmul_nvfp4.cu:766`, `src/vt/cuda/cuda_matmul_nvfp4_cutlass.cu:105`, `src/vt/cuda/cuda_matmul_fp8_cutlass.cu:95` | `tests/vt/test_cuda_backend.cpp:98`; `tests/vllm/models/test_decode_graph_sizes.cpp`; `tests/vt/test_graph_safe_scratch.cpp`; explicit 35B gate `tests/parity/test_qwen36_paged_engine.cpp:140` | [blocktable-host-cluster-cleanup.md](specs/blocktable-host-cluster-cleanup.md); [decode-graph-scratch-uaf-2026-07-18.md](specs/decode-graph-scratch-uaf-2026-07-18.md) | `PARTIAL` | **PREFILL capture REFUTED as a lever (2026-08-17, [#1161](https://github.com/mudler/vllm.cpp/issues/1161)).** vLLM's v1 default already captures prefill piecewise (`vllm/config/compilation.py:60-63,615,630` @ `555967922`) and it is in our denominator; SGLang reached the same coverage without `torch.compile` via BCG (`SGLANG-BCG` in [sglang-matrix.md](sglang-matrix.md)). Neither helps us: GB10 2026-07-09 measured prefill GPU-idle-between-launches at **3.8%** with GPU-busy >96% on both arms, and the 27B prefill gap at **92.5% non-GEMM glue GPU work** with the dominant GEMM at +0.17% and attention AHEAD. There are no launch bubbles in our prefill to collapse. Row stays `PARTIAL`; the real residuals are exec dedup ([#1162](https://github.com/mudler/vllm.cpp/issues/1162)) and the break-point seam ([#1163](https://github.com/mudler/vllm.cpp/issues/1163)). Spec [sglang-breakable-cuda-graph.md](specs/sglang-breakable-cuda-graph.md) | -| `ENG-CUDAGRAPH-DEDUP` | Graph-executable dedup: hash each captured graph's topology and re-point ONE `cudaGraphExec` with `cudaGraphExecUpdate` on a signature hit, instead of instantiating one exec per padded bucket per model. A memory and capture-time change, NOT a throughput change — a deduped replay launches the same nodes | T2 | vLLM has no analogue (its execs come from `torch.compile`); SGLang `python/sglang/srt/model_executor/runner_backend/cuda_graph_dedup_mixin.py:219-242,358` @ `f63458b5be` | UNPORTED — `cudaGraphExecUpdate` appears nowhere in `src/` or `include/`; today `src/vt/cuda/cuda_backend.cu:222-232` instantiates a fresh exec per capture, over the 7 (`max_num_seqs=32`) or 11 (64) buckets of `include/vllm/model_executor/models/decode_graph_sizes.h:32-41`, times eight drivers | owed: same-binary A/B proving a deduped replay is byte-identical to a non-deduped one; a capture-count/exec-count log line so the ratio is observable | [sglang-breakable-cuda-graph.md](specs/sglang-breakable-cuda-graph.md) | `INVENTORIED` | [#1162](https://github.com/mudler/vllm.cpp/issues/1162) | -| `ENG-CUDAGRAPH-BREAK` | One shared `vt` capture seam that accepts BREAK POINTS, so a forward containing a host-dependent op is still graphed instead of falling out entirely — and so the eight hand-rolled drivers become one. **Coverage row, not a throughput row** | T1 | mirror vLLM `CUDAGraphMode.PIECEWISE` splitting at `splitting_ops` (`vllm/config/compilation.py:60-63,517,615,630` @ `555967922`); construction from SGLang BCG `breakable_cuda_graph.py:204-241,244-260` @ `f63458b5be` (decorator + runtime stream capture, no compiler) | all-or-nothing today: `src/vllm/v1/worker/gpu/runner.cpp:1338-1341` routes only `pure_decode`; drivers `qwen3_5.h:275`, `qwen3_5_dense.h:391`, `qwen3_moe.h:117`, `qwen3.h:243`, `deepseek_v2.h:324`, `voxtral.h:126`, plus `deepseek_v4.cpp`, `laguna.cpp` | owed: reachability mutation (delete the production call site, rerun the focused gate); bit-exactness vs eager on every migrated model over MORE than one replay; the host-lifetime contract of [decode-graph-scratch-uaf-2026-07-18.md](specs/decode-graph-scratch-uaf-2026-07-18.md) enforced AT the seam | [sglang-breakable-cuda-graph.md](specs/sglang-breakable-cuda-graph.md) | `INVENTORIED` | [#1163](https://github.com/mudler/vllm.cpp/issues/1163), [#1020](https://github.com/mudler/vllm.cpp/issues/1020) | +| `ENG-CUDAGRAPH-DEDUP` | Graph-executable dedup: hash each captured graph's topology and re-point ONE `cudaGraphExec` with `cudaGraphExecUpdate` on a signature hit, instead of instantiating one exec per padded bucket per model. A memory and capture-time change, NOT a throughput change — a deduped replay launches the same nodes | T2 | vLLM has no analogue (its execs come from `torch.compile`); SGLang `python/sglang/srt/model_executor/runner_backend/cuda_graph_dedup_mixin.py:219-242,358` @ `f63458b5be` | UNPORTED — `cudaGraphExecUpdate` appears nowhere in `src/` or `include/`; today `src/vt/cuda/cuda_backend.cu:222-232` instantiates a fresh exec per capture, over the 7 (`max_num_seqs=32`) or 11 (64) buckets of `include/vllm/model_executor/models/decode_graph_sizes.h:32-41`, times NINE drivers (count corrected 2026-08-18, [#1179](https://github.com/mudler/vllm.cpp/issues/1179); `9bc4d7f44` recorded eight, missing the DFlash draft graph `src/vllm/model_executor/models/qwen3_dflash.cpp:771,870,1038,1091,1095,1106`) | owed: same-binary A/B proving a deduped replay is byte-identical to a non-deduped one; a capture-count/exec-count log line so the ratio is observable | [sglang-breakable-cuda-graph.md](specs/sglang-breakable-cuda-graph.md) | `INVENTORIED` | [#1162](https://github.com/mudler/vllm.cpp/issues/1162) | +| `ENG-CUDAGRAPH-BREAK` | One shared `vt` capture seam that accepts BREAK POINTS, so a forward containing a host-dependent op is still graphed instead of falling out entirely — and so the NINE hand-rolled drivers become one (count corrected 2026-08-18, [#1179](https://github.com/mudler/vllm.cpp/issues/1179); `9bc4d7f44` recorded eight). **Coverage AND CORRECTNESS row, not a throughput row** | T1 | mirror vLLM `CUDAGraphMode.PIECEWISE` splitting at `splitting_ops` (`vllm/config/compilation.py:60-63,517,615,630` @ `555967922`); construction from SGLang BCG `python/sglang/srt/model_executor/runner_backend_utils/breakable_cuda_graph/breakable_cuda_graph.py:204-243,246-274,309-333,335-367` @ `f63458b5be` (decorator + runtime stream capture, no compiler); its unit suite `test/registered/cuda_graph/breakable/test_breakable_cuda_graph.py:30,172,230` (305 lines, 11 unit cases) is mapped case for case in the spec's `## Tests to port` | all-or-nothing today: `src/vllm/v1/worker/gpu/runner.cpp:1338-1341` routes only `pure_decode`; drivers `qwen3_5.h:275`, `qwen3_5_dense.h:391`, `qwen3_moe.h:117`, `qwen3.h:243`, `deepseek_v2.h:324`, `voxtral.h:126`, plus `deepseek_v4.cpp`, `laguna.cpp` — and the spike found the NINTH already written, `src/vllm/model_executor/models/qwen3_dflash.cpp:771,1091`. The re-derivation is measured, not asserted: `StepDevInputs` (`src/vllm/model_executor/models/qwen3_5.cpp:3894`, the persistent DEVICE input path) exists in ONE driver and `grep -c` returns 0 in `qwen3_moe.cpp`, `qwen3.cpp`, `deepseek_v2.cpp` and `voxtral.cpp`, which is why `src/vllm/model_executor/models/qwen3.cpp:961-986` DECLINES the graph outright when the async device-token mirror is live. **That decline is why this is also a CORRECTNESS row** ([#1179](https://github.com/mudler/vllm.cpp/issues/1179)): a SHIPPED model has already lost its decode graph to the duplication, on the driver's own measurement (`depth-1, graph ON PASS 78/78`; `depth-2, graph OFF PASS 82/82`; `depth-2, graph ON FAIL, slots 1-3 degenerate`), and the fix its comment names is the sibling's `StepDevInputs`. The row still makes NO throughput claim: the prefill refutation on the `ENG-CUDAGRAPH` row (3.8% host idle, >96% GPU-busy, 92.5% glue) stands unchanged | owed: reachability mutation (delete the production call site, rerun the focused gate); bit-exactness vs eager on every migrated model over MORE than one replay; the host-lifetime contract of [decode-graph-scratch-uaf-2026-07-18.md](specs/decode-graph-scratch-uaf-2026-07-18.md) enforced AT the seam; the 11 ported SGLang unit cases; the break-function OUTPUT writeback (`replay_fn`/`_copy_output` `breakable_cuda_graph.py:231-235,172-201`, spec D9) without which a replay leaves the next segment reading capture-time data; the auxiliary-stream auto-join before every segment close (`:353-361`, spec D10), live at `src/vllm/model_executor/models/qwen3_5.cpp:6254-6255,6384` and `src/vllm/model_executor/models/laguna.cpp:2572-2576,2612` | spec [eng-cudagraph-break.md](specs/eng-cudagraph-break.md) (W0 spike DONE 2026-08-18: the existing `vt` capture vocabulary `include/vt/backend.h:208-222` expresses a SEGMENTED capture with NO new virtual, because `EndCaptureGraph` stores nothing (`src/vt/cuda/cuda_backend.cu:225-232`); a break point is expressible with one `thread_local` capture pointer plus a free function, no compiler and no decorator); analysis [sglang-breakable-cuda-graph.md](specs/sglang-breakable-cuda-graph.md) | `READY` | [#1163](https://github.com/mudler/vllm.cpp/issues/1163), [#1020](https://github.com/mudler/vllm.cpp/issues/1020) | | `ENG-CUDAGRAPH-DIFFUSION` | Capture the LTX-2.5 denoise loop (fixed shapes, many identical iterations — the ideal graph target). **BLOCKED, and the blocker is ours:** the render does almost no device compute to capture | T2 | SGLang enabled BCG on this shape AFTER our pin — LTX-2 H200 two-stage 10.75s->6.90s (`d4be483efb`), SANA 1024px -26% (`6c7498113f`), SANA denoise 0.73->0.457s (`56ef810cad`). Dated events, NOT pinned evidence; their win is mostly PyTorch host tax we do not pay | NO capture at all: `grep` for capture across `src/vllm/model_executor/models/ltx2*.cpp` returns nothing | blocked by [#1024](https://github.com/mudler/vllm.cpp/issues/1024) (GPU util **exactly 0 in 321 of 347 samples**, 1.00 core of 20 held for 17+ min after staging), [#1007](https://github.com/mudler/vllm.cpp/issues/1007) (VAE decode has no device arm), [#1087](https://github.com/mudler/vllm.cpp/issues/1087) (**57-66% of wall** is ONE resolution-CONSTANT serial host phase), [#1010](https://github.com/mudler/vllm.cpp/issues/1010) (no phase-boundary log). Decision point is a MEASUREMENT of GPU-busy vs wall once device-resident, not an implementation | [sglang-breakable-cuda-graph.md](specs/sglang-breakable-cuda-graph.md) | `INVENTORIED` | [#1164](https://github.com/mudler/vllm.cpp/issues/1164) | | `ENG-BATCH-INVARIANT` | Opt-in deterministic execution across scheduler batch sizes (`VLLM_BATCH_INVARIANT=1`): batch-invariant matmul/norm/attention/collectives plus persistent-scheduler NVFP4; production default remains off | T1 | default/env `vllm/envs.py:89,576-578`; initialization `vllm/v1/worker/gpu_worker.py:1262`; NVFP4 dispatch `csrc/libtorch_stable/quantization/fp4/nvfp4_scaled_mm_sm120_kernels.cu:212-220`; suite fixture `tests/v1/determinism/conftest.py:9-12`; operator/e2e `tests/v1/determinism/test_nvfp4_batch_invariant_scaled_mm.py`, `tests/v1/determinism/test_nvfp4_batch_invariant.py` @ `702f481` | - | [W3-C3R executed contract](specs/nvfp4-persistent-plan-cache.md#w3-c3r-batch-shape-localization-and-gate-correction-2026-07-13): production-default ours and vLLM both change outputs across batch shapes; no local opt-in implementation is claimed | `planned: specs/batch-invariant-execution.md` | `INVENTORIED` | - | | `ENG-ASYNC-SCHED` | Async/overlap scheduling (AsyncScheduler placeholders + depth-2 batch-queue step + async D2H on a copy stream); vLLM's DEFAULT at the pin — mirror obligation per B3. **Host-side machinery + runner device-input half + sampler-OUTPUT half LANDED + CPU-gated (2026-07-16):** `AsyncScheduler` placeholder accounting, `step_with_batch_queue` depth-2, `ResolveAsyncScheduling` default-ON-when-compatible + `MaxConcurrentBatches`, `VT_ASYNC_SCHED` rollback; the runner device-input path `combine_sampled_and_draft_tokens`; PLUS the sampler-OUTPUT half — `vt::Backend` event/pinned primitives (`AllocPinned`/events, CUDA cudaHostAlloc+cudaEvent, CPU sync-degeneration), `AsyncGPUModelRunnerOutput` (device sampled-id snapshot → non-blocking D2H on a copy queue + event; `get_output()` waits only that event; MAIN queue never blocked), `Sampler::forward(sampled_ids_out)` device-resident greedy, `GPUModelRunner::sample_tokens_async` + `runner_supports_async`, and the `Executor`+`step_with_batch_queue` seam resolving `get_output()` at CONSUME time. All behind `VT_ASYNC_RUNNER`/`set_async_input_combine`, default OFF. Sync path byte-identical (placeholder sites INERT while count 0; combine off; `sample_tokens_async` degenerates to sync when async off; `sampled_ids_out=nullptr`). **ENABLE-FLIP LANDED + CPU-gated (2026-07-16):** (1) `LoadedEngine` now reorders `runner_` before the scheduler and builds an `AsyncScheduler` + `max_concurrent_batches=2` when `ResolveAsyncScheduling(runner_.runner_supports_async())` resolves ON (else the byte-identical synchronous `Scheduler` + depth-1); the resolved mcb threads into `AsyncLLM`→`EngineCoreProc` (`step_with_batch_queue`) and the "Asynchronous scheduling is enabled/disabled" log mirrors vLLM for A/B audit; (2) the device combine/scatter kernel (`_combine_sampled_and_draft_tokens_kernel` + last_sampled scatter) is ported to CUDA (`src/vt/cuda/cuda_combine_tokens.cu`), main-stream-ordered on the CUDA async path so it DELETES `sample_tokens_async`'s pre-scatter `Synchronize`; the CPU backend keeps the host loop. `VT_ASYNC_RUNNER=1` engages full W3; `VT_ASYNC_SCHED=0` is the same-binary rollback. Production default (no env) stays synchronous byte-identical. **FULL W3 DGX proof RAN twice** — `f086b64` (5/5 gates PASS; c16 TPOT −5.4 ms WIN, tput neutral, TTFT +36 % = Little's-law repayment) and the 2026-07-16 re-proof on the THROUGHPUT-lever fix (persistent pooled sampled-id/pinned buffers + `Sampler` greedy scratch removing ALL per-step `cudaMalloc`/`cudaFree`/`cudaHostAlloc`/event-create from the sampled-id path, incl. the overlap-killing `cudaFree` inside `get_output`; mirrors `gpu_model_runner.py:873-878` + `async_utils.py:12-70`): token-exactness **6/6 PASS**, interleaved c16 **tput −0.32 % (gate ≥+1.5 % FAILS), TPOT −4.95 ms retained, TTFT +34.8 %** — the allocator lever is REFUTED as the tput unlock (≤0.1 % of a ~165 ms c16 step). **DEFAULT FLIPPED ON 2026-07-17** (`VT_ASYNC_RUNNER` default ON via the pure `AsyncRunnerFlagIsOn` predicate, mirroring `vllm/config/vllm.py:992-1044`): the discriminator (`6ea7856`) proved vLLM's own async pays the identical +26–31 % TTFT / −0.7 to −0.9 % tput / −2.6 to −4.3 ms TPOT envelope and W3-ON nets positive (both binding ITL-tail anomalies flip to PASS), so the "needs a throughput lever" ship-gate is RETIRED — W3 is a parity/mirror obligation with a tails+TPOT win. The flip is TOKEN-NEUTRAL (async-ON ≡ async-OFF bit-identical on DGX). `VT_ASYNC_RUNNER=0` = runner-level rollback, `VT_ASYNC_SCHED=0` = scheduler-level rollback. TTFT means rise into vLLM's async envelope BY DESIGN — the next binding grid runs async by default and its TTFT must NOT be misread as a regression. **ROBUSTNESS FIX 2026-07-20 (`discard_request_mask`):** the runner was missing vLLM's `discard_request_mask`, so `GPUModelRunner` emitted a sampled token for prefill-CHUNK requests too; under async this drained a `num_output_placeholders` never reserved (the `is_prefill_chunk` path adds none) → the `async_scheduler.cpp` `num_output_placeholders >= 0` assertion aborted on c8 + short-output (chunked prefill + preemption). FIX mirrors vLLM: `execute_model` computes `exec_state_.discard[i] = seq_len < num_tokens` (`gpu_model_runner.py:2048`); `sample_tokens` clears those rows to empty (`outputs.py:303`), the async path passes `invalid_req_indices` to `AsyncGPUModelRunnerOutput::get_output` (`gpu_model_runner.py:3625` + `outputs.py:303`). Scheduler UNCHANGED (assertion kept — it was correct once the runner honors `scheduler.py:1888-1890`). Sync/non-chunked decode byte-identical (mask all-zero); DGX 27B 235/235 + 35B 315/315, `vllm-bench` c8+short-output+chunked+kv-pressure no longer crashes, memcheck 0. Ledger [parity-ledger.md](parity-ledger.md) 2026-07-20 row | T1 | `vllm/v1/core/sched/async_scheduler.py:12`; `vllm/config/vllm.py:490,990,1038`; `vllm/v1/engine/core.py:519`; `vllm/v1/worker/gpu/input_batch.py:304-406`; `vllm/v1/worker/gpu/async_utils.py:12-70`; `vllm/v1/worker/gpu/gpu_model_runner.py:242-332`; `vllm/v1/outputs.py:298-307` | `src/vllm/v1/core/sched/async_scheduler.cpp:10,45`; placeholder plumbing `src/vllm/v1/core/sched/scheduler.cpp:148,164,605`; `src/vllm/v1/engine/core.cpp:91` (`step_with_batch_queue`, async-output seam); `src/vllm/v1/engine/core_proc.cpp:32,46`; config `include/vllm/config/scheduler.h:117,165,188`, `src/vllm/config/scheduler.cpp:12`; `include/vllm/v1/request.h:187`; runner input leaf `src/vllm/v1/worker/gpu/prepare_inputs.cpp`, `src/vllm/v1/worker/gpu/input_batch.cpp`; runner output leaf `include/vt/backend.h`+`src/vt/backend.cpp`+`src/vt/cuda/cuda_backend.cu` (event/pinned), `include/vllm/v1/worker/gpu/async_output.{h,cpp}` (`AsyncGPUModelRunnerOutput`), `src/vllm/v1/sample/sampler.cpp` (`sampled_ids_out`), `src/vllm/v1/worker/gpu/runner.cpp` (`sample_tokens_async`/`runner_supports_async`), `src/vllm/v1/executor/executor.cpp`+`include/vllm/v1/worker/gpu/model_runner_base.h` (async seam); enable-flip `include/vllm/entrypoints/model_loader.h`+`src/vllm/entrypoints/model_loader.cpp` (`runner_` before scheduler, `ResolveAsyncEnabled`/`MakeScheduler`, `AsyncScheduler`+mcb=2, log), `include/vllm/v1/engine/async_llm.h`+`src/vllm/v1/engine/async_llm.cpp` (mcb param → `EngineCoreProc`); device kernel `include/vt/cuda/combine_tokens.h`+`src/vt/cuda/cuda_combine_tokens.cu`, wired `src/vllm/v1/worker/gpu/runner.cpp` (CUDA combine/scatter branch removes the pre-sync) | `tests/vllm/v1/test_async_scheduler.cpp:1` (6 cases, 54 asserts; RED vs base Scheduler 2/6 fail); depth-2 engine cycle `tests/vllm/v1/test_engine_core_proc.cpp:479` (mcb=2, async-output seam); config resolution `tests/vllm/test_scheduler_config.cpp:75`; enable-flip construction matrix `tests/vllm/entrypoints/test_loaded_engine_dense.cpp` (runner×VT_ASYNC_SCHED → scheduler type + mcb; RED = un-flipped engine, 3/3 ON-arm asserts fail); runner input leaf `test_combine_tokens.cpp` (RED = stale → 5/7 fail), `test_input_batch.cpp`, `test_runner.cpp` (async-ON≡sync); output leaf `tests/vt/test_backend.cpp` (event/pinned contract), `tests/vllm/v1/worker/test_async_output.cpp` (materialize/flush/snapshot; RED = +1 splice), `test_runner.cpp` (`sample_tokens_async` decode ≡ sync); full CPU ctest 111/111, tools 164/164. Prior diagnostic `3812d8` six-leg control: total **1.002153×**, TTFT **0.862159×**, no GPU-time reduction (neutral for speed). **DEFAULT-FLIP (2026-07-17):** new pure CPU flag test [test_async_runner_flag.cpp](../tests/vllm/v1/worker/test_async_runner_flag.cpp) (11 asserts, default-ON/'0'-off); construction matrix [test_loaded_engine_dense.cpp](../tests/vllm/entrypoints/test_loaded_engine_dense.cpp) INVERTED (default → AsyncScheduler+mcb=2; RED verified 5 asserts fail vs un-flipped). CPU clean `-Werror` rebuild, full serial ctest **116/116**, tools **164/164**. **DGX re-confirmation** (evidence `dgx:~/work/vllm.cpp-async-flip`, CUTLASS+FA2 hard-verified, one flock): shipping default (async ON + RMSNorm-fast OFF) → **27B 235/235 + 35B 315/315** with the "Asynchronous scheduling is enabled (mcb=2)" log, and both rollback arms (`VT_ASYNC_RUNNER=0`, `VT_ASYNC_SCHED=0`) 235/235 + 315/315 log "disabled"; async arms BIT-IDENTICAL (token-neutral). Closing record [parity-ledger.md#L502](parity-ledger.md#L502) | [async-serving.md](specs/async-serving.md) | `DONE` | `6ea7856` | diff --git a/.agents/issue-index.md b/.agents/issue-index.md index fb36c663a..5de4ddcbc 100644 --- a/.agents/issue-index.md +++ b/.agents/issue-index.md @@ -358,3 +358,4 @@ rather than merged. `scripts/check-agent-record.py` gates both. | [#1169](https://github.com/mudler/vllm.cpp/issues/1169) | — | Packed GDN decode is unreachable on every MoE checkpoint for a second, independent reason: `detail::ShouldUsePackedGdnDecode` (`src/vllm/model_executor/models/qwen3_5.cpp:76-84` @ `dd8a3b0e1`) requires `e.has_packed_ba`, populated at `:4410` as `!w.in_proj_ba.Empty()`, and `in_proj_ba` is written at exactly ONE site in the tree — `src/vllm/model_executor/models/qwen3_5_dense_weights.cpp:436`, the DENSE loader. The MoE loader loads the shards split (`qwen3_5_weights.cpp:563-564`) and so does the GGUF loader (`qwen3_5_gguf_weights.cpp:1067-1070`); `qwen3_5.cpp:3220` states it in the tree ("the only path that populates `in_proj_ba`"). Consequence: removing the `dense_model` term and narrowing `GdnOutDType` (#1168) is NOT sufficient to reach packed decode on a MoE model, which corrects the premise row `GDN-MOE-BF16-OUT` started from. It matters because the 35B's geometry (`Hv=32`, `Hg=16`, `Dk=Dv=128`) has a vendored Triton AOT decode cubin for every supported architecture (`src/vt/cuda/triton_aot_vendored/*/gdn_decode_h32.*`) that is exactly the FLA kernel vLLM runs, and `.agents/kernel-matrix.md` records the contrast the vendoring exists to capture — Triton REG:205 / 0 spill against the hand CUDA port at REG:255 + STACK:48. Upstream has one physical `in_proj_ba` on every Qwen3.5/3.6 GDN layer, dense and MoE alike (`qwen_gdn_linear_attn.py:843`; `packed_modules_mapping` on the shared `Qwen3_5ForCausalLMBase`, `vllm/model_executor/models/qwen3_5.py:287-297` @ `5559679`), so the merged owner is the upstream topology and our split MoE load is the deviation. Same class of gap on `in_proj_qkvz`, which no MoE or GGUF loader builds either. Closing it also owes a repair to the stale `KERNEL-GDN-PACKED-DECODE` sentence in `.agents/kernel-matrix.md` claiming "the launcher guard rejects its `Hv=32` shape anyway": `TryTritonPackedDecode` accepts `Hv=32` and dispatches `gdn_decode_h32_default` (`src/vt/cuda/cuda_gdn.cu:5207`, `:5239`). Filed, not fixed — it is a loader change with its own resident-weight lifetime and byte-exactness argument, and the finding row is spec-only. Listed under `## Owed` in [`gdn-moe-bf16-out.md`](specs/gdn-moe-bf16-out.md) | gap | | [#1170](https://github.com/mudler/vllm.cpp/issues/1170) | — | All four GDN Triton AOT fast paths reject any geometry whose linear V-head count is not 48 or 32 — `TryTritonPackedDecode` (`src/vt/cuda/cuda_gdn.cu:5207` @ `dd8a3b0e1`), `TryTritonDeltaH` (`:5264`), `TryTritonChunkO` (`:5298`) and `TryTritonWU` (`:5361`), each reading `if (hv_n != 48 && hv_n != 32) return false;` on top of `dk == 128 && dv == 128 && hk_n == 16`. Those two are the only vendored specializations (`src/vt/cuda/triton_aot_vendored/*/gdn_{decode,deltah,chunko,wu}_h{48,32}.*`): 48 is the dense 27B and 32 is `Qwen3.6-35B-A3B`. `Qwen/Qwen3.8-2.4T-A95B` has 128 linear V-heads ([`qwen38-text-only.md`](specs/qwen38-text-only.md)) and clears every other term, so it is rejected on `hv_n` alone and runs the hand CUDA kernels on all four legs — the ones `.agents/kernel-matrix.md` measured by cuobjdump at REG:255 + STACK:48 (spilling) against the vLLM FLA cubin's REG:205 / 0 spill, which is the whole reason the vendored cubins exist and are default-on. `Qwen3.8-27B` is NOT affected: it is the `Qwen3.6-27B` geometry retrained, 48 V-heads, and hits every AOT arm. Neither reference restricts the head count — SGLang's `TritonGDNKernel` sets `supports_packed_decode` from the platform alone and takes `num_v_heads` as a runtime argument (`python/sglang/srt/layers/attention/linear/kernels/gdn_triton.py:43` @ `f63458b5be`), and `VLLM_ENABLE_FLA_PACKED_RECURRENT_DECODE` has no shape term (`vllm/envs.py:124` @ `5559679`); both JIT-compile per shape, which is the property the AOT vendoring trades away for a Python-free runtime. Closing it needs `h128` specializations vendored across the supported architectures, or a stated rule for which head counts get an AOT arm plus a visible fallback cost at the call site. Filed, not fixed: either close needs the checkpoint that motivates it, and this hardware cannot run the 2.4T (~4.8 TB bf16 against 128 GB unified memory), so the fallback cannot be measured here today. Listed under `## Owed` in [`gdn-moe-bf16-out.md`](specs/gdn-moe-bf16-out.md) | perf | | [#1171](https://github.com/mudler/vllm.cpp/issues/1171) | `KERNEL-GDN-REPLAYSSM` | GDN decode rewrites the whole `[HV,V,K]` fp32 state every step (`src/vt/cuda/cuda_gdn.cu:2393` reads the tile, `:2425` writes it back), which at the 27B shape `HV=32, V=128, K=128` is 2,097,152 bytes read plus the same written per layer, per request, per token. ReplaySSM keeps a per-slot ring of the last `L` steps' rank-1 factors `(d, k, g)`, reconstructs the state in registers, and writes it back only every `L` steps. vLLM implements the algorithm at the pin `555967922` for Mamba2 selective-state ONLY (`layers/mamba/ops/selective_state_update_replayssm_output_only.py`, ring shapes/dtypes `mamba_utils.py:84-93,202-221`, `use_replayssm` default `False` at `config/cache.py:152`, introduced `866fea2b` #48018) and it cannot reach GDN: `config/vllm.py:2318-2322` refuses any model not setting `supports_replayssm`, only `NemotronHForCausalLM` sets it (`models/nemotron_h.py:711`), `GDNAttentionMetadataBuilder` does not subclass the Mamba builder that derives the cursor (`v1/attention/backends/gdn_attn.py:82` vs `mamba_attn.py:575-638`), and the kernel hard-requires a scalar-per-head `A` (`:540-542`) with the Mamba2 `(B,C)` group structure (`:529`). Still true 877 commits past the pin. SGLang ported it to GDN at our recorded pin `f63458b5` (`layers/attention/fla/fused_recurrent_linear_replayssm.py`, whose `:50` credits vLLM; `--enable-linear-replayssm` default `False` and `--linear-replayssm-cache-len` default 16 at `server_args.py:1972-1986`; rings `memory_pool.py:465-483`; commit `a10a24e9` #28451), so the algorithm is a vLLM mirror and the GDN application is a secondary-oracle port. PAYOFF UNMEASURED HERE and deliberately not claimed: ReplaySSM removes the state WRITE and not the read, the flush step reads the checkpoint a SECOND time, so the honest state ratio is `(1+2/L)/2` = 0.5625 at `L=16` against SGLang's published 0.53x which models neither the flush re-read nor any ring read; the ring itself adds 395,264 bytes per slot per layer at the 27B shape = **+18.9% KV page**, worse than vLLM's ~7% on Nemotron because GDN's state is `V*K` while the ring is `L*(V+K)`; and SGLang's own end-to-end figure is ~2.3% TPOT at 128 concurrency on an MoE model. Neither upstream is bit-exact against its unbuffered path and neither claims to be. Motivation is the open Qwen3.8-27B bf16 decode gap (c4 total 0.918x, output 0.963x, `docs/BENCHMARKS.md:192-205`). Spec [`gdn-replayssm.md`](specs/gdn-replayssm.md) | perf | +| [#1179](https://github.com/mudler/vllm.cpp/issues/1179) | `ENG-CUDAGRAPH-BREAK` | The hand-rolled decode-graph driver count recorded in `9bc4d7f44` is **eight** and is actually **nine**, and the row it feeds was framed as coverage-only when it is also correctness. The ninth is the DFlash draft graph, file-local with no header declaration, at `src/vllm/model_executor/models/qwen3_dflash.cpp:771,870,1038,1091,1095,1106` — its own `int g_state = 0` three-state machine (`:771`), its own `VT_DFLASH_GRAPH` kill switch (`:870`) instead of the `VLLM_CPP_CUDAGRAPH` the six batched drivers read, its own invalidate-on-block-width-change (`:1038-1047`) and its own `try { EndCaptureGraph(); } catch (...) {}` drain (`:1106`). The eight-count is stated in four places, all corrected here: [`sglang-breakable-cuda-graph.md`](specs/sglang-breakable-cuda-graph.md) §4 and `## Owed`, [`.agents/engine-matrix.md`](engine-matrix.md) rows `ENG-CUDAGRAPH-BREAK` and `ENG-CUDAGRAPH-DEDUP` ("times eight drivers", which sizes #1162's signature table), and [`.agents/roadmap_v1.md`](roadmap_v1.md) track `C12`. The reframing is the substantive half: `ENG-CUDAGRAPH-BREAK` was recorded as a COVERAGE row, and the duplication has already cost a SHIPPED model its decode graph. `src/vllm/model_executor/models/qwen3.cpp:961-986` declines the decode graph outright whenever the asynchronous device-token mirror is live, on its own measured battery — `depth-1, graph ON PASS 78/78`; `depth-2, graph OFF PASS 82/82`; `depth-2, graph ON FAIL, slots 1-3 degenerate` — because `Step()` replays against the HOST `input.token_ids` and the combine has patched the DEVICE ids. The comment names the real fix as reading the identifiers at replay time from a stable device buffer, and that fix exists, in exactly one sibling driver, as `StepDevInputs` (`src/vllm/model_executor/models/qwen3_5.cpp:3894`): `grep -c StepDevInputs` returns 41 lines there and 0 in each of `qwen3_moe.cpp`, `qwen3.cpp`, `deepseek_v2.cpp` and `voxtral.cpp`. One capability, written once, unavailable to four models, with a live mitigation standing in its place. This does NOT weaken the framing rule that `ENG-CUDAGRAPH` established: the row still makes no throughput claim, and the prefill refutation (GB10 3.8% host-idle between launches, GPU-busy >96%, 27B prefill gap 92.5% non-GEMM glue) stands unchanged. Coverage AND correctness, never speed. Fixed in flow with the [`eng-cudagraph-break.md`](specs/eng-cudagraph-break.md) review repair ([#1163](https://github.com/mudler/vllm.cpp/issues/1163)) | record | diff --git a/.agents/roadmap_v1.md b/.agents/roadmap_v1.md index 879db5a67..5efd25e6a 100644 --- a/.agents/roadmap_v1.md +++ b/.agents/roadmap_v1.md @@ -149,7 +149,7 @@ live in [feature-matrix.md](feature-matrix.md). | C9 | Recurring upstream sync cycle and P1 sync tooling | 🔁 recurring; [upstream-sync.md](upstream-sync.md) | | C10 | **Qwen3.5 high-throughput lever set** (user-directed 2026-08-08, from [vllm.ai 2026-08-06 "25K tok/s/GPU"](https://vllm.ai/blog/2026-08-06-qwen35-25k-tps)): mirror the three levers upstream credits. **(a) Blackwell GDN prefill** replacing the FLA/Triton kernel (FlashInfer PR #3001, vLLM PR #40717, `--gdn-prefill-backend flashinfer`): 1.02x-5.78x on the kernel across sizes, 5.92x microbenchmark, **1.13x e2e prefill throughput, 12% lower mean TTFT**. **(b) Hybrid cache + GDN-state transfer** (vLLM PR #36687 + #37416/#37635/#37310/#41869, `VLLM_SSM_CONV_STATE_LAYOUT=DS`): ~7% intra-node H100 from cutting transferred descriptors 4284 to 1650. **(c) Race-free async scheduling** (vLLM PRs #48481/#45357, `--async-scheduling`), credited as key but not isolated | ☐ SPIKE-first. **PIN MOVE REQUIRED, and it is the first deliverable:** every referenced vLLM PR post-dates our parity pin `555967922` (0.26.0.dev0), so none of this is mirrorable until the pin advances. The spike must (1) find the earliest upstream ref containing #40717 + #36687 + #48481, (2) prove that ref BUILDS AND RUNS a greedy golden on GB10 per [`oracle-gateability`](verification.md), (3) re-capture the SACRED goldens and show ZERO drift before any lever is ported. **Honest scoping:** upstream measured a GB200/NVLink72 DISAGGREGATED cluster (1xDEP8 decode, 4-8 prefill endpoints, 8192/1024, concurrency 64-5120) - one GB10 cannot reproduce it, so **25K is NOT adopted as our bar**. Only (a) is directly portable single-GPU; (c) overlaps the `VT_ASYNC_RUNNER` W3 work already default-OFF under C6; (b) is gated on the scale-out lane | | C11 | **Krea 2 image generation** (user-directed 2026-08-08, [krea-ai/krea-2](https://github.com/krea-ai/krea-2)): text to image diffusion transformer. Two community-licensed checkpoints, `krea/krea-2-raw` (base) and `krea/krea-2-turbo` (8-step distilled, up to 2k). Reference impl is PyTorch in-repo (`mmdit.py` MultiModalDiT + `autoencoder.py` + `sampling.py`) | ☐ SPIKE-first: arch inventory + reuse map against the MiniMax-H3 DiT lane, which already owns a joint video+audio diffusion transformer, an NVFP4/GGUF DiT weight-streaming loader and the `vllm_video_*` ABI slice - a text-to-image DiT is the SIMPLER sibling and should reuse the sampler / VAE-decode / DiT-forward seams rather than start new. **No pin move needed** (nothing upstream to track). **Upstream-mirror caveat:** vLLM has no Krea 2 path, so this is a from-scratch port owing a `porting-inventory.md` section 9 entry with the krea-2 repo as reference root and its PyTorch reference as the numerical oracle | -| C12 | **SGLang Breakable CUDA Graph (BCG), evaluated and mostly REFUTED** (user-directed 2026-08-17, from a public SGLang claim: *"first engine to develop and land BCG, the full CUDA Graph, and graph memory reuse"*). Grounded in the pinned SGLang tree `v0.5.15`/`f63458b5be` rather than in the claim text: BCG captures one forward as a SEQUENCE of `torch.cuda.CUDAGraph` segments split at eager break points on the attention and recurrent calls, with no `torch.compile` (`runner_backend_utils/breakable_cuda_graph/breakable_cuda_graph.py:204-241,244-260`; `runner_backend/breakable_cuda_graph_backend.py:14-17`). **The priority claim is narrower than it reads:** vLLM's v1 default is already `FULL_AND_PIECEWISE`, splitting at `splitting_ops` (`vllm/config/compilation.py:60-63,615,630` @ `555967922`), so the CAPABILITY is in our denominator; what SGLang landed first is the CONSTRUCTION — piecewise coverage with no Dynamo/Inductor/FX. | ☐ T1 (2026-08-17, [#1161](https://github.com/mudler/vllm.cpp/issues/1161)); spec [sglang-breakable-cuda-graph.md](specs/sglang-breakable-cuda-graph.md); row `SGLANG-BCG` in [sglang-matrix.md](sglang-matrix.md). **Four outcomes, three of them not throughput.** (a) *Dropping `torch.compile`* is NO lever here — we never had a compiler; `src/vt/cuda/cuda_backend.cu:203-232` is already raw stream capture. (b) *Prefill capture* is **REFUTED** on our CUDA gate models and recorded against `ENG-CUDAGRAPH` so nobody re-derives it: GB10 2026-07-09 measured prefill GPU-idle-between-launches at **3.8%** with GPU-busy >96% on both arms, and the 27B prefill gap at **92.5% non-GEMM glue GPU work** (dominant GEMM +0.17%, attention AHEAD). No launch bubbles to collapse. (c) *Graph memory reuse* is **real and unported** — `cudaGraphExecUpdate` appears nowhere in `src/` or `include/`, while we instantiate one exec per padded bucket across eight hand-rolled drivers → `ENG-CUDAGRAPH-DEDUP` `INVENTORIED` ([#1162](https://github.com/mudler/vllm.cpp/issues/1162)). (d) The analysis surfaced a STRUCTURAL row the claim did not: our capture is all-or-nothing (`runner.cpp:1338-1341` routes only `pure_decode`) and hand-rolled eight times → `ENG-CUDAGRAPH-BREAK` `INVENTORIED` ([#1163](https://github.com/mudler/vllm.cpp/issues/1163)), a COVERAGE row, not a throughput row. Diffusion looked like the one real lever (SGLang measured LTX-2 H200 10.75s→6.90s post-pin) but is **BLOCKED by our own defects**: an LTX-2.5 render does almost no device compute to capture — GPU util **exactly 0 in 321 of 347 samples** ([#1024](https://github.com/mudler/vllm.cpp/issues/1024)), no device arm for the VAE decode ([#1007](https://github.com/mudler/vllm.cpp/issues/1007)), and **57-66% of wall** in one resolution-CONSTANT serial host phase ([#1087](https://github.com/mudler/vllm.cpp/issues/1087)) → `ENG-CUDAGRAPH-DIFFUSION` `INVENTORIED` ([#1164](https://github.com/mudler/vllm.cpp/issues/1164)) | +| C12 | **SGLang Breakable CUDA Graph (BCG), evaluated and mostly REFUTED** (user-directed 2026-08-17, from a public SGLang claim: *"first engine to develop and land BCG, the full CUDA Graph, and graph memory reuse"*). Grounded in the pinned SGLang tree `v0.5.15`/`f63458b5be` rather than in the claim text: BCG captures one forward as a SEQUENCE of `torch.cuda.CUDAGraph` segments split at eager break points on the attention and recurrent calls, with no `torch.compile` (`runner_backend_utils/breakable_cuda_graph/breakable_cuda_graph.py:204-241,244-260`; `runner_backend/breakable_cuda_graph_backend.py:14-17`). **The priority claim is narrower than it reads:** vLLM's v1 default is already `FULL_AND_PIECEWISE`, splitting at `splitting_ops` (`vllm/config/compilation.py:60-63,615,630` @ `555967922`), so the CAPABILITY is in our denominator; what SGLang landed first is the CONSTRUCTION — piecewise coverage with no Dynamo/Inductor/FX. | ☐ T1 (2026-08-17, [#1161](https://github.com/mudler/vllm.cpp/issues/1161)); spec [sglang-breakable-cuda-graph.md](specs/sglang-breakable-cuda-graph.md); row `SGLANG-BCG` in [sglang-matrix.md](sglang-matrix.md). **Four outcomes, three of them not throughput.** (a) *Dropping `torch.compile`* is NO lever here — we never had a compiler; `src/vt/cuda/cuda_backend.cu:203-232` is already raw stream capture. (b) *Prefill capture* is **REFUTED** on our CUDA gate models and recorded against `ENG-CUDAGRAPH` so nobody re-derives it: GB10 2026-07-09 measured prefill GPU-idle-between-launches at **3.8%** with GPU-busy >96% on both arms, and the 27B prefill gap at **92.5% non-GEMM glue GPU work** (dominant GEMM +0.17%, attention AHEAD). No launch bubbles to collapse. (c) *Graph memory reuse* is **real and unported** — `cudaGraphExecUpdate` appears nowhere in `src/` or `include/`, while we instantiate one exec per padded bucket across NINE hand-rolled drivers → `ENG-CUDAGRAPH-DEDUP` `INVENTORIED` ([#1162](https://github.com/mudler/vllm.cpp/issues/1162)). (d) The analysis surfaced a STRUCTURAL row the claim did not: our capture is all-or-nothing (`runner.cpp:1338-1341` routes only `pure_decode`) and hand-rolled **nine** times (count corrected 2026-08-18, [#1179](https://github.com/mudler/vllm.cpp/issues/1179): the DFlash draft graph `qwen3_dflash.cpp:771,870,1038,1091,1095,1106` was missed) → `ENG-CUDAGRAPH-BREAK` `READY` ([#1163](https://github.com/mudler/vllm.cpp/issues/1163)), a COVERAGE **and CORRECTNESS** row, not a throughput row — `qwen3.cpp:961-986` already DECLINES its decode graph while the async device-token mirror is live (`depth-2, graph ON FAIL, slots 1-3 degenerate`) because `StepDevInputs` (`qwen3_5.cpp:3894`) exists in one driver only. Diffusion looked like the one real lever (SGLang measured LTX-2 H200 10.75s→6.90s post-pin) but is **BLOCKED by our own defects**: an LTX-2.5 render does almost no device compute to capture — GPU util **exactly 0 in 321 of 347 samples** ([#1024](https://github.com/mudler/vllm.cpp/issues/1024)), no device arm for the VAE decode ([#1007](https://github.com/mudler/vllm.cpp/issues/1007)), and **57-66% of wall** in one resolution-CONSTANT serial host phase ([#1087](https://github.com/mudler/vllm.cpp/issues/1087)) → `ENG-CUDAGRAPH-DIFFUSION` `INVENTORIED` ([#1164](https://github.com/mudler/vllm.cpp/issues/1164)) | ## D. T2 (after T1, per porting-inventory.md) diff --git a/.agents/sglang-matrix.md b/.agents/sglang-matrix.md index 917b51c4d..8829af875 100644 --- a/.agents/sglang-matrix.md +++ b/.agents/sglang-matrix.md @@ -146,7 +146,7 @@ share the idea) or OUT-OF-SCOPE. | ID | SGLang surface | SGLang anchor (`file:line`) | Our mapping / anchor | Class | Notes | |---|---|---|---|---|---| -| `SGLANG-BCG` | **Breakable CUDA Graph** — one forward captured as a SEQUENCE of `torch.cuda.CUDAGraph` segments split at eager break points, with no `torch.compile`; plus per-phase backend selection (`FULL`/`BREAKABLE`/`TC_PIECEWISE`/`DISABLED`, decode defaults `FULL`, prefill defaults `BREAKABLE` on CUDA) and graph-executable dedup via `cudaGraphExecUpdate` | `runner_backend_utils/breakable_cuda_graph/breakable_cuda_graph.py:204-241` (segment split), `:244-260` (replay), `:14-23,156-169` (shared mempool + weak-ref intermediates); `runner_backend/breakable_cuda_graph_backend.py:14-17` ("No torch.compile"); break points `layers/radix_attention.py:256`, `attention_forward_methods/forward_mla.py:1092`, `layers/radix_linear_attention.py:159`, `models/nemotron_h.py:1240`, `dsa/dsa_indexer.py:2414`; config `cuda_graph_config.py:38-45,95-112`; dedup `runner_backend/cuda_graph_dedup_mixin.py:219-242,358` | our capture primitive `src/vt/cuda/cuda_backend.cu:203-232`; bucket set `include/vllm/model_executor/models/decode_graph_sizes.h:32-41`; pure-decode routing `src/vllm/v1/worker/gpu/runner.cpp:1338-1341`; eight hand-rolled drivers (`Qwen3_5DecodeGraph`, `Qwen3_5DenseDecodeGraph`, `Qwen3MoeDecodeGraph`, `Qwen3DenseDecodeGraph`, `DeepseekV2DecodeGraph`, `VoxtralDecodeGraph`, `deepseek_v4.cpp`, `laguna.cpp`) | INVENTORIED | **The CAPABILITY is vLLM-derived, not SGLang-distinct**: vLLM's v1 default is already `FULL_AND_PIECEWISE`, splitting at `splitting_ops` (`vllm/config/compilation.py:60-63,517,615,630` @ `555967922`), so it is in our benchmark denominator. What SGLang landed FIRST is the CONSTRUCTION — piecewise coverage from runtime stream capture plus a decorator, no Dynamo/Inductor/FX. That construction is the part worth mirroring, because we have no compiler either. **Prefill capture is REFUTED as a lever here** (GB10 2026-07-09: prefill GPU-idle-between-launches 3.8%, GPU-busy >96%, 27B gap 92.5% non-GEMM GLUE) — do not re-derive. Exec dedup is unported (`cudaGraphExecUpdate` appears nowhere in `src/` or `include/`). Spec [sglang-breakable-cuda-graph.md](specs/sglang-breakable-cuda-graph.md); issues [#1161](https://github.com/mudler/vllm.cpp/issues/1161), [#1162](https://github.com/mudler/vllm.cpp/issues/1162), [#1163](https://github.com/mudler/vllm.cpp/issues/1163), [#1164](https://github.com/mudler/vllm.cpp/issues/1164). | +| `SGLANG-BCG` | **Breakable CUDA Graph** — one forward captured as a SEQUENCE of `torch.cuda.CUDAGraph` segments split at eager break points, with no `torch.compile`; plus per-phase backend selection (`FULL`/`BREAKABLE`/`TC_PIECEWISE`/`DISABLED`, decode defaults `FULL`, prefill defaults `BREAKABLE` on CUDA) and graph-executable dedup via `cudaGraphExecUpdate` | `runner_backend_utils/breakable_cuda_graph/breakable_cuda_graph.py:204-241` (segment split), `:244-260` (replay), `:14-23,156-169` (shared mempool + weak-ref intermediates); `runner_backend/breakable_cuda_graph_backend.py:14-17` ("No torch.compile"); break points `layers/radix_attention.py:256`, `attention_forward_methods/forward_mla.py:1092`, `layers/radix_linear_attention.py:159`, `models/nemotron_h.py:1240`, `dsa/dsa_indexer.py:2414`; config `cuda_graph_config.py:38-45,95-112`; dedup `runner_backend/cuda_graph_dedup_mixin.py:219-242,358` | our capture primitive `src/vt/cuda/cuda_backend.cu:203-232`; bucket set `include/vllm/model_executor/models/decode_graph_sizes.h:32-41`; pure-decode routing `src/vllm/v1/worker/gpu/runner.cpp:1338-1341`; NINE hand-rolled drivers (`Qwen3_5DecodeGraph`, `Qwen3_5DenseDecodeGraph`, `Qwen3MoeDecodeGraph`, `Qwen3DenseDecodeGraph`, `DeepseekV2DecodeGraph`, `VoxtralDecodeGraph`, `deepseek_v4.cpp`, `laguna.cpp`, and the DFlash draft graph `src/vllm/model_executor/models/qwen3_dflash.cpp:771,870,1038,1091,1095,1106` — count corrected 2026-08-18, [#1179](https://github.com/mudler/vllm.cpp/issues/1179); `9bc4d7f44` recorded eight, and the DFlash driver is file-local with no header declaration, which is why it was missed) | INVENTORIED | **The CAPABILITY is vLLM-derived, not SGLang-distinct**: vLLM's v1 default is already `FULL_AND_PIECEWISE`, splitting at `splitting_ops` (`vllm/config/compilation.py:60-63,517,615,630` @ `555967922`), so it is in our benchmark denominator. What SGLang landed FIRST is the CONSTRUCTION — piecewise coverage from runtime stream capture plus a decorator, no Dynamo/Inductor/FX. That construction is the part worth mirroring, because we have no compiler either. **Prefill capture is REFUTED as a lever here** (GB10 2026-07-09: prefill GPU-idle-between-launches 3.8%, GPU-busy >96%, 27B gap 92.5% non-GEMM GLUE) — do not re-derive. Exec dedup is unported (`cudaGraphExecUpdate` appears nowhere in `src/` or `include/`). Spec [sglang-breakable-cuda-graph.md](specs/sglang-breakable-cuda-graph.md); issues [#1161](https://github.com/mudler/vllm.cpp/issues/1161), [#1162](https://github.com/mudler/vllm.cpp/issues/1162), [#1163](https://github.com/mudler/vllm.cpp/issues/1163), [#1164](https://github.com/mudler/vllm.cpp/issues/1164). | ## Structurally distinctive / research (vLLM has no native analogue) diff --git a/.agents/specs/eng-cudagraph-break.md b/.agents/specs/eng-cudagraph-break.md new file mode 100644 index 000000000..e884dc60f --- /dev/null +++ b/.agents/specs/eng-cudagraph-break.md @@ -0,0 +1,834 @@ +# ENG-CUDAGRAPH-BREAK — a shared `vt` capture seam that accepts break points + +**Row:** `ENG-CUDAGRAPH-BREAK` (engine-matrix, engine core and scheduling). +**Issue:** [#1163](https://github.com/mudler/vllm.cpp/issues/1163), with +[#1020](https://github.com/mudler/vllm.cpp/issues/1020) as one reachable instance. +**Parent analysis:** [sglang-breakable-cuda-graph.md](sglang-breakable-cuda-graph.md) `## Owed`. +**Kind:** scoping spike plus structured spec. No production code lands with this spec. + +**This is a coverage row and a CORRECTNESS row, and it is not a throughput row.** +The spike claims no speed and measures none. `## Gates` below states what would have +to be named and measured before any speed claim is admissible, and +`## Risks/decisions` D5 records why no such path is named today. + +The correctness half is not a framing device, it is already shipped damage +([#1179](https://github.com/mudler/vllm.cpp/issues/1179)). `src/vllm/model_executor/models/qwen3.cpp:961-986` +DECLINES its decode graph outright whenever the asynchronous device-token mirror is +live, and its comment records the measurement that forced it: `depth-1, graph ON +PASS 78/78`, `depth-2, graph OFF PASS 82/82`, `depth-2, graph ON FAIL, slots 1-3 +degenerate`. The comment names the real fix — read the identifiers at REPLAY time +from a stable device buffer — and that fix already exists, in the sibling driver, +as `StepDevInputs` (`qwen3_5.cpp:3894`). One driver has the capability and a shipped +model lost its decode graph because it does not. That is duplication producing a +correctness regression, not a tidiness complaint, and it is the second reason this +row exists. + +## Scope + +One shared capture seam under `vt` that captures a forward as a SEQUENCE of graph +segments split at break points, so a forward containing a host-dependent operation +is still graphed except at that operation, instead of running eager for the whole +step. The seam replaces the per-model capture drivers listed in `## Our baseline`. + +In scope: + +- The break-point primitive and the capture scope that collects segments. +- The segment container, its replay order, and its ownership of instantiated graphs. +- The lifetime contract for every buffer a segment or a break function reads. +- A staged migration of the existing drivers onto the seam, one driver per stage. +- The predicate that decides whether a step is graph-eligible, moved off + `pure_decode` and onto "eligible except at the break points". + +Out of scope, and each names its owner: + +- Graph-executable dedup through `cudaGraphExecUpdate`. That is + `ENG-CUDAGRAPH-DEDUP`, [#1162](https://github.com/mudler/vllm.cpp/issues/1162), + owned by a parallel agent. See `## Risks/decisions` D4 for the interaction. +- The LTX-2.5 denoise loop. That is `ENG-CUDAGRAPH-DIFFUSION`, + [#1164](https://github.com/mudler/vllm.cpp/issues/1164), blocked on its own defects. +- Any change to a model's numerics. Every migrated model must stay bit-exact + against its own eager forward. + +## Upstream chain + +**The boundary comes from vLLM. The construction comes from SGLang.** These are two +different questions and this spec keeps them apart. + +**Primary oracle, vLLM at pin `5559679229` (`.agents/upstream-sync.md`).** vLLM's +`CUDAGraphMode` enum defines `PIECEWISE = 1` and the composite +`FULL_AND_PIECEWISE = (FULL, PIECEWISE)` at +`/home/mudler/_git/vllm/vllm/config/compilation.py:60-63`. The docstring at +`:608-635` states that `FULL_AND_PIECEWISE` is the v1 default, that it captures a +full graph for decode batches and a piecewise graph for prefill and mixed batches, +and that piecewise mode keeps "the cudagraph incompatible ops (i.e. some attention +ops) outside the cudagraph". The split points themselves are `splitting_ops`, +declared at `:517`, and defaulted at `:1145` to `list(self._attention_ops)`. That +class-level list is at `:764-772` and names, among others, +`vllm::unified_attention_with_output`, `vllm::unified_mla_attention_with_output`, +`vllm::mamba_mixer2`, `vllm::mamba_mixer`, `vllm::short_conv`, +`vllm::linear_attention` and `vllm::qwen_gdn_attention_core`. + +So the boundary vLLM chose is: **the attention family is the break point**. Dense +attention, multi-head latent attention (MLA), the Mamba mixers, the short +convolution and the gated delta network (GDN) core. This spec adopts that boundary +and does not invent one. + +**Where vLLM's construction does not transfer.** vLLM obtains the split from +`torch.compile`: `set_splitting_ops_for_v1` at `:1123-1145` returns early unless +`self.mode == CompilationMode.VLLM_COMPILE`, and the split is applied by Dynamo at +the FX level or by an Inductor graph partition. We have no compiler, no Dynamo and +no FX graph. The boundary transfers; the mechanism cannot. + +**Secondary oracle, SGLang at pin `f63458b5be` (`.agents/oracles/sglang.md`).** +SGLang reaches the same coverage with no compiler, which is the construction this +spec ports. Verified in the pinned tree at +`/home/mudler/_git/sglang` (`git rev-parse HEAD` = +`f63458b5beaceabbd9d749b9fc956370e1b649e6`): + +Every bare `:N` anchor below is a line in the one file that implements BCG, +spelled here once in full: +`python/sglang/srt/model_executor/runner_backend_utils/breakable_cuda_graph/breakable_cuda_graph.py`. +That file is 374 lines at the pin (`wc -l` = 374), so no anchor past `:374` exists. + +| Mechanism | Anchor | +|---|---| +| Break-point wrapper | `:204-243` (`eager_on_graph`), whose `wrapper` body is `:209-241` | +| Output writeback on replay | `:231-235` (`replay_fn`), `:172-201` (`_copy_output`) | +| Active-capture context | `:63` (`_current_capture_var`, a `ContextVar`) | +| Segment container | `:246-274` (`BreakableCUDAGraph`); fields at `:251-252`; `_append_segment` at `:266-274` | +| Interleaved replay | `:255-264` | +| Capture scope enter and exit | `:309-320` (`__enter__`), `:322-333` (`__exit__`) | +| Segment open and close | `:335-350` (`_begin_new_segment`), `:352-367` (`_end_current_segment`) | +| Auxiliary-stream fork tracking | `:101-153` (the `wait_stream` hook), installed at `:310` and removed at `:332` | +| Auxiliary-stream auto-join before a segment ends | `:353-361` | +| Bare break marker | `:370-374` (`break_graph`, an empty body under the decorator) | +| Shared mempool and weak refs | `:14-23` doc, `:156-169` (`_weak_ref_if_tensor`) | +| No-compiler backend | `python/sglang/srt/model_executor/runner_backend/breakable_cuda_graph_backend.py:14-17` | + +The break points SGLang registers, all through the same one-line wrapper: + +| Site | Anchor | +|---|---| +| Dense attention | `layers/radix_attention.py:256` | +| MLA | `models/deepseek_common/attention_forward_methods/forward_mla.py:1092` | +| Linear attention | `layers/radix_linear_attention.py:159` | +| Nemotron Mamba2 | `models/nemotron_h.py:1240` | +| DeepSeek sparse-attention indexer | `layers/attention/dsa/dsa_indexer.py:2414` | +| DeepSeek V4 attention | `models/deepseek_v4.py:284` | + +That set is the same family vLLM lists in `_attention_ops`. The two oracles agree +on the boundary, which is the strongest evidence available that the boundary is +right and not an artefact of either implementation. + +## Our baseline + +### The seam that exists + +`include/vt/backend.h:208-222` already declares the whole capture vocabulary: +`SupportsGraphCapture`, `BeginCapture`, `EndCapture`, `Replay`, and the multi-graph +handle trio `EndCaptureGraph`, `ReplayGraph`, `DestroyGraph`. The base definitions +in `src/vt/backend.cpp:29-34` throw for a backend that does not implement capture, +except `DestroyGraph`, which is a no-op. + +Three backends implement it: CUDA at `src/vt/cuda/cuda_backend.cu:198,204,225,233,288`, +ROCm at `src/vt/rocm/rocm_backend.hip:248`, and Tenstorrent at +`src/vt/tenstorrent/tenstorrent_backend.cpp:75-81`, which maps the calls onto ttnn +mesh-trace capture. Vulkan and Metal return false and document why, at +`src/vt/vulkan/vulkan_backend.cpp:16` and `src/vt/metal/metal_backend.mm:13`. + +**`EndCaptureGraph` stores nothing in the backend** (`cuda_backend.cu:225-232`). It +ends the stream capture, instantiates the graph, and returns the executable as an +opaque handle the caller owns. That property is what makes a segmented capture +expressible with **no new backend virtual**: a segment is one `BeginCapture` and +`EndCaptureGraph` pair, and N break points produce N+1 such pairs on the same queue. +The spike confirmed this SHAPE in C++, on a recording backend with no GPU +attached; whether CUDA permits the re-begin mid-forward is W1's exit criterion, not +a W0 result. See `## Work breakdown` W0. + +### Defect 1: capture is all or nothing + +`src/vllm/v1/worker/gpu/runner.cpp:1341` computes the only predicate that admits a +step to a graph: + +``` +const bool pure_decode = attn_meta.num_actual_tokens == num_reqs && + gdn_meta.num_prefill_tokens == 0; +``` + +It travels to the model as `ModelForwardInput::pure_decode` +(`include/vllm/model_executor/models/model_registry.h:281`, set at +`src/vllm/v1/worker/gpu/runner.cpp:1378`), and each model's registered forward +tests it before reaching its driver, for example +`src/vllm/model_executor/models/qwen3.cpp:961`. A step that fails the predicate +runs eager in full. There is no way to express "capture this forward except at +these three calls", so coverage is a cliff and not a slope. + +[#1020](https://github.com/mudler/vllm.cpp/issues/1020) is one reachable instance: +a speculative verify step whose actual draft depth differs from the configured +`num_speculative_tokens` (`model_registry.h:288`) is admitted by no graph and falls +out to eager with no log and no counter. + +### Defect 2: the drivers, enumerated + +NINE production drivers exist. Six are batched decode drivers with a padded-bucket +ring; three are single-shape drivers. Every one of them re-derives the same +machinery. The count is nine and not the eight recorded in `9bc4d7f44` +([#1179](https://github.com/mudler/vllm.cpp/issues/1179)): the DFlash draft graph is +file-local with no header declaration, which is why the parent analysis missed it. + +| Driver | Declaration | Implementation | Shape | +|---|---|---|---| +| `Qwen3_5DecodeGraph` | `include/vllm/model_executor/models/qwen3_5.h:275` | `src/vllm/model_executor/models/qwen3_5.cpp:9540,9563,9712,9893,9918,9922` | batched, padded ring | +| `Qwen3_5DenseDecodeGraph` | `include/vllm/model_executor/models/qwen3_5_dense.h:391` | `src/vllm/model_executor/models/qwen3_5.cpp:9969,9986,10321,10340,10348` | batched, padded ring | +| `Qwen3MoeDecodeGraph` | `include/vllm/model_executor/models/qwen3_moe.h:117` | `src/vllm/model_executor/models/qwen3_moe.cpp:385,404,469,512,515,524` | batched, padded ring | +| `Qwen3DenseDecodeGraph` | `include/vllm/model_executor/models/qwen3.h:243` | `src/vllm/model_executor/models/qwen3.cpp:583,613,682,866,888,897` | batched, padded ring | +| `DeepseekV2DecodeGraph` | `include/vllm/model_executor/models/deepseek_v2.h:324` | `src/vllm/model_executor/models/deepseek_v2.cpp:887,902,963,980` | batched, padded ring | +| `VoxtralDecodeGraph` | `include/vllm/model_executor/models/voxtral.h:126` | `src/vllm/model_executor/models/voxtral.cpp:448,469,535,582,585,594` | batched, padded ring | +| DeepSeek V4 `V4Graph` | none, file-local | `src/vllm/model_executor/models/deepseek_v4.cpp:1673,1719,1898,1900,1904` | single shape, `gstate` 0/1/2 | +| Laguna decode graph | none, file-local | `src/vllm/model_executor/models/laguna.cpp:2373,2689,2691,2692,2695` | single shape | +| DFlash draft graph (the ninth) | none, file-local | `src/vllm/model_executor/models/qwen3_dflash.cpp:771,870,1038,1091,1095,1106` | single shape, `g_state` 0/1/2 | + +**What every driver re-derives.** Each item below was read in at least two drivers +and the anchors give one instance each. + +1. The environment kill switch. SIX drivers read `VLLM_CPP_CUDAGRAPH` themselves: + `qwen3_5.cpp:9540`, `qwen3_5.cpp:9969`, `qwen3_moe.cpp:385`, `qwen3.cpp:583`, + `deepseek_v2.cpp:887`, `voxtral.cpp:448`. Instrument: + `grep -rn 'std::getenv("VLLM_CPP_CUDAGRAPH")' src/` returns exactly those six. + (`grep -rn VLLM_CPP_CUDAGRAPH src/` returns seven, because `qwen3.cpp:947` + mentions the name in a comment and is not a read site.) The three single-shape + drivers each invented their own switch instead: `VT_V4_DECODE_GRAPH` at + `deepseek_v4.cpp:1680`, `VT_DFLASH_GRAPH` at `qwen3_dflash.cpp:870`, and + `VT_LAGUNA_DECODE_GRAPH` at `laguna.cpp:1684`. There is no one switch that + turns capture off. +2. The cold, warm, captured state machine. `qwen3_5.cpp:9563` carries `warm` and + `captured` flags on the slot; `deepseek_v4.cpp:1719` spells the same thing as + `int gstate = 0`; `qwen3_dflash.cpp:771` spells it as `int g_state = 0`. Three + spellings of one three-state machine. +3. The pad-to-bucket selection. SIX call sites of `PadToCaptureSize`: + `qwen3_5.cpp:9712`, `qwen3_5.cpp:10140`, `qwen3_moe.cpp:469`, `qwen3.cpp:682`, + `deepseek_v2.cpp:963`, `voxtral.cpp:535`. Instrument: `grep -rn PadToCaptureSize + src/` returns 16 lines, of which four are `#include` comments and six are prose + comments; the six above are the invocations. The function itself is already + shared, at `include/vllm/model_executor/models/decode_graph_sizes.h:47-54`, which + is the proof that extraction works and is the pattern this row extends. +4. The per-slot persistent buffer set. `qwen3_5.cpp:9563-9600` holds `token_ids`, + `positions`, the two metadata structs, `hidden`, `logits`, `aux`, `graph`, + `fa_cols`, `captured`, `warm`, `replays`, `reuse_event`, `dev` and `pin`. Every + other batched driver holds a subset of the same fields under the same name + `SizeSlot`. +5. The diagnostics pair `captured()` and `replay_count()`, SIX times. The + `replay_count()` definitions are at `qwen3_5.cpp:9680`, `qwen3_5.cpp:10110`, + `qwen3_moe.cpp:454`, `qwen3.cpp:667`, `deepseek_v2.cpp:948`, `voxtral.cpp:519`, + each preceded one line above by its `captured()` sibling + (`qwen3_5.cpp:9679`, `qwen3_5.cpp:10109`, `qwen3_moe.cpp:453`, `qwen3.cpp:666`, + `deepseek_v2.cpp:947`, `voxtral.cpp:518`). Instrument: + `grep -rn '::captured() const' src/vllm/model_executor/models/*.cpp` returns six. +6. The capture-failure recovery. `qwen3_5.cpp:9913`, `qwen3_5.cpp:10335` and + `qwen3_dflash.cpp:1106` each carry their own copy of the same + `try { g = EndCaptureGraph(); } catch (...) {}` drain, because a skipped + `EndCaptureGraph` poisons the stream permanently. +7. The invalidate-and-recapture rule on a shape change. `qwen3_5.cpp:9816-9826` + destroys the graph when the block-table column count moves; + `qwen3_dflash.cpp:1038-1047` does the same when the block width moves. + +**The divergence that proves the cost.** The persistent DEVICE input path, called +"Option A" in the tree, exists in exactly one driver. `StepDevInputs` +(`qwen3_5.cpp:3894`) appears on **41** lines of `qwen3_5.cpp` and **zero** lines of +`qwen3_moe.cpp`, `qwen3.cpp`, `deepseek_v2.cpp` and `voxtral.cpp`. Instrument: +`grep -c StepDevInputs `, which counts MATCHING LINES, not occurrences; the +occurrence count on the same file is 47 (`grep -o StepDevInputs qwen3_5.cpp | wc -l`). +An earlier draft of this spec recorded 33, which is neither number and is withdrawn. +The four zeros reproduce under both instruments and the conclusion is unchanged. The other four replay against host vectors. One of them, +`qwen3.cpp:961-986`, therefore has to DECLINE the graph outright whenever the +asynchronous device-token mirror is live, and its comment records the measured +failure that forced it: "depth-2, graph ON FAIL, slots 1-3 degenerate". The same +comment names the real fix as reading the identifiers at replay time from a stable +device buffer, which is precisely what `StepDevInputs` already does in the sibling +driver. One capability, written once, unavailable to four models. That is the cost +of nine drivers stated as a defect rather than as an aesthetic complaint. It is also +why this row is a CORRECTNESS row and not only a coverage one. + +**What is genuinely model-specific**, and must stay in the model: + +- Which operations are break points for this architecture. A GDN hybrid breaks in + different places from a dense attention stack. +- The shape key. `qwen3_5.cpp:9816` keys on the block-table column count as well + as the batch size; `qwen3_dflash.cpp:1036` keys on the draft block width. +- What the forward writes into the persistent output buffers: logits alone, or + logits plus the auxiliary hidden taps (`qwen3_5.cpp:9570-9577`). +- The pre-warm forward itself, because only the model knows its own allocation set. +- Any model-specific fusion that must run inside a segment. + +### The lifetime contract that already exists + +`src/vt/cuda/graph_safe_scratch.h:1-32` records the rule this seam must enforce +rather than re-learn: a captured graph BAKES the device pointer a grow-on-demand +scratch helper returned at capture time, so freeing that block on growth makes the +next replay read freed memory. The fix in the tree is retire-on-grow. The full +root-cause is `.agents/specs/decode-graph-scratch-uaf-2026-07-18.md`. + +The `DevicePool` (`include/vllm/model_executor/models/device_pool.h:1-38,71`) +never returns a block to the driver. That is a materially better position than +SGLang's, and `## Risks/decisions` D1 states why it does not make the hazard go +away. + +## Port map + +### The primitive + +Three types, all under `vt`, all backend-agnostic, none of them a new virtual on +`vt::Backend`. + +1. **`vt::BreakableGraph`** — the container. Holds `std::vector segments` + (each an opaque handle from `Backend::EndCaptureGraph`) and + `std::vector> break_fns` (each a seam-built replay + closure, see item 3, never a caller's raw `fn`). Invariant, asserted: + `segments.size() == break_fns.size() + 1`. `Replay(Queue&)` walks + `ReplayGraph(segments[i])` then `break_fns[i]()` for each `i`, mirroring + `breakable_cuda_graph.py:255-263`. The destructor calls `DestroyGraph` on every + segment. This is the direct analogue of `BreakableCUDAGraph` at `:246`. + +2. **`vt::GraphCaptureScope`** — a resource-acquisition-is-initialization (RAII) + scope. Its constructor sets one `thread_local GraphCaptureScope*` and calls + `Backend::BeginCapture`; its destructor calls `Backend::EndCaptureGraph`, pushes + the final segment, and clears the pointer. This is `BreakableCUDAGraphCapture` + at `:277-367`, with the destructor doing what `__exit__` does at `:322-333`. The + thread-local replaces SGLang's `ContextVar` at `:60`. It must be + `thread_local`, not a global, for the same reason SGLang chose a `ContextVar` + and CUDA chose `cudaStreamCaptureModeThreadLocal` (`cuda_backend.cu:204-206`): + capture is a property of one thread's stream, and a process-wide flag would + make an unrelated thread's forward observe a capture it is not part of. + +3. **`vt::GraphBreak(fn, out)`** — the break point. Outside a capture scope it + calls `fn` and returns, so a non-capturing forward is byte-identical to today. + Inside a scope it closes the current segment, runs `fn` once eagerly so the + outputs hold real data, appends a REPLAY CLOSURE to `break_fns`, and opens a + new segment. This is `eager_on_graph` at `:204-243`. + + **The appended function is not `fn`.** Upstream builds a `replay_fn` closure at + `:231-235` that calls the captured inner function and then writes its result + back into the capture-time destination: + + ```python + def replay_fn(): + new_out = captured_inner(*captured_args, **captured_kwargs) + return _copy_output(captured_output, new_out) + ``` + + `_copy_output` (`:172-201`) does an in-place `dst.copy_(src)` for a tensor, and + recurses field-wise over an object's `__dict__` or a dict's values; it returns + `src` only when neither side is copyable. The companion `_weak_ref_if_tensor` + (`:156-169`) is what makes the capture-time destination outlive the eager call. + + **This is load-bearing, not decorator sugar.** On replay N the eager operation + returns a FRESH allocation, whose address is not the one segment `i+1` baked at + capture time. A container that replays the raw `fn` and discards its return value + drops that guarantee silently: the next segment keeps reading the capture-time + address while the break function writes somewhere else. The failure is WRONG + NUMERICS, not a fault, which is exactly the class `## Risks/decisions` D1 records + `compute-sanitizer` cannot see. + + **Contract this seam adopts, stated at the `vt::GraphBreak` declaration.** One of + the two, and the seam must make the choice explicit rather than leave it to the + caller: + + - `GraphBreak(fn, out)` takes the destination the following segment reads, and + the seam copies `fn`'s result into `out` on every replay. This is the direct + port of `replay_fn` plus `_copy_output`. + - `GraphBreak(fn)` with no destination requires `fn` to write IN PLACE into a + persistent buffer that the model owns and that no replay reallocates. The + no-destination form is then only legal for a break function with no return + value; the bare marker below is its degenerate case. + + `vt::BreakableGraph` therefore holds `std::vector>` whose + elements are the SEAM's closures, never the caller's `fn` — and D9 records the + hazard the distinction removes. + +**The fork-join rule the scope owns.** Closing a segment is illegal while a side +stream forked inside it is still participating in the capture. Upstream solves this +inside `_end_current_segment` (`:353-361`): before `capture_end()` it walks the set +of streams forked but not rejoined and issues the join itself. Tracking that set is +the entire purpose of the `torch.cuda.Stream.wait_stream` hook (`:101-153`), which +`__enter__` installs at `:310` and `__exit__` removes at `:332`. + +**This is live for us, in both drivers W4 and W5 migrate.** Our CUDA backend reports +`SupportsAuxStream() == true` (`src/vt/cuda/cuda_backend.cu:200`, against the base +`false` at `include/vt/backend.h:166`), and two model paths fork on it. Qwen3.5's MoE +shared-expert overlap records a fork event on the main queue and makes the auxiliary +queue wait it at `src/vllm/model_executor/models/qwen3_5.cpp:6254-6255`, records the +completion event at `:6261`, and joins the main queue back at `:6384` — so any break +point placed between `:6255` and `:6384` closes a segment with the fork outstanding. +Laguna's decode-graph class owns its own auxiliary queue and event pair +(`src/vllm/model_executor/models/laguna.cpp:2362-2367`, torn down at `:2374-2378`), +forks at `:2572-2573`, records at `:2576` and joins at `:2612` — and unlike Qwen3.5's +that fork is INSIDE the captured region by construction, because the class is the +capture driver. + +**Rule: `GraphCaptureScope` owns the set of queues forked since the current segment +opened, and joins every outstanding one before it calls `EndCaptureGraph`.** We need +no `wait_stream` monkey-patch to populate that set, because our fork and join are +explicit `Backend::RecordEvent` and `Backend::QueueWaitEvent` calls on a seam we +control, not an implicit torch API. D10 records what happens if the rule is omitted. + +**Why a thread-local and not a parameter.** The break site sits inside an attention +call, several frames below the forward entry point. Threading a capture context +through every intermediate signature would touch every model. SGLang made the same +call for the same reason. The spike proved the thread-local carries the context +across three frames with no intermediate signature change; see `## Work breakdown` W0. + +**A bare marker.** `vt::GraphBreak()` with no argument splits the segment and runs +nothing, mirroring `break_graph` at `breakable_cuda_graph.py:370-374`. It is the +form a model uses when the host-dependent work already ran outside the forward. +It has no output, so the writeback contract above is vacuous for it. + +### How the boundary is registered + +vLLM registers `splitting_ops` by operation NAME because it has an FX graph to +match names against. We have neither. Our equivalent is a call at the break site +itself, exactly as SGLang's is: one line at the attention entry point, mirroring +`radix_attention.py:256` and `radix_linear_attention.py:159`. The site is the +registration. + +The break-point set is the vLLM `_attention_ops` family +(`compilation.py:764-772`), reached through our own dispatch: `dense_attn::AttnBlock` +for dense attention and MLA, and the GDN and Mamba entry points for the recurrent +families. Naming the exact call sites is W1 work, not spike work, because the set +must be established per migrated model against that model's own break inventory. + +### How segments and intermediates stay valid + +The question SGLang answers with a shared memory pool and weak references +(`breakable_cuda_graph.py:14-23,156-169`) has a different answer here, because our +allocator is different. + +- **Device intermediates.** A `DBuf` allocated during segment `i` is returned to + the `DevicePool` free list when its scope ends. The pool never returns a block to + the driver (`device_pool.h:16`), so the POINTER segment `i` baked stays mapped + for the life of the process. SGLang needs `use_count` pinning because torch's + caching allocator can release a segment to the driver; we do not. What we DO need + is protection against REUSE, which `## Risks/decisions` D1 covers. +- **Break-function inputs.** The closure must capture device pointers or `Tensor` + views that are stable across replays, never a reference to a host temporary that + dies at the end of the capturing call. This is the same contract the existing + drivers already meet for their persistent slot buffers + (`qwen3_5.cpp:9563-9600`), lifted from a per-driver convention to a seam rule. +- **Host sources of a captured upload.** Capture bakes the HOST SOURCE ADDRESS of + an upload as well as the device destination. `## Risks/decisions` D2 is the rule + the seam must enforce. + +### What is retired, and what replaces it + +The seam absorbs items 1, 2, 5, 6 and 7 of the re-derivation list, plus the +segment bookkeeping. `decode_graph_sizes.h` already owns item 3 and is unchanged. +Item 4 becomes a model-supplied persistent buffer set the seam holds but does not +interpret. Each migrated driver keeps only its shape key, its pre-warm forward, +its break-point set and its output buffers. + +## Tests to port + +vLLM's own tests do not transfer. Its piecewise split is verified through +`torch.compile` configuration tests, and we have no compiler to configure. + +**SGLang's BCG DOES carry a unit suite at the pin, and it ports.** An earlier draft +of this spec asserted there was nothing to port, which was wrong and is withdrawn. +The file is +`test/registered/cuda_graph/breakable/test_breakable_cuda_graph.py` at +`f63458b5be`, 305 lines, four classes. AGENTS.md requires the upstream tests in the +same change that ports the behavior, so the table below maps every upstream case to +the local test that owes it. Parameters, modes and failure cases are preserved; the +harness adaptation is stated once at the bottom. + +**`TestBreakableCUDAGraphBasic` (`:30`) — the capture and replay mechanism.** + +| Upstream case | Anchor | What it pins | Local test | +|---|---|---|---| +| `test_no_break_capture_replay` | `:49-63` | zero breaks captures and replays exactly like a plain graph: `y = x + 1`, refill `x` to 5.0, replay, expect 6.0 | T1: a scope with no `GraphBreak` yields 1 segment, 0 break functions, and one `Begin`/`EndCaptureGraph` pair | +| `test_single_break` | `:65-87` | one break splits into two segments and the chain composes: `x=10 -> +1=11 -> eager *2=22 -> +3=25` | T2: 1 break yields 2 segments and 1 break function, and the arithmetic chain holds | +| `test_multiple_breaks` | `:89-115` | two breaks, three segments, chained: `x=5 -> +1=6 -> +1=7 -> +1=8 -> *2=16` | T3: N breaks yield N+1 segments and N break functions, asserted at N=2 with the same values | +| `test_eager_on_graph_disabled` | `:117-129` | with the wrapper DISABLED the function is returned unchanged and runs normally | T4: `VLLM_CPP_CUDAGRAPH=0` (item 1 of `## Our baseline`) makes `GraphBreak(fn)` call `fn` and make zero backend calls | +| `test_eager_on_graph_outside_capture` | `:131-142` | outside any capture the wrapper is a pass-through | T5: with no scope active, `GraphBreak(fn)` runs `fn` and makes zero backend calls | +| `test_replay_updates_output` | `:144-169` | TWO replays with different inputs give different outputs: `3.0` then, after `x.fill_(10)`, `33.0` | T6: the multi-replay case. This is the upstream anchor for G1's "more than one replay" requirement, and the one case that can see a break function writing to a stale address | + +**`TestCopyOutput` (`:172`) — the output-writeback contract of `## Port map` §3.** +This class is why F2's writeback is not optional: upstream tests `_copy_output` +separately from the capture machinery, because it is a separate guarantee. + +| Upstream case | Anchor | What it pins | Local test | +|---|---|---|---| +| `test_tensor_copy` | `:187-192` | a tensor destination is written IN PLACE and the SAME object is returned (`assertIs(result, dst)`) | T7: `GraphBreak(fn, out)` copies into `out`; the device pointer `out` held before the call is the one holding the data after it | +| `test_dict_copy` | `:194-208` | a keyed set of destinations is copied field by field | T8: a multi-output break (logits plus an auxiliary tap, the `qwen3_5.cpp:9570-9577` shape) writes every destination | +| `test_object_copy` | `:210-223` | a struct destination copies its tensor fields in place and ASSIGNS its non-tensor fields (`dst.label == "new"`) | T9: a struct-valued break preserves the in-place/assign split | +| `test_non_tensor_fallback` | `:225-227` | with nothing copyable, `_copy_output` returns `src`, the documented fallback | T10: a break function with no device output is legal and the seam does not fabricate a copy | + +**`TestBreakGraphHelper` (`:230`).** + +| Upstream case | Anchor | What it pins | Local test | +|---|---|---|---| +| `test_break_graph_inserts_segment` | `:249-265` | the BARE marker splits the segment even though its body does nothing: `x=10 -> +1=11 -> break -> +2=13` | T11: `vt::GraphBreak()` with no argument yields 2 segments and 1 (empty) break function, and the value chain is unaffected | + +**`TestBreakableCudaGraph` (`:268`) — `test_gsm8k_accuracy` (`:288`) is DELIBERATELY +EXCLUDED from this row**, and the reason is not cost. It launches a Qwen3-8B server +with `--cuda-graph-backend-prefill=breakable` (`:279-281`) and asserts `mgsm_en >= +0.80` over 1319 examples (`:289-301`). That is a DISTRIBUTIONAL accuracy floor on a +PREFILL capture path. Both halves are wrong for us. Our gate polarity for a migrated +model is BIT-EXACTNESS against the model's own eager forward (`## Gates` G1), which +is strictly stronger than an accuracy floor and is the gate AGENTS.md requires when a +greedy path exists. And prefill capture is REFUTED as a lever here and is not what +this row builds (`## Risks/decisions` D5). The upstream case's PURPOSE — an +end-to-end model-level check that a segmented capture does not corrupt generation — +is carried by G1 against `tests/parity/test_qwen36_paged_engine.cpp:140`, per +migrated model. Excluded as redundant to a stronger gate, not as unaffordable. + +**The one unavoidable harness adaptation.** Every upstream class's `setUpClass` +raises `unittest.SkipTest` without CUDA (`:34-36`, `:177-178`, `:235-236`), so +upstream runs all eleven cases on a real device. T1 through T5, and T7 through T11, +run instead against a test backend that RECORDS the call sequence, so they gate on +this box and in continuous integration with no GPU. T6, the multi-replay case, has +both arms: the recording arm asserts the replay ORDER, and the GPU arm under G1 +asserts the VALUES. Nothing else about the upstream cases is adapted: the same break +counts, the same segment counts, the same arithmetic chains, the same in-place versus +assign split, and the same non-copyable fallback. + +The existing test surface the migration must keep green: + +| Existing test | Path | Why it binds | +|---|---|---| +| Capture-size set and pad selector | `tests/vllm/models/test_decode_graph_sizes.cpp` | the shared piece the seam must not change | +| CUDA capture capability | `tests/vt/test_cuda_backend.cpp:108` | the seam's backend contract | +| ROCm capture capability | `tests/vt/test_rocm_backend.cpp` | proves the seam is not CUDA-only | +| Tenstorrent trace capture | `tests/vt/test_tenstorrent_backend.cpp:1370,1376` | a third capture backend with different semantics | +| Retire-on-grow bookkeeping | `tests/vt/test_graph_safe_scratch.cpp` | the lifetime rule the seam enforces | +| 35B paged engine gate | `tests/parity/test_qwen36_paged_engine.cpp:140` | the model gate the first migration must hold | + +**Tests this row owes with no upstream counterpart**, each written red first, +numbered on from the ported set: + +12. **Replay ORDER, not only replay arithmetic.** Replay emits segment 0, break 0, + segment 1, break 1, and so on, in that exact order, and the order is identical on + the second and third replay. Upstream asserts the composed VALUE (T2, T3), which + a wrong order could in principle still satisfy for a commutative chain; this + asserts the sequence directly. +13. **Capture-failure drain.** A break function that throws leaves the stream + un-poisoned and the partially built `BreakableGraph` destroyed with every + already-instantiated segment released. This ports the recovery that + `qwen3_5.cpp:9913` and `qwen3_dflash.cpp:1106` each hand-rolled. Upstream has no + equivalent because Python's `finally` in `__exit__` (`:323-332`) covers it. +14. **Non-capturing backend.** On a backend where `SupportsGraphCapture()` is + false, the scope is inert and the forward runs eager. Vulkan + (`vulkan_backend.cpp:16`) and Metal (`metal_backend.mm:13`) are the live cases. + Upstream has no equivalent because BCG is CUDA and HIP only. +15. **Auxiliary-stream auto-join.** A segment that forks a queue and does not join + it before a break point is joined by the scope, per `## Port map`. This is the + port of `_end_current_segment` `:353-361`, which upstream exercises only + indirectly through its model-level integration test. +16. **Bit-exactness per migrated model**, over more than one replay. See `## Gates`. + +Tests 1 through 15 need no graphics processing unit (GPU), except T6's value arm. +They run against a test backend that records the call sequence. Test 16 needs a GPU +and an `rc` lease. + +## Gates + +**Correctness first. No performance result is accepted before the token-exact gate +for that model is established.** + +**G1, bit-exactness against eager, per migrated model, over MORE than one replay.** +For each migrated model, run the same inputs through the segmented capture path and +through the eager forward with capture disabled, and require the logits to be +bit-identical. The comparison must cover at least three consecutive replays of the +same captured graph, not one. A single replay cannot distinguish a correct segment +boundary from one that happens to read a buffer that has not yet been overwritten; +the class of defect this row's own history contains +(`.agents/specs/decode-graph-scratch-uaf-2026-07-18.md`) only appears on a LATER +replay. The existing 35B gate `tests/parity/test_qwen36_paged_engine.cpp:140` is +the model-level anchor for the first migration. + +**G2, reachability mutation.** Per `.agents/reachability.md` and the AGENTS.md +"Nothing lands dead" rule. In a scratch copy, DELETE the production call site that +routes a step into the seam — the model's registered forward entry, reached from +`ModelRegistry::Forward`, not an example and not a test — and rerun the focused +gate. The gate MUST go red. A gate that stays green without the call site measures +a class and not a capability. The reviewer restores the tree byte for byte +afterwards. Report the diff stat and the compiler status alongside the mutation +result, because a mutation that never applied and a mutation that failed to build +both read as a passing test. + +**G3, segment-count observability.** A log line or counter that reports segments +captured, break functions registered, and replays run, so the ratio is observable +in a run rather than inferred. Each driver today has a private +`replay_count()` (`qwen3_5.cpp:9680` and four siblings) and no segment count exists +at all. Without G3 there is no way to tell a two-segment capture from a fully eager +step, and "the graph ran" is exactly the claim a broken instrument fabricates. + +**G4, the eager arm is byte-identical.** With capture disabled, every migrated +model produces output bit-identical to its pre-migration eager output. This is what +makes each migration stage reversible. + +**G5, the non-CUDA arm.** The ROCm capture path +(`rocm_backend.hip:248`) runs the seam's unit tests. Tenstorrent's trace capture +(`tenstorrent_backend.cpp:75-81`) is exercised at least to the level its existing +test reaches (`tests/vt/test_tenstorrent_backend.cpp:1370`). A design that only +works on CUDA does not land. + +**Speed: what would have to be true.** No throughput gate is declared, because no +qualifying path is named. A speed claim from this row is admissible only when it +first names a path that is BOTH currently eager AND currently host-bound, and +states how the host-bound part is measured. Our prefill is neither: GB10 measured +prefill idle between launches at 3.8% with GPU-busy above 96%, and the 27B prefill +gap at 92.5% non-GEMM glue GPU work +([sglang-breakable-cuda-graph.md](sglang-breakable-cuda-graph.md) §3). Decode is +already captured and already banked its launch-overhead win. If a migration +produces a measurement, it is recorded as a coverage result plus a neutral +same-binary A/B, not as a win. + +## Dependencies + +| Dependency | State | Effect on this row | +|---|---|---| +| `vt::Backend` capture vocabulary | landed, `include/vt/backend.h:208-222` | the seam needs no new virtual | +| `DecodeGraphSizes` and `PadToCaptureSize` | landed, `decode_graph_sizes.h:47-54` | reused unchanged | +| Retire-on-grow scratch | landed, `src/vt/cuda/graph_safe_scratch.h` | the lifetime rule the seam enforces | +| `DevicePool` never-free discipline | landed, `device_pool.h:16` | replaces SGLang's mempool pinning | +| `ENG-CUDAGRAPH-DEDUP` (#1162) | in flight, parallel agent | touches the same handles; see D4 | +| `SPEC-MTP-K-GT-1` (#1020) | open | one instance this row's coverage closes | +| `ENG-CUDAGRAPH-DIFFUSION` (#1164) | blocked | explicitly excluded | +| An `rc` GPU lease | required from W1 onward | W0 needed none; every model gate does | + +## Work breakdown + +Each stage lands on its own, is reachable from a production entry point at its own +merge commit, and is revertible without touching the next stage. Each stage owes an +issue before it starts. + +**W0, the scoping spike. DONE in this change.** No production code. Its result is +recorded here and in `## Risks/decisions`. What it established: + +- The existing backend vocabulary expresses a segmented capture with no new + virtual. Verified by reading `cuda_backend.cu:225-232`: `EndCaptureGraph` stores + nothing, so `BeginCapture` and `EndCaptureGraph` compose into N+1 pairs. +- A break point is expressible in C++ with no compiler and no decorator, through + one `thread_local` capture pointer plus a free function. Proven with a + throwaway proof-of-concept, not committed: a two-layer forward with the break + site three frames below the entry produced 3 segments, 2 break functions, a + capture call log of `Begin End` three times, an identical replay order on two + consecutive replays, and a pure pass-through with zero backend calls when no + scope was active. Compiled and ran at exit status 0. +- The inventory in `## Our baseline` is the real one: NINE drivers, not the eight + the parent analysis recorded, and the persistent-device-input capability present + in exactly one of them. + +**W0 did NOT establish** that CUDA permits `cudaStreamEndCapture` followed by +`cudaStreamBeginCapture` on the same stream mid-forward with eager work between +them, on our stream configuration. This box has no NVIDIA GPU, and the fleet is +leased through `rc`. SGLang's `_end_current_segment` and `_begin_new_segment` +(`breakable_cuda_graph.py:335-365`) do exactly this on a production path at the +pinned revision, which is strong evidence but is not our measurement. **This is +W1's exit criterion and it needs a GPU lease.** It is named rather than skipped. + +**W1, the seam plus its unit gate, and one break point on one model.** Land +`vt::BreakableGraph`, `vt::GraphCaptureScope` and `vt::GraphBreak` with tests 1 +through 5 of `## Tests to port`. Register exactly one break point, on one model, +reached from `ModelRegistry::Forward`. Confirm the CUDA re-begin behavior on a +leased GPU first; if it does not hold, W1 stops and reports rather than working +around it. G2's mutation applies from this stage on, because W1 is the first stage +that can be dead. + +**W2, migrate `Qwen3DenseDecodeGraph` first.** It goes first for three reasons, +in order. It is the SMALLEST batched driver by machinery, holding no +`StepDevInputs` and no auxiliary tap buffer. It carries the DECLINE mitigation at +`qwen3.cpp:961-986`, so the migration has a documented, measured failure mode to +gate against rather than a hypothetical one. Its gate models, Qwen3-0.6B and +Qwen3-4B, are the cheapest to run of any driver's. It is deliberately NOT the 35B +MoE driver: `Qwen3_5DecodeGraph` is the richest and the most load-bearing, and +migrating it first would put the hardest correctness surface behind the least +seam experience. + +**W3, migrate the three remaining plain batched drivers.** `Qwen3MoeDecodeGraph`, +`DeepseekV2DecodeGraph` and `VoxtralDecodeGraph`. They share W2's shape, so each +is a repeat with its own G1 and G2. Landable one at a time. + +**W4, migrate the two Qwen3.5 drivers.** `Qwen3_5DecodeGraph` and +`Qwen3_5DenseDecodeGraph`. These carry `StepDevInputs`, the auxiliary taps and the +speculative-decode predicate. W4 is where the persistent device input path becomes +a seam capability rather than one driver's private code, which is what makes it +available to the four drivers that lack it. + +**W5, migrate the three single-shape drivers.** DeepSeek V4, Laguna and DFlash. +`laguna.cpp:2116-2119` already carries the note that its capture class is waiting +for exactly this seam, and it names DeepSeek V4's driver as the sibling that moves +with it. + +**W6, close the coverage gap the row exists for.** Only after W1 through W5 does +the predicate at `runner.cpp:1341` move from `pure_decode` to "eligible except at +the break points". This is LAST, not first: replacing the predicate before every +driver can survive a segmented capture would admit steps that no driver can serve. +[#1020](https://github.com/mudler/vllm.cpp/issues/1020) closes here or is +explicitly re-scoped here. + +## Risks/decisions + +**D1, the capture lifetime hazard, and why our position is different but not safe.** +A captured graph bakes the device pointer that an allocation returned at capture +time (`src/vt/cuda/graph_safe_scratch.h:9-20`). The tree already carries one +production incident from this: the 35B online-serving illegal memory access, where +a later, larger forward grew a scratch buffer and freed the block a captured graph +still referenced, root-caused in +`.agents/specs/decode-graph-scratch-uaf-2026-07-18.md`. Our `DevicePool` never +returns a block to the driver (`device_pool.h:16`), so a baked pointer stays +mapped. **That removes the free hazard and not the reuse hazard.** Between segment +`i` and segment `i+1` an eager break function runs and can take a block from the +free list that segment `i` still reads on every replay, and the result is silently +wrong numerics rather than a fault. Two segments make this strictly worse than the +single-capture case, because the window between them is host code by construction. +The seam must therefore make the intermediates a segment reads unavailable to the +free list for the life of the `BreakableGraph`, which is the same guarantee +SGLang gets from `use_count` pinning (`breakable_cuda_graph.py:14-23`) by a +different route. + +**A clean `compute-sanitizer` run is NOT evidence of safety for this class.** The +recorded incident reproduced 5 out of 5 times under normal operation and 0 out of 2 +under memcheck, because memcheck's serialization keeps the freed block from being +reused before the replay. Any W-stage that reports a clean sanitizer run as its +safety evidence has reported nothing. The evidence is G1 over more than one +replay, plus an adversarial case that allocates between segments. + +**D2, capture bakes the host source address of an upload.** `qwen3_5.h:262-268` +records that the persistent inputs are the HOST step vectors, mutated in place, +because on GB10 the host-to-device copies are capturable and a replay re-reads +from the fixed host address. A break function that captures a reference to a host +temporary, or a model that reallocates a host vector between capture and replay, +produces a replay that reads freed host memory. The seam owns this rule and must +state it at the `GraphBreak` declaration, because it is the rule a new model author +is least likely to know. + +**D3, the model-specific parts must not migrate.** The shape key, the pre-warm +forward and the output buffer set are model knowledge. A seam that tries to own +them becomes a second dispatcher with a per-model branch, which is the shape this +row exists to remove. The test of a correct split is that adding a tenth model +adds a break-point registration and a shape key, and no capture machinery. + +**D4, the interaction with `ENG-CUDAGRAPH-DEDUP` (#1162), owned in parallel.** +That row re-points ONE `cudaGraphExec` through `cudaGraphExecUpdate` on a topology +signature hit, instead of instantiating one executable per padded bucket. The two +rows touch the same handle: dedup changes what `EndCaptureGraph` hands back and +who owns it, and this row multiplies the number of such handles by the segment +count. They compose in SGLang, where `BreakableCUDAGraph._append_segment` +(`breakable_cuda_graph.py:266-274`) registers each segment with an optional +`_deduped_cuda_graph`, and `_begin_new_segment` (`:335-350`) asks for +`keep_graph=True` only when dedup is active. **Decision: this row lands first and +treats a segment handle as opaque.** `vt::BreakableGraph` must store the handle, +never inspect it, and must route every acquisition and release through +`Backend::EndCaptureGraph` and `Backend::DestroyGraph`, so #1162 can later +interpose at the backend without editing the container. Coordination is in writing +only, through this paragraph and the issues; neither agent edits the other's files. +The ordering risk if #1162 lands first is that segment count multiplies its +signature table; that is a sizing question for #1162, not a correctness question +for this row. + +**D5, the framing risk, and the decision on it.** The strongest failure mode for +this row is that somebody sells it as a speed row. It is not one, and the +supporting measurements are recorded and dated: prefill idle between launches at +3.8% with GPU-busy above 96%, and the 27B prefill gap at 92.5% non-GEMM glue GPU +work, both on GB10, both in +[sglang-breakable-cuda-graph.md](sglang-breakable-cuda-graph.md) §3. Decode already +banked its launch-overhead win. **Decision: this spec declares no throughput gate, +and `## Gates` states the two conditions any later speed claim must satisfy +first.** A negative result is regime-dependent, so the refutation is dated and +hardware-specific rather than permanent; the burden is on the claimant to name the +path and the measurement, not on this spec to pre-refute an unnamed one. + +**D6, the non-CUDA backends.** Tenstorrent implements capture as ttnn mesh-trace +(`tenstorrent_backend.cpp:18,75-81`), which is a different runtime concept from a +CUDA stream capture. A segmented capture assumes that ending a trace and starting +another on the same queue mid-forward is legal. **That assumption is UNVERIFIED on +both runtimes and this spec claims it on neither.** For CUDA, SGLang does it on a +production path at the pinned revision (`breakable_cuda_graph.py:352-367` then +`:335-350`), which is strong evidence and is not our measurement; confirming it on a +leased GPU is W1's exit criterion, in `## Work breakdown` W0's own words. For ttnn +traces there is not even that evidence. Nothing here measures the COST of a re-begin +on either runtime, so no cost claim is made; what would settle it is a same-binary +A/B of an N-segment capture against a 1-segment capture of the same forward, on the +leased GPU that W1 already needs. **Decision: the seam is +defined against the existing `vt::Backend` vocabulary and asks nothing new of any +backend, so a backend that cannot segment reports `SupportsGraphCapture()` and +runs one segment, which is exactly today's behavior.** No backend is required to +change for W1. + +**D7, a break point is a correctness boundary, not only a coverage knob.** Every +break point converts a graphed operation into a host-dispatched one, so a +mis-registered break point is invisible in a token gate and visible only as a +segment count. G3 exists for this reason. A model whose break-point set is wrong +still produces correct tokens; it produces them from more segments than it needs, +or from a graph that was never entered at all. + +**D8, this row states no runnable gate command, and that is the honest record.** +Moving the row to `READY` puts it in the gated population, where the gate-command +classifier asks whether its `## Gates` section names a command that can FAIL. It +does not, because no test exists to run until W1 lands one, so the row classifies +`gates-no-command` and the pinned runnable set does not move for this change. + +The classifier reads a backticked command out of the `## Gates` section, and the +first draft of this spec tripped it: `git diff --stat`, written as advice on how to +read a mutation result, was extracted as this row's gate command and made the row +claim gate coverage it does not have. The advice moved here and the section now +names no command. **Decision: do not manufacture a command to satisfy the +classifier, and do not re-pin the runnable set for a row with nothing to run.** W1 +earns the entry and re-pins the baseline in the same change. + +**D9, the break function's OUTPUT is a lifetime surface, and D1 and D2 do not cover +it.** D1 covers the device intermediates a SEGMENT reads and D2 covers the host +source of a captured upload. Both are about break-function INPUTS. The output is a +third case and it is the one upstream spends the most machinery on. On replay N the +eager function allocates a fresh result, so unless the seam copies that result back +into the destination the following segment baked at capture time, segment `i+1` reads +capture-time data forever while the break function writes into a buffer nobody reads. +Upstream closes it with `replay_fn` plus `_copy_output` +(`breakable_cuda_graph.py:231-235,172-201`) and keeps the destination alive with +`_weak_ref_if_tensor` (`:156-169`). **Decision: `vt::GraphBreak` states the writeback +contract at its declaration, per `## Port map` §3 — either it takes the destination +and the seam copies into it, or the break function writes in place into a persistent +model-owned buffer and returns nothing.** A `BreakableGraph` that stores the caller's +raw `fn` cannot express either, so it stores the seam's closure instead. + +This defect is invisible to every gate that runs ONE replay, and it produces wrong +numbers rather than a fault, so it is in the same detection class D1 describes: a +clean `compute-sanitizer` run says nothing about it. The ported `TestCopyOutput` +cases T7 through T10 and the multi-replay case T6 in `## Tests to port` are what pin +it, and G1's more-than-one-replay requirement is what catches it at model level. + +**D10, a segment cannot close while a forked queue is still capturing.** Upstream +auto-joins outstanding side streams inside `_end_current_segment` before +`capture_end()` (`breakable_cuda_graph.py:353-361`) and maintains the whole +`wait_stream` hook (`:101-153`, installed `:310`, removed `:332`) for no other +purpose. This is live here, not hypothetical: `SupportsAuxStream()` is `true` on +CUDA (`src/vt/cuda/cuda_backend.cu:200`) and two model paths fork on it — Qwen3.5's +MoE shared-expert overlap (`src/vllm/model_executor/models/qwen3_5.cpp:6254-6255` +fork, `:6384` join) and Laguna's decode graph, whose auxiliary queue is owned by the +capture class itself (`src/vllm/model_executor/models/laguna.cpp:2362-2367`, +`:2572-2576` fork, `:2612` join). W4 migrates the first and W5 the second. +**Decision: `GraphCaptureScope` owns the outstanding-fork set and joins it before +every `EndCaptureGraph`, and test 15 of `## Tests to port` gates it.** We need no +hook to populate that set, because our fork and join are explicit +`Backend::RecordEvent` and `Backend::QueueWaitEvent` calls rather than an implicit +torch API — which makes this cheaper for us than for SGLang, not harder. A break +point registered inside an unjoined fork window without this rule fails at +`EndCaptureGraph`, which is the one failure mode in this spec that is LOUD. + +## Now + +`READY`. The spec is committed, the design is grounded in both oracles, the +inventory is enumerated with anchors, the upstream unit suite is mapped case for +case in `## Tests to port`, and the work is decomposed into six landable stages. No +production code exists for this row. W1 needs an `rc` GPU lease to confirm its exit +criterion — that CUDA permits `cudaStreamEndCapture` followed by +`cudaStreamBeginCapture` mid-forward on our stream configuration — before +implementation starts. + +Revised 2026-08-18 after a fresh review returned `FAIL`. What changed: the upstream +test suite is ported rather than declared absent; `## Port map` §3 states the output +writeback contract that the raw-`fn` description had dropped (D9); the auxiliary +stream fork-join rule is stated and gated (D10); four enumerations in +`## Our baseline` and one `grep` count were re-derived and corrected against printed +instruments; D6 now defers to W1's exit criterion instead of asserting it; and the +row is recorded as coverage AND correctness per #1179. diff --git a/.agents/specs/sglang-breakable-cuda-graph.md b/.agents/specs/sglang-breakable-cuda-graph.md index 73bbf83b4..dd10207e4 100644 --- a/.agents/specs/sglang-breakable-cuda-graph.md +++ b/.agents/specs/sglang-breakable-cuda-graph.md @@ -108,16 +108,36 @@ only `pure_decode` batches to a graph, so one host-dependent op forces the whole eager. Coverage is a cliff, not a slope, and [#1020](https://github.com/mudler/vllm.cpp/issues/1020) is one instance of it. -Eight call sites hand-roll capture: `Qwen3_5DecodeGraph`, `Qwen3_5DenseDecodeGraph`, -`Qwen3MoeDecodeGraph`, `Qwen3DenseDecodeGraph`, `DeepseekV2DecodeGraph`, -`VoxtralDecodeGraph`, plus graph code in `deepseek_v4.cpp` and `laguna.cpp`. Each -re-derives capture, bucket padding, persistent-input threading and the pure-decode -predicate. AGENTS.md names this shape: a parallel path written by hand instead of a -shared seam. +**NINE call sites hand-roll capture**, corrected 2026-08-18 +([#1179](https://github.com/mudler/vllm.cpp/issues/1179)); this section recorded +eight when it landed in `9bc4d7f44`. They are `Qwen3_5DecodeGraph`, +`Qwen3_5DenseDecodeGraph`, `Qwen3MoeDecodeGraph`, `Qwen3DenseDecodeGraph`, +`DeepseekV2DecodeGraph`, `VoxtralDecodeGraph`, plus file-local graph code in +`deepseek_v4.cpp`, `laguna.cpp` and — the one this section missed — the DFlash draft +graph at `src/vllm/model_executor/models/qwen3_dflash.cpp:771,870,1038,1091,1095,1106`. +The full table with a declaration and implementation anchor per driver is +[eng-cudagraph-break.md](eng-cudagraph-break.md) `## Our baseline`. Each re-derives +capture, bucket padding, persistent-input threading and the pure-decode predicate. +AGENTS.md names this shape: a parallel path written by hand instead of a shared seam. + +**The duplication has already cost a shipped model its decode graph, so this is a +correctness finding and not only a coverage one.** The persistent DEVICE input path +`StepDevInputs` (`src/vllm/model_executor/models/qwen3_5.cpp:3894`) exists in exactly +one driver — `grep -c StepDevInputs` returns 41 lines in `qwen3_5.cpp` and 0 in each +of `qwen3_moe.cpp`, `qwen3.cpp`, `deepseek_v2.cpp` and `voxtral.cpp`. Because +`Qwen3DenseDecodeGraph` lacks it, `src/vllm/model_executor/models/qwen3.cpp:961-986` +DECLINES the graph outright whenever the asynchronous device-token mirror is live, +with its own measured comment: `depth-1, graph ON PASS 78/78`, `depth-2, graph OFF +PASS 82/82`, `depth-2, graph ON FAIL, slots 1-3 degenerate`. The comment names the +real fix as reading the identifiers at replay time from a stable device buffer, which +is precisely what the sibling driver already does. That is one capability, written +once, unavailable to four models, and a live mitigation standing in its place. BCG is the right construction for the fix, because vLLM gets its split from -`torch.compile` and we have no compiler. The value is **coverage and one seam instead -of eight**. It is not throughput on its own, and it must not be sold as throughput. +`torch.compile` and we have no compiler. The value is **coverage, one seam instead of +nine, and the correctness capability that seam makes shared**. It is not throughput +on its own, and it must not be sold as throughput. §3's refutation stands unchanged: +3.8% host idle in prefill, GPU-busy above 96%, and 92.5% non-GEMM glue. ## 5. Diffusion is the one shape where the lever is real, and it is blocked @@ -162,7 +182,7 @@ collapse. **Capturing now would measure nothing.** | Item | Issue | Row | |---|---|---| | Graph executable dedup via `cudaGraphExecUpdate` | [#1162](https://github.com/mudler/vllm.cpp/issues/1162) | `ENG-CUDAGRAPH-DEDUP` | -| A shared capture seam with break points, retiring the eight hand-rolled drivers | [#1163](https://github.com/mudler/vllm.cpp/issues/1163) | `ENG-CUDAGRAPH-BREAK` | +| A shared capture seam with break points, retiring the NINE hand-rolled drivers | [#1163](https://github.com/mudler/vllm.cpp/issues/1163) | `ENG-CUDAGRAPH-BREAK` | | Denoise-loop capture, blocked on #1010, #1087, #1024, #1007 | [#1164](https://github.com/mudler/vllm.cpp/issues/1164) | `ENG-CUDAGRAPH-DIFFUSION` | ## Stop conditions @@ -185,6 +205,18 @@ What was rejected and why: porting BCG as a prefill throughput lever, because ou prefill has 3.8% host idle and is 92.5% glue-GPU-bound; and scoping the diffusion capture now, because the render does no device compute to capture. +Corrected 2026-08-18 ([#1179](https://github.com/mudler/vllm.cpp/issues/1179)). Two +repairs to what this spec recorded on 2026-08-17. First, the hand-rolled driver count +is NINE, not eight: the `ENG-CUDAGRAPH-BREAK` scoping spike found the DFlash draft +graph (`src/vllm/model_executor/models/qwen3_dflash.cpp:771,870,1038,1091,1095,1106`) +that §4 had missed. Second, `ENG-CUDAGRAPH-BREAK` was recorded here as a COVERAGE row +only, and it is also a CORRECTNESS row: `qwen3.cpp:961-986` declines its decode graph +outright while the async device-token mirror is live, on a measured degeneration +(`depth-2, graph ON FAIL, slots 1-3 degenerate`), because the fix the comment names +lives in one sibling driver as `StepDevInputs` and nowhere else. A shipped model has +already lost its decode graph to this duplication. Neither repair touches the framing +rule: the row still makes no throughput claim, and §3's refutation is unchanged. + Why the defaults are what they are: `ENG-CUDAGRAPH` stays `PARTIAL` rather than moving, because nothing shipped. `SGLANG-BCG` is classified `INVENTORIED` rather than `OWED`, because the capability it names is already in the mirror source and only its