From b533de9d4057742032fa81ab94e3fbe2f899a4c3 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Tue, 18 Aug 2026 00:56:50 +0000 Subject: [PATCH 1/3] spec(ENG-CUDAGRAPH-BREAK): the capture seam has no break points, and the nine hand-rolled drivers each re-derive one Scoping spike plus the structured spec for #1163, and no production code. The row record says "large and structural, spike first", so this commit lands the design and the inventory and nothing else. Two defects, both re-grounded against the tree rather than restated. Capture is all or nothing: `src/vllm/v1/worker/gpu/runner.cpp:1341` admits a step only when `num_actual_tokens == num_reqs` and there is no prefill, and a step that fails that predicate runs eager in full. There is no way to spell "capture this forward except at these three calls", so coverage is a cliff and not a slope, and #1020 is one reachable instance. The driver count was EIGHT and is NINE. `src/vllm/model_executor/models/qwen3_dflash.cpp:771,1091` already wrote the ninth, which is the outcome the issue predicted would happen next. Seven pieces of machinery are re-derived across them, each cited at two sites in the spec, and the re-derivation has a measured cost rather than an aesthetic one: the persistent DEVICE input path `StepDevInputs` (`qwen3_5.cpp:3894`) exists in exactly one driver, `grep -c` returns 0 in the other four batched drivers, and `qwen3.cpp:961-986` therefore DECLINES the graph outright when the async device-token mirror is live, with the measured degeneration recorded in its own comment. One capability, written once, unavailable to four models. The boundary comes from vLLM and the construction comes from SGLang, and the spec keeps those apart. vLLM's `_attention_ops` (`config/compilation.py:764-772` at pin `5559679229`) is the split set and `FULL_AND_PIECEWISE` is the v1 default, so the capability is already in our denominator; but vLLM gets the split from `torch.compile` and we have no compiler. SGLang reaches the same coverage with none (`breakable_cuda_graph.py:204-241,246-263` at pin `f63458b5be`), so the mechanism is ported from there. The spike established that the existing `vt` vocabulary already expresses a segmented capture with NO new backend virtual, because `EndCaptureGraph` stores nothing (`src/vt/cuda/cuda_backend.cu:225-232`), and that a break point is expressible with one `thread_local` capture pointer plus a free function. A throwaway proof-of-concept, deliberately not committed, produced 3 segments and 2 break functions from a 2-layer forward whose break site sits three frames below the entry, replayed in a stable order twice, and made zero backend calls when no capture scope was active. It did NOT establish that CUDA permits re-beginning capture on the same stream mid-forward on our stream configuration; that needs a GPU, this box has none, and it is named as W1's exit criterion rather than skipped. Row moved `INVENTORIED` to `READY`, not `SPIKE`: the nine sections are committed and no claim is active, so no `CLAIM-*` owner would be honest. Summary counts follow. This is a COVERAGE row and the spec says so in three places. Prefill idle between launches measured 3.8% with GPU-busy above 96%, the 27B prefill gap is 92.5% non-GEMM glue GPU work, and decode already banked its launch-overhead win, so no throughput gate is declared and `## Gates` states the two conditions any later speed claim has to satisfy first. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code] --- .agents/engine-matrix.md | 6 +- .agents/specs/eng-cudagraph-break.md | 608 +++++++++++++++++++++++++++ 2 files changed, 611 insertions(+), 3 deletions(-) create mode 100644 .agents/specs/eng-cudagraph-break.md diff --git a/.agents/engine-matrix.md b/.agents/engine-matrix.md index 0b049156d..ad88f6218 100644 --- a/.agents/engine-matrix.md +++ b/.agents/engine-matrix.md @@ -38,7 +38,7 @@ forensics: roadmap_v1.md and the parity ledger. | Area | Rows | `ANCHOR-BACKFILL` | `PARTIAL` | `SPIKE` | `READY` | `ACTIVE` | `GATING` | `DONE` | `INVENTORIED` | |---|---:|---:|---:|---:|---:|---:|---:|---:|---:| -| Engine and scheduling | 30 | 6 | 3 | 1 | 3 | 6 | 2 | 2 | 7 | +| Engine and scheduling | 30 | 6 | 3 | 1 | 4 | 6 | 2 | 2 | 6 | | KV cache and memory | 23 | 7 | 3 | 2 | 3 | 2 | 2 | 0 | 4 | | Parallelism | 6 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 5 | | Sampling and generation | 15 | 4 | 2 | 0 | 0 | 4 | 0 | 1 | 4 | @@ -48,7 +48,7 @@ forensics: roadmap_v1.md and the parity ledger. | LoRA and adapters | 2 | 0 | 0 | 0 | 0 | 1 | 0 | 0 | 1 | | Long context and attention | 10 | 5 | 0 | 0 | 1 | 0 | 1 | 0 | 3 | | Loading, tokenizer, config | 10 | 3 | 3 | 0 | 0 | 1 | 1 | 1 | 1 | -| **Total** | **160** | **35** | **17** | **5** | **11** | **32** | **8** | **11** | **40** | +| **Total** | **160** | **35** | **17** | **5** | **12** | **32** | **8** | **11** | **39** | ## Engine core and scheduling @@ -61,7 +61,7 @@ forensics: roadmap_v1.md and the parity ledger. | `ENG-PREEMPT-RECOMPUTE` | FCFS tail preemption with recompute | T0 | `vllm/v1/core/sched/scheduler.py:1142`; `tests/v1/core/test_scheduler.py:930` | `src/vllm/v1/core/sched/scheduler.cpp:102,157`; `src/vllm/v1/core/sched/request_queue.cpp:36` | `tests/vllm/v1/test_scheduler.cpp:247,295`; `tests/vllm/v1/test_request_queue.cpp:91` | `planned: specs/preemption.md` | `ANCHOR-BACKFILL` | - | | `ENG-CUDAGRAPH` | Decode graph capture/replay modes (host-cluster cleanup: capture-size set derived from `max_num_seqs` mirroring vLLM `_set_cudagraph_sizes`; 2026-07-18 graph-baked-scratch use-after-free fix — the 35B c2+ online-serving IMA blocker) | T0 | `vllm/config/compilation.py:53,1319,683-684,1438-1444`; `vllm/config/vllm.py:1667-1770`; `vllm/v1/worker/gpu/cudagraph_utils.py:116`; `tests/compile/test_config.py:122,229` | `src/vt/cuda/cuda_backend.cu:76,97,105`; `include/vllm/model_executor/models/decode_graph_sizes.h`; `src/vllm/model_executor/models/qwen3_5.cpp:3754,3952`; `src/vllm/v1/worker/gpu/runner.cpp:577,597`; graph-safe scratch (retire-on-grow so graph-baked scratch pointers stay valid) `src/vt/cuda/graph_safe_scratch.h`, `src/vt/cuda/cuda_moe_marlin.cu:75`, `src/vt/cuda/cuda_matmul_nvfp4.cu:766`, `src/vt/cuda/cuda_matmul_nvfp4_cutlass.cu:105`, `src/vt/cuda/cuda_matmul_fp8_cutlass.cu:95` | `tests/vt/test_cuda_backend.cpp:98`; `tests/vllm/models/test_decode_graph_sizes.cpp`; `tests/vt/test_graph_safe_scratch.cpp`; explicit 35B gate `tests/parity/test_qwen36_paged_engine.cpp:140` | [blocktable-host-cluster-cleanup.md](specs/blocktable-host-cluster-cleanup.md); [decode-graph-scratch-uaf-2026-07-18.md](specs/decode-graph-scratch-uaf-2026-07-18.md) | `PARTIAL` | **PREFILL capture REFUTED as a lever (2026-08-17, [#1161](https://github.com/mudler/vllm.cpp/issues/1161)).** vLLM's v1 default already captures prefill piecewise (`vllm/config/compilation.py:60-63,615,630` @ `555967922`) and it is in our denominator; SGLang reached the same coverage without `torch.compile` via BCG (`SGLANG-BCG` in [sglang-matrix.md](sglang-matrix.md)). Neither helps us: GB10 2026-07-09 measured prefill GPU-idle-between-launches at **3.8%** with GPU-busy >96% on both arms, and the 27B prefill gap at **92.5% non-GEMM glue GPU work** with the dominant GEMM at +0.17% and attention AHEAD. There are no launch bubbles in our prefill to collapse. Row stays `PARTIAL`; the real residuals are exec dedup ([#1162](https://github.com/mudler/vllm.cpp/issues/1162)) and the break-point seam ([#1163](https://github.com/mudler/vllm.cpp/issues/1163)). Spec [sglang-breakable-cuda-graph.md](specs/sglang-breakable-cuda-graph.md) | | `ENG-CUDAGRAPH-DEDUP` | Graph-executable dedup: hash each captured graph's topology and re-point ONE `cudaGraphExec` with `cudaGraphExecUpdate` on a signature hit, instead of instantiating one exec per padded bucket per model. A memory and capture-time change, NOT a throughput change — a deduped replay launches the same nodes | T2 | vLLM has no analogue (its execs come from `torch.compile`); SGLang `python/sglang/srt/model_executor/runner_backend/cuda_graph_dedup_mixin.py:219-242,358` @ `f63458b5be` | UNPORTED — `cudaGraphExecUpdate` appears nowhere in `src/` or `include/`; today `src/vt/cuda/cuda_backend.cu:222-232` instantiates a fresh exec per capture, over the 7 (`max_num_seqs=32`) or 11 (64) buckets of `include/vllm/model_executor/models/decode_graph_sizes.h:32-41`, times eight drivers | owed: same-binary A/B proving a deduped replay is byte-identical to a non-deduped one; a capture-count/exec-count log line so the ratio is observable | [sglang-breakable-cuda-graph.md](specs/sglang-breakable-cuda-graph.md) | `INVENTORIED` | [#1162](https://github.com/mudler/vllm.cpp/issues/1162) | -| `ENG-CUDAGRAPH-BREAK` | One shared `vt` capture seam that accepts BREAK POINTS, so a forward containing a host-dependent op is still graphed instead of falling out entirely — and so the eight hand-rolled drivers become one. **Coverage row, not a throughput row** | T1 | mirror vLLM `CUDAGraphMode.PIECEWISE` splitting at `splitting_ops` (`vllm/config/compilation.py:60-63,517,615,630` @ `555967922`); construction from SGLang BCG `breakable_cuda_graph.py:204-241,244-260` @ `f63458b5be` (decorator + runtime stream capture, no compiler) | all-or-nothing today: `src/vllm/v1/worker/gpu/runner.cpp:1338-1341` routes only `pure_decode`; drivers `qwen3_5.h:275`, `qwen3_5_dense.h:391`, `qwen3_moe.h:117`, `qwen3.h:243`, `deepseek_v2.h:324`, `voxtral.h:126`, plus `deepseek_v4.cpp`, `laguna.cpp` | owed: reachability mutation (delete the production call site, rerun the focused gate); bit-exactness vs eager on every migrated model over MORE than one replay; the host-lifetime contract of [decode-graph-scratch-uaf-2026-07-18.md](specs/decode-graph-scratch-uaf-2026-07-18.md) enforced AT the seam | [sglang-breakable-cuda-graph.md](specs/sglang-breakable-cuda-graph.md) | `INVENTORIED` | [#1163](https://github.com/mudler/vllm.cpp/issues/1163), [#1020](https://github.com/mudler/vllm.cpp/issues/1020) | +| `ENG-CUDAGRAPH-BREAK` | One shared `vt` capture seam that accepts BREAK POINTS, so a forward containing a host-dependent op is still graphed instead of falling out entirely — and so the eight hand-rolled drivers become one. **Coverage row, not a throughput row** | T1 | mirror vLLM `CUDAGraphMode.PIECEWISE` splitting at `splitting_ops` (`vllm/config/compilation.py:60-63,517,615,630` @ `555967922`); construction from SGLang BCG `breakable_cuda_graph.py:204-241,244-260` @ `f63458b5be` (decorator + runtime stream capture, no compiler) | all-or-nothing today: `src/vllm/v1/worker/gpu/runner.cpp:1338-1341` routes only `pure_decode`; drivers `qwen3_5.h:275`, `qwen3_5_dense.h:391`, `qwen3_moe.h:117`, `qwen3.h:243`, `deepseek_v2.h:324`, `voxtral.h:126`, plus `deepseek_v4.cpp`, `laguna.cpp` — and the spike found the NINTH already written, `src/vllm/model_executor/models/qwen3_dflash.cpp:771,1091`. The re-derivation is measured, not asserted: `StepDevInputs` (`src/vllm/model_executor/models/qwen3_5.cpp:3894`, the persistent DEVICE input path) exists in ONE driver and `grep -c` returns 0 in `qwen3_moe.cpp`, `qwen3.cpp`, `deepseek_v2.cpp` and `voxtral.cpp`, which is why `src/vllm/model_executor/models/qwen3.cpp:961-986` DECLINES the graph outright when the async device-token mirror is live | owed: reachability mutation (delete the production call site, rerun the focused gate); bit-exactness vs eager on every migrated model over MORE than one replay; the host-lifetime contract of [decode-graph-scratch-uaf-2026-07-18.md](specs/decode-graph-scratch-uaf-2026-07-18.md) enforced AT the seam | spec [eng-cudagraph-break.md](specs/eng-cudagraph-break.md) (W0 spike DONE 2026-08-18: the existing `vt` capture vocabulary `include/vt/backend.h:208-222` expresses a SEGMENTED capture with NO new virtual, because `EndCaptureGraph` stores nothing (`src/vt/cuda/cuda_backend.cu:225-232`); a break point is expressible with one `thread_local` capture pointer plus a free function, no compiler and no decorator); analysis [sglang-breakable-cuda-graph.md](specs/sglang-breakable-cuda-graph.md) | `READY` | [#1163](https://github.com/mudler/vllm.cpp/issues/1163), [#1020](https://github.com/mudler/vllm.cpp/issues/1020) | | `ENG-CUDAGRAPH-DIFFUSION` | Capture the LTX-2.5 denoise loop (fixed shapes, many identical iterations — the ideal graph target). **BLOCKED, and the blocker is ours:** the render does almost no device compute to capture | T2 | SGLang enabled BCG on this shape AFTER our pin — LTX-2 H200 two-stage 10.75s->6.90s (`d4be483efb`), SANA 1024px -26% (`6c7498113f`), SANA denoise 0.73->0.457s (`56ef810cad`). Dated events, NOT pinned evidence; their win is mostly PyTorch host tax we do not pay | NO capture at all: `grep` for capture across `src/vllm/model_executor/models/ltx2*.cpp` returns nothing | blocked by [#1024](https://github.com/mudler/vllm.cpp/issues/1024) (GPU util **exactly 0 in 321 of 347 samples**, 1.00 core of 20 held for 17+ min after staging), [#1007](https://github.com/mudler/vllm.cpp/issues/1007) (VAE decode has no device arm), [#1087](https://github.com/mudler/vllm.cpp/issues/1087) (**57-66% of wall** is ONE resolution-CONSTANT serial host phase), [#1010](https://github.com/mudler/vllm.cpp/issues/1010) (no phase-boundary log). Decision point is a MEASUREMENT of GPU-busy vs wall once device-resident, not an implementation | [sglang-breakable-cuda-graph.md](specs/sglang-breakable-cuda-graph.md) | `INVENTORIED` | [#1164](https://github.com/mudler/vllm.cpp/issues/1164) | | `ENG-BATCH-INVARIANT` | Opt-in deterministic execution across scheduler batch sizes (`VLLM_BATCH_INVARIANT=1`): batch-invariant matmul/norm/attention/collectives plus persistent-scheduler NVFP4; production default remains off | T1 | default/env `vllm/envs.py:89,576-578`; initialization `vllm/v1/worker/gpu_worker.py:1262`; NVFP4 dispatch `csrc/libtorch_stable/quantization/fp4/nvfp4_scaled_mm_sm120_kernels.cu:212-220`; suite fixture `tests/v1/determinism/conftest.py:9-12`; operator/e2e `tests/v1/determinism/test_nvfp4_batch_invariant_scaled_mm.py`, `tests/v1/determinism/test_nvfp4_batch_invariant.py` @ `702f481` | - | [W3-C3R executed contract](specs/nvfp4-persistent-plan-cache.md#w3-c3r-batch-shape-localization-and-gate-correction-2026-07-13): production-default ours and vLLM both change outputs across batch shapes; no local opt-in implementation is claimed | `planned: specs/batch-invariant-execution.md` | `INVENTORIED` | - | | `ENG-ASYNC-SCHED` | Async/overlap scheduling (AsyncScheduler placeholders + depth-2 batch-queue step + async D2H on a copy stream); vLLM's DEFAULT at the pin — mirror obligation per B3. **Host-side machinery + runner device-input half + sampler-OUTPUT half LANDED + CPU-gated (2026-07-16):** `AsyncScheduler` placeholder accounting, `step_with_batch_queue` depth-2, `ResolveAsyncScheduling` default-ON-when-compatible + `MaxConcurrentBatches`, `VT_ASYNC_SCHED` rollback; the runner device-input path `combine_sampled_and_draft_tokens`; PLUS the sampler-OUTPUT half — `vt::Backend` event/pinned primitives (`AllocPinned`/events, CUDA cudaHostAlloc+cudaEvent, CPU sync-degeneration), `AsyncGPUModelRunnerOutput` (device sampled-id snapshot → non-blocking D2H on a copy queue + event; `get_output()` waits only that event; MAIN queue never blocked), `Sampler::forward(sampled_ids_out)` device-resident greedy, `GPUModelRunner::sample_tokens_async` + `runner_supports_async`, and the `Executor`+`step_with_batch_queue` seam resolving `get_output()` at CONSUME time. All behind `VT_ASYNC_RUNNER`/`set_async_input_combine`, default OFF. Sync path byte-identical (placeholder sites INERT while count 0; combine off; `sample_tokens_async` degenerates to sync when async off; `sampled_ids_out=nullptr`). **ENABLE-FLIP LANDED + CPU-gated (2026-07-16):** (1) `LoadedEngine` now reorders `runner_` before the scheduler and builds an `AsyncScheduler` + `max_concurrent_batches=2` when `ResolveAsyncScheduling(runner_.runner_supports_async())` resolves ON (else the byte-identical synchronous `Scheduler` + depth-1); the resolved mcb threads into `AsyncLLM`→`EngineCoreProc` (`step_with_batch_queue`) and the "Asynchronous scheduling is enabled/disabled" log mirrors vLLM for A/B audit; (2) the device combine/scatter kernel (`_combine_sampled_and_draft_tokens_kernel` + last_sampled scatter) is ported to CUDA (`src/vt/cuda/cuda_combine_tokens.cu`), main-stream-ordered on the CUDA async path so it DELETES `sample_tokens_async`'s pre-scatter `Synchronize`; the CPU backend keeps the host loop. `VT_ASYNC_RUNNER=1` engages full W3; `VT_ASYNC_SCHED=0` is the same-binary rollback. Production default (no env) stays synchronous byte-identical. **FULL W3 DGX proof RAN twice** — `f086b64` (5/5 gates PASS; c16 TPOT −5.4 ms WIN, tput neutral, TTFT +36 % = Little's-law repayment) and the 2026-07-16 re-proof on the THROUGHPUT-lever fix (persistent pooled sampled-id/pinned buffers + `Sampler` greedy scratch removing ALL per-step `cudaMalloc`/`cudaFree`/`cudaHostAlloc`/event-create from the sampled-id path, incl. the overlap-killing `cudaFree` inside `get_output`; mirrors `gpu_model_runner.py:873-878` + `async_utils.py:12-70`): token-exactness **6/6 PASS**, interleaved c16 **tput −0.32 % (gate ≥+1.5 % FAILS), TPOT −4.95 ms retained, TTFT +34.8 %** — the allocator lever is REFUTED as the tput unlock (≤0.1 % of a ~165 ms c16 step). **DEFAULT FLIPPED ON 2026-07-17** (`VT_ASYNC_RUNNER` default ON via the pure `AsyncRunnerFlagIsOn` predicate, mirroring `vllm/config/vllm.py:992-1044`): the discriminator (`6ea7856`) proved vLLM's own async pays the identical +26–31 % TTFT / −0.7 to −0.9 % tput / −2.6 to −4.3 ms TPOT envelope and W3-ON nets positive (both binding ITL-tail anomalies flip to PASS), so the "needs a throughput lever" ship-gate is RETIRED — W3 is a parity/mirror obligation with a tails+TPOT win. The flip is TOKEN-NEUTRAL (async-ON ≡ async-OFF bit-identical on DGX). `VT_ASYNC_RUNNER=0` = runner-level rollback, `VT_ASYNC_SCHED=0` = scheduler-level rollback. TTFT means rise into vLLM's async envelope BY DESIGN — the next binding grid runs async by default and its TTFT must NOT be misread as a regression. **ROBUSTNESS FIX 2026-07-20 (`discard_request_mask`):** the runner was missing vLLM's `discard_request_mask`, so `GPUModelRunner` emitted a sampled token for prefill-CHUNK requests too; under async this drained a `num_output_placeholders` never reserved (the `is_prefill_chunk` path adds none) → the `async_scheduler.cpp` `num_output_placeholders >= 0` assertion aborted on c8 + short-output (chunked prefill + preemption). FIX mirrors vLLM: `execute_model` computes `exec_state_.discard[i] = seq_len < num_tokens` (`gpu_model_runner.py:2048`); `sample_tokens` clears those rows to empty (`outputs.py:303`), the async path passes `invalid_req_indices` to `AsyncGPUModelRunnerOutput::get_output` (`gpu_model_runner.py:3625` + `outputs.py:303`). Scheduler UNCHANGED (assertion kept — it was correct once the runner honors `scheduler.py:1888-1890`). Sync/non-chunked decode byte-identical (mask all-zero); DGX 27B 235/235 + 35B 315/315, `vllm-bench` c8+short-output+chunked+kv-pressure no longer crashes, memcheck 0. Ledger [parity-ledger.md](parity-ledger.md) 2026-07-20 row | T1 | `vllm/v1/core/sched/async_scheduler.py:12`; `vllm/config/vllm.py:490,990,1038`; `vllm/v1/engine/core.py:519`; `vllm/v1/worker/gpu/input_batch.py:304-406`; `vllm/v1/worker/gpu/async_utils.py:12-70`; `vllm/v1/worker/gpu/gpu_model_runner.py:242-332`; `vllm/v1/outputs.py:298-307` | `src/vllm/v1/core/sched/async_scheduler.cpp:10,45`; placeholder plumbing `src/vllm/v1/core/sched/scheduler.cpp:148,164,605`; `src/vllm/v1/engine/core.cpp:91` (`step_with_batch_queue`, async-output seam); `src/vllm/v1/engine/core_proc.cpp:32,46`; config `include/vllm/config/scheduler.h:117,165,188`, `src/vllm/config/scheduler.cpp:12`; `include/vllm/v1/request.h:187`; runner input leaf `src/vllm/v1/worker/gpu/prepare_inputs.cpp`, `src/vllm/v1/worker/gpu/input_batch.cpp`; runner output leaf `include/vt/backend.h`+`src/vt/backend.cpp`+`src/vt/cuda/cuda_backend.cu` (event/pinned), `include/vllm/v1/worker/gpu/async_output.{h,cpp}` (`AsyncGPUModelRunnerOutput`), `src/vllm/v1/sample/sampler.cpp` (`sampled_ids_out`), `src/vllm/v1/worker/gpu/runner.cpp` (`sample_tokens_async`/`runner_supports_async`), `src/vllm/v1/executor/executor.cpp`+`include/vllm/v1/worker/gpu/model_runner_base.h` (async seam); enable-flip `include/vllm/entrypoints/model_loader.h`+`src/vllm/entrypoints/model_loader.cpp` (`runner_` before scheduler, `ResolveAsyncEnabled`/`MakeScheduler`, `AsyncScheduler`+mcb=2, log), `include/vllm/v1/engine/async_llm.h`+`src/vllm/v1/engine/async_llm.cpp` (mcb param → `EngineCoreProc`); device kernel `include/vt/cuda/combine_tokens.h`+`src/vt/cuda/cuda_combine_tokens.cu`, wired `src/vllm/v1/worker/gpu/runner.cpp` (CUDA combine/scatter branch removes the pre-sync) | `tests/vllm/v1/test_async_scheduler.cpp:1` (6 cases, 54 asserts; RED vs base Scheduler 2/6 fail); depth-2 engine cycle `tests/vllm/v1/test_engine_core_proc.cpp:479` (mcb=2, async-output seam); config resolution `tests/vllm/test_scheduler_config.cpp:75`; enable-flip construction matrix `tests/vllm/entrypoints/test_loaded_engine_dense.cpp` (runner×VT_ASYNC_SCHED → scheduler type + mcb; RED = un-flipped engine, 3/3 ON-arm asserts fail); runner input leaf `test_combine_tokens.cpp` (RED = stale → 5/7 fail), `test_input_batch.cpp`, `test_runner.cpp` (async-ON≡sync); output leaf `tests/vt/test_backend.cpp` (event/pinned contract), `tests/vllm/v1/worker/test_async_output.cpp` (materialize/flush/snapshot; RED = +1 splice), `test_runner.cpp` (`sample_tokens_async` decode ≡ sync); full CPU ctest 111/111, tools 164/164. Prior diagnostic `3812d8` six-leg control: total **1.002153×**, TTFT **0.862159×**, no GPU-time reduction (neutral for speed). **DEFAULT-FLIP (2026-07-17):** new pure CPU flag test [test_async_runner_flag.cpp](../tests/vllm/v1/worker/test_async_runner_flag.cpp) (11 asserts, default-ON/'0'-off); construction matrix [test_loaded_engine_dense.cpp](../tests/vllm/entrypoints/test_loaded_engine_dense.cpp) INVERTED (default → AsyncScheduler+mcb=2; RED verified 5 asserts fail vs un-flipped). CPU clean `-Werror` rebuild, full serial ctest **116/116**, tools **164/164**. **DGX re-confirmation** (evidence `dgx:~/work/vllm.cpp-async-flip`, CUTLASS+FA2 hard-verified, one flock): shipping default (async ON + RMSNorm-fast OFF) → **27B 235/235 + 35B 315/315** with the "Asynchronous scheduling is enabled (mcb=2)" log, and both rollback arms (`VT_ASYNC_RUNNER=0`, `VT_ASYNC_SCHED=0`) 235/235 + 315/315 log "disabled"; async arms BIT-IDENTICAL (token-neutral). Closing record [parity-ledger.md#L502](parity-ledger.md#L502) | [async-serving.md](specs/async-serving.md) | `DONE` | `6ea7856` | diff --git a/.agents/specs/eng-cudagraph-break.md b/.agents/specs/eng-cudagraph-break.md new file mode 100644 index 000000000..1f84b2ef4 --- /dev/null +++ b/.agents/specs/eng-cudagraph-break.md @@ -0,0 +1,608 @@ +# ENG-CUDAGRAPH-BREAK — a shared `vt` capture seam that accepts break points + +**Row:** `ENG-CUDAGRAPH-BREAK` (engine-matrix, engine core and scheduling). +**Issue:** [#1163](https://github.com/mudler/vllm.cpp/issues/1163), with +[#1020](https://github.com/mudler/vllm.cpp/issues/1020) as one reachable instance. +**Parent analysis:** [sglang-breakable-cuda-graph.md](sglang-breakable-cuda-graph.md) `## Owed`. +**Kind:** scoping spike plus structured spec. No production code lands with this spec. + +**This is a coverage row, not a throughput row.** The spike claims no speed and +measures none. `## Gates` below states what would have to be named and measured +before any speed claim is admissible, and `## Risks/decisions` D5 records why no +such path is named today. + +## Scope + +One shared capture seam under `vt` that captures a forward as a SEQUENCE of graph +segments split at break points, so a forward containing a host-dependent operation +is still graphed except at that operation, instead of running eager for the whole +step. The seam replaces the per-model capture drivers listed in `## Our baseline`. + +In scope: + +- The break-point primitive and the capture scope that collects segments. +- The segment container, its replay order, and its ownership of instantiated graphs. +- The lifetime contract for every buffer a segment or a break function reads. +- A staged migration of the existing drivers onto the seam, one driver per stage. +- The predicate that decides whether a step is graph-eligible, moved off + `pure_decode` and onto "eligible except at the break points". + +Out of scope, and each names its owner: + +- Graph-executable dedup through `cudaGraphExecUpdate`. That is + `ENG-CUDAGRAPH-DEDUP`, [#1162](https://github.com/mudler/vllm.cpp/issues/1162), + owned by a parallel agent. See `## Risks/decisions` D4 for the interaction. +- The LTX-2.5 denoise loop. That is `ENG-CUDAGRAPH-DIFFUSION`, + [#1164](https://github.com/mudler/vllm.cpp/issues/1164), blocked on its own defects. +- Any change to a model's numerics. Every migrated model must stay bit-exact + against its own eager forward. + +## Upstream chain + +**The boundary comes from vLLM. The construction comes from SGLang.** These are two +different questions and this spec keeps them apart. + +**Primary oracle, vLLM at pin `5559679229` (`.agents/upstream-sync.md`).** vLLM's +`CUDAGraphMode` enum defines `PIECEWISE = 1` and the composite +`FULL_AND_PIECEWISE = (FULL, PIECEWISE)` at +`/home/mudler/_git/vllm/vllm/config/compilation.py:60-63`. The docstring at +`:608-635` states that `FULL_AND_PIECEWISE` is the v1 default, that it captures a +full graph for decode batches and a piecewise graph for prefill and mixed batches, +and that piecewise mode keeps "the cudagraph incompatible ops (i.e. some attention +ops) outside the cudagraph". The split points themselves are `splitting_ops`, +declared at `:517`, and defaulted at `:1145` to `list(self._attention_ops)`. That +class-level list is at `:764-772` and names, among others, +`vllm::unified_attention_with_output`, `vllm::unified_mla_attention_with_output`, +`vllm::mamba_mixer2`, `vllm::mamba_mixer`, `vllm::short_conv`, +`vllm::linear_attention` and `vllm::qwen_gdn_attention_core`. + +So the boundary vLLM chose is: **the attention family is the break point**. Dense +attention, multi-head latent attention (MLA), the Mamba mixers, the short +convolution and the gated delta network (GDN) core. This spec adopts that boundary +and does not invent one. + +**Where vLLM's construction does not transfer.** vLLM obtains the split from +`torch.compile`: `set_splitting_ops_for_v1` at `:1123-1145` returns early unless +`self.mode == CompilationMode.VLLM_COMPILE`, and the split is applied by Dynamo at +the FX level or by an Inductor graph partition. We have no compiler, no Dynamo and +no FX graph. The boundary transfers; the mechanism cannot. + +**Secondary oracle, SGLang at pin `f63458b5be` (`.agents/oracles/sglang.md`).** +SGLang reaches the same coverage with no compiler, which is the construction this +spec ports. Verified in the pinned tree at +`/home/mudler/_git/sglang` (`git rev-parse HEAD` = +`f63458b5beaceabbd9d749b9fc956370e1b649e6`): + +| Mechanism | Anchor in `python/sglang/srt/` | +|---|---| +| Break-point wrapper | `.../breakable_cuda_graph/breakable_cuda_graph.py:204-241` (`eager_on_graph`) | +| Active-capture context | `:60` (`_current_capture_var`, a `ContextVar`) | +| Segment container | `:246-253` (`BreakableCUDAGraph`, `_segments` plus `_break_fns`) | +| Interleaved replay | `:255-263` | +| Capture scope enter and exit | `:307-331` (`__enter__`, `__exit__`) | +| Segment open and close | `:335-350`, `:352-365` | +| Bare break marker | `:370-375` (`break_graph`, an empty body under the decorator) | +| Shared mempool and weak refs | `:14-23` doc, `:156-169` (`_weak_ref_if_tensor`) | +| No-compiler backend | `.../runner_backend/breakable_cuda_graph_backend.py:14-17` | + +The break points SGLang registers, all through the same one-line wrapper: + +| Site | Anchor | +|---|---| +| Dense attention | `layers/radix_attention.py:256` | +| MLA | `models/deepseek_common/attention_forward_methods/forward_mla.py:1092` | +| Linear attention | `layers/radix_linear_attention.py:159` | +| Nemotron Mamba2 | `models/nemotron_h.py:1240` | +| DeepSeek sparse-attention indexer | `layers/attention/dsa/dsa_indexer.py:2414` | +| DeepSeek V4 attention | `models/deepseek_v4.py:284` | + +That set is the same family vLLM lists in `_attention_ops`. The two oracles agree +on the boundary, which is the strongest evidence available that the boundary is +right and not an artefact of either implementation. + +## Our baseline + +### The seam that exists + +`include/vt/backend.h:208-222` already declares the whole capture vocabulary: +`SupportsGraphCapture`, `BeginCapture`, `EndCapture`, `Replay`, and the multi-graph +handle trio `EndCaptureGraph`, `ReplayGraph`, `DestroyGraph`. The base definitions +in `src/vt/backend.cpp:29-34` throw for a backend that does not implement capture, +except `DestroyGraph`, which is a no-op. + +Three backends implement it: CUDA at `src/vt/cuda/cuda_backend.cu:198,204,225,233,288`, +ROCm at `src/vt/rocm/rocm_backend.hip:248`, and Tenstorrent at +`src/vt/tenstorrent/tenstorrent_backend.cpp:75-81`, which maps the calls onto ttnn +mesh-trace capture. Vulkan and Metal return false and document why, at +`src/vt/vulkan/vulkan_backend.cpp:16` and `src/vt/metal/metal_backend.mm:13`. + +**`EndCaptureGraph` stores nothing in the backend** (`cuda_backend.cu:225-232`). It +ends the stream capture, instantiates the graph, and returns the executable as an +opaque handle the caller owns. That property is what makes a segmented capture +expressible with **no new backend virtual**: a segment is one `BeginCapture` and +`EndCaptureGraph` pair, and N break points produce N+1 such pairs on the same queue. +The spike confirmed this shape end to end; see `## Work breakdown` W0. + +### Defect 1: capture is all or nothing + +`src/vllm/v1/worker/gpu/runner.cpp:1341` computes the only predicate that admits a +step to a graph: + +``` +const bool pure_decode = attn_meta.num_actual_tokens == num_reqs && + gdn_meta.num_prefill_tokens == 0; +``` + +It travels to the model as `ModelForwardInput::pure_decode` +(`include/vllm/model_executor/models/model_registry.h:281`, set at +`src/vllm/v1/worker/gpu/runner.cpp:1378`), and each model's registered forward +tests it before reaching its driver, for example +`src/vllm/model_executor/models/qwen3.cpp:961`. A step that fails the predicate +runs eager in full. There is no way to express "capture this forward except at +these three calls", so coverage is a cliff and not a slope. + +[#1020](https://github.com/mudler/vllm.cpp/issues/1020) is one reachable instance: +a speculative verify step whose actual draft depth differs from the configured +`num_speculative_tokens` (`model_registry.h:288`) is admitted by no graph and falls +out to eager with no log and no counter. + +### Defect 2: the drivers, enumerated + +Eight production drivers exist, and a ninth already landed. Six are batched decode +drivers with a padded-bucket ring; three are single-shape drivers. Every one of +them re-derives the same machinery. + +| Driver | Declaration | Implementation | Shape | +|---|---|---|---| +| `Qwen3_5DecodeGraph` | `include/vllm/model_executor/models/qwen3_5.h:275` | `src/vllm/model_executor/models/qwen3_5.cpp:9540,9563,9712,9893,9918,9922` | batched, padded ring | +| `Qwen3_5DenseDecodeGraph` | `include/vllm/model_executor/models/qwen3_5_dense.h:391` | `src/vllm/model_executor/models/qwen3_5.cpp:9974,9986,10321,10340,10348` | batched, padded ring | +| `Qwen3MoeDecodeGraph` | `include/vllm/model_executor/models/qwen3_moe.h:117` | `src/vllm/model_executor/models/qwen3_moe.cpp:385,404,469,512,515,524` | batched, padded ring | +| `Qwen3DenseDecodeGraph` | `include/vllm/model_executor/models/qwen3.h:243` | `src/vllm/model_executor/models/qwen3.cpp:583,613,682,866,888,897` | batched, padded ring | +| `DeepseekV2DecodeGraph` | `include/vllm/model_executor/models/deepseek_v2.h:324` | `src/vllm/model_executor/models/deepseek_v2.cpp:887,902,963,980` | batched, padded ring | +| `VoxtralDecodeGraph` | `include/vllm/model_executor/models/voxtral.h:126` | `src/vllm/model_executor/models/voxtral.cpp:448,469,535,582,585,594` | batched, padded ring | +| DeepSeek V4 `V4Graph` | none, file-local | `src/vllm/model_executor/models/deepseek_v4.cpp:1673,1719,1898,1900,1904` | single shape, `gstate` 0/1/2 | +| Laguna decode graph | none, file-local | `src/vllm/model_executor/models/laguna.cpp:2373,2689,2691,2692,2695` | single shape | +| DFlash draft graph (the ninth) | none, file-local | `src/vllm/model_executor/models/qwen3_dflash.cpp:771,870,1038,1091,1095,1106` | single shape, `g_state` 0/1/2 | + +**What every driver re-derives.** Each item below was read in at least two drivers +and the anchors give one instance each. + +1. The environment kill switch. Five drivers read `VLLM_CPP_CUDAGRAPH` themselves: + `qwen3_5.cpp:9540`, `qwen3_5.cpp:9974`, `qwen3_moe.cpp:385`, `qwen3.cpp:583`, + `deepseek_v2.cpp:887`, `voxtral.cpp:448`. The three single-shape drivers each + invented their own instead: `VT_V4_DECODE_GRAPH` at `deepseek_v4.cpp:1680`, + `VT_DFLASH_GRAPH` at `qwen3_dflash.cpp:870`, and the Laguna cluster at + `laguna.cpp:104,116,140`. There is no one switch that turns capture off. +2. The cold, warm, captured state machine. `qwen3_5.cpp:9563` carries `warm` and + `captured` flags on the slot; `deepseek_v4.cpp:1719` spells the same thing as + `int gstate = 0`; `qwen3_dflash.cpp:771` spells it as `int g_state = 0`. Three + spellings of one three-state machine. +3. The pad-to-bucket selection. Five call sites of `PadToCaptureSize`: + `qwen3_5.cpp:9712`, `qwen3_moe.cpp:469`, `qwen3.cpp:682`, `deepseek_v2.cpp:963`, + `voxtral.cpp:535`. The function itself is already shared, at + `include/vllm/model_executor/models/decode_graph_sizes.h:47-56`, which is the + proof that extraction works and is the pattern this row extends. +4. The per-slot persistent buffer set. `qwen3_5.cpp:9563-9600` holds `token_ids`, + `positions`, the two metadata structs, `hidden`, `logits`, `aux`, `graph`, + `fa_cols`, `captured`, `warm`, `replays`, `reuse_event`, `dev` and `pin`. Every + other batched driver holds a subset of the same fields under the same name + `SizeSlot`. +5. The diagnostics pair `captured()` and `replay_count()`, at `qwen3_5.cpp:9680`, + `qwen3_moe.cpp:454`, `qwen3.cpp:667`, `deepseek_v2.cpp:948`, `voxtral.cpp:519`. +6. The capture-failure recovery. `qwen3_5.cpp:9913`, `qwen3_5.cpp:10335` and + `qwen3_dflash.cpp:1106` each carry their own copy of the same + `try { g = EndCaptureGraph(); } catch (...) {}` drain, because a skipped + `EndCaptureGraph` poisons the stream permanently. +7. The invalidate-and-recapture rule on a shape change. `qwen3_5.cpp:9816-9826` + destroys the graph when the block-table column count moves; + `qwen3_dflash.cpp:1038-1047` does the same when the block width moves. + +**The divergence that proves the cost.** The persistent DEVICE input path, called +"Option A" in the tree, exists in exactly one driver. `StepDevInputs` +(`qwen3_5.cpp:3894`) is referenced 33 times in `qwen3_5.cpp` and **zero** times in +`qwen3_moe.cpp`, `qwen3.cpp`, `deepseek_v2.cpp` and `voxtral.cpp` (`grep -c`, all +four returned `0`). The other four replay against host vectors. One of them, +`qwen3.cpp:961-986`, therefore has to DECLINE the graph outright whenever the +asynchronous device-token mirror is live, and its comment records the measured +failure that forced it: "depth-2, graph ON FAIL, slots 1-3 degenerate". The same +comment names the real fix as reading the identifiers at replay time from a stable +device buffer, which is precisely what `StepDevInputs` already does in the sibling +driver. One capability, written once, unavailable to four models. That is the cost +of eight drivers stated as a defect rather than as an aesthetic complaint. + +**What is genuinely model-specific**, and must stay in the model: + +- Which operations are break points for this architecture. A GDN hybrid breaks in + different places from a dense attention stack. +- The shape key. `qwen3_5.cpp:9816` keys on the block-table column count as well + as the batch size; `qwen3_dflash.cpp:1036` keys on the draft block width. +- What the forward writes into the persistent output buffers: logits alone, or + logits plus the auxiliary hidden taps (`qwen3_5.cpp:9570-9577`). +- The pre-warm forward itself, because only the model knows its own allocation set. +- Any model-specific fusion that must run inside a segment. + +### The lifetime contract that already exists + +`src/vt/cuda/graph_safe_scratch.h:1-32` records the rule this seam must enforce +rather than re-learn: a captured graph BAKES the device pointer a grow-on-demand +scratch helper returned at capture time, so freeing that block on growth makes the +next replay read freed memory. The fix in the tree is retire-on-grow. The full +root-cause is `.agents/specs/decode-graph-scratch-uaf-2026-07-18.md`. + +The `DevicePool` (`include/vllm/model_executor/models/device_pool.h:1-38,71`) +never returns a block to the driver. That is a materially better position than +SGLang's, and `## Risks/decisions` D1 states why it does not make the hazard go +away. + +## Port map + +### The primitive + +Three types, all under `vt`, all backend-agnostic, none of them a new virtual on +`vt::Backend`. + +1. **`vt::BreakableGraph`** — the container. Holds `std::vector segments` + (each an opaque handle from `Backend::EndCaptureGraph`) and + `std::vector> break_fns`. Invariant, asserted: + `segments.size() == break_fns.size() + 1`. `Replay(Queue&)` walks + `ReplayGraph(segments[i])` then `break_fns[i]()` for each `i`, mirroring + `breakable_cuda_graph.py:255-263`. The destructor calls `DestroyGraph` on every + segment. This is the direct analogue of `BreakableCUDAGraph` at `:246`. + +2. **`vt::GraphCaptureScope`** — a resource-acquisition-is-initialization (RAII) + scope. Its constructor sets one `thread_local GraphCaptureScope*` and calls + `Backend::BeginCapture`; its destructor calls `Backend::EndCaptureGraph`, pushes + the final segment, and clears the pointer. This is `BreakableCUDAGraphCapture` + at `:277-331`, with the destructor doing what `__exit__` does at `:320-331`. The + thread-local replaces SGLang's `ContextVar` at `:60`. It must be + `thread_local`, not a global, for the same reason SGLang chose a `ContextVar` + and CUDA chose `cudaStreamCaptureModeThreadLocal` (`cuda_backend.cu:204-206`): + capture is a property of one thread's stream, and a process-wide flag would + make an unrelated thread's forward observe a capture it is not part of. + +3. **`vt::GraphBreak(fn)`** — the break point. Outside a capture scope it calls + `fn` and returns, so a non-capturing forward is byte-identical to today. Inside + a scope it closes the current segment, runs `fn` once eagerly so the outputs + hold real data, appends `fn` to `break_fns`, and opens a new segment. This is + `eager_on_graph` at `:204-241`, minus the decorator syntax we cannot write. + +**Why a thread-local and not a parameter.** The break site sits inside an attention +call, several frames below the forward entry point. Threading a capture context +through every intermediate signature would touch every model. SGLang made the same +call for the same reason. The spike proved the thread-local carries the context +across three frames with no intermediate signature change; see `## Work breakdown` W0. + +**A bare marker.** `vt::GraphBreak()` with no argument splits the segment and runs +nothing, mirroring `break_graph` at `breakable_cuda_graph.py:370-375`. It is the +form a model uses when the host-dependent work already ran outside the forward. + +### How the boundary is registered + +vLLM registers `splitting_ops` by operation NAME because it has an FX graph to +match names against. We have neither. Our equivalent is a call at the break site +itself, exactly as SGLang's is: one line at the attention entry point, mirroring +`radix_attention.py:256` and `radix_linear_attention.py:159`. The site is the +registration. + +The break-point set is the vLLM `_attention_ops` family +(`compilation.py:764-772`), reached through our own dispatch: `dense_attn::AttnBlock` +for dense attention and MLA, and the GDN and Mamba entry points for the recurrent +families. Naming the exact call sites is W1 work, not spike work, because the set +must be established per migrated model against that model's own break inventory. + +### How segments and intermediates stay valid + +The question SGLang answers with a shared memory pool and weak references +(`breakable_cuda_graph.py:14-23,156-169`) has a different answer here, because our +allocator is different. + +- **Device intermediates.** A `DBuf` allocated during segment `i` is returned to + the `DevicePool` free list when its scope ends. The pool never returns a block to + the driver (`device_pool.h:16`), so the POINTER segment `i` baked stays mapped + for the life of the process. SGLang needs `use_count` pinning because torch's + caching allocator can release a segment to the driver; we do not. What we DO need + is protection against REUSE, which `## Risks/decisions` D1 covers. +- **Break-function inputs.** The closure must capture device pointers or `Tensor` + views that are stable across replays, never a reference to a host temporary that + dies at the end of the capturing call. This is the same contract the existing + drivers already meet for their persistent slot buffers + (`qwen3_5.cpp:9563-9600`), lifted from a per-driver convention to a seam rule. +- **Host sources of a captured upload.** Capture bakes the HOST SOURCE ADDRESS of + an upload as well as the device destination. `## Risks/decisions` D2 is the rule + the seam must enforce. + +### What is retired, and what replaces it + +The seam absorbs items 1, 2, 5, 6 and 7 of the re-derivation list, plus the +segment bookkeeping. `decode_graph_sizes.h` already owns item 3 and is unchanged. +Item 4 becomes a model-supplied persistent buffer set the seam holds but does not +interpret. Each migrated driver keeps only its shape key, its pre-warm forward, +its break-point set and its output buffers. + +## Tests to port + +vLLM's own tests do not transfer. Its piecewise split is verified through +`torch.compile` configuration tests, and we have no compiler to configure. SGLang's +BCG carries no unit test at the pinned revision that exercises the segment +container in isolation. So the primary tests here are written against OUR seam, and +the anchors above are the behavioral specification they encode. + +The existing test surface the migration must keep green: + +| Existing test | Path | Why it binds | +|---|---|---| +| Capture-size set and pad selector | `tests/vllm/models/test_decode_graph_sizes.cpp` | the shared piece the seam must not change | +| CUDA capture capability | `tests/vt/test_cuda_backend.cpp:108` | the seam's backend contract | +| ROCm capture capability | `tests/vt/test_rocm_backend.cpp` | proves the seam is not CUDA-only | +| Tenstorrent trace capture | `tests/vt/test_tenstorrent_backend.cpp:1370,1376` | a third capture backend with different semantics | +| Retire-on-grow bookkeeping | `tests/vt/test_graph_safe_scratch.cpp` | the lifetime rule the seam enforces | +| 35B paged engine gate | `tests/parity/test_qwen36_paged_engine.cpp:140` | the model gate the first migration must hold | + +New tests this row owes, each written red first: + +1. **Segment arithmetic.** N break points in one capture scope produce exactly + N+1 segments and N break functions. Red first by asserting N segments. +2. **Replay order.** Replay emits segment 0, break 0, segment 1, break 1, and so + on, in that exact order, and the order is identical on the second and third + replay. This is the multi-replay requirement stated in `## Gates`. +3. **Pass-through outside capture.** With no scope active, `vt::GraphBreak(fn)` + runs `fn` and makes zero backend calls. Red first by asserting a backend call. +4. **Capture-failure drain.** A break function that throws leaves the stream + un-poisoned and the partially built `BreakableGraph` destroyed with every + already-instantiated segment released. This ports the recovery that + `qwen3_5.cpp:9913` and `qwen3_dflash.cpp:1106` each hand-rolled. +5. **Non-capturing backend.** On a backend where `SupportsGraphCapture()` is + false, the scope is inert and the forward runs eager. Vulkan + (`vulkan_backend.cpp:16`) and Metal (`metal_backend.mm:13`) are the live cases. +6. **Bit-exactness per migrated model**, over more than one replay. See `## Gates`. + +Tests 1 through 5 need no graphics processing unit (GPU). They run against a test +backend that records the call sequence. Test 6 needs a GPU and an `rc` lease. + +## Gates + +**Correctness first. No performance result is accepted before the token-exact gate +for that model is established.** + +**G1, bit-exactness against eager, per migrated model, over MORE than one replay.** +For each migrated model, run the same inputs through the segmented capture path and +through the eager forward with capture disabled, and require the logits to be +bit-identical. The comparison must cover at least three consecutive replays of the +same captured graph, not one. A single replay cannot distinguish a correct segment +boundary from one that happens to read a buffer that has not yet been overwritten; +the class of defect this row's own history contains +(`.agents/specs/decode-graph-scratch-uaf-2026-07-18.md`) only appears on a LATER +replay. The existing 35B gate `tests/parity/test_qwen36_paged_engine.cpp:140` is +the model-level anchor for the first migration. + +**G2, reachability mutation.** Per `.agents/reachability.md` and the AGENTS.md +"Nothing lands dead" rule. In a scratch copy, DELETE the production call site that +routes a step into the seam — the model's registered forward entry, reached from +`ModelRegistry::Forward`, not an example and not a test — and rerun the focused +gate. The gate MUST go red. A gate that stays green without the call site measures +a class and not a capability. The reviewer restores the tree byte for byte +afterwards. Report the diff stat and the compiler status alongside the mutation +result, because a mutation that never applied and a mutation that failed to build +both read as a passing test. + +**G3, segment-count observability.** A log line or counter that reports segments +captured, break functions registered, and replays run, so the ratio is observable +in a run rather than inferred. Each driver today has a private +`replay_count()` (`qwen3_5.cpp:9680` and four siblings) and no segment count exists +at all. Without G3 there is no way to tell a two-segment capture from a fully eager +step, and "the graph ran" is exactly the claim a broken instrument fabricates. + +**G4, the eager arm is byte-identical.** With capture disabled, every migrated +model produces output bit-identical to its pre-migration eager output. This is what +makes each migration stage reversible. + +**G5, the non-CUDA arm.** The ROCm capture path +(`rocm_backend.hip:248`) runs the seam's unit tests. Tenstorrent's trace capture +(`tenstorrent_backend.cpp:75-81`) is exercised at least to the level its existing +test reaches (`tests/vt/test_tenstorrent_backend.cpp:1370`). A design that only +works on CUDA does not land. + +**Speed: what would have to be true.** No throughput gate is declared, because no +qualifying path is named. A speed claim from this row is admissible only when it +first names a path that is BOTH currently eager AND currently host-bound, and +states how the host-bound part is measured. Our prefill is neither: GB10 measured +prefill idle between launches at 3.8% with GPU-busy above 96%, and the 27B prefill +gap at 92.5% non-GEMM glue GPU work +([sglang-breakable-cuda-graph.md](sglang-breakable-cuda-graph.md) §3). Decode is +already captured and already banked its launch-overhead win. If a migration +produces a measurement, it is recorded as a coverage result plus a neutral +same-binary A/B, not as a win. + +## Dependencies + +| Dependency | State | Effect on this row | +|---|---|---| +| `vt::Backend` capture vocabulary | landed, `include/vt/backend.h:208-222` | the seam needs no new virtual | +| `DecodeGraphSizes` and `PadToCaptureSize` | landed, `decode_graph_sizes.h:47-56` | reused unchanged | +| Retire-on-grow scratch | landed, `src/vt/cuda/graph_safe_scratch.h` | the lifetime rule the seam enforces | +| `DevicePool` never-free discipline | landed, `device_pool.h:16` | replaces SGLang's mempool pinning | +| `ENG-CUDAGRAPH-DEDUP` (#1162) | in flight, parallel agent | touches the same handles; see D4 | +| `SPEC-MTP-K-GT-1` (#1020) | open | one instance this row's coverage closes | +| `ENG-CUDAGRAPH-DIFFUSION` (#1164) | blocked | explicitly excluded | +| An `rc` GPU lease | required from W1 onward | W0 needed none; every model gate does | + +## Work breakdown + +Each stage lands on its own, is reachable from a production entry point at its own +merge commit, and is revertible without touching the next stage. Each stage owes an +issue before it starts. + +**W0, the scoping spike. DONE in this change.** No production code. Its result is +recorded here and in `## Risks/decisions`. What it established: + +- The existing backend vocabulary expresses a segmented capture with no new + virtual. Verified by reading `cuda_backend.cu:225-232`: `EndCaptureGraph` stores + nothing, so `BeginCapture` and `EndCaptureGraph` compose into N+1 pairs. +- A break point is expressible in C++ with no compiler and no decorator, through + one `thread_local` capture pointer plus a free function. Proven with a + throwaway proof-of-concept, not committed: a two-layer forward with the break + site three frames below the entry produced 3 segments, 2 break functions, a + capture call log of `Begin End` three times, an identical replay order on two + consecutive replays, and a pure pass-through with zero backend calls when no + scope was active. Compiled and ran at exit status 0. +- The inventory in `## Our baseline` is the real one: eight drivers plus a ninth + already written, and the persistent-device-input capability present in exactly + one of them. + +**W0 did NOT establish** that CUDA permits `cudaStreamEndCapture` followed by +`cudaStreamBeginCapture` on the same stream mid-forward with eager work between +them, on our stream configuration. This box has no NVIDIA GPU, and the fleet is +leased through `rc`. SGLang's `_end_current_segment` and `_begin_new_segment` +(`breakable_cuda_graph.py:335-365`) do exactly this on a production path at the +pinned revision, which is strong evidence but is not our measurement. **This is +W1's exit criterion and it needs a GPU lease.** It is named rather than skipped. + +**W1, the seam plus its unit gate, and one break point on one model.** Land +`vt::BreakableGraph`, `vt::GraphCaptureScope` and `vt::GraphBreak` with tests 1 +through 5 of `## Tests to port`. Register exactly one break point, on one model, +reached from `ModelRegistry::Forward`. Confirm the CUDA re-begin behavior on a +leased GPU first; if it does not hold, W1 stops and reports rather than working +around it. G2's mutation applies from this stage on, because W1 is the first stage +that can be dead. + +**W2, migrate `Qwen3DenseDecodeGraph` first.** It goes first for three reasons, +in order. It is the SMALLEST batched driver by machinery, holding no +`StepDevInputs` and no auxiliary tap buffer. It carries the DECLINE mitigation at +`qwen3.cpp:961-986`, so the migration has a documented, measured failure mode to +gate against rather than a hypothetical one. Its gate models, Qwen3-0.6B and +Qwen3-4B, are the cheapest to run of any driver's. It is deliberately NOT the 35B +MoE driver: `Qwen3_5DecodeGraph` is the richest and the most load-bearing, and +migrating it first would put the hardest correctness surface behind the least +seam experience. + +**W3, migrate the three remaining plain batched drivers.** `Qwen3MoeDecodeGraph`, +`DeepseekV2DecodeGraph` and `VoxtralDecodeGraph`. They share W2's shape, so each +is a repeat with its own G1 and G2. Landable one at a time. + +**W4, migrate the two Qwen3.5 drivers.** `Qwen3_5DecodeGraph` and +`Qwen3_5DenseDecodeGraph`. These carry `StepDevInputs`, the auxiliary taps and the +speculative-decode predicate. W4 is where the persistent device input path becomes +a seam capability rather than one driver's private code, which is what makes it +available to the four drivers that lack it. + +**W5, migrate the three single-shape drivers.** DeepSeek V4, Laguna and DFlash. +`laguna.cpp:2116-2119` already carries the note that its capture class is waiting +for exactly this seam, and it names DeepSeek V4's driver as the sibling that moves +with it. + +**W6, close the coverage gap the row exists for.** Only after W1 through W5 does +the predicate at `runner.cpp:1341` move from `pure_decode` to "eligible except at +the break points". This is LAST, not first: replacing the predicate before every +driver can survive a segmented capture would admit steps that no driver can serve. +[#1020](https://github.com/mudler/vllm.cpp/issues/1020) closes here or is +explicitly re-scoped here. + +## Risks/decisions + +**D1, the capture lifetime hazard, and why our position is different but not safe.** +A captured graph bakes the device pointer that an allocation returned at capture +time (`src/vt/cuda/graph_safe_scratch.h:9-20`). The tree already carries one +production incident from this: the 35B online-serving illegal memory access, where +a later, larger forward grew a scratch buffer and freed the block a captured graph +still referenced, root-caused in +`.agents/specs/decode-graph-scratch-uaf-2026-07-18.md`. Our `DevicePool` never +returns a block to the driver (`device_pool.h:16`), so a baked pointer stays +mapped. **That removes the free hazard and not the reuse hazard.** Between segment +`i` and segment `i+1` an eager break function runs and can take a block from the +free list that segment `i` still reads on every replay, and the result is silently +wrong numerics rather than a fault. Two segments make this strictly worse than the +single-capture case, because the window between them is host code by construction. +The seam must therefore make the intermediates a segment reads unavailable to the +free list for the life of the `BreakableGraph`, which is the same guarantee +SGLang gets from `use_count` pinning (`breakable_cuda_graph.py:14-23`) by a +different route. + +**A clean `compute-sanitizer` run is NOT evidence of safety for this class.** The +recorded incident reproduced 5 out of 5 times under normal operation and 0 out of 2 +under memcheck, because memcheck's serialization keeps the freed block from being +reused before the replay. Any W-stage that reports a clean sanitizer run as its +safety evidence has reported nothing. The evidence is G1 over more than one +replay, plus an adversarial case that allocates between segments. + +**D2, capture bakes the host source address of an upload.** `qwen3_5.h:262-268` +records that the persistent inputs are the HOST step vectors, mutated in place, +because on GB10 the host-to-device copies are capturable and a replay re-reads +from the fixed host address. A break function that captures a reference to a host +temporary, or a model that reallocates a host vector between capture and replay, +produces a replay that reads freed host memory. The seam owns this rule and must +state it at the `GraphBreak` declaration, because it is the rule a new model author +is least likely to know. + +**D3, the model-specific parts must not migrate.** The shape key, the pre-warm +forward and the output buffer set are model knowledge. A seam that tries to own +them becomes a second dispatcher with a per-model branch, which is the shape this +row exists to remove. The test of a correct split is that adding a tenth model +adds a break-point registration and a shape key, and no capture machinery. + +**D4, the interaction with `ENG-CUDAGRAPH-DEDUP` (#1162), owned in parallel.** +That row re-points ONE `cudaGraphExec` through `cudaGraphExecUpdate` on a topology +signature hit, instead of instantiating one executable per padded bucket. The two +rows touch the same handle: dedup changes what `EndCaptureGraph` hands back and +who owns it, and this row multiplies the number of such handles by the segment +count. They compose in SGLang, where `BreakableCUDAGraph._append_segment` +(`breakable_cuda_graph.py:266-276`) registers each segment with an optional +`_deduped_cuda_graph`, and `_begin_new_segment` (`:335-350`) asks for +`keep_graph=True` only when dedup is active. **Decision: this row lands first and +treats a segment handle as opaque.** `vt::BreakableGraph` must store the handle, +never inspect it, and must route every acquisition and release through +`Backend::EndCaptureGraph` and `Backend::DestroyGraph`, so #1162 can later +interpose at the backend without editing the container. Coordination is in writing +only, through this paragraph and the issues; neither agent edits the other's files. +The ordering risk if #1162 lands first is that segment count multiplies its +signature table; that is a sizing question for #1162, not a correctness question +for this row. + +**D5, the framing risk, and the decision on it.** The strongest failure mode for +this row is that somebody sells it as a speed row. It is not one, and the +supporting measurements are recorded and dated: prefill idle between launches at +3.8% with GPU-busy above 96%, and the 27B prefill gap at 92.5% non-GEMM glue GPU +work, both on GB10, both in +[sglang-breakable-cuda-graph.md](sglang-breakable-cuda-graph.md) §3. Decode already +banked its launch-overhead win. **Decision: this spec declares no throughput gate, +and `## Gates` states the two conditions any later speed claim must satisfy +first.** A negative result is regime-dependent, so the refutation is dated and +hardware-specific rather than permanent; the burden is on the claimant to name the +path and the measurement, not on this spec to pre-refute an unnamed one. + +**D6, the non-CUDA backends.** Tenstorrent implements capture as ttnn mesh-trace +(`tenstorrent_backend.cpp:18,75-81`), which is a different runtime concept from a +CUDA stream capture. A segmented capture assumes that ending a trace and starting +another on the same queue mid-forward is legal and cheap. That holds for CUDA by +SGLang's production use and is UNVERIFIED for ttnn traces. **Decision: the seam is +defined against the existing `vt::Backend` vocabulary and asks nothing new of any +backend, so a backend that cannot segment reports `SupportsGraphCapture()` and +runs one segment, which is exactly today's behavior.** No backend is required to +change for W1. + +**D8, this row states no runnable gate command, and that is the honest record.** +Moving the row to `READY` puts it in the gated population, where the gate-command +classifier asks whether its `## Gates` section names a command that can FAIL. It +does not, because no test exists to run until W1 lands one, so the row classifies +`gates-no-command` and the pinned runnable set does not move for this change. + +The classifier reads a backticked command out of the `## Gates` section, and the +first draft of this spec tripped it: `git diff --stat`, written as advice on how to +read a mutation result, was extracted as this row's gate command and made the row +claim gate coverage it does not have. The advice moved here and the section now +names no command. **Decision: do not manufacture a command to satisfy the +classifier, and do not re-pin the runnable set for a row with nothing to run.** W1 +earns the entry and re-pins the baseline in the same change. + +**D7, a break point is a correctness boundary, not only a coverage knob.** Every +break point converts a graphed operation into a host-dispatched one, so a +mis-registered break point is invisible in a token gate and visible only as a +segment count. G3 exists for this reason. A model whose break-point set is wrong +still produces correct tokens; it produces them from more segments than it needs, +or from a graph that was never entered at all. + +## Now + +`READY`. The spec is committed, the design is grounded in both oracles, the +inventory is enumerated with anchors, and the work is decomposed into six landable +stages. No production code exists for this row. W1 needs an `rc` GPU lease to +confirm its exit criterion before implementation starts. From e714dc25e67e16362ffb32060475cf58a10c6368 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Tue, 18 Aug 2026 01:40:59 +0000 Subject: [PATCH 2/3] fix(ENG-CUDAGRAPH-BREAK): the upstream suite the spec said did not exist is 305 lines, and the primitive it described drops the output writeback (#1163, #1179) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A fresh review returned FAIL on the spec-only change. Two findings were load-bearing and both were verified against the pinned SGLang tree at f63458b5be before repair. `## Tests to port` asserted that "SGLang's BCG carries no unit test at the pinned revision that exercises the segment container in isolation". That file is test/registered/cuda_graph/breakable/test_breakable_cuda_graph.py, 305 lines, and it carries eleven unit cases across TestBreakableCUDAGraphBasic (:30), TestCopyOutput (:172) and TestBreakGraphHelper (:230). A section that asserts there is nothing to port deletes the obligation the section exists for. Every case is now mapped to the local test that owes it, with the one harness adaptation stated once: upstream skips without CUDA, and ten of the eleven run here against a recording backend. test_gsm8k_accuracy is deliberately excluded, with the reason, because it is a distributional accuracy floor on a prefill path and G1's bit-exactness is strictly stronger. `## Port map` §3 said eager_on_graph "appends `fn` to break_fns". It does not. It appends a replay_fn closure (:231-235) that calls the captured inner function and then writes the result back into the capture-time destination through _copy_output (:172-201), with _weak_ref_if_tensor (:156-169) keeping that destination alive. The writeback is the guarantee, not sugar: on replay N the eager op returns a fresh allocation whose address is not the one the next segment baked, so a container replaying the raw fn leaves segment i+1 reading capture-time data while the break function writes elsewhere. That is wrong numerics rather than a fault, the same detection class D1 records compute-sanitizer cannot see. The contract is now stated at the GraphBreak declaration, pinned by the ported TestCopyOutput cases, and recorded as D9. Third, the auxiliary-stream fork was absent everywhere. Upstream auto-joins side streams before closing a segment (:353-361) and maintains the whole wait_stream hook (:101-153, installed :310, removed :332) for that alone. It is live here: SupportsAuxStream() is true on CUDA (cuda_backend.cu:200) and both drivers this row migrates in W4 and W5 fork on it, at qwen3_5.cpp:6254 and laguna.cpp:2572. Stated on GraphCaptureScope, gated by a new test, and recorded as D10. Four enumerations in `## Our baseline` and one grep count were re-derived with printed instruments and corrected. Six drivers read VLLM_CPP_CUDAGRAPH, not five; the Qwen3.5 dense getenv is :9969, not :9974; Laguna's capture switch is VT_LAGUNA_DECODE_GRAPH at :1684, not the KV cluster at :104,116,140, none of which is a capture switch; PadToCaptureSize has six call sites and the diagnostics pair six definitions. The pattern was that the Qwen3.5 dense driver was dropped from three of four enumerations though W4 migrates it. StepDevInputs is on 41 lines of qwen3_5.cpp and occurs 47 times, not the 33 the spec recorded, which is neither number; the four zeros reproduce under both instruments and the conclusion survives. D6 asserted the CUDA mid-forward re-begin "holds" and is "cheap" while W0 said the same fact is not our measurement. D6 now defers to W1's exit criterion in W0's own words, drops the unmeasured cost claim, and names the A/B that would settle it. Anchors that landed inside the right construct but not on it are now exact, the D-entries run D1 through D10 in order, and the upstream path is spelled once in full rather than abbreviated in a table header. Riding with it, #1179: the driver count recorded in 9bc4d7f44 is eight and is actually nine, corrected in all four places that state it. The DFlash draft graph (qwen3_dflash.cpp:771,870,1038,1091,1095,1106) is file-local with no header declaration, which is why it was missed. The substantive half is the reframing. ENG-CUDAGRAPH-BREAK was recorded as a coverage row and it is also a correctness row: qwen3.cpp:961-986 declines its decode graph outright while the async device-token mirror is live, on its own measured battery, because the fix its comment names lives in one sibling driver as StepDevInputs and nowhere else. A shipped model has already lost its decode graph to this duplication. The framing rule is unchanged: coverage and correctness, never speed, and the prefill refutation stands. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code] --- .agents/engine-matrix.md | 4 +- .agents/issue-index.md | 1 + .agents/roadmap_v1.md | 2 +- .agents/specs/eng-cudagraph-break.md | 375 +++++++++++++++---- .agents/specs/sglang-breakable-cuda-graph.md | 50 ++- 5 files changed, 344 insertions(+), 88 deletions(-) diff --git a/.agents/engine-matrix.md b/.agents/engine-matrix.md index ad88f6218..4c42aa9f4 100644 --- a/.agents/engine-matrix.md +++ b/.agents/engine-matrix.md @@ -60,8 +60,8 @@ forensics: roadmap_v1.md and the parity ledger. | `KV-PREFIX-MATCH-UNIT` | `--prefix-match-unit` (config `prefix_match_unit`): the finest token boundary a prefix-cache hit can land on == the `hash_block_size`/"prefix match unit" the block hasher uses. NEW in 0.26 (absent at the prior `e24d1b24`/0.25.0 pin). For a HYBRID/multi-group model the resolver `resolve_kv_cache_block_sizes` computes `hash_block_size = prefix_match_unit if set else gcd(group_block_sizes)` (scheduler block size = `lcm`), letting matching land FINER than a physical block (e.g. 16/32 tokens inside a 1024-token block) provided every group block size is divisible by it; single-group (dense) models ignore the knob. Backs off to the scheduler block size when no prefix-cache/connector consumer is active or a mamba group diverges from `cache_block_size` (mamba_cache_mode != "align"); throws on a non-divisible unit. **W0 spike + W1 resolver LANDED 2026-07-28 (`CLAIM-PREFIX-MATCH-UNIT`, NOT pushed):** `resolve_kv_cache_block_sizes` ported 1:1 (explicit-parameter signature vs upstream's `VllmConfig`, our config surface is threaded), RED-first unit-gated (default gcd `!=` `=16` override). `PARTIAL`: the config/CLI/ABI field (W2), the scheduler threading of a resolved `hash_block_size != block_size` + mamba partial-tail stop (W3, needs the `KV-BLOCK-POOL` align path that still throws), and the benchmark (W4) are deferred. Default path byte-identical (single-group inert; scheduler still passes `block_size`). | T1 | `vllm/engine/arg_utils.py:696,1222,1940`; `vllm/config/cache.py:56-67`; resolver `vllm/v1/core/kv_cache_utils.py:626-688`; hasher `:691-748`; call site `vllm/v1/engine/core.py:154`; scheduler `vllm/v1/core/sched/scheduler.py:76,268-270,282,312-318`; fine-grained view `vllm/v1/core/single_type_kv_cache_manager.py:683,697` | resolver `src/vllm/v1/core/kv_cache_utils.cpp:638` (`resolve_kv_cache_block_sizes`), decl `include/vllm/v1/core/kv_cache_utils.h`; hash_block_size already plumbed `get_request_block_hasher` `src/vllm/v1/core/kv_cache_utils.cpp:577`; DEFERRED align path throws `src/vllm/v1/core/block_pool.cpp:93,220` (shared with `KV-BLOCK-POOL`) | `tests/vllm/v1/test_prefix_match_unit.cpp:64,88,99,119,129,145,164,186` 8/8 (29 assertions): single-group inert + DCP scale, multi-group default=gcd, `=16` override finer-than-default (RED), finer-than-1024-block, non-divisible throws, no-consumer back-off + connector-alone re-enable, mamba non-align back-off vs align gcd, hasher-granularity RED (coarse 2 vs fine 4 hashes); [parity-ledger.md](parity-ledger.md) | [prefix-match-unit.md](specs/prefix-match-unit.md) | `PARTIAL` | `CLAIM-PREFIX-MATCH-UNIT` | | `ENG-PREEMPT-RECOMPUTE` | FCFS tail preemption with recompute | T0 | `vllm/v1/core/sched/scheduler.py:1142`; `tests/v1/core/test_scheduler.py:930` | `src/vllm/v1/core/sched/scheduler.cpp:102,157`; `src/vllm/v1/core/sched/request_queue.cpp:36` | `tests/vllm/v1/test_scheduler.cpp:247,295`; `tests/vllm/v1/test_request_queue.cpp:91` | `planned: specs/preemption.md` | `ANCHOR-BACKFILL` | - | | `ENG-CUDAGRAPH` | Decode graph capture/replay modes (host-cluster cleanup: capture-size set derived from `max_num_seqs` mirroring vLLM `_set_cudagraph_sizes`; 2026-07-18 graph-baked-scratch use-after-free fix — the 35B c2+ online-serving IMA blocker) | T0 | `vllm/config/compilation.py:53,1319,683-684,1438-1444`; `vllm/config/vllm.py:1667-1770`; `vllm/v1/worker/gpu/cudagraph_utils.py:116`; `tests/compile/test_config.py:122,229` | `src/vt/cuda/cuda_backend.cu:76,97,105`; `include/vllm/model_executor/models/decode_graph_sizes.h`; `src/vllm/model_executor/models/qwen3_5.cpp:3754,3952`; `src/vllm/v1/worker/gpu/runner.cpp:577,597`; graph-safe scratch (retire-on-grow so graph-baked scratch pointers stay valid) `src/vt/cuda/graph_safe_scratch.h`, `src/vt/cuda/cuda_moe_marlin.cu:75`, `src/vt/cuda/cuda_matmul_nvfp4.cu:766`, `src/vt/cuda/cuda_matmul_nvfp4_cutlass.cu:105`, `src/vt/cuda/cuda_matmul_fp8_cutlass.cu:95` | `tests/vt/test_cuda_backend.cpp:98`; `tests/vllm/models/test_decode_graph_sizes.cpp`; `tests/vt/test_graph_safe_scratch.cpp`; explicit 35B gate `tests/parity/test_qwen36_paged_engine.cpp:140` | [blocktable-host-cluster-cleanup.md](specs/blocktable-host-cluster-cleanup.md); [decode-graph-scratch-uaf-2026-07-18.md](specs/decode-graph-scratch-uaf-2026-07-18.md) | `PARTIAL` | **PREFILL capture REFUTED as a lever (2026-08-17, [#1161](https://github.com/mudler/vllm.cpp/issues/1161)).** vLLM's v1 default already captures prefill piecewise (`vllm/config/compilation.py:60-63,615,630` @ `555967922`) and it is in our denominator; SGLang reached the same coverage without `torch.compile` via BCG (`SGLANG-BCG` in [sglang-matrix.md](sglang-matrix.md)). Neither helps us: GB10 2026-07-09 measured prefill GPU-idle-between-launches at **3.8%** with GPU-busy >96% on both arms, and the 27B prefill gap at **92.5% non-GEMM glue GPU work** with the dominant GEMM at +0.17% and attention AHEAD. There are no launch bubbles in our prefill to collapse. Row stays `PARTIAL`; the real residuals are exec dedup ([#1162](https://github.com/mudler/vllm.cpp/issues/1162)) and the break-point seam ([#1163](https://github.com/mudler/vllm.cpp/issues/1163)). Spec [sglang-breakable-cuda-graph.md](specs/sglang-breakable-cuda-graph.md) | -| `ENG-CUDAGRAPH-DEDUP` | Graph-executable dedup: hash each captured graph's topology and re-point ONE `cudaGraphExec` with `cudaGraphExecUpdate` on a signature hit, instead of instantiating one exec per padded bucket per model. A memory and capture-time change, NOT a throughput change — a deduped replay launches the same nodes | T2 | vLLM has no analogue (its execs come from `torch.compile`); SGLang `python/sglang/srt/model_executor/runner_backend/cuda_graph_dedup_mixin.py:219-242,358` @ `f63458b5be` | UNPORTED — `cudaGraphExecUpdate` appears nowhere in `src/` or `include/`; today `src/vt/cuda/cuda_backend.cu:222-232` instantiates a fresh exec per capture, over the 7 (`max_num_seqs=32`) or 11 (64) buckets of `include/vllm/model_executor/models/decode_graph_sizes.h:32-41`, times eight drivers | owed: same-binary A/B proving a deduped replay is byte-identical to a non-deduped one; a capture-count/exec-count log line so the ratio is observable | [sglang-breakable-cuda-graph.md](specs/sglang-breakable-cuda-graph.md) | `INVENTORIED` | [#1162](https://github.com/mudler/vllm.cpp/issues/1162) | -| `ENG-CUDAGRAPH-BREAK` | One shared `vt` capture seam that accepts BREAK POINTS, so a forward containing a host-dependent op is still graphed instead of falling out entirely — and so the eight hand-rolled drivers become one. **Coverage row, not a throughput row** | T1 | mirror vLLM `CUDAGraphMode.PIECEWISE` splitting at `splitting_ops` (`vllm/config/compilation.py:60-63,517,615,630` @ `555967922`); construction from SGLang BCG `breakable_cuda_graph.py:204-241,244-260` @ `f63458b5be` (decorator + runtime stream capture, no compiler) | all-or-nothing today: `src/vllm/v1/worker/gpu/runner.cpp:1338-1341` routes only `pure_decode`; drivers `qwen3_5.h:275`, `qwen3_5_dense.h:391`, `qwen3_moe.h:117`, `qwen3.h:243`, `deepseek_v2.h:324`, `voxtral.h:126`, plus `deepseek_v4.cpp`, `laguna.cpp` — and the spike found the NINTH already written, `src/vllm/model_executor/models/qwen3_dflash.cpp:771,1091`. The re-derivation is measured, not asserted: `StepDevInputs` (`src/vllm/model_executor/models/qwen3_5.cpp:3894`, the persistent DEVICE input path) exists in ONE driver and `grep -c` returns 0 in `qwen3_moe.cpp`, `qwen3.cpp`, `deepseek_v2.cpp` and `voxtral.cpp`, which is why `src/vllm/model_executor/models/qwen3.cpp:961-986` DECLINES the graph outright when the async device-token mirror is live | owed: reachability mutation (delete the production call site, rerun the focused gate); bit-exactness vs eager on every migrated model over MORE than one replay; the host-lifetime contract of [decode-graph-scratch-uaf-2026-07-18.md](specs/decode-graph-scratch-uaf-2026-07-18.md) enforced AT the seam | spec [eng-cudagraph-break.md](specs/eng-cudagraph-break.md) (W0 spike DONE 2026-08-18: the existing `vt` capture vocabulary `include/vt/backend.h:208-222` expresses a SEGMENTED capture with NO new virtual, because `EndCaptureGraph` stores nothing (`src/vt/cuda/cuda_backend.cu:225-232`); a break point is expressible with one `thread_local` capture pointer plus a free function, no compiler and no decorator); analysis [sglang-breakable-cuda-graph.md](specs/sglang-breakable-cuda-graph.md) | `READY` | [#1163](https://github.com/mudler/vllm.cpp/issues/1163), [#1020](https://github.com/mudler/vllm.cpp/issues/1020) | +| `ENG-CUDAGRAPH-DEDUP` | Graph-executable dedup: hash each captured graph's topology and re-point ONE `cudaGraphExec` with `cudaGraphExecUpdate` on a signature hit, instead of instantiating one exec per padded bucket per model. A memory and capture-time change, NOT a throughput change — a deduped replay launches the same nodes | T2 | vLLM has no analogue (its execs come from `torch.compile`); SGLang `python/sglang/srt/model_executor/runner_backend/cuda_graph_dedup_mixin.py:219-242,358` @ `f63458b5be` | UNPORTED — `cudaGraphExecUpdate` appears nowhere in `src/` or `include/`; today `src/vt/cuda/cuda_backend.cu:222-232` instantiates a fresh exec per capture, over the 7 (`max_num_seqs=32`) or 11 (64) buckets of `include/vllm/model_executor/models/decode_graph_sizes.h:32-41`, times NINE drivers (count corrected 2026-08-18, [#1179](https://github.com/mudler/vllm.cpp/issues/1179); `9bc4d7f44` recorded eight, missing the DFlash draft graph `src/vllm/model_executor/models/qwen3_dflash.cpp:771,870,1038,1091,1095,1106`) | owed: same-binary A/B proving a deduped replay is byte-identical to a non-deduped one; a capture-count/exec-count log line so the ratio is observable | [sglang-breakable-cuda-graph.md](specs/sglang-breakable-cuda-graph.md) | `INVENTORIED` | [#1162](https://github.com/mudler/vllm.cpp/issues/1162) | +| `ENG-CUDAGRAPH-BREAK` | One shared `vt` capture seam that accepts BREAK POINTS, so a forward containing a host-dependent op is still graphed instead of falling out entirely — and so the NINE hand-rolled drivers become one (count corrected 2026-08-18, [#1179](https://github.com/mudler/vllm.cpp/issues/1179); `9bc4d7f44` recorded eight). **Coverage AND CORRECTNESS row, not a throughput row** | T1 | mirror vLLM `CUDAGraphMode.PIECEWISE` splitting at `splitting_ops` (`vllm/config/compilation.py:60-63,517,615,630` @ `555967922`); construction from SGLang BCG `python/sglang/srt/model_executor/runner_backend_utils/breakable_cuda_graph/breakable_cuda_graph.py:204-243,246-274,309-333,335-367` @ `f63458b5be` (decorator + runtime stream capture, no compiler); its unit suite `test/registered/cuda_graph/breakable/test_breakable_cuda_graph.py:30,172,230` (305 lines, 11 unit cases) is mapped case for case in the spec's `## Tests to port` | all-or-nothing today: `src/vllm/v1/worker/gpu/runner.cpp:1338-1341` routes only `pure_decode`; drivers `qwen3_5.h:275`, `qwen3_5_dense.h:391`, `qwen3_moe.h:117`, `qwen3.h:243`, `deepseek_v2.h:324`, `voxtral.h:126`, plus `deepseek_v4.cpp`, `laguna.cpp` — and the spike found the NINTH already written, `src/vllm/model_executor/models/qwen3_dflash.cpp:771,1091`. The re-derivation is measured, not asserted: `StepDevInputs` (`src/vllm/model_executor/models/qwen3_5.cpp:3894`, the persistent DEVICE input path) exists in ONE driver and `grep -c` returns 0 in `qwen3_moe.cpp`, `qwen3.cpp`, `deepseek_v2.cpp` and `voxtral.cpp`, which is why `src/vllm/model_executor/models/qwen3.cpp:961-986` DECLINES the graph outright when the async device-token mirror is live. **That decline is why this is also a CORRECTNESS row** ([#1179](https://github.com/mudler/vllm.cpp/issues/1179)): a SHIPPED model has already lost its decode graph to the duplication, on the driver's own measurement (`depth-1, graph ON PASS 78/78`; `depth-2, graph OFF PASS 82/82`; `depth-2, graph ON FAIL, slots 1-3 degenerate`), and the fix its comment names is the sibling's `StepDevInputs`. The row still makes NO throughput claim: the prefill refutation on the `ENG-CUDAGRAPH` row (3.8% host idle, >96% GPU-busy, 92.5% glue) stands unchanged | owed: reachability mutation (delete the production call site, rerun the focused gate); bit-exactness vs eager on every migrated model over MORE than one replay; the host-lifetime contract of [decode-graph-scratch-uaf-2026-07-18.md](specs/decode-graph-scratch-uaf-2026-07-18.md) enforced AT the seam; the 11 ported SGLang unit cases; the break-function OUTPUT writeback (`replay_fn`/`_copy_output` `breakable_cuda_graph.py:231-235,172-201`, spec D9) without which a replay leaves the next segment reading capture-time data; the auxiliary-stream auto-join before every segment close (`:353-361`, spec D10), live at `src/vllm/model_executor/models/qwen3_5.cpp:6254-6255,6384` and `src/vllm/model_executor/models/laguna.cpp:2572-2576,2612` | spec [eng-cudagraph-break.md](specs/eng-cudagraph-break.md) (W0 spike DONE 2026-08-18: the existing `vt` capture vocabulary `include/vt/backend.h:208-222` expresses a SEGMENTED capture with NO new virtual, because `EndCaptureGraph` stores nothing (`src/vt/cuda/cuda_backend.cu:225-232`); a break point is expressible with one `thread_local` capture pointer plus a free function, no compiler and no decorator); analysis [sglang-breakable-cuda-graph.md](specs/sglang-breakable-cuda-graph.md) | `READY` | [#1163](https://github.com/mudler/vllm.cpp/issues/1163), [#1020](https://github.com/mudler/vllm.cpp/issues/1020) | | `ENG-CUDAGRAPH-DIFFUSION` | Capture the LTX-2.5 denoise loop (fixed shapes, many identical iterations — the ideal graph target). **BLOCKED, and the blocker is ours:** the render does almost no device compute to capture | T2 | SGLang enabled BCG on this shape AFTER our pin — LTX-2 H200 two-stage 10.75s->6.90s (`d4be483efb`), SANA 1024px -26% (`6c7498113f`), SANA denoise 0.73->0.457s (`56ef810cad`). Dated events, NOT pinned evidence; their win is mostly PyTorch host tax we do not pay | NO capture at all: `grep` for capture across `src/vllm/model_executor/models/ltx2*.cpp` returns nothing | blocked by [#1024](https://github.com/mudler/vllm.cpp/issues/1024) (GPU util **exactly 0 in 321 of 347 samples**, 1.00 core of 20 held for 17+ min after staging), [#1007](https://github.com/mudler/vllm.cpp/issues/1007) (VAE decode has no device arm), [#1087](https://github.com/mudler/vllm.cpp/issues/1087) (**57-66% of wall** is ONE resolution-CONSTANT serial host phase), [#1010](https://github.com/mudler/vllm.cpp/issues/1010) (no phase-boundary log). Decision point is a MEASUREMENT of GPU-busy vs wall once device-resident, not an implementation | [sglang-breakable-cuda-graph.md](specs/sglang-breakable-cuda-graph.md) | `INVENTORIED` | [#1164](https://github.com/mudler/vllm.cpp/issues/1164) | | `ENG-BATCH-INVARIANT` | Opt-in deterministic execution across scheduler batch sizes (`VLLM_BATCH_INVARIANT=1`): batch-invariant matmul/norm/attention/collectives plus persistent-scheduler NVFP4; production default remains off | T1 | default/env `vllm/envs.py:89,576-578`; initialization `vllm/v1/worker/gpu_worker.py:1262`; NVFP4 dispatch `csrc/libtorch_stable/quantization/fp4/nvfp4_scaled_mm_sm120_kernels.cu:212-220`; suite fixture `tests/v1/determinism/conftest.py:9-12`; operator/e2e `tests/v1/determinism/test_nvfp4_batch_invariant_scaled_mm.py`, `tests/v1/determinism/test_nvfp4_batch_invariant.py` @ `702f481` | - | [W3-C3R executed contract](specs/nvfp4-persistent-plan-cache.md#w3-c3r-batch-shape-localization-and-gate-correction-2026-07-13): production-default ours and vLLM both change outputs across batch shapes; no local opt-in implementation is claimed | `planned: specs/batch-invariant-execution.md` | `INVENTORIED` | - | | `ENG-ASYNC-SCHED` | Async/overlap scheduling (AsyncScheduler placeholders + depth-2 batch-queue step + async D2H on a copy stream); vLLM's DEFAULT at the pin — mirror obligation per B3. **Host-side machinery + runner device-input half + sampler-OUTPUT half LANDED + CPU-gated (2026-07-16):** `AsyncScheduler` placeholder accounting, `step_with_batch_queue` depth-2, `ResolveAsyncScheduling` default-ON-when-compatible + `MaxConcurrentBatches`, `VT_ASYNC_SCHED` rollback; the runner device-input path `combine_sampled_and_draft_tokens`; PLUS the sampler-OUTPUT half — `vt::Backend` event/pinned primitives (`AllocPinned`/events, CUDA cudaHostAlloc+cudaEvent, CPU sync-degeneration), `AsyncGPUModelRunnerOutput` (device sampled-id snapshot → non-blocking D2H on a copy queue + event; `get_output()` waits only that event; MAIN queue never blocked), `Sampler::forward(sampled_ids_out)` device-resident greedy, `GPUModelRunner::sample_tokens_async` + `runner_supports_async`, and the `Executor`+`step_with_batch_queue` seam resolving `get_output()` at CONSUME time. All behind `VT_ASYNC_RUNNER`/`set_async_input_combine`, default OFF. Sync path byte-identical (placeholder sites INERT while count 0; combine off; `sample_tokens_async` degenerates to sync when async off; `sampled_ids_out=nullptr`). **ENABLE-FLIP LANDED + CPU-gated (2026-07-16):** (1) `LoadedEngine` now reorders `runner_` before the scheduler and builds an `AsyncScheduler` + `max_concurrent_batches=2` when `ResolveAsyncScheduling(runner_.runner_supports_async())` resolves ON (else the byte-identical synchronous `Scheduler` + depth-1); the resolved mcb threads into `AsyncLLM`→`EngineCoreProc` (`step_with_batch_queue`) and the "Asynchronous scheduling is enabled/disabled" log mirrors vLLM for A/B audit; (2) the device combine/scatter kernel (`_combine_sampled_and_draft_tokens_kernel` + last_sampled scatter) is ported to CUDA (`src/vt/cuda/cuda_combine_tokens.cu`), main-stream-ordered on the CUDA async path so it DELETES `sample_tokens_async`'s pre-scatter `Synchronize`; the CPU backend keeps the host loop. `VT_ASYNC_RUNNER=1` engages full W3; `VT_ASYNC_SCHED=0` is the same-binary rollback. Production default (no env) stays synchronous byte-identical. **FULL W3 DGX proof RAN twice** — `f086b64` (5/5 gates PASS; c16 TPOT −5.4 ms WIN, tput neutral, TTFT +36 % = Little's-law repayment) and the 2026-07-16 re-proof on the THROUGHPUT-lever fix (persistent pooled sampled-id/pinned buffers + `Sampler` greedy scratch removing ALL per-step `cudaMalloc`/`cudaFree`/`cudaHostAlloc`/event-create from the sampled-id path, incl. the overlap-killing `cudaFree` inside `get_output`; mirrors `gpu_model_runner.py:873-878` + `async_utils.py:12-70`): token-exactness **6/6 PASS**, interleaved c16 **tput −0.32 % (gate ≥+1.5 % FAILS), TPOT −4.95 ms retained, TTFT +34.8 %** — the allocator lever is REFUTED as the tput unlock (≤0.1 % of a ~165 ms c16 step). **DEFAULT FLIPPED ON 2026-07-17** (`VT_ASYNC_RUNNER` default ON via the pure `AsyncRunnerFlagIsOn` predicate, mirroring `vllm/config/vllm.py:992-1044`): the discriminator (`6ea7856`) proved vLLM's own async pays the identical +26–31 % TTFT / −0.7 to −0.9 % tput / −2.6 to −4.3 ms TPOT envelope and W3-ON nets positive (both binding ITL-tail anomalies flip to PASS), so the "needs a throughput lever" ship-gate is RETIRED — W3 is a parity/mirror obligation with a tails+TPOT win. The flip is TOKEN-NEUTRAL (async-ON ≡ async-OFF bit-identical on DGX). `VT_ASYNC_RUNNER=0` = runner-level rollback, `VT_ASYNC_SCHED=0` = scheduler-level rollback. TTFT means rise into vLLM's async envelope BY DESIGN — the next binding grid runs async by default and its TTFT must NOT be misread as a regression. **ROBUSTNESS FIX 2026-07-20 (`discard_request_mask`):** the runner was missing vLLM's `discard_request_mask`, so `GPUModelRunner` emitted a sampled token for prefill-CHUNK requests too; under async this drained a `num_output_placeholders` never reserved (the `is_prefill_chunk` path adds none) → the `async_scheduler.cpp` `num_output_placeholders >= 0` assertion aborted on c8 + short-output (chunked prefill + preemption). FIX mirrors vLLM: `execute_model` computes `exec_state_.discard[i] = seq_len < num_tokens` (`gpu_model_runner.py:2048`); `sample_tokens` clears those rows to empty (`outputs.py:303`), the async path passes `invalid_req_indices` to `AsyncGPUModelRunnerOutput::get_output` (`gpu_model_runner.py:3625` + `outputs.py:303`). Scheduler UNCHANGED (assertion kept — it was correct once the runner honors `scheduler.py:1888-1890`). Sync/non-chunked decode byte-identical (mask all-zero); DGX 27B 235/235 + 35B 315/315, `vllm-bench` c8+short-output+chunked+kv-pressure no longer crashes, memcheck 0. Ledger [parity-ledger.md](parity-ledger.md) 2026-07-20 row | T1 | `vllm/v1/core/sched/async_scheduler.py:12`; `vllm/config/vllm.py:490,990,1038`; `vllm/v1/engine/core.py:519`; `vllm/v1/worker/gpu/input_batch.py:304-406`; `vllm/v1/worker/gpu/async_utils.py:12-70`; `vllm/v1/worker/gpu/gpu_model_runner.py:242-332`; `vllm/v1/outputs.py:298-307` | `src/vllm/v1/core/sched/async_scheduler.cpp:10,45`; placeholder plumbing `src/vllm/v1/core/sched/scheduler.cpp:148,164,605`; `src/vllm/v1/engine/core.cpp:91` (`step_with_batch_queue`, async-output seam); `src/vllm/v1/engine/core_proc.cpp:32,46`; config `include/vllm/config/scheduler.h:117,165,188`, `src/vllm/config/scheduler.cpp:12`; `include/vllm/v1/request.h:187`; runner input leaf `src/vllm/v1/worker/gpu/prepare_inputs.cpp`, `src/vllm/v1/worker/gpu/input_batch.cpp`; runner output leaf `include/vt/backend.h`+`src/vt/backend.cpp`+`src/vt/cuda/cuda_backend.cu` (event/pinned), `include/vllm/v1/worker/gpu/async_output.{h,cpp}` (`AsyncGPUModelRunnerOutput`), `src/vllm/v1/sample/sampler.cpp` (`sampled_ids_out`), `src/vllm/v1/worker/gpu/runner.cpp` (`sample_tokens_async`/`runner_supports_async`), `src/vllm/v1/executor/executor.cpp`+`include/vllm/v1/worker/gpu/model_runner_base.h` (async seam); enable-flip `include/vllm/entrypoints/model_loader.h`+`src/vllm/entrypoints/model_loader.cpp` (`runner_` before scheduler, `ResolveAsyncEnabled`/`MakeScheduler`, `AsyncScheduler`+mcb=2, log), `include/vllm/v1/engine/async_llm.h`+`src/vllm/v1/engine/async_llm.cpp` (mcb param → `EngineCoreProc`); device kernel `include/vt/cuda/combine_tokens.h`+`src/vt/cuda/cuda_combine_tokens.cu`, wired `src/vllm/v1/worker/gpu/runner.cpp` (CUDA combine/scatter branch removes the pre-sync) | `tests/vllm/v1/test_async_scheduler.cpp:1` (6 cases, 54 asserts; RED vs base Scheduler 2/6 fail); depth-2 engine cycle `tests/vllm/v1/test_engine_core_proc.cpp:479` (mcb=2, async-output seam); config resolution `tests/vllm/test_scheduler_config.cpp:75`; enable-flip construction matrix `tests/vllm/entrypoints/test_loaded_engine_dense.cpp` (runner×VT_ASYNC_SCHED → scheduler type + mcb; RED = un-flipped engine, 3/3 ON-arm asserts fail); runner input leaf `test_combine_tokens.cpp` (RED = stale → 5/7 fail), `test_input_batch.cpp`, `test_runner.cpp` (async-ON≡sync); output leaf `tests/vt/test_backend.cpp` (event/pinned contract), `tests/vllm/v1/worker/test_async_output.cpp` (materialize/flush/snapshot; RED = +1 splice), `test_runner.cpp` (`sample_tokens_async` decode ≡ sync); full CPU ctest 111/111, tools 164/164. Prior diagnostic `3812d8` six-leg control: total **1.002153×**, TTFT **0.862159×**, no GPU-time reduction (neutral for speed). **DEFAULT-FLIP (2026-07-17):** new pure CPU flag test [test_async_runner_flag.cpp](../tests/vllm/v1/worker/test_async_runner_flag.cpp) (11 asserts, default-ON/'0'-off); construction matrix [test_loaded_engine_dense.cpp](../tests/vllm/entrypoints/test_loaded_engine_dense.cpp) INVERTED (default → AsyncScheduler+mcb=2; RED verified 5 asserts fail vs un-flipped). CPU clean `-Werror` rebuild, full serial ctest **116/116**, tools **164/164**. **DGX re-confirmation** (evidence `dgx:~/work/vllm.cpp-async-flip`, CUTLASS+FA2 hard-verified, one flock): shipping default (async ON + RMSNorm-fast OFF) → **27B 235/235 + 35B 315/315** with the "Asynchronous scheduling is enabled (mcb=2)" log, and both rollback arms (`VT_ASYNC_RUNNER=0`, `VT_ASYNC_SCHED=0`) 235/235 + 315/315 log "disabled"; async arms BIT-IDENTICAL (token-neutral). Closing record [parity-ledger.md#L502](parity-ledger.md#L502) | [async-serving.md](specs/async-serving.md) | `DONE` | `6ea7856` | diff --git a/.agents/issue-index.md b/.agents/issue-index.md index fb36c663a..5de4ddcbc 100644 --- a/.agents/issue-index.md +++ b/.agents/issue-index.md @@ -358,3 +358,4 @@ rather than merged. `scripts/check-agent-record.py` gates both. | [#1169](https://github.com/mudler/vllm.cpp/issues/1169) | — | Packed GDN decode is unreachable on every MoE checkpoint for a second, independent reason: `detail::ShouldUsePackedGdnDecode` (`src/vllm/model_executor/models/qwen3_5.cpp:76-84` @ `dd8a3b0e1`) requires `e.has_packed_ba`, populated at `:4410` as `!w.in_proj_ba.Empty()`, and `in_proj_ba` is written at exactly ONE site in the tree — `src/vllm/model_executor/models/qwen3_5_dense_weights.cpp:436`, the DENSE loader. The MoE loader loads the shards split (`qwen3_5_weights.cpp:563-564`) and so does the GGUF loader (`qwen3_5_gguf_weights.cpp:1067-1070`); `qwen3_5.cpp:3220` states it in the tree ("the only path that populates `in_proj_ba`"). Consequence: removing the `dense_model` term and narrowing `GdnOutDType` (#1168) is NOT sufficient to reach packed decode on a MoE model, which corrects the premise row `GDN-MOE-BF16-OUT` started from. It matters because the 35B's geometry (`Hv=32`, `Hg=16`, `Dk=Dv=128`) has a vendored Triton AOT decode cubin for every supported architecture (`src/vt/cuda/triton_aot_vendored/*/gdn_decode_h32.*`) that is exactly the FLA kernel vLLM runs, and `.agents/kernel-matrix.md` records the contrast the vendoring exists to capture — Triton REG:205 / 0 spill against the hand CUDA port at REG:255 + STACK:48. Upstream has one physical `in_proj_ba` on every Qwen3.5/3.6 GDN layer, dense and MoE alike (`qwen_gdn_linear_attn.py:843`; `packed_modules_mapping` on the shared `Qwen3_5ForCausalLMBase`, `vllm/model_executor/models/qwen3_5.py:287-297` @ `5559679`), so the merged owner is the upstream topology and our split MoE load is the deviation. Same class of gap on `in_proj_qkvz`, which no MoE or GGUF loader builds either. Closing it also owes a repair to the stale `KERNEL-GDN-PACKED-DECODE` sentence in `.agents/kernel-matrix.md` claiming "the launcher guard rejects its `Hv=32` shape anyway": `TryTritonPackedDecode` accepts `Hv=32` and dispatches `gdn_decode_h32_default` (`src/vt/cuda/cuda_gdn.cu:5207`, `:5239`). Filed, not fixed — it is a loader change with its own resident-weight lifetime and byte-exactness argument, and the finding row is spec-only. Listed under `## Owed` in [`gdn-moe-bf16-out.md`](specs/gdn-moe-bf16-out.md) | gap | | [#1170](https://github.com/mudler/vllm.cpp/issues/1170) | — | All four GDN Triton AOT fast paths reject any geometry whose linear V-head count is not 48 or 32 — `TryTritonPackedDecode` (`src/vt/cuda/cuda_gdn.cu:5207` @ `dd8a3b0e1`), `TryTritonDeltaH` (`:5264`), `TryTritonChunkO` (`:5298`) and `TryTritonWU` (`:5361`), each reading `if (hv_n != 48 && hv_n != 32) return false;` on top of `dk == 128 && dv == 128 && hk_n == 16`. Those two are the only vendored specializations (`src/vt/cuda/triton_aot_vendored/*/gdn_{decode,deltah,chunko,wu}_h{48,32}.*`): 48 is the dense 27B and 32 is `Qwen3.6-35B-A3B`. `Qwen/Qwen3.8-2.4T-A95B` has 128 linear V-heads ([`qwen38-text-only.md`](specs/qwen38-text-only.md)) and clears every other term, so it is rejected on `hv_n` alone and runs the hand CUDA kernels on all four legs — the ones `.agents/kernel-matrix.md` measured by cuobjdump at REG:255 + STACK:48 (spilling) against the vLLM FLA cubin's REG:205 / 0 spill, which is the whole reason the vendored cubins exist and are default-on. `Qwen3.8-27B` is NOT affected: it is the `Qwen3.6-27B` geometry retrained, 48 V-heads, and hits every AOT arm. Neither reference restricts the head count — SGLang's `TritonGDNKernel` sets `supports_packed_decode` from the platform alone and takes `num_v_heads` as a runtime argument (`python/sglang/srt/layers/attention/linear/kernels/gdn_triton.py:43` @ `f63458b5be`), and `VLLM_ENABLE_FLA_PACKED_RECURRENT_DECODE` has no shape term (`vllm/envs.py:124` @ `5559679`); both JIT-compile per shape, which is the property the AOT vendoring trades away for a Python-free runtime. Closing it needs `h128` specializations vendored across the supported architectures, or a stated rule for which head counts get an AOT arm plus a visible fallback cost at the call site. Filed, not fixed: either close needs the checkpoint that motivates it, and this hardware cannot run the 2.4T (~4.8 TB bf16 against 128 GB unified memory), so the fallback cannot be measured here today. Listed under `## Owed` in [`gdn-moe-bf16-out.md`](specs/gdn-moe-bf16-out.md) | perf | | [#1171](https://github.com/mudler/vllm.cpp/issues/1171) | `KERNEL-GDN-REPLAYSSM` | GDN decode rewrites the whole `[HV,V,K]` fp32 state every step (`src/vt/cuda/cuda_gdn.cu:2393` reads the tile, `:2425` writes it back), which at the 27B shape `HV=32, V=128, K=128` is 2,097,152 bytes read plus the same written per layer, per request, per token. ReplaySSM keeps a per-slot ring of the last `L` steps' rank-1 factors `(d, k, g)`, reconstructs the state in registers, and writes it back only every `L` steps. vLLM implements the algorithm at the pin `555967922` for Mamba2 selective-state ONLY (`layers/mamba/ops/selective_state_update_replayssm_output_only.py`, ring shapes/dtypes `mamba_utils.py:84-93,202-221`, `use_replayssm` default `False` at `config/cache.py:152`, introduced `866fea2b` #48018) and it cannot reach GDN: `config/vllm.py:2318-2322` refuses any model not setting `supports_replayssm`, only `NemotronHForCausalLM` sets it (`models/nemotron_h.py:711`), `GDNAttentionMetadataBuilder` does not subclass the Mamba builder that derives the cursor (`v1/attention/backends/gdn_attn.py:82` vs `mamba_attn.py:575-638`), and the kernel hard-requires a scalar-per-head `A` (`:540-542`) with the Mamba2 `(B,C)` group structure (`:529`). Still true 877 commits past the pin. SGLang ported it to GDN at our recorded pin `f63458b5` (`layers/attention/fla/fused_recurrent_linear_replayssm.py`, whose `:50` credits vLLM; `--enable-linear-replayssm` default `False` and `--linear-replayssm-cache-len` default 16 at `server_args.py:1972-1986`; rings `memory_pool.py:465-483`; commit `a10a24e9` #28451), so the algorithm is a vLLM mirror and the GDN application is a secondary-oracle port. PAYOFF UNMEASURED HERE and deliberately not claimed: ReplaySSM removes the state WRITE and not the read, the flush step reads the checkpoint a SECOND time, so the honest state ratio is `(1+2/L)/2` = 0.5625 at `L=16` against SGLang's published 0.53x which models neither the flush re-read nor any ring read; the ring itself adds 395,264 bytes per slot per layer at the 27B shape = **+18.9% KV page**, worse than vLLM's ~7% on Nemotron because GDN's state is `V*K` while the ring is `L*(V+K)`; and SGLang's own end-to-end figure is ~2.3% TPOT at 128 concurrency on an MoE model. Neither upstream is bit-exact against its unbuffered path and neither claims to be. Motivation is the open Qwen3.8-27B bf16 decode gap (c4 total 0.918x, output 0.963x, `docs/BENCHMARKS.md:192-205`). Spec [`gdn-replayssm.md`](specs/gdn-replayssm.md) | perf | +| [#1179](https://github.com/mudler/vllm.cpp/issues/1179) | `ENG-CUDAGRAPH-BREAK` | The hand-rolled decode-graph driver count recorded in `9bc4d7f44` is **eight** and is actually **nine**, and the row it feeds was framed as coverage-only when it is also correctness. The ninth is the DFlash draft graph, file-local with no header declaration, at `src/vllm/model_executor/models/qwen3_dflash.cpp:771,870,1038,1091,1095,1106` — its own `int g_state = 0` three-state machine (`:771`), its own `VT_DFLASH_GRAPH` kill switch (`:870`) instead of the `VLLM_CPP_CUDAGRAPH` the six batched drivers read, its own invalidate-on-block-width-change (`:1038-1047`) and its own `try { EndCaptureGraph(); } catch (...) {}` drain (`:1106`). The eight-count is stated in four places, all corrected here: [`sglang-breakable-cuda-graph.md`](specs/sglang-breakable-cuda-graph.md) §4 and `## Owed`, [`.agents/engine-matrix.md`](engine-matrix.md) rows `ENG-CUDAGRAPH-BREAK` and `ENG-CUDAGRAPH-DEDUP` ("times eight drivers", which sizes #1162's signature table), and [`.agents/roadmap_v1.md`](roadmap_v1.md) track `C12`. The reframing is the substantive half: `ENG-CUDAGRAPH-BREAK` was recorded as a COVERAGE row, and the duplication has already cost a SHIPPED model its decode graph. `src/vllm/model_executor/models/qwen3.cpp:961-986` declines the decode graph outright whenever the asynchronous device-token mirror is live, on its own measured battery — `depth-1, graph ON PASS 78/78`; `depth-2, graph OFF PASS 82/82`; `depth-2, graph ON FAIL, slots 1-3 degenerate` — because `Step()` replays against the HOST `input.token_ids` and the combine has patched the DEVICE ids. The comment names the real fix as reading the identifiers at replay time from a stable device buffer, and that fix exists, in exactly one sibling driver, as `StepDevInputs` (`src/vllm/model_executor/models/qwen3_5.cpp:3894`): `grep -c StepDevInputs` returns 41 lines there and 0 in each of `qwen3_moe.cpp`, `qwen3.cpp`, `deepseek_v2.cpp` and `voxtral.cpp`. One capability, written once, unavailable to four models, with a live mitigation standing in its place. This does NOT weaken the framing rule that `ENG-CUDAGRAPH` established: the row still makes no throughput claim, and the prefill refutation (GB10 3.8% host-idle between launches, GPU-busy >96%, 27B prefill gap 92.5% non-GEMM glue) stands unchanged. Coverage AND correctness, never speed. Fixed in flow with the [`eng-cudagraph-break.md`](specs/eng-cudagraph-break.md) review repair ([#1163](https://github.com/mudler/vllm.cpp/issues/1163)) | record | diff --git a/.agents/roadmap_v1.md b/.agents/roadmap_v1.md index 879db5a67..5efd25e6a 100644 --- a/.agents/roadmap_v1.md +++ b/.agents/roadmap_v1.md @@ -149,7 +149,7 @@ live in [feature-matrix.md](feature-matrix.md). | C9 | Recurring upstream sync cycle and P1 sync tooling | 🔁 recurring; [upstream-sync.md](upstream-sync.md) | | C10 | **Qwen3.5 high-throughput lever set** (user-directed 2026-08-08, from [vllm.ai 2026-08-06 "25K tok/s/GPU"](https://vllm.ai/blog/2026-08-06-qwen35-25k-tps)): mirror the three levers upstream credits. **(a) Blackwell GDN prefill** replacing the FLA/Triton kernel (FlashInfer PR #3001, vLLM PR #40717, `--gdn-prefill-backend flashinfer`): 1.02x-5.78x on the kernel across sizes, 5.92x microbenchmark, **1.13x e2e prefill throughput, 12% lower mean TTFT**. **(b) Hybrid cache + GDN-state transfer** (vLLM PR #36687 + #37416/#37635/#37310/#41869, `VLLM_SSM_CONV_STATE_LAYOUT=DS`): ~7% intra-node H100 from cutting transferred descriptors 4284 to 1650. **(c) Race-free async scheduling** (vLLM PRs #48481/#45357, `--async-scheduling`), credited as key but not isolated | ☐ SPIKE-first. **PIN MOVE REQUIRED, and it is the first deliverable:** every referenced vLLM PR post-dates our parity pin `555967922` (0.26.0.dev0), so none of this is mirrorable until the pin advances. The spike must (1) find the earliest upstream ref containing #40717 + #36687 + #48481, (2) prove that ref BUILDS AND RUNS a greedy golden on GB10 per [`oracle-gateability`](verification.md), (3) re-capture the SACRED goldens and show ZERO drift before any lever is ported. **Honest scoping:** upstream measured a GB200/NVLink72 DISAGGREGATED cluster (1xDEP8 decode, 4-8 prefill endpoints, 8192/1024, concurrency 64-5120) - one GB10 cannot reproduce it, so **25K is NOT adopted as our bar**. Only (a) is directly portable single-GPU; (c) overlaps the `VT_ASYNC_RUNNER` W3 work already default-OFF under C6; (b) is gated on the scale-out lane | | C11 | **Krea 2 image generation** (user-directed 2026-08-08, [krea-ai/krea-2](https://github.com/krea-ai/krea-2)): text to image diffusion transformer. Two community-licensed checkpoints, `krea/krea-2-raw` (base) and `krea/krea-2-turbo` (8-step distilled, up to 2k). Reference impl is PyTorch in-repo (`mmdit.py` MultiModalDiT + `autoencoder.py` + `sampling.py`) | ☐ SPIKE-first: arch inventory + reuse map against the MiniMax-H3 DiT lane, which already owns a joint video+audio diffusion transformer, an NVFP4/GGUF DiT weight-streaming loader and the `vllm_video_*` ABI slice - a text-to-image DiT is the SIMPLER sibling and should reuse the sampler / VAE-decode / DiT-forward seams rather than start new. **No pin move needed** (nothing upstream to track). **Upstream-mirror caveat:** vLLM has no Krea 2 path, so this is a from-scratch port owing a `porting-inventory.md` section 9 entry with the krea-2 repo as reference root and its PyTorch reference as the numerical oracle | -| C12 | **SGLang Breakable CUDA Graph (BCG), evaluated and mostly REFUTED** (user-directed 2026-08-17, from a public SGLang claim: *"first engine to develop and land BCG, the full CUDA Graph, and graph memory reuse"*). Grounded in the pinned SGLang tree `v0.5.15`/`f63458b5be` rather than in the claim text: BCG captures one forward as a SEQUENCE of `torch.cuda.CUDAGraph` segments split at eager break points on the attention and recurrent calls, with no `torch.compile` (`runner_backend_utils/breakable_cuda_graph/breakable_cuda_graph.py:204-241,244-260`; `runner_backend/breakable_cuda_graph_backend.py:14-17`). **The priority claim is narrower than it reads:** vLLM's v1 default is already `FULL_AND_PIECEWISE`, splitting at `splitting_ops` (`vllm/config/compilation.py:60-63,615,630` @ `555967922`), so the CAPABILITY is in our denominator; what SGLang landed first is the CONSTRUCTION — piecewise coverage with no Dynamo/Inductor/FX. | ☐ T1 (2026-08-17, [#1161](https://github.com/mudler/vllm.cpp/issues/1161)); spec [sglang-breakable-cuda-graph.md](specs/sglang-breakable-cuda-graph.md); row `SGLANG-BCG` in [sglang-matrix.md](sglang-matrix.md). **Four outcomes, three of them not throughput.** (a) *Dropping `torch.compile`* is NO lever here — we never had a compiler; `src/vt/cuda/cuda_backend.cu:203-232` is already raw stream capture. (b) *Prefill capture* is **REFUTED** on our CUDA gate models and recorded against `ENG-CUDAGRAPH` so nobody re-derives it: GB10 2026-07-09 measured prefill GPU-idle-between-launches at **3.8%** with GPU-busy >96% on both arms, and the 27B prefill gap at **92.5% non-GEMM glue GPU work** (dominant GEMM +0.17%, attention AHEAD). No launch bubbles to collapse. (c) *Graph memory reuse* is **real and unported** — `cudaGraphExecUpdate` appears nowhere in `src/` or `include/`, while we instantiate one exec per padded bucket across eight hand-rolled drivers → `ENG-CUDAGRAPH-DEDUP` `INVENTORIED` ([#1162](https://github.com/mudler/vllm.cpp/issues/1162)). (d) The analysis surfaced a STRUCTURAL row the claim did not: our capture is all-or-nothing (`runner.cpp:1338-1341` routes only `pure_decode`) and hand-rolled eight times → `ENG-CUDAGRAPH-BREAK` `INVENTORIED` ([#1163](https://github.com/mudler/vllm.cpp/issues/1163)), a COVERAGE row, not a throughput row. Diffusion looked like the one real lever (SGLang measured LTX-2 H200 10.75s→6.90s post-pin) but is **BLOCKED by our own defects**: an LTX-2.5 render does almost no device compute to capture — GPU util **exactly 0 in 321 of 347 samples** ([#1024](https://github.com/mudler/vllm.cpp/issues/1024)), no device arm for the VAE decode ([#1007](https://github.com/mudler/vllm.cpp/issues/1007)), and **57-66% of wall** in one resolution-CONSTANT serial host phase ([#1087](https://github.com/mudler/vllm.cpp/issues/1087)) → `ENG-CUDAGRAPH-DIFFUSION` `INVENTORIED` ([#1164](https://github.com/mudler/vllm.cpp/issues/1164)) | +| C12 | **SGLang Breakable CUDA Graph (BCG), evaluated and mostly REFUTED** (user-directed 2026-08-17, from a public SGLang claim: *"first engine to develop and land BCG, the full CUDA Graph, and graph memory reuse"*). Grounded in the pinned SGLang tree `v0.5.15`/`f63458b5be` rather than in the claim text: BCG captures one forward as a SEQUENCE of `torch.cuda.CUDAGraph` segments split at eager break points on the attention and recurrent calls, with no `torch.compile` (`runner_backend_utils/breakable_cuda_graph/breakable_cuda_graph.py:204-241,244-260`; `runner_backend/breakable_cuda_graph_backend.py:14-17`). **The priority claim is narrower than it reads:** vLLM's v1 default is already `FULL_AND_PIECEWISE`, splitting at `splitting_ops` (`vllm/config/compilation.py:60-63,615,630` @ `555967922`), so the CAPABILITY is in our denominator; what SGLang landed first is the CONSTRUCTION — piecewise coverage with no Dynamo/Inductor/FX. | ☐ T1 (2026-08-17, [#1161](https://github.com/mudler/vllm.cpp/issues/1161)); spec [sglang-breakable-cuda-graph.md](specs/sglang-breakable-cuda-graph.md); row `SGLANG-BCG` in [sglang-matrix.md](sglang-matrix.md). **Four outcomes, three of them not throughput.** (a) *Dropping `torch.compile`* is NO lever here — we never had a compiler; `src/vt/cuda/cuda_backend.cu:203-232` is already raw stream capture. (b) *Prefill capture* is **REFUTED** on our CUDA gate models and recorded against `ENG-CUDAGRAPH` so nobody re-derives it: GB10 2026-07-09 measured prefill GPU-idle-between-launches at **3.8%** with GPU-busy >96% on both arms, and the 27B prefill gap at **92.5% non-GEMM glue GPU work** (dominant GEMM +0.17%, attention AHEAD). No launch bubbles to collapse. (c) *Graph memory reuse* is **real and unported** — `cudaGraphExecUpdate` appears nowhere in `src/` or `include/`, while we instantiate one exec per padded bucket across NINE hand-rolled drivers → `ENG-CUDAGRAPH-DEDUP` `INVENTORIED` ([#1162](https://github.com/mudler/vllm.cpp/issues/1162)). (d) The analysis surfaced a STRUCTURAL row the claim did not: our capture is all-or-nothing (`runner.cpp:1338-1341` routes only `pure_decode`) and hand-rolled **nine** times (count corrected 2026-08-18, [#1179](https://github.com/mudler/vllm.cpp/issues/1179): the DFlash draft graph `qwen3_dflash.cpp:771,870,1038,1091,1095,1106` was missed) → `ENG-CUDAGRAPH-BREAK` `READY` ([#1163](https://github.com/mudler/vllm.cpp/issues/1163)), a COVERAGE **and CORRECTNESS** row, not a throughput row — `qwen3.cpp:961-986` already DECLINES its decode graph while the async device-token mirror is live (`depth-2, graph ON FAIL, slots 1-3 degenerate`) because `StepDevInputs` (`qwen3_5.cpp:3894`) exists in one driver only. Diffusion looked like the one real lever (SGLang measured LTX-2 H200 10.75s→6.90s post-pin) but is **BLOCKED by our own defects**: an LTX-2.5 render does almost no device compute to capture — GPU util **exactly 0 in 321 of 347 samples** ([#1024](https://github.com/mudler/vllm.cpp/issues/1024)), no device arm for the VAE decode ([#1007](https://github.com/mudler/vllm.cpp/issues/1007)), and **57-66% of wall** in one resolution-CONSTANT serial host phase ([#1087](https://github.com/mudler/vllm.cpp/issues/1087)) → `ENG-CUDAGRAPH-DIFFUSION` `INVENTORIED` ([#1164](https://github.com/mudler/vllm.cpp/issues/1164)) | ## D. T2 (after T1, per porting-inventory.md) diff --git a/.agents/specs/eng-cudagraph-break.md b/.agents/specs/eng-cudagraph-break.md index 1f84b2ef4..747fd8ea8 100644 --- a/.agents/specs/eng-cudagraph-break.md +++ b/.agents/specs/eng-cudagraph-break.md @@ -6,10 +6,22 @@ **Parent analysis:** [sglang-breakable-cuda-graph.md](sglang-breakable-cuda-graph.md) `## Owed`. **Kind:** scoping spike plus structured spec. No production code lands with this spec. -**This is a coverage row, not a throughput row.** The spike claims no speed and -measures none. `## Gates` below states what would have to be named and measured -before any speed claim is admissible, and `## Risks/decisions` D5 records why no -such path is named today. +**This is a coverage row and a CORRECTNESS row, and it is not a throughput row.** +The spike claims no speed and measures none. `## Gates` below states what would have +to be named and measured before any speed claim is admissible, and +`## Risks/decisions` D5 records why no such path is named today. + +The correctness half is not a framing device, it is already shipped damage +([#1179](https://github.com/mudler/vllm.cpp/issues/1179)). `src/vllm/model_executor/models/qwen3.cpp:961-986` +DECLINES its decode graph outright whenever the asynchronous device-token mirror is +live, and its comment records the measurement that forced it: `depth-1, graph ON +PASS 78/78`, `depth-2, graph OFF PASS 82/82`, `depth-2, graph ON FAIL, slots 1-3 +degenerate`. The comment names the real fix — read the identifiers at REPLAY time +from a stable device buffer — and that fix already exists, in the sibling driver, +as `StepDevInputs` (`qwen3_5.cpp:3894`). One driver has the capability and a shipped +model lost its decode graph because it does not. That is duplication producing a +correctness regression, not a tidiness complaint, and it is the second reason this +row exists. ## Scope @@ -73,17 +85,25 @@ spec ports. Verified in the pinned tree at `/home/mudler/_git/sglang` (`git rev-parse HEAD` = `f63458b5beaceabbd9d749b9fc956370e1b649e6`): -| Mechanism | Anchor in `python/sglang/srt/` | +Every bare `:N` anchor below is a line in the one file that implements BCG, +spelled here once in full: +`python/sglang/srt/model_executor/runner_backend_utils/breakable_cuda_graph/breakable_cuda_graph.py`. +That file is 374 lines at the pin (`wc -l` = 374), so no anchor past `:374` exists. + +| Mechanism | Anchor | |---|---| -| Break-point wrapper | `.../breakable_cuda_graph/breakable_cuda_graph.py:204-241` (`eager_on_graph`) | -| Active-capture context | `:60` (`_current_capture_var`, a `ContextVar`) | -| Segment container | `:246-253` (`BreakableCUDAGraph`, `_segments` plus `_break_fns`) | -| Interleaved replay | `:255-263` | -| Capture scope enter and exit | `:307-331` (`__enter__`, `__exit__`) | -| Segment open and close | `:335-350`, `:352-365` | -| Bare break marker | `:370-375` (`break_graph`, an empty body under the decorator) | +| Break-point wrapper | `:204-243` (`eager_on_graph`), whose `wrapper` body is `:209-241` | +| Output writeback on replay | `:231-235` (`replay_fn`), `:172-201` (`_copy_output`) | +| Active-capture context | `:63` (`_current_capture_var`, a `ContextVar`) | +| Segment container | `:246-274` (`BreakableCUDAGraph`); fields at `:251-252`; `_append_segment` at `:266-274` | +| Interleaved replay | `:255-264` | +| Capture scope enter and exit | `:309-320` (`__enter__`), `:322-333` (`__exit__`) | +| Segment open and close | `:335-350` (`_begin_new_segment`), `:352-367` (`_end_current_segment`) | +| Auxiliary-stream fork tracking | `:101-153` (the `wait_stream` hook), installed at `:310` and removed at `:332` | +| Auxiliary-stream auto-join before a segment ends | `:353-361` | +| Bare break marker | `:370-374` (`break_graph`, an empty body under the decorator) | | Shared mempool and weak refs | `:14-23` doc, `:156-169` (`_weak_ref_if_tensor`) | -| No-compiler backend | `.../runner_backend/breakable_cuda_graph_backend.py:14-17` | +| No-compiler backend | `python/sglang/srt/model_executor/runner_backend/breakable_cuda_graph_backend.py:14-17` | The break points SGLang registers, all through the same one-line wrapper: @@ -121,7 +141,9 @@ ends the stream capture, instantiates the graph, and returns the executable as a opaque handle the caller owns. That property is what makes a segmented capture expressible with **no new backend virtual**: a segment is one `BeginCapture` and `EndCaptureGraph` pair, and N break points produce N+1 such pairs on the same queue. -The spike confirmed this shape end to end; see `## Work breakdown` W0. +The spike confirmed this SHAPE in C++, on a recording backend with no GPU +attached; whether CUDA permits the re-begin mid-forward is W1's exit criterion, not +a W0 result. See `## Work breakdown` W0. ### Defect 1: capture is all or nothing @@ -155,7 +177,7 @@ them re-derives the same machinery. | Driver | Declaration | Implementation | Shape | |---|---|---|---| | `Qwen3_5DecodeGraph` | `include/vllm/model_executor/models/qwen3_5.h:275` | `src/vllm/model_executor/models/qwen3_5.cpp:9540,9563,9712,9893,9918,9922` | batched, padded ring | -| `Qwen3_5DenseDecodeGraph` | `include/vllm/model_executor/models/qwen3_5_dense.h:391` | `src/vllm/model_executor/models/qwen3_5.cpp:9974,9986,10321,10340,10348` | batched, padded ring | +| `Qwen3_5DenseDecodeGraph` | `include/vllm/model_executor/models/qwen3_5_dense.h:391` | `src/vllm/model_executor/models/qwen3_5.cpp:9969,9986,10321,10340,10348` | batched, padded ring | | `Qwen3MoeDecodeGraph` | `include/vllm/model_executor/models/qwen3_moe.h:117` | `src/vllm/model_executor/models/qwen3_moe.cpp:385,404,469,512,515,524` | batched, padded ring | | `Qwen3DenseDecodeGraph` | `include/vllm/model_executor/models/qwen3.h:243` | `src/vllm/model_executor/models/qwen3.cpp:583,613,682,866,888,897` | batched, padded ring | | `DeepseekV2DecodeGraph` | `include/vllm/model_executor/models/deepseek_v2.h:324` | `src/vllm/model_executor/models/deepseek_v2.cpp:887,902,963,980` | batched, padded ring | @@ -167,28 +189,39 @@ them re-derives the same machinery. **What every driver re-derives.** Each item below was read in at least two drivers and the anchors give one instance each. -1. The environment kill switch. Five drivers read `VLLM_CPP_CUDAGRAPH` themselves: - `qwen3_5.cpp:9540`, `qwen3_5.cpp:9974`, `qwen3_moe.cpp:385`, `qwen3.cpp:583`, - `deepseek_v2.cpp:887`, `voxtral.cpp:448`. The three single-shape drivers each - invented their own instead: `VT_V4_DECODE_GRAPH` at `deepseek_v4.cpp:1680`, - `VT_DFLASH_GRAPH` at `qwen3_dflash.cpp:870`, and the Laguna cluster at - `laguna.cpp:104,116,140`. There is no one switch that turns capture off. +1. The environment kill switch. SIX drivers read `VLLM_CPP_CUDAGRAPH` themselves: + `qwen3_5.cpp:9540`, `qwen3_5.cpp:9969`, `qwen3_moe.cpp:385`, `qwen3.cpp:583`, + `deepseek_v2.cpp:887`, `voxtral.cpp:448`. Instrument: + `grep -rn 'std::getenv("VLLM_CPP_CUDAGRAPH")' src/` returns exactly those six. + (`grep -rn VLLM_CPP_CUDAGRAPH src/` returns seven, because `qwen3.cpp:947` + mentions the name in a comment and is not a read site.) The three single-shape + drivers each invented their own switch instead: `VT_V4_DECODE_GRAPH` at + `deepseek_v4.cpp:1680`, `VT_DFLASH_GRAPH` at `qwen3_dflash.cpp:870`, and + `VT_LAGUNA_DECODE_GRAPH` at `laguna.cpp:1684`. There is no one switch that + turns capture off. 2. The cold, warm, captured state machine. `qwen3_5.cpp:9563` carries `warm` and `captured` flags on the slot; `deepseek_v4.cpp:1719` spells the same thing as `int gstate = 0`; `qwen3_dflash.cpp:771` spells it as `int g_state = 0`. Three spellings of one three-state machine. -3. The pad-to-bucket selection. Five call sites of `PadToCaptureSize`: - `qwen3_5.cpp:9712`, `qwen3_moe.cpp:469`, `qwen3.cpp:682`, `deepseek_v2.cpp:963`, - `voxtral.cpp:535`. The function itself is already shared, at - `include/vllm/model_executor/models/decode_graph_sizes.h:47-56`, which is the - proof that extraction works and is the pattern this row extends. +3. The pad-to-bucket selection. SIX call sites of `PadToCaptureSize`: + `qwen3_5.cpp:9712`, `qwen3_5.cpp:10140`, `qwen3_moe.cpp:469`, `qwen3.cpp:682`, + `deepseek_v2.cpp:963`, `voxtral.cpp:535`. Instrument: `grep -rn PadToCaptureSize + src/` returns 16 lines, of which four are `#include` comments and six are prose + comments; the six above are the invocations. The function itself is already + shared, at `include/vllm/model_executor/models/decode_graph_sizes.h:47-54`, which + is the proof that extraction works and is the pattern this row extends. 4. The per-slot persistent buffer set. `qwen3_5.cpp:9563-9600` holds `token_ids`, `positions`, the two metadata structs, `hidden`, `logits`, `aux`, `graph`, `fa_cols`, `captured`, `warm`, `replays`, `reuse_event`, `dev` and `pin`. Every other batched driver holds a subset of the same fields under the same name `SizeSlot`. -5. The diagnostics pair `captured()` and `replay_count()`, at `qwen3_5.cpp:9680`, - `qwen3_moe.cpp:454`, `qwen3.cpp:667`, `deepseek_v2.cpp:948`, `voxtral.cpp:519`. +5. The diagnostics pair `captured()` and `replay_count()`, SIX times. The + `replay_count()` definitions are at `qwen3_5.cpp:9680`, `qwen3_5.cpp:10110`, + `qwen3_moe.cpp:454`, `qwen3.cpp:667`, `deepseek_v2.cpp:948`, `voxtral.cpp:519`, + each preceded one line above by its `captured()` sibling + (`qwen3_5.cpp:9679`, `qwen3_5.cpp:10109`, `qwen3_moe.cpp:453`, `qwen3.cpp:666`, + `deepseek_v2.cpp:947`, `voxtral.cpp:518`). Instrument: + `grep -rn '::captured() const' src/vllm/model_executor/models/*.cpp` returns six. 6. The capture-failure recovery. `qwen3_5.cpp:9913`, `qwen3_5.cpp:10335` and `qwen3_dflash.cpp:1106` each carry their own copy of the same `try { g = EndCaptureGraph(); } catch (...) {}` drain, because a skipped @@ -199,9 +232,12 @@ and the anchors give one instance each. **The divergence that proves the cost.** The persistent DEVICE input path, called "Option A" in the tree, exists in exactly one driver. `StepDevInputs` -(`qwen3_5.cpp:3894`) is referenced 33 times in `qwen3_5.cpp` and **zero** times in -`qwen3_moe.cpp`, `qwen3.cpp`, `deepseek_v2.cpp` and `voxtral.cpp` (`grep -c`, all -four returned `0`). The other four replay against host vectors. One of them, +(`qwen3_5.cpp:3894`) appears on **41** lines of `qwen3_5.cpp` and **zero** lines of +`qwen3_moe.cpp`, `qwen3.cpp`, `deepseek_v2.cpp` and `voxtral.cpp`. Instrument: +`grep -c StepDevInputs `, which counts MATCHING LINES, not occurrences; the +occurrence count on the same file is 47 (`grep -o StepDevInputs qwen3_5.cpp | wc -l`). +An earlier draft of this spec recorded 33, which is neither number and is withdrawn. +The four zeros reproduce under both instruments and the conclusion is unchanged. The other four replay against host vectors. One of them, `qwen3.cpp:961-986`, therefore has to DECLINE the graph outright whenever the asynchronous device-token mirror is live, and its comment records the measured failure that forced it: "depth-2, graph ON FAIL, slots 1-3 degenerate". The same @@ -243,7 +279,8 @@ Three types, all under `vt`, all backend-agnostic, none of them a new virtual on 1. **`vt::BreakableGraph`** — the container. Holds `std::vector segments` (each an opaque handle from `Backend::EndCaptureGraph`) and - `std::vector> break_fns`. Invariant, asserted: + `std::vector> break_fns` (each a seam-built replay + closure, see item 3, never a caller's raw `fn`). Invariant, asserted: `segments.size() == break_fns.size() + 1`. `Replay(Queue&)` walks `ReplayGraph(segments[i])` then `break_fns[i]()` for each `i`, mirroring `breakable_cuda_graph.py:255-263`. The destructor calls `DestroyGraph` on every @@ -253,18 +290,83 @@ Three types, all under `vt`, all backend-agnostic, none of them a new virtual on scope. Its constructor sets one `thread_local GraphCaptureScope*` and calls `Backend::BeginCapture`; its destructor calls `Backend::EndCaptureGraph`, pushes the final segment, and clears the pointer. This is `BreakableCUDAGraphCapture` - at `:277-331`, with the destructor doing what `__exit__` does at `:320-331`. The + at `:277-367`, with the destructor doing what `__exit__` does at `:322-333`. The thread-local replaces SGLang's `ContextVar` at `:60`. It must be `thread_local`, not a global, for the same reason SGLang chose a `ContextVar` and CUDA chose `cudaStreamCaptureModeThreadLocal` (`cuda_backend.cu:204-206`): capture is a property of one thread's stream, and a process-wide flag would make an unrelated thread's forward observe a capture it is not part of. -3. **`vt::GraphBreak(fn)`** — the break point. Outside a capture scope it calls - `fn` and returns, so a non-capturing forward is byte-identical to today. Inside - a scope it closes the current segment, runs `fn` once eagerly so the outputs - hold real data, appends `fn` to `break_fns`, and opens a new segment. This is - `eager_on_graph` at `:204-241`, minus the decorator syntax we cannot write. +3. **`vt::GraphBreak(fn, out)`** — the break point. Outside a capture scope it + calls `fn` and returns, so a non-capturing forward is byte-identical to today. + Inside a scope it closes the current segment, runs `fn` once eagerly so the + outputs hold real data, appends a REPLAY CLOSURE to `break_fns`, and opens a + new segment. This is `eager_on_graph` at `:204-243`. + + **The appended function is not `fn`.** Upstream builds a `replay_fn` closure at + `:231-235` that calls the captured inner function and then writes its result + back into the capture-time destination: + + ```python + def replay_fn(): + new_out = captured_inner(*captured_args, **captured_kwargs) + return _copy_output(captured_output, new_out) + ``` + + `_copy_output` (`:172-201`) does an in-place `dst.copy_(src)` for a tensor, and + recurses field-wise over an object's `__dict__` or a dict's values; it returns + `src` only when neither side is copyable. The companion `_weak_ref_if_tensor` + (`:156-169`) is what makes the capture-time destination outlive the eager call. + + **This is load-bearing, not decorator sugar.** On replay N the eager operation + returns a FRESH allocation, whose address is not the one segment `i+1` baked at + capture time. A container that replays the raw `fn` and discards its return value + drops that guarantee silently: the next segment keeps reading the capture-time + address while the break function writes somewhere else. The failure is WRONG + NUMERICS, not a fault, which is exactly the class `## Risks/decisions` D1 records + `compute-sanitizer` cannot see. + + **Contract this seam adopts, stated at the `vt::GraphBreak` declaration.** One of + the two, and the seam must make the choice explicit rather than leave it to the + caller: + + - `GraphBreak(fn, out)` takes the destination the following segment reads, and + the seam copies `fn`'s result into `out` on every replay. This is the direct + port of `replay_fn` plus `_copy_output`. + - `GraphBreak(fn)` with no destination requires `fn` to write IN PLACE into a + persistent buffer that the model owns and that no replay reallocates. The + no-destination form is then only legal for a break function with no return + value; the bare marker below is its degenerate case. + + `vt::BreakableGraph` therefore holds `std::vector>` whose + elements are the SEAM's closures, never the caller's `fn` — and D9 records the + hazard the distinction removes. + +**The fork-join rule the scope owns.** Closing a segment is illegal while a side +stream forked inside it is still participating in the capture. Upstream solves this +inside `_end_current_segment` (`:353-361`): before `capture_end()` it walks the set +of streams forked but not rejoined and issues the join itself. Tracking that set is +the entire purpose of the `torch.cuda.Stream.wait_stream` hook (`:101-153`), which +`__enter__` installs at `:310` and `__exit__` removes at `:332`. + +**This is live for us, in both drivers W4 and W5 migrate.** Our CUDA backend reports +`SupportsAuxStream() == true` (`src/vt/cuda/cuda_backend.cu:200`, against the base +`false` at `include/vt/backend.h:166`), and two model paths fork on it. Qwen3.5's MoE +shared-expert overlap records a fork event on the main queue and makes the auxiliary +queue wait it at `src/vllm/model_executor/models/qwen3_5.cpp:6254-6255`, records the +completion event at `:6261`, and joins the main queue back at `:6384` — so any break +point placed between `:6255` and `:6384` closes a segment with the fork outstanding. +Laguna's decode-graph class owns its own auxiliary queue and event pair +(`src/vllm/model_executor/models/laguna.cpp:2362-2367`, torn down at `:2374-2378`), +forks at `:2572-2573`, records at `:2576` and joins at `:2612` — and unlike Qwen3.5's +that fork is INSIDE the captured region by construction, because the class is the +capture driver. + +**Rule: `GraphCaptureScope` owns the set of queues forked since the current segment +opened, and joins every outstanding one before it calls `EndCaptureGraph`.** We need +no `wait_stream` monkey-patch to populate that set, because our fork and join are +explicit `Backend::RecordEvent` and `Backend::QueueWaitEvent` calls on a seam we +control, not an implicit torch API. D10 records what happens if the rule is omitted. **Why a thread-local and not a parameter.** The break site sits inside an attention call, several frames below the forward entry point. Threading a capture context @@ -273,8 +375,9 @@ call for the same reason. The spike proved the thread-local carries the context across three frames with no intermediate signature change; see `## Work breakdown` W0. **A bare marker.** `vt::GraphBreak()` with no argument splits the segment and runs -nothing, mirroring `break_graph` at `breakable_cuda_graph.py:370-375`. It is the +nothing, mirroring `break_graph` at `breakable_cuda_graph.py:370-374`. It is the form a model uses when the host-dependent work already ran outside the forward. +It has no output, so the writeback contract above is vacuous for it. ### How the boundary is registered @@ -322,10 +425,67 @@ its break-point set and its output buffers. ## Tests to port vLLM's own tests do not transfer. Its piecewise split is verified through -`torch.compile` configuration tests, and we have no compiler to configure. SGLang's -BCG carries no unit test at the pinned revision that exercises the segment -container in isolation. So the primary tests here are written against OUR seam, and -the anchors above are the behavioral specification they encode. +`torch.compile` configuration tests, and we have no compiler to configure. + +**SGLang's BCG DOES carry a unit suite at the pin, and it ports.** An earlier draft +of this spec asserted there was nothing to port, which was wrong and is withdrawn. +The file is +`test/registered/cuda_graph/breakable/test_breakable_cuda_graph.py` at +`f63458b5be`, 305 lines, four classes. AGENTS.md requires the upstream tests in the +same change that ports the behavior, so the table below maps every upstream case to +the local test that owes it. Parameters, modes and failure cases are preserved; the +harness adaptation is stated once at the bottom. + +**`TestBreakableCUDAGraphBasic` (`:30`) — the capture and replay mechanism.** + +| Upstream case | Anchor | What it pins | Local test | +|---|---|---|---| +| `test_no_break_capture_replay` | `:49-63` | zero breaks captures and replays exactly like a plain graph: `y = x + 1`, refill `x` to 5.0, replay, expect 6.0 | T1: a scope with no `GraphBreak` yields 1 segment, 0 break functions, and one `Begin`/`EndCaptureGraph` pair | +| `test_single_break` | `:65-87` | one break splits into two segments and the chain composes: `x=10 -> +1=11 -> eager *2=22 -> +3=25` | T2: 1 break yields 2 segments and 1 break function, and the arithmetic chain holds | +| `test_multiple_breaks` | `:89-115` | two breaks, three segments, chained: `x=5 -> +1=6 -> +1=7 -> +1=8 -> *2=16` | T3: N breaks yield N+1 segments and N break functions, asserted at N=2 with the same values | +| `test_eager_on_graph_disabled` | `:117-129` | with the wrapper DISABLED the function is returned unchanged and runs normally | T4: `VLLM_CPP_CUDAGRAPH=0` (item 1 of `## Our baseline`) makes `GraphBreak(fn)` call `fn` and make zero backend calls | +| `test_eager_on_graph_outside_capture` | `:131-142` | outside any capture the wrapper is a pass-through | T5: with no scope active, `GraphBreak(fn)` runs `fn` and makes zero backend calls | +| `test_replay_updates_output` | `:144-169` | TWO replays with different inputs give different outputs: `3.0` then, after `x.fill_(10)`, `33.0` | T6: the multi-replay case. This is the upstream anchor for G1's "more than one replay" requirement, and the one case that can see a break function writing to a stale address | + +**`TestCopyOutput` (`:172`) — the output-writeback contract of `## Port map` §3.** +This class is why F2's writeback is not optional: upstream tests `_copy_output` +separately from the capture machinery, because it is a separate guarantee. + +| Upstream case | Anchor | What it pins | Local test | +|---|---|---|---| +| `test_tensor_copy` | `:187-192` | a tensor destination is written IN PLACE and the SAME object is returned (`assertIs(result, dst)`) | T7: `GraphBreak(fn, out)` copies into `out`; the device pointer `out` held before the call is the one holding the data after it | +| `test_dict_copy` | `:194-208` | a keyed set of destinations is copied field by field | T8: a multi-output break (logits plus an auxiliary tap, the `qwen3_5.cpp:9570-9577` shape) writes every destination | +| `test_object_copy` | `:210-223` | a struct destination copies its tensor fields in place and ASSIGNS its non-tensor fields (`dst.label == "new"`) | T9: a struct-valued break preserves the in-place/assign split | +| `test_non_tensor_fallback` | `:225-227` | with nothing copyable, `_copy_output` returns `src`, the documented fallback | T10: a break function with no device output is legal and the seam does not fabricate a copy | + +**`TestBreakGraphHelper` (`:230`).** + +| Upstream case | Anchor | What it pins | Local test | +|---|---|---|---| +| `test_break_graph_inserts_segment` | `:249-265` | the BARE marker splits the segment even though its body does nothing: `x=10 -> +1=11 -> break -> +2=13` | T11: `vt::GraphBreak()` with no argument yields 2 segments and 1 (empty) break function, and the value chain is unaffected | + +**`TestBreakableCudaGraph` (`:268`) — `test_gsm8k_accuracy` (`:288`) is DELIBERATELY +EXCLUDED from this row**, and the reason is not cost. It launches a Qwen3-8B server +with `--cuda-graph-backend-prefill=breakable` (`:279-281`) and asserts `mgsm_en >= +0.80` over 1319 examples (`:289-301`). That is a DISTRIBUTIONAL accuracy floor on a +PREFILL capture path. Both halves are wrong for us. Our gate polarity for a migrated +model is BIT-EXACTNESS against the model's own eager forward (`## Gates` G1), which +is strictly stronger than an accuracy floor and is the gate AGENTS.md requires when a +greedy path exists. And prefill capture is REFUTED as a lever here and is not what +this row builds (`## Risks/decisions` D5). The upstream case's PURPOSE — an +end-to-end model-level check that a segmented capture does not corrupt generation — +is carried by G1 against `tests/parity/test_qwen36_paged_engine.cpp:140`, per +migrated model. Excluded as redundant to a stronger gate, not as unaffordable. + +**The one unavoidable harness adaptation.** Every upstream class's `setUpClass` +raises `unittest.SkipTest` without CUDA (`:34-36`, `:177-178`, `:235-236`), so +upstream runs all eleven cases on a real device. T1 through T5, and T7 through T11, +run instead against a test backend that RECORDS the call sequence, so they gate on +this box and in continuous integration with no GPU. T6, the multi-replay case, has +both arms: the recording arm asserts the replay ORDER, and the GPU arm under G1 +asserts the VALUES. Nothing else about the upstream cases is adapted: the same break +counts, the same segment counts, the same arithmetic chains, the same in-place versus +assign split, and the same non-copyable fallback. The existing test surface the migration must keep green: @@ -338,26 +498,32 @@ The existing test surface the migration must keep green: | Retire-on-grow bookkeeping | `tests/vt/test_graph_safe_scratch.cpp` | the lifetime rule the seam enforces | | 35B paged engine gate | `tests/parity/test_qwen36_paged_engine.cpp:140` | the model gate the first migration must hold | -New tests this row owes, each written red first: - -1. **Segment arithmetic.** N break points in one capture scope produce exactly - N+1 segments and N break functions. Red first by asserting N segments. -2. **Replay order.** Replay emits segment 0, break 0, segment 1, break 1, and so - on, in that exact order, and the order is identical on the second and third - replay. This is the multi-replay requirement stated in `## Gates`. -3. **Pass-through outside capture.** With no scope active, `vt::GraphBreak(fn)` - runs `fn` and makes zero backend calls. Red first by asserting a backend call. -4. **Capture-failure drain.** A break function that throws leaves the stream - un-poisoned and the partially built `BreakableGraph` destroyed with every - already-instantiated segment released. This ports the recovery that - `qwen3_5.cpp:9913` and `qwen3_dflash.cpp:1106` each hand-rolled. -5. **Non-capturing backend.** On a backend where `SupportsGraphCapture()` is - false, the scope is inert and the forward runs eager. Vulkan - (`vulkan_backend.cpp:16`) and Metal (`metal_backend.mm:13`) are the live cases. -6. **Bit-exactness per migrated model**, over more than one replay. See `## Gates`. - -Tests 1 through 5 need no graphics processing unit (GPU). They run against a test -backend that records the call sequence. Test 6 needs a GPU and an `rc` lease. +**Tests this row owes with no upstream counterpart**, each written red first, +numbered on from the ported set: + +12. **Replay ORDER, not only replay arithmetic.** Replay emits segment 0, break 0, + segment 1, break 1, and so on, in that exact order, and the order is identical on + the second and third replay. Upstream asserts the composed VALUE (T2, T3), which + a wrong order could in principle still satisfy for a commutative chain; this + asserts the sequence directly. +13. **Capture-failure drain.** A break function that throws leaves the stream + un-poisoned and the partially built `BreakableGraph` destroyed with every + already-instantiated segment released. This ports the recovery that + `qwen3_5.cpp:9913` and `qwen3_dflash.cpp:1106` each hand-rolled. Upstream has no + equivalent because Python's `finally` in `__exit__` (`:323-332`) covers it. +14. **Non-capturing backend.** On a backend where `SupportsGraphCapture()` is + false, the scope is inert and the forward runs eager. Vulkan + (`vulkan_backend.cpp:16`) and Metal (`metal_backend.mm:13`) are the live cases. + Upstream has no equivalent because BCG is CUDA and HIP only. +15. **Auxiliary-stream auto-join.** A segment that forks a queue and does not join + it before a break point is joined by the scope, per `## Port map`. This is the + port of `_end_current_segment` `:353-361`, which upstream exercises only + indirectly through its model-level integration test. +16. **Bit-exactness per migrated model**, over more than one replay. See `## Gates`. + +Tests 1 through 15 need no graphics processing unit (GPU), except T6's value arm. +They run against a test backend that records the call sequence. Test 16 needs a GPU +and an `rc` lease. ## Gates @@ -418,7 +584,7 @@ same-binary A/B, not as a win. | Dependency | State | Effect on this row | |---|---|---| | `vt::Backend` capture vocabulary | landed, `include/vt/backend.h:208-222` | the seam needs no new virtual | -| `DecodeGraphSizes` and `PadToCaptureSize` | landed, `decode_graph_sizes.h:47-56` | reused unchanged | +| `DecodeGraphSizes` and `PadToCaptureSize` | landed, `decode_graph_sizes.h:47-54` | reused unchanged | | Retire-on-grow scratch | landed, `src/vt/cuda/graph_safe_scratch.h` | the lifetime rule the seam enforces | | `DevicePool` never-free discipline | landed, `device_pool.h:16` | replaces SGLang's mempool pinning | | `ENG-CUDAGRAPH-DEDUP` (#1162) | in flight, parallel agent | touches the same handles; see D4 | @@ -545,7 +711,7 @@ signature hit, instead of instantiating one executable per padded bucket. The tw rows touch the same handle: dedup changes what `EndCaptureGraph` hands back and who owns it, and this row multiplies the number of such handles by the segment count. They compose in SGLang, where `BreakableCUDAGraph._append_segment` -(`breakable_cuda_graph.py:266-276`) registers each segment with an optional +(`breakable_cuda_graph.py:266-274`) registers each segment with an optional `_deduped_cuda_graph`, and `_begin_new_segment` (`:335-350`) asks for `keep_graph=True` only when dedup is active. **Decision: this row lands first and treats a segment handle as opaque.** `vt::BreakableGraph` must store the handle, @@ -572,13 +738,27 @@ path and the measurement, not on this spec to pre-refute an unnamed one. **D6, the non-CUDA backends.** Tenstorrent implements capture as ttnn mesh-trace (`tenstorrent_backend.cpp:18,75-81`), which is a different runtime concept from a CUDA stream capture. A segmented capture assumes that ending a trace and starting -another on the same queue mid-forward is legal and cheap. That holds for CUDA by -SGLang's production use and is UNVERIFIED for ttnn traces. **Decision: the seam is +another on the same queue mid-forward is legal. **That assumption is UNVERIFIED on +both runtimes and this spec claims it on neither.** For CUDA, SGLang does it on a +production path at the pinned revision (`breakable_cuda_graph.py:352-367` then +`:335-350`), which is strong evidence and is not our measurement; confirming it on a +leased GPU is W1's exit criterion, in `## Work breakdown` W0's own words. For ttnn +traces there is not even that evidence. Nothing here measures the COST of a re-begin +on either runtime, so no cost claim is made; what would settle it is a same-binary +A/B of an N-segment capture against a 1-segment capture of the same forward, on the +leased GPU that W1 already needs. **Decision: the seam is defined against the existing `vt::Backend` vocabulary and asks nothing new of any backend, so a backend that cannot segment reports `SupportsGraphCapture()` and runs one segment, which is exactly today's behavior.** No backend is required to change for W1. +**D7, a break point is a correctness boundary, not only a coverage knob.** Every +break point converts a graphed operation into a host-dispatched one, so a +mis-registered break point is invisible in a token gate and visible only as a +segment count. G3 exists for this reason. A model whose break-point set is wrong +still produces correct tokens; it produces them from more segments than it needs, +or from a graph that was never entered at all. + **D8, this row states no runnable gate command, and that is the honest record.** Moving the row to `READY` puts it in the gated population, where the gate-command classifier asks whether its `## Gates` section names a command that can FAIL. It @@ -593,16 +773,59 @@ names no command. **Decision: do not manufacture a command to satisfy the classifier, and do not re-pin the runnable set for a row with nothing to run.** W1 earns the entry and re-pins the baseline in the same change. -**D7, a break point is a correctness boundary, not only a coverage knob.** Every -break point converts a graphed operation into a host-dispatched one, so a -mis-registered break point is invisible in a token gate and visible only as a -segment count. G3 exists for this reason. A model whose break-point set is wrong -still produces correct tokens; it produces them from more segments than it needs, -or from a graph that was never entered at all. +**D9, the break function's OUTPUT is a lifetime surface, and D1 and D2 do not cover +it.** D1 covers the device intermediates a SEGMENT reads and D2 covers the host +source of a captured upload. Both are about break-function INPUTS. The output is a +third case and it is the one upstream spends the most machinery on. On replay N the +eager function allocates a fresh result, so unless the seam copies that result back +into the destination the following segment baked at capture time, segment `i+1` reads +capture-time data forever while the break function writes into a buffer nobody reads. +Upstream closes it with `replay_fn` plus `_copy_output` +(`breakable_cuda_graph.py:231-235,172-201`) and keeps the destination alive with +`_weak_ref_if_tensor` (`:156-169`). **Decision: `vt::GraphBreak` states the writeback +contract at its declaration, per `## Port map` §3 — either it takes the destination +and the seam copies into it, or the break function writes in place into a persistent +model-owned buffer and returns nothing.** A `BreakableGraph` that stores the caller's +raw `fn` cannot express either, so it stores the seam's closure instead. + +This defect is invisible to every gate that runs ONE replay, and it produces wrong +numbers rather than a fault, so it is in the same detection class D1 describes: a +clean `compute-sanitizer` run says nothing about it. The ported `TestCopyOutput` +cases T7 through T10 and the multi-replay case T6 in `## Tests to port` are what pin +it, and G1's more-than-one-replay requirement is what catches it at model level. + +**D10, a segment cannot close while a forked queue is still capturing.** Upstream +auto-joins outstanding side streams inside `_end_current_segment` before +`capture_end()` (`breakable_cuda_graph.py:353-361`) and maintains the whole +`wait_stream` hook (`:101-153`, installed `:310`, removed `:332`) for no other +purpose. This is live here, not hypothetical: `SupportsAuxStream()` is `true` on +CUDA (`src/vt/cuda/cuda_backend.cu:200`) and two model paths fork on it — Qwen3.5's +MoE shared-expert overlap (`src/vllm/model_executor/models/qwen3_5.cpp:6254-6255` +fork, `:6384` join) and Laguna's decode graph, whose auxiliary queue is owned by the +capture class itself (`src/vllm/model_executor/models/laguna.cpp:2362-2367`, +`:2572-2576` fork, `:2612` join). W4 migrates the first and W5 the second. +**Decision: `GraphCaptureScope` owns the outstanding-fork set and joins it before +every `EndCaptureGraph`, and test 15 of `## Tests to port` gates it.** We need no +hook to populate that set, because our fork and join are explicit +`Backend::RecordEvent` and `Backend::QueueWaitEvent` calls rather than an implicit +torch API — which makes this cheaper for us than for SGLang, not harder. A break +point registered inside an unjoined fork window without this rule fails at +`EndCaptureGraph`, which is the one failure mode in this spec that is LOUD. ## Now `READY`. The spec is committed, the design is grounded in both oracles, the -inventory is enumerated with anchors, and the work is decomposed into six landable -stages. No production code exists for this row. W1 needs an `rc` GPU lease to -confirm its exit criterion before implementation starts. +inventory is enumerated with anchors, the upstream unit suite is mapped case for +case in `## Tests to port`, and the work is decomposed into six landable stages. No +production code exists for this row. W1 needs an `rc` GPU lease to confirm its exit +criterion — that CUDA permits `cudaStreamEndCapture` followed by +`cudaStreamBeginCapture` mid-forward on our stream configuration — before +implementation starts. + +Revised 2026-08-18 after a fresh review returned `FAIL`. What changed: the upstream +test suite is ported rather than declared absent; `## Port map` §3 states the output +writeback contract that the raw-`fn` description had dropped (D9); the auxiliary +stream fork-join rule is stated and gated (D10); four enumerations in +`## Our baseline` and one `grep` count were re-derived and corrected against printed +instruments; D6 now defers to W1's exit criterion instead of asserting it; and the +row is recorded as coverage AND correctness per #1179. diff --git a/.agents/specs/sglang-breakable-cuda-graph.md b/.agents/specs/sglang-breakable-cuda-graph.md index 73bbf83b4..dd10207e4 100644 --- a/.agents/specs/sglang-breakable-cuda-graph.md +++ b/.agents/specs/sglang-breakable-cuda-graph.md @@ -108,16 +108,36 @@ only `pure_decode` batches to a graph, so one host-dependent op forces the whole eager. Coverage is a cliff, not a slope, and [#1020](https://github.com/mudler/vllm.cpp/issues/1020) is one instance of it. -Eight call sites hand-roll capture: `Qwen3_5DecodeGraph`, `Qwen3_5DenseDecodeGraph`, -`Qwen3MoeDecodeGraph`, `Qwen3DenseDecodeGraph`, `DeepseekV2DecodeGraph`, -`VoxtralDecodeGraph`, plus graph code in `deepseek_v4.cpp` and `laguna.cpp`. Each -re-derives capture, bucket padding, persistent-input threading and the pure-decode -predicate. AGENTS.md names this shape: a parallel path written by hand instead of a -shared seam. +**NINE call sites hand-roll capture**, corrected 2026-08-18 +([#1179](https://github.com/mudler/vllm.cpp/issues/1179)); this section recorded +eight when it landed in `9bc4d7f44`. They are `Qwen3_5DecodeGraph`, +`Qwen3_5DenseDecodeGraph`, `Qwen3MoeDecodeGraph`, `Qwen3DenseDecodeGraph`, +`DeepseekV2DecodeGraph`, `VoxtralDecodeGraph`, plus file-local graph code in +`deepseek_v4.cpp`, `laguna.cpp` and — the one this section missed — the DFlash draft +graph at `src/vllm/model_executor/models/qwen3_dflash.cpp:771,870,1038,1091,1095,1106`. +The full table with a declaration and implementation anchor per driver is +[eng-cudagraph-break.md](eng-cudagraph-break.md) `## Our baseline`. Each re-derives +capture, bucket padding, persistent-input threading and the pure-decode predicate. +AGENTS.md names this shape: a parallel path written by hand instead of a shared seam. + +**The duplication has already cost a shipped model its decode graph, so this is a +correctness finding and not only a coverage one.** The persistent DEVICE input path +`StepDevInputs` (`src/vllm/model_executor/models/qwen3_5.cpp:3894`) exists in exactly +one driver — `grep -c StepDevInputs` returns 41 lines in `qwen3_5.cpp` and 0 in each +of `qwen3_moe.cpp`, `qwen3.cpp`, `deepseek_v2.cpp` and `voxtral.cpp`. Because +`Qwen3DenseDecodeGraph` lacks it, `src/vllm/model_executor/models/qwen3.cpp:961-986` +DECLINES the graph outright whenever the asynchronous device-token mirror is live, +with its own measured comment: `depth-1, graph ON PASS 78/78`, `depth-2, graph OFF +PASS 82/82`, `depth-2, graph ON FAIL, slots 1-3 degenerate`. The comment names the +real fix as reading the identifiers at replay time from a stable device buffer, which +is precisely what the sibling driver already does. That is one capability, written +once, unavailable to four models, and a live mitigation standing in its place. BCG is the right construction for the fix, because vLLM gets its split from -`torch.compile` and we have no compiler. The value is **coverage and one seam instead -of eight**. It is not throughput on its own, and it must not be sold as throughput. +`torch.compile` and we have no compiler. The value is **coverage, one seam instead of +nine, and the correctness capability that seam makes shared**. It is not throughput +on its own, and it must not be sold as throughput. §3's refutation stands unchanged: +3.8% host idle in prefill, GPU-busy above 96%, and 92.5% non-GEMM glue. ## 5. Diffusion is the one shape where the lever is real, and it is blocked @@ -162,7 +182,7 @@ collapse. **Capturing now would measure nothing.** | Item | Issue | Row | |---|---|---| | Graph executable dedup via `cudaGraphExecUpdate` | [#1162](https://github.com/mudler/vllm.cpp/issues/1162) | `ENG-CUDAGRAPH-DEDUP` | -| A shared capture seam with break points, retiring the eight hand-rolled drivers | [#1163](https://github.com/mudler/vllm.cpp/issues/1163) | `ENG-CUDAGRAPH-BREAK` | +| A shared capture seam with break points, retiring the NINE hand-rolled drivers | [#1163](https://github.com/mudler/vllm.cpp/issues/1163) | `ENG-CUDAGRAPH-BREAK` | | Denoise-loop capture, blocked on #1010, #1087, #1024, #1007 | [#1164](https://github.com/mudler/vllm.cpp/issues/1164) | `ENG-CUDAGRAPH-DIFFUSION` | ## Stop conditions @@ -185,6 +205,18 @@ What was rejected and why: porting BCG as a prefill throughput lever, because ou prefill has 3.8% host idle and is 92.5% glue-GPU-bound; and scoping the diffusion capture now, because the render does no device compute to capture. +Corrected 2026-08-18 ([#1179](https://github.com/mudler/vllm.cpp/issues/1179)). Two +repairs to what this spec recorded on 2026-08-17. First, the hand-rolled driver count +is NINE, not eight: the `ENG-CUDAGRAPH-BREAK` scoping spike found the DFlash draft +graph (`src/vllm/model_executor/models/qwen3_dflash.cpp:771,870,1038,1091,1095,1106`) +that §4 had missed. Second, `ENG-CUDAGRAPH-BREAK` was recorded here as a COVERAGE row +only, and it is also a CORRECTNESS row: `qwen3.cpp:961-986` declines its decode graph +outright while the async device-token mirror is live, on a measured degeneration +(`depth-2, graph ON FAIL, slots 1-3 degenerate`), because the fix the comment names +lives in one sibling driver as `StepDevInputs` and nowhere else. A shipped model has +already lost its decode graph to this duplication. Neither repair touches the framing +rule: the row still makes no throughput claim, and §3's refutation is unchanged. + Why the defaults are what they are: `ENG-CUDAGRAPH` stays `PARTIAL` rather than moving, because nothing shipped. `SGLANG-BCG` is classified `INVENTORIED` rather than `OWED`, because the capability it names is already in the mirror source and only its From a647abbe970a3720319cac7aadc240a0c922128b Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Tue, 18 Aug 2026 01:45:25 +0000 Subject: [PATCH 3/3] fix(ENG-CUDAGRAPH-BREAK): a fifth surface still said eight, and the spec I had just corrected said it three more times (#1179) The #1179 sweep named four locations. It missed `.agents/sglang-matrix.md:149`, where the `SGLANG-BCG` row's local-mapping column enumerates the drivers by name and stopped at `laguna.cpp`. Re-running the sweep across all of `.agents/` then found three more inside `eng-cudagraph-break.md` itself: the `## Our baseline` heading sentence, the cost sentence that closes the divergence paragraph, and the W0 inventory bullet. All four now read nine, each naming the DFlash draft graph (`src/vllm/model_executor/models/qwen3_dflash.cpp:771,870,1038,1091,1095,1106`) and why it was missed: it is file-local with no header declaration, so an enumeration built from the model headers cannot see it. `.agents/issue-index.md:354` (#1162) and `:355` (#1163) also still say eight and are deliberately NOT edited. That file is append-only and carries `merge=union`, so an edited row is duplicated rather than merged. The #1179 row appended at `:361` is the correction, and it names the DEDUP row's "times eight drivers" sizing explicitly so a reader of either row reaches it. Instrument, run from the worktree after the edits: `grep -rniE '\beight\b' .agents/ | grep -viE 'completed/' | grep -iE 'driver|hand-rolled|capture|seam|cudagraph'`. Every surviving hit in the capture domain is either one of my own corrective clauses ("`9bc4d7f44` recorded eight") or one of the two append-only index rows above. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code] --- .agents/sglang-matrix.md | 2 +- .agents/specs/eng-cudagraph-break.md | 17 ++++++++++------- 2 files changed, 11 insertions(+), 8 deletions(-) diff --git a/.agents/sglang-matrix.md b/.agents/sglang-matrix.md index 917b51c4d..8829af875 100644 --- a/.agents/sglang-matrix.md +++ b/.agents/sglang-matrix.md @@ -146,7 +146,7 @@ share the idea) or OUT-OF-SCOPE. | ID | SGLang surface | SGLang anchor (`file:line`) | Our mapping / anchor | Class | Notes | |---|---|---|---|---|---| -| `SGLANG-BCG` | **Breakable CUDA Graph** — one forward captured as a SEQUENCE of `torch.cuda.CUDAGraph` segments split at eager break points, with no `torch.compile`; plus per-phase backend selection (`FULL`/`BREAKABLE`/`TC_PIECEWISE`/`DISABLED`, decode defaults `FULL`, prefill defaults `BREAKABLE` on CUDA) and graph-executable dedup via `cudaGraphExecUpdate` | `runner_backend_utils/breakable_cuda_graph/breakable_cuda_graph.py:204-241` (segment split), `:244-260` (replay), `:14-23,156-169` (shared mempool + weak-ref intermediates); `runner_backend/breakable_cuda_graph_backend.py:14-17` ("No torch.compile"); break points `layers/radix_attention.py:256`, `attention_forward_methods/forward_mla.py:1092`, `layers/radix_linear_attention.py:159`, `models/nemotron_h.py:1240`, `dsa/dsa_indexer.py:2414`; config `cuda_graph_config.py:38-45,95-112`; dedup `runner_backend/cuda_graph_dedup_mixin.py:219-242,358` | our capture primitive `src/vt/cuda/cuda_backend.cu:203-232`; bucket set `include/vllm/model_executor/models/decode_graph_sizes.h:32-41`; pure-decode routing `src/vllm/v1/worker/gpu/runner.cpp:1338-1341`; eight hand-rolled drivers (`Qwen3_5DecodeGraph`, `Qwen3_5DenseDecodeGraph`, `Qwen3MoeDecodeGraph`, `Qwen3DenseDecodeGraph`, `DeepseekV2DecodeGraph`, `VoxtralDecodeGraph`, `deepseek_v4.cpp`, `laguna.cpp`) | INVENTORIED | **The CAPABILITY is vLLM-derived, not SGLang-distinct**: vLLM's v1 default is already `FULL_AND_PIECEWISE`, splitting at `splitting_ops` (`vllm/config/compilation.py:60-63,517,615,630` @ `555967922`), so it is in our benchmark denominator. What SGLang landed FIRST is the CONSTRUCTION — piecewise coverage from runtime stream capture plus a decorator, no Dynamo/Inductor/FX. That construction is the part worth mirroring, because we have no compiler either. **Prefill capture is REFUTED as a lever here** (GB10 2026-07-09: prefill GPU-idle-between-launches 3.8%, GPU-busy >96%, 27B gap 92.5% non-GEMM GLUE) — do not re-derive. Exec dedup is unported (`cudaGraphExecUpdate` appears nowhere in `src/` or `include/`). Spec [sglang-breakable-cuda-graph.md](specs/sglang-breakable-cuda-graph.md); issues [#1161](https://github.com/mudler/vllm.cpp/issues/1161), [#1162](https://github.com/mudler/vllm.cpp/issues/1162), [#1163](https://github.com/mudler/vllm.cpp/issues/1163), [#1164](https://github.com/mudler/vllm.cpp/issues/1164). | +| `SGLANG-BCG` | **Breakable CUDA Graph** — one forward captured as a SEQUENCE of `torch.cuda.CUDAGraph` segments split at eager break points, with no `torch.compile`; plus per-phase backend selection (`FULL`/`BREAKABLE`/`TC_PIECEWISE`/`DISABLED`, decode defaults `FULL`, prefill defaults `BREAKABLE` on CUDA) and graph-executable dedup via `cudaGraphExecUpdate` | `runner_backend_utils/breakable_cuda_graph/breakable_cuda_graph.py:204-241` (segment split), `:244-260` (replay), `:14-23,156-169` (shared mempool + weak-ref intermediates); `runner_backend/breakable_cuda_graph_backend.py:14-17` ("No torch.compile"); break points `layers/radix_attention.py:256`, `attention_forward_methods/forward_mla.py:1092`, `layers/radix_linear_attention.py:159`, `models/nemotron_h.py:1240`, `dsa/dsa_indexer.py:2414`; config `cuda_graph_config.py:38-45,95-112`; dedup `runner_backend/cuda_graph_dedup_mixin.py:219-242,358` | our capture primitive `src/vt/cuda/cuda_backend.cu:203-232`; bucket set `include/vllm/model_executor/models/decode_graph_sizes.h:32-41`; pure-decode routing `src/vllm/v1/worker/gpu/runner.cpp:1338-1341`; NINE hand-rolled drivers (`Qwen3_5DecodeGraph`, `Qwen3_5DenseDecodeGraph`, `Qwen3MoeDecodeGraph`, `Qwen3DenseDecodeGraph`, `DeepseekV2DecodeGraph`, `VoxtralDecodeGraph`, `deepseek_v4.cpp`, `laguna.cpp`, and the DFlash draft graph `src/vllm/model_executor/models/qwen3_dflash.cpp:771,870,1038,1091,1095,1106` — count corrected 2026-08-18, [#1179](https://github.com/mudler/vllm.cpp/issues/1179); `9bc4d7f44` recorded eight, and the DFlash driver is file-local with no header declaration, which is why it was missed) | INVENTORIED | **The CAPABILITY is vLLM-derived, not SGLang-distinct**: vLLM's v1 default is already `FULL_AND_PIECEWISE`, splitting at `splitting_ops` (`vllm/config/compilation.py:60-63,517,615,630` @ `555967922`), so it is in our benchmark denominator. What SGLang landed FIRST is the CONSTRUCTION — piecewise coverage from runtime stream capture plus a decorator, no Dynamo/Inductor/FX. That construction is the part worth mirroring, because we have no compiler either. **Prefill capture is REFUTED as a lever here** (GB10 2026-07-09: prefill GPU-idle-between-launches 3.8%, GPU-busy >96%, 27B gap 92.5% non-GEMM GLUE) — do not re-derive. Exec dedup is unported (`cudaGraphExecUpdate` appears nowhere in `src/` or `include/`). Spec [sglang-breakable-cuda-graph.md](specs/sglang-breakable-cuda-graph.md); issues [#1161](https://github.com/mudler/vllm.cpp/issues/1161), [#1162](https://github.com/mudler/vllm.cpp/issues/1162), [#1163](https://github.com/mudler/vllm.cpp/issues/1163), [#1164](https://github.com/mudler/vllm.cpp/issues/1164). | ## Structurally distinctive / research (vLLM has no native analogue) diff --git a/.agents/specs/eng-cudagraph-break.md b/.agents/specs/eng-cudagraph-break.md index 747fd8ea8..e884dc60f 100644 --- a/.agents/specs/eng-cudagraph-break.md +++ b/.agents/specs/eng-cudagraph-break.md @@ -170,9 +170,11 @@ out to eager with no log and no counter. ### Defect 2: the drivers, enumerated -Eight production drivers exist, and a ninth already landed. Six are batched decode -drivers with a padded-bucket ring; three are single-shape drivers. Every one of -them re-derives the same machinery. +NINE production drivers exist. Six are batched decode drivers with a padded-bucket +ring; three are single-shape drivers. Every one of them re-derives the same +machinery. The count is nine and not the eight recorded in `9bc4d7f44` +([#1179](https://github.com/mudler/vllm.cpp/issues/1179)): the DFlash draft graph is +file-local with no header declaration, which is why the parent analysis missed it. | Driver | Declaration | Implementation | Shape | |---|---|---|---| @@ -244,7 +246,8 @@ failure that forced it: "depth-2, graph ON FAIL, slots 1-3 degenerate". The same comment names the real fix as reading the identifiers at replay time from a stable device buffer, which is precisely what `StepDevInputs` already does in the sibling driver. One capability, written once, unavailable to four models. That is the cost -of eight drivers stated as a defect rather than as an aesthetic complaint. +of nine drivers stated as a defect rather than as an aesthetic complaint. It is also +why this row is a CORRECTNESS row and not only a coverage one. **What is genuinely model-specific**, and must stay in the model: @@ -611,9 +614,9 @@ recorded here and in `## Risks/decisions`. What it established: capture call log of `Begin End` three times, an identical replay order on two consecutive replays, and a pure pass-through with zero backend calls when no scope was active. Compiled and ran at exit status 0. -- The inventory in `## Our baseline` is the real one: eight drivers plus a ninth - already written, and the persistent-device-input capability present in exactly - one of them. +- The inventory in `## Our baseline` is the real one: NINE drivers, not the eight + the parent analysis recorded, and the persistent-device-input capability present + in exactly one of them. **W0 did NOT establish** that CUDA permits `cudaStreamEndCapture` followed by `cudaStreamBeginCapture` on the same stream mid-forward with eager work between