feat(ENG-CUDAGRAPH-BREAK): a forward can be captured as SEGMENTS split at break points, and CUDA permits the re-begin W0 could not test (#1192, #1163)

[localai-bot]

W1 of the break-point capture seam. Lands vt::BreakableGraph,
vt::GraphCaptureScope and vt::GraphBreak, ports SGLang's unit suite case for
case, and registers exactly ONE break point on ONE model.

Coverage and correctness. Not speed, and no throughput claim is made anywhere
in this change. GB10 measured prefill idle between launches at 3.8% with
GPU-busy above 96%, and the 27B prefill gap at 92.5% non-GEMM glue GPU work;
decode is already captured and already banked its launch-overhead win. The
correctness half is shipped damage: qwen3.cpp declines its decode graph
outright whenever the async device-token mirror is live, because the fix its own
comment names lives in one sibling driver and nowhere else.

The exit criterion, answered before anything was built

W0 deliberately did NOT establish that CUDA permits cudaStreamEndCapture
followed by cudaStreamBeginCapture on the same stream mid-forward with eager
work between them, on our stream configuration. SGLang does exactly this on a
production path at the pin, which is strong evidence and was not our measurement.

It is now. On orin:gpu0 through an rc lease, driver 12060, under
cudaStreamCaptureModeThreadLocal — the mode src/vt/cuda/cuda_backend.cu:204
uses: segment, host-dependent eager break, RE-BEGIN, segment, bare zero-work
re-begin, segment; then three replays with fresh inputs, 0 mismatches on each.
The bare re-begin (breakable_cuda_graph.py:370-374) is legal too.

The first probe REFUSED, and the refusal was the probe's. dlsym on a bare
symbol name binds libcuda's LEGACY v1 entry points, which are not
capture-aware: cuMemcpyDtoDAsync (v1) returned
CUDA_ERROR_STREAM_CAPTURE_UNSUPPORTED inside a legal capture, and
cuStreamBeginCapture (v1) takes no capture-mode argument, so the probe believed
it was testing thread-local mode while testing global. Preferring _v3 blindly
then bound cuCtxCreate_v3, which takes two more parameters, and context
creation failed CUDA_ERROR_INVALID_DEVICE. Both readings presented as a verdict
about the design rather than about the instrument.

The boundary is vLLM's; only the construction is SGLang's

vLLM's v1 default splits at splitting_ops, defaulted to the attention family
(vllm/config/compilation.py:517,764-772,1145 @ 5559679229). That boundary
transfers; its mechanism, Dynamo and FX, cannot. So the registration form is
SGLang's — one line at the site, as layers/radix_attention.py:256 @
f63458b5be — and the site IS the registration. The break point registered here
is the dense attention entry of Qwen3ForCausalLM, in the destination-carrying
form, because AttnBlock returns a fresh pooled buffer on every call and the
in-place form's contract is one that site cannot meet.

The writeback is the part a container gets wrong

Upstream does not append the caller's function. eager_on_graph appends a
replay_fn closure that calls the captured inner function and writes the result
back into the capture-time destination through _copy_output (:231-235,
:172-201). On replay N the eager operation returns a FRESH allocation whose
address is not the one the next segment baked, so a container replaying the raw
function leaves segment i+1 reading capture-time data while the break writes
elsewhere — wrong numerics, not a fault, the class a clean compute-sanitizer
run cannot see. BreakableGraph therefore stores the seam's closures, never the
caller's function, and CopyOutput is a customization point reproducing all four
of upstream's branches, including the non-copyable fallback that fabricates no
copy. T7 through T10 pin it; T10 asserts the ABSENCE of the writeback on purpose.

The destination is a vt::BreakSlot<T>, not a caller reference, and that is
the fix for the finding below. Upstream holds captured_output BY VALUE with its
storage pinned by the segment mempool (:156-169). A C++ seam taking Out& gets
the opposite: the natural call site declares the destination as a local of the
function containing the break, which dies on the next return. The slot's cell
is owned by the replay closure, so the rule holds by construction, and the
inline-until-pinned storage keeps the pass-through path allocation-free.

What the review found, and what closed it

A fresh review of 30b9b43ba returned FAIL. All findings are repaired in
cba969857, each red first. They are listed here because each is a class of
defect W2 through W5 can repeat.

Finding	What was wrong	What closed it
HIGH-1	the production break point's destination was a function-local std::optional<DBuf>: both scoped layers registered the SAME stack slot, in a frame gone before the scope closed, and the pooled block went back on the DevicePool free list	vt::BreakSlot<T> the seam owns; the reference-taking overload is GONE, so the obvious call site cannot express the defect
HIGH-2	interleaved replay was claimed, spec'd and NOT gated: batching the loop left the suite green, because segments and break markers went into two independent sequences	break markers go into the backend's OWN log; one trace asserts ReplayGraph break0 ReplayGraph break1 ReplayGraph
MEDIUM-1	renaming CopyOutput dropped the site into the non-copyable fallback (the D9 wrong-numerics path) with both suites green	static_assert in the destination form, an explicit vt::NoWriteback{} opt-in, and the model's overload moved to a header so the G2 gate asserts which branch is selected
MEDIUM-2	the VLLM_CPP_CUDAGRAPH=0 kill switch was ungated; the existing case substituted the other conjunct	a CHILD PROCESS arm, because the switch is read once per process; a sentinel file defeats "0 cases ran, SUCCESS!"
MEDIUM-3	the drain was owed as BEHAVIOUR, not only as a test: the destructor's catch guarded a throwing EndCaptureGraph alone, so a mid-capture exception left a partial capture reporting captured() == true	std::uncaught_exceptions() against the depth at scope entry; three gated arms
MEDIUM-4	segment == break + 1 was called "asserted" in three records and asserted nowhere; re-entry gave seg == brk, whose last break Replay drops	refused at scope open and at Replay
MEDIUM-5	nesting was accepted and traced a second BeginCapture onto an already-capturing stream	refused before any backend call; prev_ deleted with the shape that implied nesting was contemplated
MEDIUM-6	the exit-criterion probe was not committed and had no recipe or hash	scripts/probe_cudagraph_rebegin.c and .cu committed, recipe and sha256 in the record; the leased run's own stdout was NOT retained and the record says so
MEDIUM-7	fidelity was overstated: T3 had lost upstream's chain and T1 its post-replay value, because a recording backend cannot re-execute a segment	the backend SIMULATES the graph, so every upstream chain and post-replay assertion ports literally; the two remaining deviations are named
LOW	CopyOutput sized from dst alone; Reset() left replays_ stale; the suite was not hermetic under VLLM_CPP_CUDAGRAPH=0; form numbering conflicted; engine-matrix owed: listed three delivered items; ## Owed named a tracker that dies at merge	agreement VT_CHECKs, replays_ cleared, a named precondition per case, one naming scheme, the owed: cell re-derived, and #1163 named as the staged-slice tracker

Gates

tests/vt/test_breakable_graph.cpp, 24 cases / 163 assertions, exit 0 — T1
through T11 of the spec's ## Tests to port with their arithmetic chains and
post-replay assertions intact, plus the replay-order case (12), the
capture-failure drain (13, all three arms), the non-capturing backend (14), the
lifetime-rule case, the aliasing refusal with its two controls, the nesting and
re-entry refusals, and the ownership case that proves every segment is released
through Backend::DestroyGraph, so #1162's dedup can interpose at the backend.

tests/vllm/models/test_qwen3_break_point.cpp is the G2 reachability gate: it
drives the production Qwen3DenseModel::Forward with a scope open and counts
num_hidden_layers + 1 segments. Deleting the one call site takes that to 1 and
the gate to RED, which a unit test constructing the container by hand could not
have asked. The same case holds G4 by comparing the logits BIT FOR BIT against
the unscoped forward: 500 values, 0 differing.

Staged slice, named rather than implied

GraphCaptureScope and BreakableGraph are not yet ENTERED from a production
step, because no driver opens a scope until W2 migrates Qwen3DenseDecodeGraph.
The break point itself is on the production path — every forward executes it and
takes the pass-through arm, which is byte-identical to the line it replaced and
makes zero backend calls. ## Owed in the spec lists this with W2 as owner,
alongside the auxiliary-stream auto-join (D10, W4 and W5), the ROCm and
Tenstorrent arms (G5, W3), G1 bit-exactness on a real GPU over more than one
replay (W2), and D1's INPUT half (W2). The capture-failure drain is no longer
owed: it landed here as behaviour and as three gated arms, and the two records
that still said otherwise — the engine-matrix spec cell and the issue-index
row — now say so too. ## Owed gains one residual the scoped re-review
surfaced, with W2 as owner: an exception CAUGHT INSIDE the capture scope leaves
the rest of the forward uncaptured while captured() stays true, because
nothing is unwinding at scope exit for the drain to see. The staged slice is
tracked by #1163, the parent, because #1192 closes with this change and a
tracker that dies at merge tracks nothing.

The aliasing half of HIGH-1, gated after the re-review

BreakSlot closed the LIFETIME half of lifetime rule 1 and this spec claimed
the whole rule was "unexpressible to violate". It was not. One slot reused for
TWO break points in one capture compiled: PinForCapture returns the cell it
already made, so both replay closures bound to the SAME address, the earlier
writeback was overwritten, and any segment that baked the earlier destination
read the later break's data. The production site is correct today because its
slot is a RunLayer local, and W2 through W5 add nine more callers.

GraphCaptureScope::AppendBreak now takes the destination as a REQUIRED
parameter — no GraphBreak form can register without stating where it lands,
including a form added later — and throws when a second break in the same
capture names a cell already registered. The identity is the CELL and not the
SLOT, measured rather than reasoned: the production slot is a per-call local,
every layer's slot occupies the same STACK address, and slot-address identity
refused the correct program on layer 2 and reddened the G2 gate. The
non-copyable fallback registers no destination, because it pins no cell and its
replay writes back nowhere. Red-first: the case failed on the unrepaired head
with the aliased capture ACCEPTED at break_count() == 2 into one slot.

Mutation	Compiles	Result
delete the refusal	yes, 0 errors	RED — 1 of 24 cases, only the new one; G2 stays green, which is why nothing caught this before
make the refusal over-fire	yes, 0 errors	RED — 2 unit cases AND the G2 reachability gate, so the controls are not vacuous

Evidence

Command	Exit
./build/tests/test_breakable_graph	0 — 24 cases / 163 assertions
./build/tests/test_qwen3_break_point	0 — 2 cases / 516 assertions
python3 scripts/check-agent-record.py	0 — ENGINE=162 MODEL=377 QUANT=82 KERNEL=52 BACKEND=85
python3 scripts/check-public-doc-tables.py	0
python3 scripts/check-issue-index-append-only.py --base origin/main --head HEAD	0
python3 scripts/check-commit-trailers.py --range origin/main..HEAD	0
python3 scripts/check-commit-style.py --range origin/main..HEAD	0
./build/tests/test_decode_graph_sizes	0 — 478 assertions (the shared piece the seam must not change)
./build/tests/test_graph_safe_scratch	0 — 4 assertions (the lifetime rule the seam enforces)
./build/tests/test_qwen3_forward	0 — 1575 assertions (the model the break point is registered on)
./build/tests/test_qwen3_moe_forward	0 — 504 assertions

The RED came first: with the header written and the source not yet in the CMake
source list, test_breakable_graph failed to link on
vt::BreakableGraph::~BreakableGraph() and
vt::GraphCaptureScope::~GraphCaptureScope(), exit 1.

Negative mutation

Every mutation asserted its anchor before applying, printed its COMPILE status
and its exit status, and was restored byte for byte, because a mutation that
never applied and a mutation that failed to build both read as a passing test.
The repair pass added ten, on top of the six the first head carried. Every one
went RED or failed to compile for the intended reason.

#	Mutation	Compiled	Result
N1	HIGH-1 — the destination is a pointer into the slot's inline storage instead of the owned cell	OK	RED, 2 cases / 1 assertion
N2	HIGH-2 — replay ALL segments, then ALL break functions	OK	RED, 5 cases / 10 assertions
N3	MEDIUM-1 — rename the model's CopyOutput overload (a typo, a move, a shadow)	COMPILE ERROR at two sites: the seam's static_assert and the G2 gate's	red before it can run
N4	MEDIUM-2 — drop GraphCaptureEnabled() from the active_ conjunction	OK	RED, 1 case / 2 assertions
N5	MEDIUM-3 — drop the unwinding drain from the scope destructor	OK	RED, 2 cases / 7 assertions
N6	MEDIUM-4 — drop the re-entry refusal	OK	RED, 1 case / 2 assertions
N7	MEDIUM-5 — drop the nesting refusal	OK	RED, 1 case / 4 assertions
N8	LOW — Reset() leaves replays_ stale	OK	RED, 1 case / 1 assertion
N9	LOW — CopyOutput sizes from dst alone, no agreement check	OK	RED, 1 case / 4 assertions
N10	G2 reachability — delete the one production vt::GraphBreak call site in qwen3.cpp	OK	RED, 2/2 cases; the unit suite stays GREEN

N10 is the one that matters for "Nothing lands dead". The deleted call site still
COMPILES and the seam's own unit suite stays green without it — which is exactly
the false green the rule exists to catch. test_qwen3_break_point is what goes
red, because it counts segments coming out of a real model forward.

N1 is the one that matters for HIGH-1. It is the defect the first head shipped,
expressed on the new API, and the lifetime case reds on it — where before, no
case could ask the question at all.

Two known reds, characterised rather than hidden

scripts/check-doc-checkpoint.py iterates NON-MERGE COMMITS individually, and on
this branch the lifecycle move (READY to ACTIVE) landed one commit before the
docs/BENCHMARKS.md and docs/USAGE.md edits it obliges. Per commit that is a
FAIL. It is not a fail on what actually lands: this repository squash-merges, so
main receives the branch as ONE commit.

Measured rather than argued. A detached scratch commit carrying this exact tree
as a single commit on top of origin/main:

check-doc-checkpoint.py --base origin/main --head <squash-sim>   -> exit 0
check-doc-checkpoint.py --base origin/main --head HEAD           -> exit 1

The branch history is left alone because rewriting it was outside this task's
authority, and because faking a revert to satisfy a per-commit gate would be
worse than the red — it would put a false transition in the history the gate
exists to keep honest.

The two findings that gate raised on the earlier commit were BOTH real and are
both repaired here: the row's owed: cell cited another spec by link BEFORE the
Spike/spec column, so spec_for_row read the wrong spec as this row's own; and
the lifecycle move genuinely did owe docs/BENCHMARKS.md, while CMakeLists.txt
genuinely did owe docs/USAGE.md.

tests/scripts/test_cpu_x86_llamacpp_floor.py also reds on this box, and the
failing sub-test MOVES between runs (test_a_contended_leg_is_discarded..., then
test_the_quiet_gate_does_not_see_the_harnesss_own_process_tree, which printed
NO_QUIET_WINDOW after 15s busy=114% load=22.05). That is the harness measuring
box load from other sessions, which is #618, not this change.

What is NOT claimed

No throughput result. No GPU bit-exactness over a replayed segmented capture.
No auxiliary-stream auto-join. No ROCm or Tenstorrent arm. Each is listed under
## Owed in the spec with the stage that owns it.

Closes #1192. Parent #1163.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [Claude Code]