record(ENG-CUDAGRAPH-DEDUP): the device A/B ran, and the fold it was filed for NEVER HAPPENS (#1162, #1184, #1226) - #1227
Merged
Conversation
…filed for NEVER HAPPENS (#1162, #1184, #1226) The owed W4 gate ran on `dgx:gpu0` on 2026-08-18 (GB10, driver 580.173.02, nvcc 13.0.88, `rc` job f88d484b, one binary, `VT_ASYNC_RUNNER=0` and `VT_DECODE_GRAPH_STATS=1` throughout, the only variable `VT_CUDA_GRAPH_DEDUP`). It came back with two opposite verdicts, and this is the record of both. No engine code changes here. The gated commit is `72de552c8`, not the merge `2a976eb9f`. The row squashed, so the gated tree is not an ancestor of what landed; what carries the claim across that gap is that all four dedup sources are byte-identical at the two commits, which was verified rather than assumed. CORRECTNESS PASSES, and #1184 is closed by this run rather than by the CPU suite that could never see it: 12/12 cells exit 0, zero `invalid device function` and zero `engine-fatal` in every cell log where the pre-fix head `e4ce5571a` died after exactly one replay, ON replays as often as OFF (60=60, 33=33, 43=43), and `--output-token-ids` is identical across 10/10 comparisons with the three OFF/OFF controls passing first and the three workloads hashing to three different values, so the identity is not vacuous. THE BENEFIT IS REFUTED for exactly the case the row was filed for. `N == M` in every ON cell, now over 2 and 3 distinct padded buckets per process where the first attempt managed one and could prove nothing, with the registry count climbing 1→1, 2→2, 3→3. The cause was pre-registered before the run and is structural: `AppendKernelPayload` hashes the grid and block dimensions and the memcpy payload hashes the copy extent, so the padded batch dimension is in the KEY, no candidate group forms, and `cudaGraphExecUpdate` is never attempted. That contradicts the row's own header comment, and SGLang keys the same fields, so whatever folds upstream is not decode buckets either. No throughput or memory number is recorded on any axis, for two independent reasons either of which suffices: the clocks were not pinned, and the ON arm allocated exactly as many executables as OFF, so there is no delta to claim. THE MATRIX ROW STAYS `ACTIVE`, argued rather than inherited. `DONE` would owe an `## Outcome` claiming a delivered capability, and the capability measurably folds nothing on the only driver exercised; banking that as done is how a refuted premise becomes a number people quote. `PARTIAL` means the implementation is also known to omit upstream behavior, and we omit none — we key the same fields SGLang keys, faithfully. The gap is a design question about the key, which is a live hypothesis with an owner, and that is what `ACTIVE` means. The summary-table counts are therefore unchanged, because no state moved. The open question is filed as #1226 and deliberately NOT decided here: a coarser key that keeps the function addresses and the topology but drops the launch dimensions and the memcpy extents would let two padded buckets form a candidate group at all, and because the registry probes with `cudaGraphExecUpdate` BEFORE it folds, a wrong grouping costs one wasted probe and a private executable rather than a wrong replay. That makes it a cost question needing its own spec and red-first evidence. "Unreachable with THIS key" is not "unreachable", and AGENTS.md forbids declaring a ceiling. Four honest gaps ride into every surface rather than into none: per-shape replay counts are unavailable because the driver prints a total, so workload B's ~30-per-shape is arithmetic and is labelled as arithmetic; the driver's "N captured size(s)" line counts slots, not captures; the container's own cuBLASLt was never re-tested at CUDA 13.0 because the staged cu130 prefix was probed first and worked, so the shim's original reason may no longer apply; and only the Qwen3 dense decode driver was exercised. The supporting `orin:gpu0` lane is BLOCKED, cleanly: the Jetson 540.4.0 driver cannot run a CUDA 13 runtime (`cudaGetDeviceCount err=35`), and a CUDA 12.x toolkit is the untried route. Checkers run from this worktree: `check-agent-record.py` OK (ENGINE=162 unchanged), `check-public-doc-tables.py` OK, `check-issue-index-append-only.py --base origin/main --head HEAD` OK. `agent-preflight.sh` was NOT run: it has been killed repeatedly on this box, so the component checkers were run instead and are named here. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The owed W4 gate ran on
dgx:gpu0on 2026-08-18 (GB10, driver 580.173.02, nvcc13.0.88,
rcjob f88d484b, one binary,VT_ASYNC_RUNNER=0andVT_DECODE_GRAPH_STATS=1throughout, the only variableVT_CUDA_GRAPH_DEDUP).It came back with two opposite verdicts, and this is the record of both. No
engine code changes here.
The gated commit is
72de552c8, not the merge2a976eb9f. The row squashed, sothe gated tree is not an ancestor of what landed; what carries the claim across
that gap is that all four dedup sources are byte-identical at the two commits,
which was verified rather than assumed.
CORRECTNESS PASSES, and #1184 is closed by this run rather than by the CPU suite
that could never see it: 12/12 cells exit 0, zero
invalid device functionandzero
engine-fatalin every cell log where the pre-fix heade4ce5571adiedafter exactly one replay, ON replays as often as OFF (60=60, 33=33, 43=43), and
--output-token-idsis identical across 10/10 comparisons with the threeOFF/OFF controls passing first and the three workloads hashing to three
different values, so the identity is not vacuous.
THE BENEFIT IS REFUTED for exactly the case the row was filed for.
N == Minevery ON cell, now over 2 and 3 distinct padded buckets per process where the
first attempt managed one and could prove nothing, with the registry count
climbing 1→1, 2→2, 3→3. The cause was pre-registered before the run and is
structural:
AppendKernelPayloadhashes the grid and block dimensions and thememcpy payload hashes the copy extent, so the padded batch dimension is in the
KEY, no candidate group forms, and
cudaGraphExecUpdateis never attempted.That contradicts the row's own header comment, and SGLang keys the same fields,
so whatever folds upstream is not decode buckets either.
No throughput or memory number is recorded on any axis, for two independent
reasons either of which suffices: the clocks were not pinned, and the ON arm
allocated exactly as many executables as OFF, so there is no delta to claim.
THE MATRIX ROW STAYS
ACTIVE, argued rather than inherited.DONEwould owe an## Outcomeclaiming a delivered capability, and the capability measurablyfolds nothing on the only driver exercised; banking that as done is how a
refuted premise becomes a number people quote.
PARTIALmeans theimplementation is also known to omit upstream behavior, and we omit none — we
key the same fields SGLang keys, faithfully. The gap is a design question about
the key, which is a live hypothesis with an owner, and that is what
ACTIVEmeans. The summary-table counts are therefore unchanged, because no state moved.
The open question is filed as #1226 and deliberately NOT decided here: a coarser
key that keeps the function addresses and the topology but drops the launch
dimensions and the memcpy extents would let two padded buckets form a candidate
group at all, and because the registry probes with
cudaGraphExecUpdateBEFOREit folds, a wrong grouping costs one wasted probe and a private executable
rather than a wrong replay. That makes it a cost question needing its own spec
and red-first evidence. "Unreachable with THIS key" is not "unreachable", and
AGENTS.md forbids declaring a ceiling.
Four honest gaps ride into every surface rather than into none: per-shape replay
counts are unavailable because the driver prints a total, so workload B's
~30-per-shape is arithmetic and is labelled as arithmetic; the driver's "N
captured size(s)" line counts slots, not captures; the container's own cuBLASLt
was never re-tested at CUDA 13.0 because the staged cu130 prefix was probed
first and worked, so the shim's original reason may no longer apply; and only
the Qwen3 dense decode driver was exercised. The supporting
orin:gpu0lane isBLOCKED, cleanly: the Jetson 540.4.0 driver cannot run a CUDA 13 runtime
(
cudaGetDeviceCount err=35), and a CUDA 12.x toolkit is the untried route.Checkers run from this worktree:
check-agent-record.pyOK (ENGINE=162unchanged),
check-public-doc-tables.pyOK,check-issue-index-append-only.py --base origin/main --head HEADOK.agent-preflight.shwas NOT run: it has been killed repeatedly on this box, sothe component checkers were run instead and are named here.
Records only.
docs/BENCHMARKS.mdmoves the axis fromPENDINGtoMEASURED, NEGATIVEwith no throughput or memory number;docs/STATUS.mddropsthe now-false "A/B owed" clause; the spec gains
## Outcomeand a rewritten## Owed; the engine matrix row records the split verdict and keepsACTIVE;.agents/issue-index.mdgains one appended row;.agents/benchmark-record.mdgains the full recipe, hashes, cell table and caveats.
Closes #1184
Refs #1162
Refs #1226
FOLLOWING_AGENTS_PROTOCOL
Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [Claude Code]