Skip to content

record(ENG-CUDAGRAPH-DEDUP): the device A/B ran, and the fold it was filed for NEVER HAPPENS (#1162, #1184, #1226) - #1227

Merged
localai-bot merged 1 commit into
mainfrom
row/ENG-CUDAGRAPH-DEDUP-RESULT
Aug 18, 2026
Merged

record(ENG-CUDAGRAPH-DEDUP): the device A/B ran, and the fold it was filed for NEVER HAPPENS (#1162, #1184, #1226)#1227
localai-bot merged 1 commit into
mainfrom
row/ENG-CUDAGRAPH-DEDUP-RESULT

Conversation

@localai-bot

Copy link
Copy Markdown
Collaborator

The owed W4 gate ran on dgx:gpu0 on 2026-08-18 (GB10, driver 580.173.02, nvcc
13.0.88, rc job f88d484b, one binary, VT_ASYNC_RUNNER=0 and
VT_DECODE_GRAPH_STATS=1 throughout, the only variable VT_CUDA_GRAPH_DEDUP).
It came back with two opposite verdicts, and this is the record of both. No
engine code changes here.

The gated commit is 72de552c8, not the merge 2a976eb9f. The row squashed, so
the gated tree is not an ancestor of what landed; what carries the claim across
that gap is that all four dedup sources are byte-identical at the two commits,
which was verified rather than assumed.

CORRECTNESS PASSES, and #1184 is closed by this run rather than by the CPU suite
that could never see it: 12/12 cells exit 0, zero invalid device function and
zero engine-fatal in every cell log where the pre-fix head e4ce5571a died
after exactly one replay, ON replays as often as OFF (60=60, 33=33, 43=43), and
--output-token-ids is identical across 10/10 comparisons with the three
OFF/OFF controls passing first and the three workloads hashing to three
different values, so the identity is not vacuous.

THE BENEFIT IS REFUTED for exactly the case the row was filed for. N == M in
every ON cell, now over 2 and 3 distinct padded buckets per process where the
first attempt managed one and could prove nothing, with the registry count
climbing 1→1, 2→2, 3→3. The cause was pre-registered before the run and is
structural: AppendKernelPayload hashes the grid and block dimensions and the
memcpy payload hashes the copy extent, so the padded batch dimension is in the
KEY, no candidate group forms, and cudaGraphExecUpdate is never attempted.
That contradicts the row's own header comment, and SGLang keys the same fields,
so whatever folds upstream is not decode buckets either.

No throughput or memory number is recorded on any axis, for two independent
reasons either of which suffices: the clocks were not pinned, and the ON arm
allocated exactly as many executables as OFF, so there is no delta to claim.

THE MATRIX ROW STAYS ACTIVE, argued rather than inherited. DONE would owe an
## Outcome claiming a delivered capability, and the capability measurably
folds nothing on the only driver exercised; banking that as done is how a
refuted premise becomes a number people quote. PARTIAL means the
implementation is also known to omit upstream behavior, and we omit none — we
key the same fields SGLang keys, faithfully. The gap is a design question about
the key, which is a live hypothesis with an owner, and that is what ACTIVE
means. The summary-table counts are therefore unchanged, because no state moved.

The open question is filed as #1226 and deliberately NOT decided here: a coarser
key that keeps the function addresses and the topology but drops the launch
dimensions and the memcpy extents would let two padded buckets form a candidate
group at all, and because the registry probes with cudaGraphExecUpdate BEFORE
it folds, a wrong grouping costs one wasted probe and a private executable
rather than a wrong replay. That makes it a cost question needing its own spec
and red-first evidence. "Unreachable with THIS key" is not "unreachable", and
AGENTS.md forbids declaring a ceiling.

Four honest gaps ride into every surface rather than into none: per-shape replay
counts are unavailable because the driver prints a total, so workload B's
~30-per-shape is arithmetic and is labelled as arithmetic; the driver's "N
captured size(s)" line counts slots, not captures; the container's own cuBLASLt
was never re-tested at CUDA 13.0 because the staged cu130 prefix was probed
first and worked, so the shim's original reason may no longer apply; and only
the Qwen3 dense decode driver was exercised. The supporting orin:gpu0 lane is
BLOCKED, cleanly: the Jetson 540.4.0 driver cannot run a CUDA 13 runtime
(cudaGetDeviceCount err=35), and a CUDA 12.x toolkit is the untried route.

Checkers run from this worktree: check-agent-record.py OK (ENGINE=162
unchanged), check-public-doc-tables.py OK,
check-issue-index-append-only.py --base origin/main --head HEAD OK.
agent-preflight.sh was NOT run: it has been killed repeatedly on this box, so
the component checkers were run instead and are named here.

Records only. docs/BENCHMARKS.md moves the axis from PENDING to
MEASURED, NEGATIVE with no throughput or memory number; docs/STATUS.md drops
the now-false "A/B owed" clause; the spec gains ## Outcome and a rewritten
## Owed; the engine matrix row records the split verdict and keeps ACTIVE;
.agents/issue-index.md gains one appended row; .agents/benchmark-record.md
gains the full recipe, hashes, cell table and caveats.

Closes #1184
Refs #1162
Refs #1226

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [Claude Code]

…filed for NEVER HAPPENS (#1162, #1184, #1226)

The owed W4 gate ran on `dgx:gpu0` on 2026-08-18 (GB10, driver 580.173.02, nvcc
13.0.88, `rc` job f88d484b, one binary, `VT_ASYNC_RUNNER=0` and
`VT_DECODE_GRAPH_STATS=1` throughout, the only variable `VT_CUDA_GRAPH_DEDUP`).
It came back with two opposite verdicts, and this is the record of both. No
engine code changes here.

The gated commit is `72de552c8`, not the merge `2a976eb9f`. The row squashed, so
the gated tree is not an ancestor of what landed; what carries the claim across
that gap is that all four dedup sources are byte-identical at the two commits,
which was verified rather than assumed.

CORRECTNESS PASSES, and #1184 is closed by this run rather than by the CPU suite
that could never see it: 12/12 cells exit 0, zero `invalid device function` and
zero `engine-fatal` in every cell log where the pre-fix head `e4ce5571a` died
after exactly one replay, ON replays as often as OFF (60=60, 33=33, 43=43), and
`--output-token-ids` is identical across 10/10 comparisons with the three
OFF/OFF controls passing first and the three workloads hashing to three
different values, so the identity is not vacuous.

THE BENEFIT IS REFUTED for exactly the case the row was filed for. `N == M` in
every ON cell, now over 2 and 3 distinct padded buckets per process where the
first attempt managed one and could prove nothing, with the registry count
climbing 1→1, 2→2, 3→3. The cause was pre-registered before the run and is
structural: `AppendKernelPayload` hashes the grid and block dimensions and the
memcpy payload hashes the copy extent, so the padded batch dimension is in the
KEY, no candidate group forms, and `cudaGraphExecUpdate` is never attempted.
That contradicts the row's own header comment, and SGLang keys the same fields,
so whatever folds upstream is not decode buckets either.

No throughput or memory number is recorded on any axis, for two independent
reasons either of which suffices: the clocks were not pinned, and the ON arm
allocated exactly as many executables as OFF, so there is no delta to claim.

THE MATRIX ROW STAYS `ACTIVE`, argued rather than inherited. `DONE` would owe an
`## Outcome` claiming a delivered capability, and the capability measurably
folds nothing on the only driver exercised; banking that as done is how a
refuted premise becomes a number people quote. `PARTIAL` means the
implementation is also known to omit upstream behavior, and we omit none — we
key the same fields SGLang keys, faithfully. The gap is a design question about
the key, which is a live hypothesis with an owner, and that is what `ACTIVE`
means. The summary-table counts are therefore unchanged, because no state moved.

The open question is filed as #1226 and deliberately NOT decided here: a coarser
key that keeps the function addresses and the topology but drops the launch
dimensions and the memcpy extents would let two padded buckets form a candidate
group at all, and because the registry probes with `cudaGraphExecUpdate` BEFORE
it folds, a wrong grouping costs one wasted probe and a private executable
rather than a wrong replay. That makes it a cost question needing its own spec
and red-first evidence. "Unreachable with THIS key" is not "unreachable", and
AGENTS.md forbids declaring a ceiling.

Four honest gaps ride into every surface rather than into none: per-shape replay
counts are unavailable because the driver prints a total, so workload B's
~30-per-shape is arithmetic and is labelled as arithmetic; the driver's "N
captured size(s)" line counts slots, not captures; the container's own cuBLASLt
was never re-tested at CUDA 13.0 because the staged cu130 prefix was probed
first and worked, so the shim's original reason may no longer apply; and only
the Qwen3 dense decode driver was exercised. The supporting `orin:gpu0` lane is
BLOCKED, cleanly: the Jetson 540.4.0 driver cannot run a CUDA 13 runtime
(`cudaGetDeviceCount err=35`), and a CUDA 12.x toolkit is the untried route.

Checkers run from this worktree: `check-agent-record.py` OK (ENGINE=162
unchanged), `check-public-doc-tables.py` OK,
`check-issue-index-append-only.py --base origin/main --head HEAD` OK.
`agent-preflight.sh` was NOT run: it has been killed repeatedly on this box, so
the component checkers were run instead and are named here.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [Claude Code]
@localai-bot
localai-bot merged commit fe24a30 into main Aug 18, 2026
0 of 15 checks passed
@localai-bot
localai-bot deleted the row/ENG-CUDAGRAPH-DEDUP-RESULT branch August 18, 2026 12:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

VT_CUDA_GRAPH_DEDUP=1 dies after one replay: the signature walk latches a CUDA error that the next unrelated kernel reports as its own

2 participants