Claim
tests/parity/test_op_parity.cpp closes its walker's input set with ClassifyGoldenManifest plus a NonOpGoldenDirs() allowlist (landed in #755 / PR #761). The source comment at :1862 claims:
this set cannot rot into a mute exclusion list, and the cases >= N floors ...
That claim is false, and the floors bound nothing. On main today:
:2252 CHECK(op_goldens > 0); <- 65 manifests; passes with 1
:2330 CHECK(cases >= 46); <- only 46 of 65 op manifests reach ++cases on the CPU pass
:2333 CHECK(non_op >= 1);
:2819 CHECK(cases >= 26); <- CUDA pass, against ~61 actual
Nineteen op manifests continue before the counter under backend gates — qwen36_{embed,norm,gdn_layer,fullattn_layer,logits}_* (15), qwen3_5_mtp_head_* (2), gdn-packed-decode-oracle, qwen36_gguf_35b. For any of those, deleting the op key and adding the directory to NonOpGoldenDirs() removes it from gating while the suite stays green.
Proof
Demonstrated during the fresh review of PR #761 against qwen36_logits_35b — the full-model M0-exit gate. With its op key removed and its directory listed:
[doctest] test cases: 12 | 12 passed | 0 failed | 0 skipped
[doctest] assertions: 142 | 142 passed | 0 failed |
[doctest] Status: SUCCESS! exit 0
Byte-identical to the unmutated baseline. The only trace was a MESSAGE reading 64 op manifests, 2 declared non-op that no assertion reads — and a message no assertion reads is not a gate.
Why it is on main
The fresh review of PR #761 returned FAIL on exactly this finding. The repair was dispatched but its agent terminated on an API session limit before landing, and the PR merged with the finding unrepaired. The follow-up #853 fixed a different review finding (a throwing manifest aborting the pass); this one was not addressed.
To be fair to that change: the three other guards it added do work and were verified by mutation — an unlisted manifest with no op fails by name without aborting, a listed directory that does declare an op fails, and a non-string/null op (the shape that originally threw) is handled. The defect is confined to the floors.
What done looks like
The reviewer's one-line option is CHECK(op_goldens >= 65) at :2252. It closes the hole, but a hardcoded total in a shared test file makes that line a lock — every future PR adding a golden must edit it, and concurrent golden-adding rows will conflict there, which AGENTS.md's Records section explicitly warns against.
Prefer a summing invariant. Count every depth-1 manifest declaring a string op as op_goldens, and account for each one: it ran (cases), or it was skipped for a declared, named reason — a PendingRunnerOps() entry, a backend gate, a missing fixture. Assert the buckets sum to op_goldens. Then adding a golden needs no edit here, and deleting an op key changes op_goldens so the sum no longer balances.
If the skip reasons turn out not to be enumerable at that point in the code, say so with file:line and fall back to the exact count — but justify the fallback rather than defaulting to it.
The mutation that proves the fix is not "does the guard fire on garbage". It is "can I remove a REAL op golden from gating and stay green" — i.e. re-run the qwen36_logits_35b mutation above and require it to RED. Also fix the :1862 comment, which currently asserts the guarantee that does not hold.
Related: #755, #776, #853.
Claim
tests/parity/test_op_parity.cppcloses its walker's input set withClassifyGoldenManifestplus aNonOpGoldenDirs()allowlist (landed in #755 / PR #761). The source comment at:1862claims:That claim is false, and the floors bound nothing. On
maintoday:Nineteen op manifests
continuebefore the counter under backend gates —qwen36_{embed,norm,gdn_layer,fullattn_layer,logits}_*(15),qwen3_5_mtp_head_*(2),gdn-packed-decode-oracle,qwen36_gguf_35b. For any of those, deleting theopkey and adding the directory toNonOpGoldenDirs()removes it from gating while the suite stays green.Proof
Demonstrated during the fresh review of PR #761 against
qwen36_logits_35b— the full-model M0-exit gate. With itsopkey removed and its directory listed:Byte-identical to the unmutated baseline. The only trace was a
MESSAGEreading64 op manifests, 2 declared non-opthat no assertion reads — and a message no assertion reads is not a gate.Why it is on main
The fresh review of PR #761 returned FAIL on exactly this finding. The repair was dispatched but its agent terminated on an API session limit before landing, and the PR merged with the finding unrepaired. The follow-up #853 fixed a different review finding (a throwing manifest aborting the pass); this one was not addressed.
To be fair to that change: the three other guards it added do work and were verified by mutation — an unlisted manifest with no
opfails by name without aborting, a listed directory that does declare anopfails, and a non-string/nullop(the shape that originally threw) is handled. The defect is confined to the floors.What done looks like
The reviewer's one-line option is
CHECK(op_goldens >= 65)at:2252. It closes the hole, but a hardcoded total in a shared test file makes that line a lock — every future PR adding a golden must edit it, and concurrent golden-adding rows will conflict there, which AGENTS.md's Records section explicitly warns against.Prefer a summing invariant. Count every depth-1 manifest declaring a string
opasop_goldens, and account for each one: it ran (cases), or it was skipped for a declared, named reason — aPendingRunnerOps()entry, a backend gate, a missing fixture. Assert the buckets sum toop_goldens. Then adding a golden needs no edit here, and deleting anopkey changesop_goldensso the sum no longer balances.If the skip reasons turn out not to be enumerable at that point in the code, say so with
file:lineand fall back to the exact count — but justify the fallback rather than defaulting to it.The mutation that proves the fix is not "does the guard fire on garbage". It is "can I remove a REAL op golden from gating and stay green" — i.e. re-run the
qwen36_logits_35bmutation above and require it to RED. Also fix the:1862comment, which currently asserts the guarantee that does not hold.Related: #755, #776, #853.