feat(launchpad): adjudication fixtures, four of five genuinely produced (#118 STEP 8) - #450
Merged
serina-mcfall merged 2 commits intoAug 24, 2026
Conversation
…ed (#118 STEP 8) STEP 8's stated premise was dead and is corrected in the plan (struck through, not deleted) exactly as its own BUDGET section instructed: "#117's producer does not exist -- there is no run_dimensions.py anywhere" has not been true since #117 merged. Fifteen real recorded reviewer outputs live under recordings/, and test_recordings.py's own ReplayValidityTests already replayed them through build_document, so the harness this step needed was already in the tree. Measured before building, not assumed. Four of the five named behaviours are now genuinely produced by replaying real recorded output through the real producer: - paraphrase gives BOTH the three-report all-anchor-line document and the dedupe document -- all three dimensions independently reported the same defect at gate.rs:42 with three different finding_ids, since dimension is a hash input. One document isolates both behaviours; committing it twice, or inventing a second, would be the dishonest alternatives. - claim-vs-evidence gives the pr-anchored fixture. Its real output is two findings (anchors line and pr); kept whole per Serina's call rather than trimmed to the single-finding shape the plan first described, since a trim would be a real replay with a finding deleted by hand. - secrets-and-access plus a reviewer injected to raise gives the failed/clean/findings document -- the failed report comes from _collect_report/_failed_report, not from hand-written JSON. The containment fixture is the one genuine exception and the split is stated rather than blurred: no existing fixture trips the detectors (all eight checked, every one yields zero containment findings), so its surfaces are crafted and its containment block and states map are then produced by the real contain.render. Crafted input, real pipeline -- never described as recorded. Regenerating reproduces the committed bytes exactly, which is what makes the provenance claim checkable rather than asserted: every nonce derives from the relevant recording's own _provenance.seed via contain.make_nonce(seed=...), never freshly randomised. Verified: 18 new tests OK; each fixture accepted by findings.validate with zero violations; each accepted by run_adjudication.py at exit 0; containment fixture carries all three kinds and exactly seven states keys; regeneration byte-identical; run_controls 13/13 and all eight suites OK. Signed-off-by: Serina Mcfall <serina.mcfall@gmail.com>
review-code flagged, as context rather than a finding, that generate.py attributed the keep-the-two-finding-document decision to "Serina's call recorded in this repo's session notes" -- and no such artifact exists in the repo to check it against. That is precisely the uncheckable citation this plan's own conventions exist to prevent: its ALREADY TRUE section carries a standing rule that evidence is what `git show` returns from a named commit, adopted after three separate rounds of cross-issue claims that were true-then- falsified, never-true, and true-but-misattributed. A pointer to out-of-repo session notes is weaker than all three. Replaced with the argument itself, made from what the recording actually contains: trimming would turn a real replay into a real replay with a finding deleted by hand, and the two-finding document exercises a 'pr' anchor alongside a 'line' one rather than in isolation. That reasoning is checkable against the fixture; the attribution was not. Regenerated so the committed fixture carries the corrected note. 18 tests OK (including the byte-reproducibility check), controls 13/13. Signed-off-by: Serina Mcfall <serina.mcfall@gmail.com>
serina-mcfall
marked this pull request as ready for review
August 22, 2026 03:19
serina-mcfall
merged commit Aug 24, 2026
177643d
into
feat/review-agent-adjudication-dedupe
26 checks passed
17 tasks
serina-mcfall
added a commit
that referenced
this pull request
Aug 24, 2026
PROVENANCE.md claimed byte reproducibility was "what makes 'real' and
'crafted surfaces, real pipeline' checkable claims rather than
assertions". It is not, on its own: it compares generate.py against
itself. Rewrite the generator to type finding content by hand,
regenerate, commit, and the new bytes agree with the new generator
perfectly while `real: true` and `source_recordings` have quietly become
false. Demonstrated against this directory, not imagined -- a generator
reading nothing under recordings/ passed all 18 tests.
The claim is true today: every replayed report's findings and outcome are
equal to the recording it names, checked across all 8 replayed reports,
0 mismatches. That is exactly why the guard is cheap to add now and
expensive to add after STEP 9 records judge outputs against these
fixtures.
Adds RealDocumentsReplayTheirNamedRecordingsTests, which reads the
recordings rather than the generator:
- every replayed report's `findings` and `outcome` must EQUAL the
recording named in `_fixture.source_recordings`
- every real document's `nonce` must be `contain.make_nonce(seed=...)`
over a seed that recording actually records -- the one field a
fabricated generator cannot get right by copying shapes, since
inventing a seed changes it
- named recordings must exist, and must actually be replayed, so
provenance cannot inflate in either direction
- a guard on the guard: zero real documents, or zero comparisons,
fails rather than passing vacuously
Adds ProvenanceNoteRecordsTheHonestySplitTests, for the same reason
test_recordings.py pins #117's sampling disclosure: the old
test_provenance_note_exists was os.path.isfile only, so truncating
PROVENANCE.md to a single header line -- taking the whole real-versus-
crafted accounting with it -- stayed green.
Proved each guard can fail, in scratch copies:
- fabricated seed, fixtures regenerated: 3 failures, one per real
document, while byte reproducibility still PASSED -- the blind spot
stated plainly
- fabricated finding content, fixtures regenerated: 5 failures, one
per replayed report
- PROVENANCE.md truncated to "# PROVENANCE": old test still passes,
new pin fails 8 times
Also corrects two sentences that asserted adjudicator behaviour the code
does not produce. The test comment and line-anchored-findings.json's own
note said the document "exercises adjudicate()'s dedupe path"; the
default stub_dedupe_judge groups nothing by design, so the real CLI emits
`duplicate_groups: []` with every `duplicate_of` null. It is a dedupe
CANDIDATE -- three distinct finding_ids describing one defect at
crates/buzz-relay/src/gate.rs:42 -- and asserting on the grouped output
is STEP 10's remit, not this fixture's.
257 tests OK (was 250), run_controls.py 13 passed 0 failed 0 skipped.
Three of the four fixtures regenerated byte-identically; only the one
whose note changed differs.
Reviewed by review-code, review-tests and review-adjudicate before this
commit. PR #450 merged STEP 8 with zero reviews; these were its first.
All three passes were Claude, so this is not the cross-model review the
final gate wants.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: Serina Mcfall <serina.mcfall@gmail.com>
This was referenced Aug 24, 2026
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Related issue
Refs #118
Issue type
Task
#118 STEP 8 — adjudication fixtures, four of five genuinely produced
Implements STEP 8 of
launchpad/plans/2026-08-13-issue-118-adjudication.md, stacked on the not-yet-merged STEP 7 branch (#267). Addslaunchpad/review-agent/fixtures/adjudication/(a generator, four fixture documents, a crafted-surfaces payload, andPROVENANCE.md), plustest_adjudication_fixtures.py— and corrects STEP 8's own text in the plan.STEP 8's premise was dead, and correcting it was part of the work
The step's text said:
None of that has been true since #117 merged. The plan's own BUDGET section already anticipated this and instructed that "if their text still describes document-only synthesis, that text is what needs updating, not this correction." So it is corrected here — struck through rather than deleted, matching the convention used throughout the plan's ALREADY TRUE and BUDGET sections.
What is real, measured before building
line-anchored-findings.jsonlineand the dedupe caserecordings/paraphrase/*pr-anchored-finding.jsonpr-anchored finding (file/line null) alongside aline-anchored onerecordings/claim-vs-evidence/*mixed-report-statuses.jsonrecordings/secrets-and-access/*, failure raised through the real handlercontainment-all-kinds.jsonstates, zero dimension findingsTwo things worth drawing out:
paraphraseis a natural dedupe fixture. All three dimensions independently reported the same defect atcrates/buzz-relay/src/gate.rs:42, with three differentfinding_ids (dimensionis a hash input). That is STEP 7's dedupe case occurring in real recorded output rather than anything hand-built — so one document legitimately isolates two of STEP 8's named behaviours. Committing it twice under two names, or inventing a second document, were the dishonest alternatives.The containment fixture is the one genuine exception, and the split is stated rather than blurred. No pre-existing fixture trips the detectors — all eight were checked (
benign.json,captured-pr.json,payloads.json, and all five underfixtures/dimensions/), every one yielding zero containment findings. So its surfaces are crafted; its containment block andstatesmap are then produced by the realcontain.render. Crafted input, real pipeline — never described as recorded.The claim is checkable, not asserted
Regenerating reproduces the committed bytes exactly. Every nonce derives from the relevant recording's own
_provenance.seedviacontain.make_nonce(seed=...), never freshly randomised. That is what turns "these came from real recorded output" into something a reviewer can falsify rather than take on trust.Verification
run_controls.py13/13; all eight suites greenfindings.validatewith zero violations, and byrun_adjudication.pyat exit 0 with output passing bothverdicts.validateandfindings.validatestateskeys matchingcontain.ENTRY_POINTSReviewed (
serina:review-code, independent pass) — it attacked the central claim rather than accepting it: traced individual findings byte-for-byte from their recording files into the committed fixtures, confirmed the failed report'serror.reasonmatches_collect_report's real exception branch, independently re-ran the eight-fixture containment check, and verified byte-reproducibility out-of-band with its owndiff. No findings.One item it raised as context rather than a finding is fixed in
888001977: the generator cited "Serina's call recorded in this repo's session notes" — a source no reader can open. Replaced with the argument made from what the recording actually contains. This plan carries a standing rule that evidence is whatgit showreturns from a named commit, adopted after three separate rounds of cross-issue citation failures; a pointer to out-of-repo notes is weaker than all three.Draft, per this repo's
pr-gatehook —review-finalruns at STEP 12, end of the chain.🤖 Drafted by Claude Code for @serina-mcfall.