Skip to content

feat(launchpad): adjudication fixtures, four of five genuinely produced (#118 STEP 8) - #450

Merged
serina-mcfall merged 2 commits into
feat/review-agent-adjudication-dedupefrom
feat/review-agent-adjudication-fixtures
Aug 24, 2026
Merged

feat(launchpad): adjudication fixtures, four of five genuinely produced (#118 STEP 8)#450
serina-mcfall merged 2 commits into
feat/review-agent-adjudication-dedupefrom
feat/review-agent-adjudication-fixtures

Conversation

@serina-mcfall

Copy link
Copy Markdown

Related issue

Refs #118

Issue type

Task

#118 STEP 8 — adjudication fixtures, four of five genuinely produced

Implements STEP 8 of launchpad/plans/2026-08-13-issue-118-adjudication.md, stacked on the not-yet-merged STEP 7 branch (#267). Adds launchpad/review-agent/fixtures/adjudication/ (a generator, four fixture documents, a crafted-surfaces payload, and PROVENANCE.md), plus test_adjudication_fixtures.py — and corrects STEP 8's own text in the plan.

STEP 8's premise was dead, and correcting it was part of the work

The step's text said:

"THESE ARE SYNTHESISED, NOT RECORDED, AND THAT IS A KNOWN WEAKNESS. #117's producer does not exist — there is no run_dimensions.py anywhere... Until then the suite's coverage of real producer output is zero."

None of that has been true since #117 merged. The plan's own BUDGET section already anticipated this and instructed that "if their text still describes document-only synthesis, that text is what needs updating, not this correction." So it is corrected here — struck through rather than deleted, matching the convention used throughout the plan's ALREADY TRUE and BUDGET sections.

What is real, measured before building

fixture isolates provenance
line-anchored-findings.json three reports, one finding per dimension, all anchor line and the dedupe case real — replays recordings/paraphrase/*
pr-anchored-finding.json a pr-anchored finding (file/line null) alongside a line-anchored one real — replays recordings/claim-vs-evidence/*
mixed-report-statuses.json one failed, one clean, one with findings real — replays recordings/secrets-and-access/*, failure raised through the real handler
containment-all-kinds.json all three containment kinds, seven-key states, zero dimension findings crafted surfaces, real pipeline

Two things worth drawing out:

paraphrase is a natural dedupe fixture. All three dimensions independently reported the same defect at crates/buzz-relay/src/gate.rs:42, with three different finding_ids (dimension is a hash input). That is STEP 7's dedupe case occurring in real recorded output rather than anything hand-built — so one document legitimately isolates two of STEP 8's named behaviours. Committing it twice under two names, or inventing a second document, were the dishonest alternatives.

The containment fixture is the one genuine exception, and the split is stated rather than blurred. No pre-existing fixture trips the detectors — all eight were checked (benign.json, captured-pr.json, payloads.json, and all five under fixtures/dimensions/), every one yielding zero containment findings. So its surfaces are crafted; its containment block and states map are then produced by the real contain.render. Crafted input, real pipeline — never described as recorded.

The claim is checkable, not asserted

Regenerating reproduces the committed bytes exactly. Every nonce derives from the relevant recording's own _provenance.seed via contain.make_nonce(seed=...), never freshly randomised. That is what turns "these came from real recorded output" into something a reviewer can falsify rather than take on trust.

Verification

  • 18 new tests; run_controls.py 13/13; all eight suites green
  • Each fixture accepted by findings.validate with zero violations, and by run_adjudication.py at exit 0 with output passing both verdicts.validate and findings.validate
  • Containment fixture: all three kinds, exactly seven states keys matching contain.ENTRY_POINTS

Reviewed (serina:review-code, independent pass) — it attacked the central claim rather than accepting it: traced individual findings byte-for-byte from their recording files into the committed fixtures, confirmed the failed report's error.reason matches _collect_report's real exception branch, independently re-ran the eight-fixture containment check, and verified byte-reproducibility out-of-band with its own diff. No findings.

One item it raised as context rather than a finding is fixed in 888001977: the generator cited "Serina's call recorded in this repo's session notes" — a source no reader can open. Replaced with the argument made from what the recording actually contains. This plan carries a standing rule that evidence is what git show returns from a named commit, adopted after three separate rounds of cross-issue citation failures; a pointer to out-of-repo notes is weaker than all three.

Draft, per this repo's pr-gate hook — review-final runs at STEP 12, end of the chain.

🤖 Drafted by Claude Code for @serina-mcfall.

…ed (#118 STEP 8)

STEP 8's stated premise was dead and is corrected in the plan (struck
through, not deleted) exactly as its own BUDGET section instructed:
"#117's producer does not exist -- there is no run_dimensions.py
anywhere" has not been true since #117 merged. Fifteen real recorded
reviewer outputs live under recordings/, and test_recordings.py's own
ReplayValidityTests already replayed them through build_document, so the
harness this step needed was already in the tree.

Measured before building, not assumed. Four of the five named behaviours
are now genuinely produced by replaying real recorded output through the
real producer:

- paraphrase gives BOTH the three-report all-anchor-line document and the
  dedupe document -- all three dimensions independently reported the same
  defect at gate.rs:42 with three different finding_ids, since dimension
  is a hash input. One document isolates both behaviours; committing it
  twice, or inventing a second, would be the dishonest alternatives.
- claim-vs-evidence gives the pr-anchored fixture. Its real output is two
  findings (anchors line and pr); kept whole per Serina's call rather
  than trimmed to the single-finding shape the plan first described,
  since a trim would be a real replay with a finding deleted by hand.
- secrets-and-access plus a reviewer injected to raise gives the
  failed/clean/findings document -- the failed report comes from
  _collect_report/_failed_report, not from hand-written JSON.

The containment fixture is the one genuine exception and the split is
stated rather than blurred: no existing fixture trips the detectors (all
eight checked, every one yields zero containment findings), so its
surfaces are crafted and its containment block and states map are then
produced by the real contain.render. Crafted input, real pipeline --
never described as recorded.

Regenerating reproduces the committed bytes exactly, which is what makes
the provenance claim checkable rather than asserted: every nonce derives
from the relevant recording's own _provenance.seed via
contain.make_nonce(seed=...), never freshly randomised.

Verified: 18 new tests OK; each fixture accepted by findings.validate
with zero violations; each accepted by run_adjudication.py at exit 0;
containment fixture carries all three kinds and exactly seven states
keys; regeneration byte-identical; run_controls 13/13 and all eight
suites OK.

Signed-off-by: Serina Mcfall <serina.mcfall@gmail.com>
review-code flagged, as context rather than a finding, that generate.py
attributed the keep-the-two-finding-document decision to "Serina's call
recorded in this repo's session notes" -- and no such artifact exists in
the repo to check it against.

That is precisely the uncheckable citation this plan's own conventions
exist to prevent: its ALREADY TRUE section carries a standing rule that
evidence is what `git show` returns from a named commit, adopted after
three separate rounds of cross-issue claims that were true-then-
falsified, never-true, and true-but-misattributed. A pointer to
out-of-repo session notes is weaker than all three.

Replaced with the argument itself, made from what the recording actually
contains: trimming would turn a real replay into a real replay with a
finding deleted by hand, and the two-finding document exercises a 'pr'
anchor alongside a 'line' one rather than in isolation. That reasoning
is checkable against the fixture; the attribution was not.

Regenerated so the committed fixture carries the corrected note.
18 tests OK (including the byte-reproducibility check), controls 13/13.

Signed-off-by: Serina Mcfall <serina.mcfall@gmail.com>
@serina-mcfall
serina-mcfall marked this pull request as ready for review August 22, 2026 03:19
@serina-mcfall
serina-mcfall merged commit 177643d into feat/review-agent-adjudication-dedupe Aug 24, 2026
26 checks passed
@serina-mcfall
serina-mcfall deleted the feat/review-agent-adjudication-fixtures branch August 24, 2026 00:34
serina-mcfall added a commit that referenced this pull request Aug 24, 2026
PROVENANCE.md claimed byte reproducibility was "what makes 'real' and
'crafted surfaces, real pipeline' checkable claims rather than
assertions". It is not, on its own: it compares generate.py against
itself. Rewrite the generator to type finding content by hand,
regenerate, commit, and the new bytes agree with the new generator
perfectly while `real: true` and `source_recordings` have quietly become
false. Demonstrated against this directory, not imagined -- a generator
reading nothing under recordings/ passed all 18 tests.

The claim is true today: every replayed report's findings and outcome are
equal to the recording it names, checked across all 8 replayed reports,
0 mismatches. That is exactly why the guard is cheap to add now and
expensive to add after STEP 9 records judge outputs against these
fixtures.

Adds RealDocumentsReplayTheirNamedRecordingsTests, which reads the
recordings rather than the generator:

  - every replayed report's `findings` and `outcome` must EQUAL the
    recording named in `_fixture.source_recordings`
  - every real document's `nonce` must be `contain.make_nonce(seed=...)`
    over a seed that recording actually records -- the one field a
    fabricated generator cannot get right by copying shapes, since
    inventing a seed changes it
  - named recordings must exist, and must actually be replayed, so
    provenance cannot inflate in either direction
  - a guard on the guard: zero real documents, or zero comparisons,
    fails rather than passing vacuously

Adds ProvenanceNoteRecordsTheHonestySplitTests, for the same reason
test_recordings.py pins #117's sampling disclosure: the old
test_provenance_note_exists was os.path.isfile only, so truncating
PROVENANCE.md to a single header line -- taking the whole real-versus-
crafted accounting with it -- stayed green.

Proved each guard can fail, in scratch copies:

  - fabricated seed, fixtures regenerated: 3 failures, one per real
    document, while byte reproducibility still PASSED -- the blind spot
    stated plainly
  - fabricated finding content, fixtures regenerated: 5 failures, one
    per replayed report
  - PROVENANCE.md truncated to "# PROVENANCE": old test still passes,
    new pin fails 8 times

Also corrects two sentences that asserted adjudicator behaviour the code
does not produce. The test comment and line-anchored-findings.json's own
note said the document "exercises adjudicate()'s dedupe path"; the
default stub_dedupe_judge groups nothing by design, so the real CLI emits
`duplicate_groups: []` with every `duplicate_of` null. It is a dedupe
CANDIDATE -- three distinct finding_ids describing one defect at
crates/buzz-relay/src/gate.rs:42 -- and asserting on the grouped output
is STEP 10's remit, not this fixture's.

257 tests OK (was 250), run_controls.py 13 passed 0 failed 0 skipped.
Three of the four fixtures regenerated byte-identically; only the one
whose note changed differs.

Reviewed by review-code, review-tests and review-adjudicate before this
commit. PR #450 merged STEP 8 with zero reviews; these were its first.
All three passes were Claude, so this is not the cross-model review the
final gate wants.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: Serina Mcfall <serina.mcfall@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant