feat(launchpad): adjudicator.md -- what the judge is told (#118 STEP 5)

[serina-mcfall]

Summary

Implements STEP 5 of launchpad/plans/2026-08-13-issue-118-adjudication.md: adjudicator.md, the document stating what the judge is told. Depends only on STEP 1 (ADJUDICATION.md, merged via #256), so it branches directly off launchpad rather than stacking on the in-flight STEPs 2-4. Docs-only.

Related issue

Refs #118

Issue type

Task

---

Agent provenance

Field	Value
Harness / provider	Claude Code
Model	claude-opus-5
Session reference	N/A - the harness exposes no shareable run id
Initiating human	@serina-mcfall

Objective

Add launchpad/review-agent/adjudicator.md — a prompt-definition file, sibling to ADJUDICATION.md/FINDINGS.md/CONTAINMENT.md and in the same normative voice.

Impacted components

• launchpad/review-agent/adjudicator.md

Approach and rejected alternatives

States the judge's scope and four prohibitions: do not restate a claim as its own evidence, do not hunt for new defects, do not emit approval-shaped output, and do not refute for want of evidence (UNPROVEN is the fail-closed default). Includes a worked contrast between a restatement and an independent check, because the distinction is the one a reader is most likely to get wrong, and states that anchor pr (file and line both null) is a legitimate finding shape rather than malformed.

Rejected: building a behavioural test that the prohibitions hold under a live model. Whether an exclusion clause survives contact with a real adjudicator is a property of output, which needs STEP 9's recorded before/after pairs — and tagging this step against STEP 9 would make the dependency circular. This step's own plan text says its done-when is deliberately textual, and no harness was expected here.

Rejected on the notes prohibition: leaving it as originally written. See Escalations — it mandated a channel the runner discards.

Verification

Command run:

cd launchpad/review-agent
python3 -m unittest discover -s . -p 'test_*.py'

Raw output:

.........................................................................................................................
----------------------------------------------------------------------
Ran 121 tests in 0.585s

OK

Docs-only, so that run confirms nothing regressed rather than proving anything about this diff — nothing in it touches launchpad/review-agent/*.py.

The notes claim this PR now corrects, probed on the STEP 3 branch where the runner lives — a judge returning a notes key alongside a valid verdict:

adjudication.notes: []
verdicts.validate: 0 violations
findings.validate: 0 violations

The value is discarded and both contract checks pass, so nothing catches the loss.

• Tests or checks were run and the raw output is pasted above
• The diff is confined to the scope of the linked issue
• No secrets, keys, tokens or hostnames were added to tracked files

Not verified

Nothing here is verified against a live model, by design. Every claim in this document is about what a judge is instructed to do; whether an instruction changes model behaviour is unmeasured and unmeasurable until STEP 9 exists. Treat this as a specification, not as evidence of behaviour.

The worked restatement-vs-independent-check contrast is illustrative, not sampled. It was written to make the distinction legible, not drawn from real adjudicator output — so it may not resemble the failure mode a real judge actually exhibits.

The four prohibitions have no mechanical enforcement. Prohibition 3 (no approval-shaped output) is the only one with any: verdicts.py's _FORBIDDEN_KEYS walk. And that walk lives in verdicts.validate, which neither adjudicate() nor main() calls on its own output before printing — so it is a validator a downstream caller may run, not an enforcement in the producer. The conclusion still holds (the runner writes only known keys, so it cannot produce a forbidden one), but "no such key can exist" would be stronger than the code supports. Prohibitions 1, 2 and 4 are textual only.

Security implications

Adds one markdown file; no change to exposure. What it records is security-relevant: prohibition 3 is the document half of "escalate, never approve", which is the property that keeps an agent from approving work — forbidden by launchpad/AGENTS.md rule 1. Prohibition 4 makes UNPROVEN the default, so a judge that cannot establish something escalates rather than clearing it.

The notes amendment closes a small integrity gap rather than opening one. Instructing a judge to write into a channel that silently discards its input meant a genuine new observation had exactly one remaining outlet — verdict_evidence, the field with no structural guard. The amendment says so, and says not to stretch that field instead.

Escalations

1. This PR's central claim was wrong as originally written, and is corrected here. Prohibition 2 told the judge to "record it in adjudication.notes". ADJUDICATION.md declares that field and verdicts.py carries it, which made the instruction look supported — but run_adjudication.py hardcodes notes=[] and its judge protocol never reads the key. Adjudicated across feat(launchpad): run_adjudication.py -- the adjudication CLI (#118 STEP 3) #263 and feat(launchpad): dedupe via an injectable second judge (#118 STEP 7) #267 as one cross-step drift. Resolved on this side rather than by plumbing notes through the protocol, because collecting and attributing notes is STEP 6/7's design and building it here would pre-empt a decision those steps own. Decision taken by @serina-mcfall.
2. A new observation now has nowhere to go from this stage. That is the honest consequence of deferring the channel, and it is stated in the document rather than papered over. If it turns out to matter before STEP 6/7, the fix is to plumb notes, not to relax prohibition 2.
3. The run_adjudication.py citation is pinned to 2be90c629 rather than to a branch name, after a Low finding on the citation-rot risk. That SHA is on feat(launchpad): nonce check and stages manifest (#118 STEP 4) #264, which is not merged — so the citation will need re-pinning if feat(launchpad): nonce check and stages manifest (#118 STEP 4) #264's history is rewritten.

🤖 Drafted by Claude Code (claude-opus-5) for @serina-mcfall.

[ciaran-slow]

Review pipeline — PR #265

Stages run: review-code, review-tests, review-a11y, review-adjudicate, review-final.
Plan read first: launchpad/plans/2026-08-13-issue-118-adjudication.md, STEP 5 at :565.

Not applicable, declared rather than faked:

• review-a11y — one markdown document. The plan's own LEFT OUT section says accessibility is out of scope for task: adjudication pass over every reported finding #118 and states why: "The deliverable is a definition plus a CLI printing JSON to stdout — no UI, no interactive control, nothing to announce, no focus to manage." I agree and am not manufacturing a finding against it.
• review-tests — no test files in the diff. STEP 5's done-when is textual on purpose and the document says so at :10-15, with the behavioural proof correctly deferred to STEP 9's recordings. The plan explains why tagging this step against 9 would be circular. Reporting "no tests" here would be reporting the plan's own deliberate design.
• check-ledger.sh — the plan uses STEP N headings, not ### Task N:, and there is no .superpowers/sdd/ ledger; the checker exits 1 on its vacuity guard. I walked the step graph by hand.

STEP 5's done-when, clause by clause

Clause	Result
the file exists	✅ launchpad/review-agent/adjudicator.md, 133 lines
states all four exclusions with the reason for each	✅ :34, :61, :74, :89 — each with its own rationale, not a bare rule
states UNPROVEN as the answer to insufficient evidence, in those words	✅ :91 — "Not-established is UNPROVEN, never REFUTED"
evidence must be the adjudicator's own, with a worked contrast	✅ :41-59
states anchor pr is legitimate	✅ :99-123
names no model	✅ grepped for claude/gpt/sonnet/opus/haiku/gemini/llama/anthropic/openai — no match

All six met. I checked each mechanically rather than reading for impression.

---

Findings

1. High — the document normatively directs the judge to write to adjudication.notes, and no code path lets it

launchpad/review-agent/adjudicator.md:66 against launchpad/review-agent/run_adjudication.py:864 (on the #263→#267 chain)

Exclusion 2 is unambiguous: "If you notice a genuinely new defect while adjudicating one that was reported, it does not become a finding: record it in adjudication.notes and never in reports[].findings." The reasoning is right, and the field is real — ADJUDICATION.md:137 declares it ("array of free-text notes") and verdicts.py:93 carries it as a dataclass field with a default_factory=list.

The problem is the plumbing. The judge protocol enumerates exactly which keys it honours, and notes is not one of them (run_adjudication.py:226-234):

the judge protocol: judge(finding, input_document) -> dict with at least {"verdict": ..., "verdict_evidence": ...}, and MAY also carry severity (STEP 6's re-rating) plus, when re-rating, severity_reason

_run_judge_safely forwards verdict, verdict_evidence, severity and severity_reason and nothing else. And notes=[] at :864 is the only assignment to that field anywhere in the runner — it is hardcoded empty on every run.

Concrete failure: a live judge follows exclusion 2 exactly as written. It notices a real new defect while adjudicating a reported one, and tries to record it where the document tells it to. The runner discards it — there is no key for it to arrive in. The judge's only remaining outlet is verdict_evidence, which is the field exclusion 1 exists to keep free of anything but independent evidence for that specific finding, and which the plan's own OPEN item names as the escalate-only hole with no mechanical guard:

"an approving sentence inside verdict_evidence publishes as this stage's own words… a keyword filter over model prose is the narrow guard this agent exists to find elsewhere."

So the instruction, followed faithfully, pushes free text into precisely the field that has no guard — and the document's own exclusion 3 is the only thing standing there. Nothing is wrong today because stub_judge never emits a note; this becomes live the moment STEP 9's recordings or any real model runs, and STEP 9 is the next planned step.

The contrast makes it clear this is an oversight rather than a decision. The runner plumbs severity and severity_reason carefully — :73-76 even generates a default reason when the judge omits one, because verdicts.validate requires it. notes got the field and the documentation but not the wire.

Fix, either end: add notes to the judge protocol and append what a judge returns to Adjudication.notes — a few lines, symmetric with how severity_reason is already handled — or, if the channel is genuinely deferred, say so in adjudicator.md:66 so a judge is not instructed to use a route that does not exist. The first is better; the second is honest.

Reported here rather than on #267 because the instruction is what makes it wrong, and this is the PR that introduces the instruction. Counted once. I have cross-referenced it from #267's review rather than filing it twice.

2. Low — missing by:agent label

No labels on this PR. Its body should carry an Agent provenance block, and launchpad/AGENTS.md §5 rule 3 requires by:agent on every agent-authored PR. Of the nine open PRs only #262 has it.

gh pr edit 265 --repo launchpad-26/buzz --add-label by:agent

---

What I looked for and did not find

The plan's own history made two checks worth more than the rest here, and both came back clean.

• The commit citation at :113-114 is accurate. This is the check I expected to fail, because the plan records getting this exact class of claim wrong three times and derived a standing rule from it: "the evidence is what git show returns from a named commit, and the citation carries that commit." The document cites 2be90c629 for _location_description branching on anchor before touching file/line. Verified: git cat-file -t 2be90c629 → commit; git merge-base --is-ancestor 2be90c629 pr/264 → true; and git show 2be90c629:launchpad/review-agent/run_adjudication.py puts _location_description at :268 with anchor = finding.get("anchor") / if anchor == "pr": return "the whole pull request (no file or line anchor)" at :275-277 — anchor first, file and line never read on that branch, exactly as claimed. The citation-rot rule was followed, including naming the commit rather than the branch and saying why.
• Every sibling reference resolves. ADJUDICATION.md, FINDINGS.md and CONTAINMENT.md are all present on launchpad. I also checked the specific claim at :76 — that ADJUDICATION.md's "Escalate, never approve" section states three concrete prohibitions — against ADJUDICATION.md:88-94, which does exactly that, including approved / mergeable / verdict: OK by name.
• A severity_reason gap I expected and did not find. The document mentions severity_reason only in passing (:81) and never tells the judge that a re-rating requires one — which would have let a compliant judge produce output verdicts.validate rejects. The runner closes it: :73-76 supplies "the judge's own if it gave one, else a generated default, since verdicts.validate requires a reason whenever severity != reported_severity regardless of direction." Not a finding.
• An OPEN item answered sideways. The plan reserves whether verdict_evidence's free text needs a second mechanism. Exclusion 3 at :83-87 restates that exposure and explicitly declines to solve it — "a control that grepped free text for approving phrasing would be a keyword filter, the kind of narrow guard the review dimensions exist to find in other people's code, not a fix this stage can rely on." It leaves the decision open rather than quietly taking it. That is the right handling and I checked for the opposite.
• Scope creep into other steps. None. The document defines instructions and nothing else; it does not specify the schema (STEP 1/2's job) or the runner's behaviour.
• Injection surface. The document is a prompt, and it instructs the judge to read the repository at head_sha — the change as code — never the author's prose as a fresh surface, consistent with CONTAINMENT.md's prohibition and the plan's LEFT OUT.

Merge readiness

A reader of #118 STEP 5 would find every clause of its done-when met, with the reasoning for each exclusion stated rather than asserted. The worked contrast at :41-59 is the strongest part of the document: it distinguishes restatement from independent evidence by a criterion that is actually operable — "whether the evidence could have been written by someone who never opened the file" — rather than by exhortation.

One merge-order fact, not a finding. This PR's base is launchpad, and run_adjudication.py does not exist there — I verified its absence. The document cites that file at :112-118 and discloses the situation honestly ("the unmerged sibling branch… PR #264", "still under active revision until it merges"), so I am not reporting it as an undisclosed dangling reference. But if #265 merges ahead of #261→#263→#264, launchpad carries a normative document citing a function no file on launchpad defines, and a reader grepping for _location_description cannot tell whether the rule is enforced or aspirational. Landing the chain first, or #265 last, costs nothing and avoids that window.

Finding 1 is the substantive one, and its fix most likely belongs in the runner rather than in this document.

What I could not check: whether these clauses change what a model actually produces. That is STEP 9's job by the plan's own design, and the document says so itself. I read wording, not behaviour, and no part of this review claims otherwise.

Independence and tools

Independent of the work under review: I did not write it. Not independent across pipeline stages — one context ran the reviewers, the adjudicator and the final pass, where the skills call for a fresh context per stage. Both findings are self-adjudicated. Treat that as a limit on this report.

Tools actually held and used: Bash (git, git grep, git cat-file, git merge-base, gh, gh api), Read, Edit, Write. No Grep or Glob tool was available in this session.

Nothing found at Blocker.

CONFIRMED	High	launchpad/review-agent/adjudicator.md:66	directs the judge to adjudication.notes; run_adjudication.py:864 hardcodes it empty and the judge protocol has no such key
CONFIRMED	Low	PR #265 (labels)	missing required by:agent label

Handed 2 findings, confirmed 2, refuted 0, merged 0 — finding 1 spans this PR and #267 and is counted once, here. No reviewer report arrived without its REVIEW COMPLETE marker, because all stages ran in one context; stated as a limit, not a pass. I did not author any of the work under review.

ADJUDICATION COMPLETE

REVIEW COMPLETE

---

Per launchpad/AGENTS.md §5 rule 1 — an agent drafts and raises, never approves or clears. This is a report, not an approval; the merge decision is @ciaran-slow's.

[ciaran-slow]

Comment review recording the pipeline result. Not an approval and not a change-request — the merge decision is unchanged by this.

Reviewed via the full pipeline — detail in my comment on this PR.

The document itself needs no change. I checked all six of STEP 5's done-when clauses mechanically and every one is met: four exclusions each with its reason, UNPROVEN stated in those words, a worked restatement-vs-independent-check contrast, anchor pr declared legitimate, and no model named anywhere (grepped for every vendor and family). The worked contrast at :41-59 is the strongest part — it distinguishes restatement from real evidence by a criterion that is actually operable: "whether the evidence could have been written by someone who never opened the file."

I also expected the commit citation to be stale and it is not. 2be90c629 exists, is an ancestor of #264, and _location_description at that commit does branch on anchor before touching file/line, exactly as claimed. Given the plan records getting this class of claim wrong three times and deriving a standing rule from it, that is worth naming.

Two holds, neither a defect in this diff: a dependency and a merge-order constraint.

1. :66 is contingent on #263. It normatively instructs the judge to record new observations in adjudication.notes, and the runner provides no channel — notes=[] is hardcoded and the judge protocol has no such key. Probed: a judge returning notes has it silently dropped, leaving verdict_evidence as its only outlet, which is the field the plan's OPEN item flags as unguarded. If #263 gains the wire, this line is correct as written and needs nothing. If the channel is deferred instead, this line needs to say so. Either way the resolution happens on #263, so this PR should not merge ahead of that decision.

2. Merge after #264. This PR's base is launchpad, and run_adjudication.py does not exist there — I verified. The document cites it at :112-118 and discloses that honestly, so it is not a hidden dangling reference, but landing ahead of #261→#263→#264 leaves launchpad carrying a normative document citing a function no file on launchpad defines.

[benmitchell11]

Independent review — first pass on this one, no standing review to cross-check against.

Checked out feat/review-agent-adjudicator-md and ran the suite: Ran 121 tests ... OK, matching the PR body's own claim of "confirms nothing regressed" (this is a docs-only change, correctly not claiming to prove anything about the diff itself).

Stacking note, for the record rather than as a problem: this PR's base is launchpad directly, not feat/review-agent-adjudication-nonce (STEP 4) like the review-in-step-order assignment would suggest. The PR body states this is deliberate ("Depends only on STEP 1 ... so it branches directly off launchpad rather than stacking on the in-flight STEPs 2-4") and that holds up — it's a pure markdown addition with no code dependency on STEPs 2-4. Worth flagging only because it means this PR does not carry #264's nonce/stages fixes or #263's whitespace-evidence fix, so a reader diffing branches shouldn't expect them here.

Content review:

• The four prohibitions are each given a stated reason, and prohibition 1's worked contrast (restatement vs. independent check) is genuinely useful — it's concrete enough that a reader could apply the test to a real judge transcript.
• Prohibition 3 ("no approval-shaped output") is honest about being only partially mechanical: the doc itself says _FORBIDDEN_KEYS lives in verdicts.validate, which nothing calls automatically on the runner's own output before printing, so "no such key can exist" is stated as not proven, only "the runner writes only known keys, so it cannot produce a forbidden one today." That's the right amount of honesty for a spec document and matches what I saw reading run_adjudication.py directly on the other branches — adjudicate()/main() indeed never call verdicts.validate on their own output.
• The notes amendment (commit 05a960478) is real and matches what it claims: the original text told the judge to "record it in adjudication.notes", and this PR corrects that to say the channel is deferred and not to use verdict_evidence as a substitute dumping ground either. I confirmed independently (reading run_adjudication.py on feat/review-agent-adjudication-run/-escalate/-dedupe) that notes=[] is hardcoded and the judge protocol has no notes key — so this document's corrected claim is accurate. Worth noting for the author: #267's standing review flagged this exact cross-step drift and suggested "resolution belongs on #263" — this PR resolves it from the adjudicator.md side instead, which is arguably the more natural place (it's the instruction that was wrong), but the two PRs' resolutions should be cross-referenced so a reviewer of #267 doesn't think it's still open.
• The anchor: "pr" section's citation to run_adjudication.py pins a specific commit (2be90c629) on the still-unmerged feat/review-agent-adjudication-nonce branch rather than citing the branch alone, explicitly to avoid citation rot before that branch merges — I confirmed that commit exists on that branch's history and does contain the _location_description function as described (anchor branched on first, pr case never falls through to a None:None format string).

No blockers. Clean.

[serina-mcfall]

Not an independent review — I wrote part of this PR

Flagging a conflict rather than posting a verdict, because a review is worth nothing if the reviewer wrote the thing.

Commit 05a960478 on this branch — "stop mandating a notes channel the runner discards" — was written by me earlier in this session, acting for @serina-mcfall. It amends adjudicator.md's prohibition 2, which had told a judge to record new observations in adjudication.notes while run_adjudication.py hardcodes that field empty and its judge protocol never reads the key.

So I cannot assess this PR's central document. I would be marking my own work, and the one thing that made the earlier adjudication of this stack worth anything was that a fresh context checked what the authoring context had claimed.

What I can say without judging the content:

• All checks pass; mergeable: MERGEABLE.
• The three commits are e0276ec8d (the original document), afe121a56 (pinning the run_adjudication.py citation to a SHA), and 05a960478 (mine).
• ADR/numbering and placement are not at issue here — this is a single file under launchpad/review-agent/.
• No private-tooling paths appear in the diff.

What it needs: a reviewer who did not write 05a960478. Per the cohort's own practice of cross-model final review — one vendor's agent reviewing another's output — the natural fit is whoever has been reviewing the rest of the #118 stack, since they already hold the context on how notes flows through STEP 3 and STEP 6/7 and can judge whether deferring the channel was the right call versus plumbing it.

The specific question a real reviewer should press on: the deferral I wrote resolves the contradiction by telling the judge not to rely on the channel, rather than by making the channel work. That was @serina-mcfall's explicit decision between three options, and it is defensible — but it means a judge that genuinely notices a new defect has nowhere to put it until STEP 6/7, and the document now says so plainly. Whether that is acceptable, or whether notes should have been plumbed through the protocol instead, is a judgement I am the wrong party to certify.