Bind judge provenance to the verdict it describes - #166
Merged
Conversation
Peer review of #164 (gate f4abb02c) found that re-judging a case through the Codex runner could keep the Claude runner's provenance sidecar: prepare_audit removed verdict.json but left verdict.meta.json, the Codex runner never touched the sidecar, and the freezer preferred it over codex.log. A later freeze would have miscounted judges. - prepare_audit unlinks verdict.meta.json together with a stale verdict. - Both runners write verdict.meta.json with the verdict's sha256 and the runner name (the Codex runner now writes one too, with the model from its log header), and remove any leftover sidecar before judging. - The freezer counts a sidecar only when its hash matches the case's current verdict.json; otherwise it reads codex.log, else "unknown". - scripts/backfill_verdict_provenance.py binds existing sidecars (only when the recorded judging time sits within tolerance of the verdict file's mtime) and writes Codex sidecars from codex.log. Run on the audit tree: 350 bound, 318 written; the tally reproduces the frozen manifest (350 Claude Opus 5, 318 GPT-5.6 Sol). - Tests: sidecar binding rules; prepare_audit cleanup; a cross-runner re-judge through both runner scripts with fake CLIs. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Gate review of #166 (round 1): the backfill's timestamp tolerance could bind a legacy Claude sidecar to a Codex verdict written minutes later, and the freezer's codex.log fallback could attribute a current hash-less Claude verdict to an older Codex attempt. - Backfill binds a legacy sidecar only when the verdict file dates from its judging time AND no codex.log was written after that time; a verdict with no sidecar at all can only have come from the Codex runner, which is the evidence for writing its Codex sidecar. - The freezer reads codex.log only when the case has no sidecar and the log was written alongside the verdict (600 s); a stale or hash-less sidecar beside an unrelated log counts as unknown. - Tests for both rules; the real audit tree's tally still reproduces the frozen manifest (350 / 318) and the backfill is idempotent (668 already). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Follow-up to #164's peer review (gate f4abb02c, round 1): judge provenance could go stale when a case is re-judged through the other runner.
prepare_auditremovedverdict.jsonbut leftverdict.meta.json; the Codex runner replaced the verdict without touching the sidecar; the freezer preferred the sidecar overcodex.log. A later freeze would have attributed a Sol verdict to Opus.Changes
prepare_auditunlinksverdict.meta.jsontogether with a stale verdict.verdict.meta.jsonbound to the verdict by sha256, with the runner name; the Codex runner now writes one too (model from its log header) and both remove any leftover sidecar before judging.scripts/freeze_snapshot.py:verdict_provenance()accepts a sidecar only when its hash matches the case's currentverdict.json; otherwise the tally readscodex.log, else counts the case asunknown.scripts/backfill_verdict_provenance.pybinds existing sidecars (only when the recorded judging time is within tolerance of the verdict file's mtime) and writes Codex sidecars fromcodex.log. Run on the audit tree: 350 bound, 318 written, 0 left alone;audit_judge_provenance()reproduces the frozen manifest block exactly (350 Claude Opus 5, 318 GPT-5.6 Sol). A timing audit of the tree found no case whose Codex log postdates a Claude verdict, so the published tally was correct.No frozen artifact, score, or manuscript changes.
Review round 1 (Astra)
Two legacy-handling gaps fixed: the backfill now binds a legacy Claude sidecar only when the verdict file dates from its judging time and no
codex.logwas written afterwards (timestamp proximity alone is not ownership), and the freezer'scodex.logfallback applies only to a case with no sidecar whose log was written alongside the verdict; a stale or hash-less sidecar beside an unrelated log counts asunknown. Tests cover both; the real tree's tally is unchanged (350 / 318) and the backfill is idempotent.Tests
tests/test_judge_provenance.py: sidecar binding rules (fresh, stale, legacy without hash, Codex sidecar); a cross-runner re-judge that runs both runner scripts with fakeclaudeandcodexCLIs and checks the sidecar and the tally follow the current verdict.tests/test_audit.py: re-preparing a changed case drops the sidecar with the verdict.uv run pytest -m "not slow": 802 passed, 5 skipped.🤖 Generated with Claude Code