Skip to content

fixes from the fourth adversarial review - #13

Merged
mnkprs merged 1 commit into
mainfrom
codex4-fixes
Aug 30, 2026
Merged

fixes from the fourth adversarial review#13
mnkprs merged 1 commit into
mainfrom
codex4-fixes

Conversation

@mnkprs

@mnkprs mnkprs commented Aug 30, 2026

Copy link
Copy Markdown
Owner

Why

A fourth adversarial review of the delivered v0.1.0 found seven issues: a repeat-run guard that only compared variants, a reproduction command downloading a moving dataset revision, headline numbers not labelled as development-set results, two narrative claims contradicted by the committed predictions, a report command that fails on a clean clone without saying how to fix it, a lay-out contract requiring a field the generated file omitted, and a wrong claim about which variants use calibration data. All seven were verified against the code and data before fixing.

What changed

  • src/report.ts: --final-repeat now must match the final run's variant, model and case universe (n / flagged / usable), with a message naming every mismatch; both table titles carry "development set". src/report.test.ts adds a CLI-level rejection test (27 tests total). --markdown output re-verified byte-identical to README §4 after the title change.
  • src/lib/audit.ts: the fail-closed workspace error now names the fix (npm run data:workspaces).
  • src/show.ts: human-labels.md gains a difficulty row; examples/astropy__astropy-12544/ regenerated.
  • README: the top result and both §4 headings are labelled as development-set numbers (no held-out evaluation), making the §6c claim that every headline figure carries this caveat true; §2d's baseline row and §6's challenging-case narrative now match the committed predictions (underspecified=1 came from v1, v2, v4 and the first v5 run; the repeats, v6 and v7 stayed at 2); §2c notes the report needs cloned workspaces.
  • REPRODUCE.md: parquet download pinned to the CI revision e48e2bd1… with curl --fail; calibration attributed to the v4–v6 experiments, not the v3 default; §5 lists report.ts among the commands needing workspaces.
  • skills/fairtask-eval/SKILL.md: same pinned parquet URL; the report row and the gotcha state the workspace prerequisite.
  • trajectories/v3-verify/astropy__astropy-12544.md: committed the deterministic re-render (running the renderer no longer dirties the tree).

…t, dev-set labels, narrative corrections

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@mnkprs
mnkprs merged commit 494b723 into main Aug 30, 2026
2 checks passed
@mnkprs
mnkprs deleted the codex4-fixes branch August 30, 2026 22:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant