fixes from the fourth adversarial review - #13
Merged
Merged
Conversation
…t, dev-set labels, narrative corrections Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
A fourth adversarial review of the delivered v0.1.0 found seven issues: a repeat-run guard that only compared variants, a reproduction command downloading a moving dataset revision, headline numbers not labelled as development-set results, two narrative claims contradicted by the committed predictions, a report command that fails on a clean clone without saying how to fix it, a lay-out contract requiring a field the generated file omitted, and a wrong claim about which variants use calibration data. All seven were verified against the code and data before fixing.
What changed
src/report.ts:--final-repeatnow must match the final run's variant, model and case universe (n / flagged / usable), with a message naming every mismatch; both table titles carry "development set".src/report.test.tsadds a CLI-level rejection test (27 tests total).--markdownoutput re-verified byte-identical to README §4 after the title change.src/lib/audit.ts: the fail-closed workspace error now names the fix (npm run data:workspaces).src/show.ts:human-labels.mdgains adifficultyrow;examples/astropy__astropy-12544/regenerated.underspecified=1came from v1, v2, v4 and the first v5 run; the repeats, v6 and v7 stayed at 2); §2c notes the report needs cloned workspaces.e48e2bd1…withcurl --fail; calibration attributed to the v4–v6 experiments, not the v3 default; §5 listsreport.tsamong the commands needing workspaces.skills/fairtask-eval/SKILL.md: same pinned parquet URL; the report row and the gotcha state the workspace prerequisite.trajectories/v3-verify/astropy__astropy-12544.md: committed the deterministic re-render (running the renderer no longer dirties the tree).