fix(skill-evals): retry a dropped grader verdict, and correct the step 1f expectations - #1341
Merged
Merged
Conversation
…p 1f expectations Two of the cases left red by #1340 were the fixture disagreeing with the documented process rather than the model getting it wrong, and verifying that turned up a third problem in the runner itself. **A verdict the grader drops is no longer a failure.** `batch_grade_prose_fields` sends one rubric prompt covering every pair, and the grader intermittently omits a path from a larger batch. That pair was then recorded as `(False, "grader did not return a verdict")` — a FAIL indistinguishable in the report from a real mismatch, for a reason that has nothing to do with the candidate output. It is the mechanism behind cases that appear to flap between runs of unchanged code. The dropped subset is now re-asked once; silence that survives the retry is still reported, and a verdict the grader actively gives is never re-asked, so a NO cannot become a YES by asking twice. The structural-assertion path already drew this distinction with a tri-state verdict; the prose path now does too. **`case-7` expected the wrong step.** The process table in `gather.md` — the table this step tells the model to cross-reference — puts "Archive URL captured" at `14 → 15`, and the step's output spec says in as many words that when two consecutive rows both apply the answer is the inclusive hyphenated range. The tracker in this fixture is still open with the archive URL populated, so it sits on that row. The model answered `"14-15"` on every run; the fixture asserted `"14"`. **`case-5` asserted a field the spec calls explanatory.** `key_signals` exists so a reviewer can see why a step was chosen, and which facts are worth naming is a judgement call. The model named five signals that subsume the expected three and the case failed on the difference. Cases 1, 2 and 6 in this same step already omit the field, so 3, 4 and 5 now match the step's own majority and the spec records that it is emitted but not graded. Same fragility #1340 removed from the guardrails `evidence` field. `case-7`'s `step_description` is also shortened to the one-line shape the spec asks for; the first attempt enumerated every action in the combined apply, which made a prose-graded field fail on whichever sub-action the model left out. The verifying run is 43/45 with no dropped verdicts, against 42/45 and two dropped verdicts before. The two that remain are the grader's judgement on free text — one on whether "valid" implies "CVE-worthy", one on whether naming the confirmation step over-describes the action — and both have passed on other runs. They are left alone deliberately: rewording a fixture until a rubric happens to agree tunes the test to the model. Narrowing what the prose fields assert is the real fix, and it wants deciding on its own terms rather than inside this change. Generated-by: Claude Opus 5
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
batch_grade_prose_fieldssends one prompt covering every pair; when the grader omits a path from a larger batch, that pair was recorded as(False, "grader did not return a verdict")— a FAIL indistinguishable from a real mismatch, for a reason unrelated to the candidate output. This is the mechanism behind cases that appear to flap between runs of unchanged code.Type of change
.claude/skills/<name>/) — eval fixtures updated belowtools/*/withpyproject.toml)prek, workflows, validators)Test plan
prek run --all-filespasses (34 hooks, 0 failures)uv run pytestontools/skill-evals— 174 tests pass, including two new regression tests for the retrysecurity-issue-syncfull suite: 43/45 with no dropped verdicts, against 42/45 with two dropped verdicts beforestep-1f-process-stepon its own: 7/7RFC-AI-0004 compliance
Linked issues
Follows #1340.
Notes for reviewers
The runner change is the substantive one. The structural-assertion path already distinguished "no verdict" from "no match" with a tri-state; the prose path did not. It now re-asks the dropped subset once, and silence surviving the retry is still reported rather than dropped.
case-7expected the wrong step. The process table ingather.md— the table this step tells the model to cross-reference — puts "Archive URL captured" at14 → 15, and the step's output spec says explicitly that when two consecutive rows both apply the answer is the inclusive hyphenated range. The fixture's tracker is open with the archive URL populated, so it sits on that row. The model answered"14-15"on every run.case-5asserted a field the spec calls explanatory.key_signalsexists so a reviewer can see why a step was chosen. Cases 1, 2 and 6 in this same step already omit it from their expectations; 3, 4 and 5 now match, and the spec records that the field is emitted but not graded.Two cases remain red on the verifying run and are deliberately untouched:
step-1f/case-3(whether "valid" implies "CVE-worthy") andstep-2b/case-5(whether naming the confirmation step over-describes the action). Both are the grader's judgement on free text, and both have passed on other runs. Rewording a fixture until a rubric happens to agree tunes the test to the model. The real fix is narrowing what the prose fields assert, which wants deciding on its own terms rather than inside this PR.For the record, on
case-7I first guessed the half-applied label flip was the cause and edited the fixture — the model's answer did not change, so I reverted it. The defect was simply that the fixture contradicted the table it cross-references.