fixes from the fourth adversarial review

README.md

@@ -29,7 +29,7 @@ It is a Claude Agent SDK pipeline (TypeScript) that reads the actual repository,
 verifies every piece of evidence they cite against the code, and returns a scored verdict with quotes a reviewer can
 check. Its output is measured against **OpenAI's public human annotations** for the same tasks.
 
-**Result on 30 human-annotated cases (same cases, same deciding model for every system):** on the headline
+**Result on 30 human-annotated cases (same cases and same deciding model for every system — a development set: these same 30 cases were used to diagnose failures and design every iteration, and there is no held-out evaluation):** on the headline
 metric — agreement with the human usable/flag decision — the agent pipeline **does not reliably beat the one-prompt
 baseline**: the baseline scores 67% on two independent runs, the agent variants 60–70% with a ±2-case run-to-run
 spread. What the agent changes is measurable elsewhere: the evidence it cites is **verifiable 100% of the time**
@@ -355,7 +355,7 @@ a complete command; rows with the same script differ by one flag.
 | `npm run score -- baseline v3-verify --detail` | Adds one row per instance: human scores, predicted scores, decision, cost. |
 | `npm run score -- baseline v3-verify --common` | Restricts every run to the instances all listed runs scored (for comparing partial runs). |
 | `npm run score -- baseline --json` | The summary as JSON instead of the table. |
-| `node src/report.ts --baseline baseline --final v3-verify` | Prints the headline comparison of those two runs as an aligned terminal table (the same rows as §4). |
+| `node src/report.ts --baseline baseline --final v3-verify` | Prints the headline comparison of those two runs as an aligned terminal table (the same rows as §4). Needs cloned workspaces (`npm run data:workspaces`, ~20 s) — the bad-evidence rows re-verify every quote against the repositories. |
 | `node src/report.ts --baseline baseline --final v3-verify --markdown` | The same as a Markdown pipe table — what `scripts/finalize-report.py` pastes into this README. |
 | `node src/report.ts --baseline baseline --final v5-cheap-probes --final-repeat v5-rerun` | Shows the final configuration as "first run · repeat run". The repeat must be a run of the same configuration; the script refuses otherwise. |
 | `node src/report.ts --baseline baseline --final v3-verify --runs baseline,v1-context,v3-verify` | Adds the all-systems table with those runs as rows. |
@@ -409,7 +409,7 @@ the corrective turn). The representative ones, one per agent and one per kind of
 
 | Agent / situation | Trajectory | What to look at |
 |---|---|---|
-| **Baseline** — one prompt, no tools | [`trajectories/baseline/astropy__astropy-12544.md`](trajectories/baseline/astropy__astropy-12544.md) | Instructions and task prompt, then the verdict straight away. It flags the task (humans: flag, `false_negative` 3) but also scores the issue 2 where humans gave 0 — the right decision for half the right reasons; every later configuration keeps the flag and the same over-score on the issue axis. |
+| **Baseline** — one prompt, no tools | [`trajectories/baseline/astropy__astropy-12544.md`](trajectories/baseline/astropy__astropy-12544.md) | Instructions and task prompt, then the verdict straight away. It flags the task (humans: flag, `false_negative` 3) but also scores the issue 2 where humans gave 0 — the right decision for half the right reasons; every configuration keeps the flag, while the issue-axis score wobbles between 1 and 2 across runs (§6). |
 | **Single agent with repository tools** (v1) | [`trajectories/v1-context/astropy__astropy-12544.md`](trajectories/v1-context/astropy__astropy-12544.md) | The same prompt plus `Read`/`Grep`/`Glob`: what it reads, and how reading the code made it *more* lenient (§5, Iteration 1). |
 | **Judge + `spec-probe` + `test-probe`** (v3, the final default) | [`trajectories/v3-verify/astropy__astropy-12544.md`](trajectories/v3-verify/astropy__astropy-12544.md) | Instructions for all three agents (`### Subagent spec-probe`, `### Subagent test-probe`); the judge dispatching both in parallel; each probe's `Read`/`Grep` calls and the file contents that came back; the two probe reports with evidence; the judge re-opening the cited lines itself ("Now spot-checking the strongest claims myself"); the final verdict; `✅ Verification passed`. |
 | **Verifier feedback and a corrective turn** | [`trajectories/v4-calibrated/scikit-learn__scikit-learn-11574.md`](trajectories/v4-calibrated/scikit-learn__scikit-learn-11574.md) · [`trajectories/v5-cheap-probes/sphinx-doc__sphinx-8284.md`](trajectories/v5-cheap-probes/sphinx-doc__sphinx-8284.md) | `⛔ Verification failed (attempt 1)`: the verifier names the evidence item whose quote is not in the test patch, the exact feedback text sent to the judge, the judge's re-check with `Grep`, and the corrected verdict that passes. These are the only two corrective turns in 300 evaluation runs — the announcement that quotes will be checked did the work (§5, Iteration 3). |
@@ -453,7 +453,7 @@ matched to the runs that used it.
 
 ## 4. Results
 
-### Headline comparison (30 cases, same cases and same deciding model for both)
+### Headline comparison (30 cases, development set — same cases and same deciding model for both)
 
 | Metric | Simple baseline (`baseline`) | Agent solution (`v3-verify`) | Change |
 |---|---|---|---|
@@ -469,7 +469,7 @@ matched to the runs that used it.
 | Cost per task (USD, list price) | $0.14 | $0.95 | +$0.81 ▼ |
 | Challenging case (`astropy__astropy-12544`) | correct (us=2 fn=3 flag) | correct (us=2 fn=3 flag) | human: us=0 fn=3 flag |
 
-### All systems on the same 30 cases
+### All systems on the same 30 cases (development set)
 
 | Run | Decision acc. | κ | TPR / TNR | Missed / false alarms | Both axes | Bad evidence | Cost/task | Time/task |
 |---|---|---|---|---|---|---|---|---|
@@ -501,8 +501,9 @@ lines of a flagged task.
 the axis. Humans: issue clear (0), tests unfair (3). Every configuration found the test-axis problem with the right
 evidence — the `mask_invalid` assertions quoted from the test patch against the issue's own `mask` wording — but the
 baseline and the default pipeline (`v3-verify`) also scored `underspecified = 2`, arguing that the issue leaves the
-single-table and memmap behaviour open; only the Sonnet-probe configurations (`v5`, `v7`) put it at 1, on the human
-side of the threshold. Right decision, one axis debatable — which is why the "both axes correct" metric exists.
+single-table and memmap behaviour open. Four runs put it at 1, on the human side of the threshold (`v1`, `v2`, `v4`
+and the first `v5` run); the baseline repeat, `v5-rerun`, `v6` and `v7` stayed at 2. Right decision, one axis
+debatable — and unstable from run to run — which is why the "both axes correct" metric exists.
 
 
 ---

REPRODUCE.md

@@ -38,13 +38,13 @@ Two public inputs, both already committed under `data/`:
   `npm run data:annotations` re-downloads it from the pinned source commit and refuses a mismatching file.
 - `data/eval/instances.json` — the fixed evaluation set: 30 instances (seed `20260828`), each with its issue text,
   gold patch, test patch, FAIL_TO_PASS list and the human labels. `data/eval/calibration.json` holds the annotator
-  notes for the *other* 1,600+ instances, grouped by repository (used by the final variant; contains no eval instance).
+  notes for the *other* 1,600+ instances, grouped by repository (used only by the `v4`–`v6` experimental configurations, not by the `v3-verify` default; contains no eval instance).
 
 To rebuild them from scratch (needs the 12 MB SWE-bench parquet, not committed):
 
 ```bash
-curl -L -o data/raw/swebench_test.parquet \
-  "https://huggingface.co/datasets/princeton-nlp/SWE-bench/resolve/main/data/test-00000-of-00001.parquet"
+curl --fail -L -o data/raw/swebench_test.parquet \
+  "https://huggingface.co/datasets/princeton-nlp/SWE-bench/resolve/e48e2bd1e9fecd5bbd641e9414ac59da9f2e69f6/data/test-00000-of-00001.parquet"
 npm run data:eval-set        # -> data/eval/instances.json  (prints the 30 instances and strata)
 npm run data:calibration     # -> data/eval/calibration.json
 ```
@@ -116,8 +116,8 @@ of them with `scripts/render-trajectories.sh <run-id>`.
 
 ## 5. Evaluate
 
-`npm run score` needs only the committed results. `evidence-audit`, `code-check` and `finalize-report.py` re-read the
-repositories at the base commit, so run `npm run data:workspaces` first (they refuse to run otherwise).
+`npm run score` needs only the committed results. `report.ts`, `evidence-audit`, `code-check` and `finalize-report.py`
+re-read the repositories at the base commit, so run `npm run data:workspaces` first (they refuse to run otherwise).
 
 ```bash
 npm run score -- baseline baseline-rerun v1-context v2-specialists v3-verify v4-calibrated v5-cheap-probes v5-rerun v6-target-aware v7-sonnet-nocal

examples/astropy__astropy-12544/human-labels.md

@@ -6,6 +6,7 @@ Source: data/raw/ensembled_annotations_public.csv (provenance in data/raw/SOURCE
 |---|---|
 | underspecified | 0 |
 | false_negative | 3 |
+| difficulty | 1-4 hours |
 | filter_out | true |
 
 ## Annotator note — issue

skills/fairtask-eval/SKILL.md

@@ -21,9 +21,9 @@ except `data` and `annotations`, which download pinned public files. None calls
 
    | Request mentions | Run from the repository root |
    |---|---|
-   | the evaluation set, the thirty cases, strata, which tasks | `npm run data:eval-set` (needs `data/raw/swebench_test.parquet`; if missing, run `curl -L -o data/raw/swebench_test.parquet https://huggingface.co/datasets/princeton-nlp/SWE-bench/resolve/main/data/test-00000-of-00001.parquet` first and say you did) |
+   | the evaluation set, the thirty cases, strata, which tasks | `npm run data:eval-set` (needs `data/raw/swebench_test.parquet`; if missing, run `curl -L -o data/raw/swebench_test.parquet https://huggingface.co/datasets/princeton-nlp/SWE-bench/resolve/e48e2bd1e9fecd5bbd641e9414ac59da9f2e69f6/data/test-00000-of-00001.parquet` first and say you did) |
    | score, accuracy, kappa, recall, a run id, compare runs | `npm run score -- <run ids…>` (default: `baseline v3-verify`; add `--detail` for per-instance rows) |
-   | the report, the headline table, baseline versus final | `node src/report.ts --baseline baseline --final v3-verify --runs <all run ids>` — the run ids are the directory names under `results/`; the default (`v3-verify`) has no repeat run, so no `--final-repeat` (the script refuses a repeat of a different configuration). Prints aligned tables; `--markdown` gives the README's pipe tables |
+   | the report, the headline table, baseline versus final | `node src/report.ts --baseline baseline --final v3-verify --runs <all run ids>` — the run ids are the directory names under `results/`; the default (`v3-verify`) has no repeat run, so no `--final-repeat` (the script refuses a repeat of a different configuration). Needs cloned workspaces for the bad-evidence rows — run `npm run data:workspaces` (~20 s) first if `workspaces/` is missing. Prints aligned tables; `--markdown` gives the README's pipe tables |
    | evidence audit, bad evidence, fabricated quotes | `npm run audit -- <run ids…>` (needs cloned workspaces: `npm run data:workspaces` first, ~20 s) |
    | code check, novel identifiers, pre-check | `npm run code-check` (needs workspaces) |
    | provenance, annotations, checksum, data source | `npm run data:annotations -- --check` |
@@ -47,7 +47,7 @@ except `data` and `annotations`, which download pinned public files. None calls
 - Committed run ids are complete: `npm run run -- --run-id baseline` does nothing. Reproduction runs use fresh ids
   (`baseline-repro`), cost money and need `claude login` or `ANTHROPIC_API_KEY`; this skill does not start them.
   Point the user at `REPRODUCE.md` instead.
-- `audit`, `code-check` and `finalize-report.py` refuse to run without workspaces at the right commits; that is
-  intended. Run `npm run data:workspaces`, do not work around it.
+- `report.ts`, `audit`, `code-check` and `finalize-report.py` refuse to run without workspaces at the right commits;
+  that is intended. Run `npm run data:workspaces`, do not work around it.
 - `data:eval-set` prints the table; it also rewrites `data/eval/instances.json` deterministically. In a clean checkout
   `git status` stays clean afterwards — if it does not, say so, because that is a reproducibility failure.

src/lib/audit.ts

@@ -28,7 +28,7 @@ export function requireWorkspaces(ids: Iterable<string>): void {
       if (git("status", "--porcelain", "--untracked-files=all") !== "") problems.push(`${id}: working tree has modified or untracked files`);
     } catch (e) { problems.push(`${id}: ${(e as Error).message.split("\n")[0]}`); }
   }
-  if (problems.length) throw new Error(`${problems.length} workspace(s) cannot be used for verification:\n  ${problems.slice(0, 5).join("\n  ")}${problems.length > 5 ? "\n  …" : ""}\nRun \`npm run data:workspaces\` to (re)clone them at the base commit.`);
+  if (problems.length) throw new Error(`${problems.length} workspace(s) cannot be used for verification (fix: npm run data:workspaces):\n  ${problems.slice(0, 5).join("\n  ")}${problems.length > 5 ? "\n  …" : ""}\nRun \`npm run data:workspaces\` to (re)clone them at the base commit.`);
 }
 
 export function auditRun(runId: string): Audit {

src/report.test.ts

@@ -0,0 +1,10 @@
+import { test } from "node:test";
+import assert from "node:assert/strict";
+import { execFileSync } from "node:child_process";
+
+test("report refuses a --final-repeat of a different configuration", () => {
+  assert.throws(
+    () => execFileSync(process.execPath, ["src/report.ts", "--baseline", "baseline", "--final", "v3-verify", "--final-repeat", "v5-rerun"], { encoding: "utf8", stdio: "pipe" }),
+    (e: unknown) => String((e as { stderr?: string }).stderr).includes("is not a repeat of") && String((e as { stderr?: string }).stderr).includes("variant"),
+  );
+});

src/report.ts

@@ -23,7 +23,13 @@ const delta = (a: number, b: number, fmt: (x: number) => string, betterHigh = tr
 const base = load(args.get("baseline") ?? "baseline");
 const fin = load(args.get("final") ?? "v3-verify");
 const rep = args.get("final-repeat") ? load(args.get("final-repeat")!) : undefined;
-if (rep && rep.variant !== fin.variant) throw new Error(`--final-repeat ${rep.run_id} is a run of variant "${rep.variant}", not of the final configuration "${fin.variant}" (${fin.run_id}); a repeat must be the same configuration run again`);
+if (rep) {
+  const mismatch: string[] = [];
+  if (rep.variant !== fin.variant) mismatch.push(`variant "${rep.variant}" vs "${fin.variant}"`);
+  if (rep.model !== fin.model) mismatch.push(`model "${rep.model}" vs "${fin.model}"`);
+  if (rep.n !== fin.n || rep.n_flag !== fin.n_flag || rep.n_usable !== fin.n_usable) mismatch.push(`case universe ${rep.n} (${rep.n_flag}/${rep.n_usable}) vs ${fin.n} (${fin.n_flag}/${fin.n_usable})`);
+  if (mismatch.length) throw new Error(`--final-repeat ${rep.run_id} is not a repeat of ${fin.run_id}: ${mismatch.join("; ")}. A repeat must be the same configuration on the same cases run again.`);
+}
 const format: TableFormat = args.get("markdown") ? "markdown" : "table";
 /** Show "first / repeat" when a repeat run of the final configuration exists. */
 const both = (f: (s: Summary) => string) => (rep ? `${f(fin)} · ${f(rep)}` : f(fin));
@@ -44,12 +50,12 @@ const headline: string[][] = [
   [`Cost per task (USD, list price)`, `${usd(base.mean_cost_usd)}`, `${usd(fin.mean_cost_usd)}`, `${delta(base.mean_cost_usd, fin.mean_cost_usd, usd, false)}`],
   [`Challenging case (\`${fin.challenging?.instance_id}\`)`, `${base.challenging?.correct ? "correct" : "wrong"} (${base.challenging?.pred})`, `${fin.challenging?.correct ? "correct" : "wrong"} (${fin.challenging?.pred})`, `human: ${fin.challenging?.human}`],
 ];
-console.log(renderTitle(`Headline comparison (${base.n} cases, same cases and same deciding model for both${rep ? "; final shown as first run · repeat run" : ""})`, format));
+console.log(renderTitle(`Headline comparison (${base.n} cases, development set — same cases and same deciding model for both${rep ? "; final shown as first run · repeat run" : ""})`, format));
 console.log(renderTable(["Metric", `Simple baseline (\`${base.run_id}\`)`, `Agent solution (\`${fin.run_id}\`)`, "Change"], headline, format));
 
 const runs = (args.get("runs") ?? "").split(",").filter(Boolean).map(load);
 if (runs.length) {
-  console.log("\n" + renderTitle(`All systems on the same ${base.n} cases`, format));
+  console.log("\n" + renderTitle(`All systems on the same ${base.n} cases (development set)`, format));
   console.log(renderTable(
     ["Run", "Decision acc.", "κ", "TPR / TNR", "Missed / false alarms", "Both axes", "Bad evidence", "Cost/task", "Time/task"],
     runs.map((r) => [`\`${r.run_id}\``, pct(r.decision_accuracy), r.kappa.toFixed(2), `${pct(r.tpr)} / ${pct(r.tnr)}`, `${r.missed_problems} / ${r.false_alarms}`, pct(r.both_axes_correct), badRate(r.run_id), usd(r.mean_cost_usd), `${Math.round(r.mean_duration_s)} s`]),

src/show.ts

@@ -28,5 +28,5 @@ const links = [
 writeFileSync(`${dir}/issue.md`, `# ${id} — the issue text (all a solver sees)\n\n${links}\n\n---\n\n${inst.problem_statement.trim()}\n`);
 writeFileSync(`${dir}/test.patch`, inst.test_patch);
 writeFileSync(`${dir}/gold.patch`, inst.patch);
-writeFileSync(`${dir}/human-labels.md`, `# ${id} — human labels (OpenAI, 2024; max over three annotators)\n\nSource: data/raw/ensembled_annotations_public.csv (provenance in data/raw/SOURCES.md)${pr ? `; task: https://github.com/${inst.repo}/pull/${pr}` : ""}\n\n| axis | score |\n|---|---|\n| underspecified | ${inst.human.underspecified} |\n| false_negative | ${inst.human.false_negative} |\n| filter_out | ${inst.human.filter_out} |\n\n## Annotator note — issue\n\n${inst.human.underspecified_notes.trim()}\n\n## Annotator note — tests\n\n${inst.human.false_negative_notes.trim()}\n\n## Graded tests (FAIL_TO_PASS)\n\n${inst.FAIL_TO_PASS.map((t) => `- \`${t}\``).join("\n")}\n`);
+writeFileSync(`${dir}/human-labels.md`, `# ${id} — human labels (OpenAI, 2024; max over three annotators)\n\nSource: data/raw/ensembled_annotations_public.csv (provenance in data/raw/SOURCES.md)${pr ? `; task: https://github.com/${inst.repo}/pull/${pr}` : ""}\n\n| axis | score |\n|---|---|\n| underspecified | ${inst.human.underspecified} |\n| false_negative | ${inst.human.false_negative} |\n| difficulty | ${inst.human.difficulty} |\n| filter_out | ${inst.human.filter_out} |\n\n## Annotator note — issue\n\n${inst.human.underspecified_notes.trim()}\n\n## Annotator note — tests\n\n${inst.human.false_negative_notes.trim()}\n\n## Graded tests (FAIL_TO_PASS)\n\n${inst.FAIL_TO_PASS.map((t) => `- \`${t}\``).join("\n")}\n`);
 console.log(`wrote ${dir}/{issue.md,test.patch,gold.patch,human-labels.md}`);