From e304735a9e47c760f556ccb14d300b67a8296ecd Mon Sep 17 00:00:00 2001 From: Manos Kaparos <> Date: Sat, 29 Aug 2026 21:49:19 +0300 Subject: [PATCH] fairtask-eval: lay out an instance by quoting the decisive lines, not by pasting the files Co-Authored-By: Claude Fable 5 --- README.md | 2 +- skills/fairtask-eval/SKILL.md | 8 +++++++- 2 files changed, 8 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 83be8b3..190ac52 100644 --- a/README.md +++ b/README.md @@ -313,7 +313,7 @@ at `REPRODUCE.md` instead. | `/fairtask-eval the headline report, baseline versus final` | `node src/report.ts …` — the two tables in §4, as aligned terminal tables. | | `/fairtask-eval audit evidence for v3-verify` | `npm run audit -- v3-verify` — share of cited quotes that do not exist where cited (needs `npm run data:workspaces`, which it tells you to run). | | `/fairtask-eval check the annotation provenance` | `npm run data:annotations -- --check` — SHA-256 of the committed file against the pinned source. | -| `/fairtask-eval lay out astropy__astropy-12544` | `npm run show -- astropy__astropy-12544` — writes `examples//` and reads back the issue and the test patch. | +| `/fairtask-eval lay out astropy__astropy-12544` | `npm run show -- astropy__astropy-12544` — writes `examples//`, then shows the human labels and quotes, with `file:line`, what the issue asks for and what the graded tests require. | | `/fairtask-eval what did the agent do on astropy__astropy-12544 in v3-verify` | renders and opens that run's trajectory. | ### What the skills may and may not do diff --git a/skills/fairtask-eval/SKILL.md b/skills/fairtask-eval/SKILL.md index bc5dfe4..8d35de0 100644 --- a/skills/fairtask-eval/SKILL.md +++ b/skills/fairtask-eval/SKILL.md @@ -27,10 +27,16 @@ except `data` and `annotations`, which download pinned public files. None calls | evidence audit, bad evidence, fabricated quotes | `npm run audit -- ` (needs cloned workspaces: `npm run data:workspaces` first, ~20 s) | | code check, novel identifiers, pre-check | `npm run code-check` (needs workspaces) | | provenance, annotations, checksum, data source | `npm run data:annotations -- --check` | - | show an instance, lay out, issue text, test patch of `` | `npm run show -- ` then read back `examples//issue.md`, `test.patch`, `human-labels.md` | + | show an instance, lay out, issue text, test patch of `` | `npm run show -- `, then open `examples//human-labels.md`, `issue.md` and `test.patch` yourself and present them as described in step 3 — do not paste the files | | trajectory, what the agent did on `` in run `` | `npm run trajectory -- trajectories//.jsonl` and open the rendered `.md` | 3. **Show the output verbatim** in a code block. Do not round, reorder or summarise numbers before the block. + Exception — **lay out an instance**: the command only writes files. Show its one `wrote …` line, then the human + labels from `human-labels.md` as one line (`underspecified=N, false_negative=N, difficulty, filter_out`), then the + lines that decide the case, quoted exactly with `file:line` — what the issue asks for (from `issue.md`), and what + the graded tests require that the issue does not say (from `test.patch`; the `+` lines of the FAIL_TO_PASS tests). + Two to four quotes, no more; point to `examples//` for the full text. Done when a reader can see the + discrepancy (or its absence) without opening the files. 4. **Add one or two sentences** on how to read it — which row is the primary metric (decision accuracy), that TPR/TNR are over scored cases only, that "bad evidence" is the share of cited quotes not found where cited. For the evaluation set, name the four strata and the count per stratum. Stop there; the reader asked for the artifact, not