Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -313,7 +313,7 @@ at `REPRODUCE.md` instead.
| `/fairtask-eval the headline report, baseline versus final` | `node src/report.ts …` — the two tables in §4, as aligned terminal tables. |
| `/fairtask-eval audit evidence for v3-verify` | `npm run audit -- v3-verify` — share of cited quotes that do not exist where cited (needs `npm run data:workspaces`, which it tells you to run). |
| `/fairtask-eval check the annotation provenance` | `npm run data:annotations -- --check` — SHA-256 of the committed file against the pinned source. |
| `/fairtask-eval lay out astropy__astropy-12544` | `npm run show -- astropy__astropy-12544` — writes `examples/<id>/` and reads back the issue and the test patch. |
| `/fairtask-eval lay out astropy__astropy-12544` | `npm run show -- astropy__astropy-12544` — writes `examples/<id>/`, then shows the human labels and quotes, with `file:line`, what the issue asks for and what the graded tests require. |
| `/fairtask-eval what did the agent do on astropy__astropy-12544 in v3-verify` | renders and opens that run's trajectory. |

### What the skills may and may not do
Expand Down
8 changes: 7 additions & 1 deletion skills/fairtask-eval/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,10 +27,16 @@ except `data` and `annotations`, which download pinned public files. None calls
| evidence audit, bad evidence, fabricated quotes | `npm run audit -- <run ids…>` (needs cloned workspaces: `npm run data:workspaces` first, ~20 s) |
| code check, novel identifiers, pre-check | `npm run code-check` (needs workspaces) |
| provenance, annotations, checksum, data source | `npm run data:annotations -- --check` |
| show an instance, lay out, issue text, test patch of `<id>` | `npm run show -- <instance_id>` then read back `examples/<instance_id>/issue.md`, `test.patch`, `human-labels.md` |
| show an instance, lay out, issue text, test patch of `<id>` | `npm run show -- <instance_id>`, then open `examples/<instance_id>/human-labels.md`, `issue.md` and `test.patch` yourself and present them as described in step 3 — do not paste the files |
| trajectory, what the agent did on `<id>` in run `<run>` | `npm run trajectory -- trajectories/<run>/<instance_id>.jsonl` and open the rendered `.md` |

3. **Show the output verbatim** in a code block. Do not round, reorder or summarise numbers before the block.
Exception — **lay out an instance**: the command only writes files. Show its one `wrote …` line, then the human
labels from `human-labels.md` as one line (`underspecified=N, false_negative=N, difficulty, filter_out`), then the
lines that decide the case, quoted exactly with `file:line` — what the issue asks for (from `issue.md`), and what
the graded tests require that the issue does not say (from `test.patch`; the `+` lines of the FAIL_TO_PASS tests).
Two to four quotes, no more; point to `examples/<instance_id>/` for the full text. Done when a reader can see the
discrepancy (or its absence) without opening the files.
4. **Add one or two sentences** on how to read it — which row is the primary metric (decision accuracy), that
TPR/TNR are over scored cases only, that "bad evidence" is the share of cited quotes not found where cited. For the
evaluation set, name the four strata and the count per stratum. Stop there; the reader asked for the artifact, not
Expand Down