From 8985ad3e3d784f2aec462e9383744194342e69e4 Mon Sep 17 00:00:00 2001 From: Manos Kaparos <> Date: Sat, 29 Aug 2026 16:54:20 +0300 Subject: [PATCH] readme: agent interface as its own section; command reference with one row per invocation Co-Authored-By: Claude Fable 5 --- README.md | 204 ++++++++++++++++++++++++++++++++++++++---------------- 1 file changed, 144 insertions(+), 60 deletions(-) diff --git a/README.md b/README.md index 26e60f4..db80628 100644 --- a/README.md +++ b/README.md @@ -48,7 +48,7 @@ git clone https://github.com/mnkprs/fairtask && cd fairtask && npm ci # the command fetches the task from Hugging Face and shallow-clones the repository at the right commit. npm run screen -- --swebench django__django-11099 -# screen your own task (JSON format in §2b) +# screen your own task (JSON format in §2a) npm run screen -- --task my-task.json # or use it from inside your agent: /fairtask @@ -63,8 +63,8 @@ Django pull request #11099. Full command reference: [§2c](#2c-command-reference | I want to… | Read | |---|---| -| Screen my own tasks | [§2b · Use it on your own tasks](#2b-use-it-on-your-own-tasks) | -| Use it from inside an agent (skill / plugin) | [§2b · From inside your agent](#use-it-from-inside-your-agent) | +| Screen my own tasks from a shell | [§2a · Use it from a shell](#2a-use-it-from-a-shell) | +| Use it from inside an agent (the agentic interface) | [§2b · Use it from inside your agent](#2b-use-it-from-inside-your-agent) | | Look up a command and its flags | [§2c · Command reference](#2c-command-reference) | | Reproduce every number from a clean machine | [REPRODUCE.md](REPRODUCE.md) | | Understand the problem and who has it | [§1 · Who has this problem](#1-who-has-this-problem-and-what-is-the-bottleneck) | @@ -172,9 +172,9 @@ read-only, workspaces are throwaway shallow clones) and no external action is ta --- -## 2b. Use it on your own tasks +## 2a. Use it from a shell -fairtask screens one task at a time; the evaluation harness in §3–§5 is built on top of the same command. +fairtask screens one task at a time; the evaluation harness in §3–§5 is built on top of the same command. If you work inside an agent rather than a terminal, start with [§2b](#2b-use-it-from-inside-your-agent) instead. ```bash git clone https://github.com/mnkprs/fairtask && cd fairtask && npm ci @@ -234,80 +234,164 @@ with `--variant v5-cheap-probes`. Verdicts are recommendations for a human revie of any flagged task before dropping it — see §6 for why the labels the pipeline was tuned against are themselves imperfect. -### Use it from inside your agent +--- -The same engine is packaged as an **agent skill** (`skills/fairtask/`), installable from the open skills registry: +## 2b. Use it from inside your agent + +This is the interface the project is built for. The people who screen tasks in 2026 do it inside agents — a +Claude Code or Codex session that already has the repository open, the PR in front of it, and the rest of the +task-authoring workflow around it. fairtask ships as **two agent skills** on top of the same engine, so the screening +and the evaluation happen in the conversation, and the numbers a session reports come from the scripts, not from +the model's memory. + +### Install once + +| Where you work | Install | What you get | +|---|---|---| +| Any harness that reads the open skills registry (Claude Code, Codex, Cursor, OpenCode, …) | `npx skills add mnkprs/fairtask` | both skills, installed into that harness's skills directory | +| Claude Code, as a plugin | `/plugin marketplace add mnkprs/fairtask` then `/plugin install fairtask@fairtask` | both skills (plugin manifest in `.claude-plugin/`) | +| Codex, as a plugin | the repository carries `.codex-plugin/plugin.json` | both skills | + +Both skills need Node ≥ 22.18 and git. `/fairtask` also needs a Claude login or `ANTHROPIC_API_KEY`, because it runs +the model; `/fairtask-eval` needs neither. + +### `/fairtask` — screen a task + +Give it any of three things; it does the rest and reports in the conversation. + +| You type | What happens | +|---|---| +| `/fairtask django__django-11099` | Fetches that SWE-bench instance from Hugging Face, shallow-clones Django at the task's base commit, runs the judge and the two probes, verifies every quoted line, and reports the verdict. | +| `/fairtask my-task.json` | Same for your own task file (`repo`, `base_commit`, `problem_statement`, `patch`, `test_patch`, `FAIL_TO_PASS`). | +| `/fairtask astropy/astropy 12544` | A pull request. The skill first runs `scripts/task-from-pr.sh`, which builds the task file from the PR with `gh` (base commit, the linked issue's text, code diff, test diff, added tests). It **fails closed** — exits with reasons — if the PR has no linked issue, no test file, or no graded test it can confirm; the skill reports those reasons and stops unless you explicitly accept a provisional screening. Then it screens the task. | + +What comes back, verbatim from the session that produced `examples/psf__requests-2317/`: -```bash -npx skills add mnkprs/fairtask # skills.sh: installs the `fairtask` skill into Claude Code / Codex / Cursor / … -/plugin marketplace add mnkprs/fairtask # Claude Code plugin marketplace … -/plugin install fairtask@fairtask # … then the plugin (the /fairtask skill) ``` -The repository also carries a Codex plugin manifest (`.codex-plugin/`). Packaging follows the conventions of -[ECC](https://github.com/affaan-m/ECC) — plugin manifest, marketplace file, `metadata.origin` in the skill — -so the same skill installs the same way on every harness ECC supports. - -Then, in a session: `/fairtask django__django-11099`, `/fairtask task.json`, or `/fairtask astropy/astropy 12544`. -A second skill, `/fairtask-eval`, runs the evaluation tooling from inside the session — `/fairtask-eval show the -evaluation set`, `/fairtask-eval score baseline v3-verify`, `/fairtask-eval lay out astropy__astropy-12544` — and -prints the scripts' output verbatim, so the numbers in a conversation come from the code, not from memory. -The screening skill: `/fairtask …` -(a pull request — `skills/fairtask/scripts/task-from-pr.sh` turns it into a task: base commit, linked issue text, -code diff, test diff, added test functions). The skill runs `npm run screen` from `$FAIRTASK_HOME` (cloning the -engine on first use), reports the verdict with every evidence item and its location, and — when the engine cannot -run — falls back to a **manual mode** in which the agent itself follows the two probe procedures in -`skills/fairtask/references/method.md` and self-checks its quotes. This is how a task-authoring session would use -it: screen the PR you are about to turn into a task, before you spend an hour on it. +USABLE underspecified=0 false_negative=1 confidence=4/5 (evidence verified) -## 2c. Command reference +Issue specification (0): the issue names the exact file and line (requests/sessions.py: method = builtin_str(method)), +the wrong behaviour (b'GET' becomes the literal "b'GET'" under Python 3, yielding 404), and the desired behaviour … -Every command is an npm script; flags go after `--`. Paths are relative to the repository root. "Run id" is the name of -a directory under `results/` (the committed ones: `baseline`, `baseline-rerun`, `v1-context`, `v2-specialists`, -`v3-verify`, `v4-calibrated`, `v5-cheap-probes`, `v5-rerun`, `v6-target-aware`, `v7-sonnet-nocal`). An **instance id** -is a SWE-bench task name, `__-`. +Test scope (1): the new test asserts only that requests.request(b'GET', …).ok, so any normalisation strategy passes … -### Screening one task (the product) +Evidence (3): + [underspecified] repo requests/sessions.py:L21 + › from .utils import to_key_val_list, default_headers, to_native_string + [false_negative] repo test_requests.py:L140-L142 + › def test_HTTP_302_ALLOW_REDIRECT_GET(self): r = requests.get(httpbin('redirect', '1')) … + [false_negative] test_patch test_requests.py + › r = requests.request(b'GET', httpbin('get')); assert r.ok -| Command | What it does | Flags | -|---|---|---| -| `npm run screen -- --swebench ` | Fetches the task from Hugging Face, shallow-clones the repository at its base commit into `workspaces//repo`, runs the pipeline, prints the verdict, writes `screenings//verdict.json` + trajectory. | `--dataset ` (default `princeton-nlp/SWE-bench`; e.g. `princeton-nlp/SWE-bench_Verified`) · `--variant ` (default `v3-verify`; `v5-cheap-probes` = Sonnet probes, `v6-target-aware` = high recall) · `--model ` (default `claude-opus-5`) · `--out ` (default `screenings`) · `--allow-unconfirmed` (screen a task file the PR script marked unconfirmed) | -| `npm run screen -- --task ` | Same, for your own task. Required fields: `repo`, `base_commit`, `problem_statement`, `patch`, `test_patch`; optional `instance_id`, `FAIL_TO_PASS`. | same as above | -| `skills/fairtask/scripts/task-from-pr.sh [--lenient] > task.json` | Builds a task file from a pull request with `gh`. Exits 3 when the PR has no linked issue, no test file, or no confirmable FAIL_TO_PASS; `--lenient` emits it marked `_status: unconfirmed`. | `--lenient` | +$0.49 · 195s · 8 turns +``` -### Evaluation (needs no API key: reads the committed results) +The skill then says what to do with it: for FLAG, open the cited lines and decide; for USABLE, nothing. It presents +every verdict as a recommendation for a human reviewer, never as ground truth — on the evaluation set this pipeline +agrees with expert humans about as often as a one-prompt baseline; what it adds is evidence you can check. -| Command | What it does | Flags | -|---|---|---| -| `npm run score -- […]` | Metrics of each run against the human labels, side by side; writes `results//summary.json`. | `--detail` (per-instance rows) · `--json` · `--common` (restrict to instances every listed run scored) | -| `node src/report.ts --baseline --final ` | The README tables (headline comparison + all-systems table) from the summaries. | `--final-repeat ` (show first run · repeat) · `--runs ` (rows of the all-systems table) | -| `npm run audit -- …` | Post-hoc verifier: share of cited evidence that does not exist where cited. Needs workspaces. | — | -| `npm run code-check` | Zero-LLM pre-check: identifiers required by graded tests, introduced by the gold patch, absent from issue and repository; TPR/TNR vs the human label. Needs workspaces. | — | -| `npm run show -- ` | Lays out one evaluation instance as readable files in `examples//` (issue, test patch, gold patch, human labels, links). | `--out ` | -| `npm run trajectory -- ` | Renders a trajectory as Markdown next to it. | `--full` (no truncation of tool output) | -| `scripts/finalize-report.py …` | Re-scores and regenerates the README/REPRODUCE tables. Needs workspaces. | — | +How it finds the engine: `FAIRTASK_HOME` if set, else `~/.fairtask`; if neither exists it clones **the pinned +release tag** (`--branch v0.1.0`), never a moving branch, runs `npm ci`, and says so. If the engine cannot run at all +(no Node, no credentials), the skill switches to **manual mode**: it performs the two probes' procedures itself with +its own read-only tools, following `skills/fairtask/references/method.md`, re-opens every location it cites to +confirm the quote, and states in the report that the quotes were self-checked rather than machine-verified. -### Running the pipeline over the evaluation set (needs `claude login` or `ANTHROPIC_API_KEY`) +### `/fairtask-eval` — reproduce and inspect the evaluation -| Command | What it does | Flags | -|---|---|---| -| `npm run run -- --variant --run-id ` | Runs one system over all 30 instances (resumable; refuses to resume a run id with a different configuration). Variants: `baseline`, `v1-context`, `v2-specialists`, `v3-verify`, `v4-calibrated`, `v5-cheap-probes`, `v6-target-aware`, `v7-sonnet-nocal`. | `--model ` · `--only ` · `--concurrency ` (default 3) · `--force` (overwrite the run id) · `--retry-errors` (re-run errored instances) | +Everything a judge or reviewer might want to see, run from inside the session with the output printed verbatim. +No model calls; offline except the two pinned data downloads; it refuses to start paid reproduction runs and points +at `REPRODUCE.md` instead. -### Data (public inputs; committed outputs are byte-exact reproductions) +| You type | What it runs, and shows | +|---|---| +| `/fairtask-eval show the evaluation set` | `npm run data:eval-set` — the 30-row table (instance, stratum, human scores, difficulty, sizes) and the per-stratum and per-repository counts. | +| `/fairtask-eval score baseline v3-verify` | `npm run score -- baseline v3-verify` — decision accuracy, κ, TPR/TNR, missed/false alarms, per-axis agreement, cost and time, side by side. | +| `/fairtask-eval the headline report, baseline versus final` | `node src/report.ts …` — the two tables in §4. | +| `/fairtask-eval audit evidence for v3-verify` | `npm run audit -- v3-verify` — share of cited quotes that do not exist where cited (needs `npm run data:workspaces`, which it tells you to run). | +| `/fairtask-eval check the annotation provenance` | `npm run data:annotations -- --check` — SHA-256 of the committed file against the pinned source. | +| `/fairtask-eval lay out astropy__astropy-12544` | `npm run show -- astropy__astropy-12544` — writes `examples//` and reads back the issue and the test patch. | +| `/fairtask-eval what did the agent do on astropy__astropy-12544 in v3-verify` | renders and opens that run's trajectory. | + +### What the skills may and may not do + +- The screening agents get read-only tools (`Read`, `Grep`, `Glob`) and a hook that denies any path outside the task's + repository — they cannot read the evaluation labels, other tasks, or your files. Only the two declared probes can be + dispatched. Nothing is written to the repository under review; workspaces are throwaway shallow clones. +- `/fairtask` spends money (about $1 per task at list price; $0.55 with `--variant v5-cheap-probes`) and says so. + `/fairtask-eval` spends nothing. +- A verdict is triage, not a decision. The human reviewer adjudicates flagged tasks from the cited lines. -| Command | What it does | Flags | -|---|---|---| -| `npm run data:annotations` | Downloads OpenAI's annotation CSV from its pinned mirror commit; refuses to write it unless the SHA-256 matches. | `--check` (verify the committed copy offline) | -| `npm run data:eval-set` | Rebuilds `data/eval/instances.json` (the 30 cases) from the SWE-bench parquet + annotations and prints the table. | `--n ` (default 30) · `--seed ` (default 20260828) | -| `npm run data:calibration` | Rebuilds `data/eval/calibration.json` (annotator notes for non-evaluation instances, per repository). | — | -| `npm run data:workspaces` | Shallow-clones the 30 repositories at their base commits (~20 s). | `--only ` · `--jobs ` (default 4) | +## 2c. Command reference + +Every command is an npm script. The `--` after `npm run