From 4e532b874288c0ed4269d87bb225060c00d0a519 Mon Sep 17 00:00:00 2001 From: Manos Kaparos <> Date: Sat, 29 Aug 2026 17:00:56 +0300 Subject: [PATCH] readme: agent trajectories section; reproduce: expected output for screen and score Co-Authored-By: Claude Fable 5 --- README.md | 45 +++++++++++++++++++++++++++++++++++++++------ REPRODUCE.md | 39 +++++++++++++++++++++++++++++++++++---- 2 files changed, 74 insertions(+), 10 deletions(-) diff --git a/README.md b/README.md index db80628..6924500 100644 --- a/README.md +++ b/README.md @@ -66,6 +66,7 @@ Django pull request #11099. Full command reference: [§2c](#2c-command-reference | Screen my own tasks from a shell | [§2a · Use it from a shell](#2a-use-it-from-a-shell) | | Use it from inside an agent (the agentic interface) | [§2b · Use it from inside your agent](#2b-use-it-from-inside-your-agent) | | Look up a command and its flags | [§2c · Command reference](#2c-command-reference) | +| Follow what each agent did, tool by tool, including retries | [§2d · Agent trajectories](#2d-agent-trajectories) | | Reproduce every number from a clean machine | [REPRODUCE.md](REPRODUCE.md) | | Understand the problem and who has it | [§1 · Who has this problem](#1-who-has-this-problem-and-what-is-the-bottleneck) | | See how the pipeline works | [§2 · What the solution does](#2-what-the-solution-does) | @@ -265,23 +266,26 @@ Give it any of three things; it does the rest and reports in the conversation. | `/fairtask my-task.json` | Same for your own task file (`repo`, `base_commit`, `problem_statement`, `patch`, `test_patch`, `FAIL_TO_PASS`). | | `/fairtask astropy/astropy 12544` | A pull request. The skill first runs `scripts/task-from-pr.sh`, which builds the task file from the PR with `gh` (base commit, the linked issue's text, code diff, test diff, added tests). It **fails closed** — exits with reasons — if the PR has no linked issue, no test file, or no graded test it can confirm; the skill reports those reasons and stops unless you explicitly accept a provisional screening. Then it screens the task. | -What comes back, verbatim from the session that produced `examples/psf__requests-2317/`: +What comes back — the engine's own output for the committed example `examples/psf__requests-2317/` (the skill +reads it from `verdict.json` and repeats it in the conversation, then says what to do with it): ``` USABLE underspecified=0 false_negative=1 confidence=4/5 (evidence verified) -Issue specification (0): the issue names the exact file and line (requests/sessions.py: method = builtin_str(method)), -the wrong behaviour (b'GET' becomes the literal "b'GET'" under Python 3, yielding 404), and the desired behaviour … +Issue specification (0): The issue names the exact file and line (`requests/sessions.py`: `method = builtin_str(method)`), the exact wrong behavior (b'GET' becomes the literal string "b'GET'" under Python 3, yielding 404), and the desired behavior ("if requests handled the method value being a binary string, we wouldn't have any problem"). I confirmed the buggy line exists at requests/sessions.py:428 and that the canonical helper `to_native_string` is already imported in the very same module at line 21 and already used a few lines above at line 136 (`prepared_request.url = to_native_string(url)`), so the conventional fix is directly signposted by existing code. There is essentially no ambiguity about what a successful solution looks like. -Test scope (1): the new test asserts only that requests.request(b'GET', …).ok, so any normalisation strategy passes … +Test scope (1): The new test `test_encoded_methods` is implementation-agnostic: it only does `r = requests.request(b'GET', httpbin('get')); assert r.ok`. It never inspects the method's type, never references `to_native_string`, and imposes no naming or message constraints, so a fix via `to_native_string`, `method.decode('ascii')`, or an `isinstance(method, bytes)` branch all pass equally. The only residual risk is that the FAIL_TO_PASS list bundles seven pre-existing, network-dependent tests (test_HTTP_302_ALLOW_REDIRECT_GET, test_POSTBIN_GET_POST_FILES, test_basicauth_with_netrc, test_json_param_post_content_type_works, etc.) that hit `HTTPBIN = os.environ.get('HTTPBIN_URL', 'http://httpbin.org/')`. I spot-checked these (e.g. test_requests.py:140-144 calls `requests.get(httpbin('redirect', '1'))` with a plain string method) and they do not touch the patched code path at all, so they are neutral between competing correct solutions rather than biased toward the gold patch — they add environment flakiness risk equal for gold and for any reasonable fix, which keeps this at 1 rather than 2. Evidence (3): [underspecified] repo requests/sessions.py:L21 + The helper needed for the fix, to_native_string, is already imported in sessions.py at the base commit, making the conventional fix directly discoverable from the issue's pointer to the buggy line. › from .utils import to_key_val_list, default_headers, to_native_string [false_negative] repo test_requests.py:L140-L142 - › def test_HTTP_302_ALLOW_REDIRECT_GET(self): r = requests.get(httpbin('redirect', '1')) … + The extra FAIL_TO_PASS tests are pre-existing network tests using plain-string methods that cannot depend on the bytes-method fix; they only add environment flakiness, equally for any solution. + › def test_HTTP_302_ALLOW_REDIRECT_GET(self): r = requests.get(httpbin('redirect', '1')) assert r.status_code == 200 [false_negative] test_patch test_requests.py - › r = requests.request(b'GET', httpbin('get')); assert r.ok + The new test asserts only that the request succeeds, imposing no constraint on how bytes methods are normalized. + › r = requests.request(b'GET', httpbin('get')) assert r.ok $0.49 · 195s · 8 turns ``` @@ -393,6 +397,35 @@ a complete command; rows with the same script differ by one flag. | `npm test` | 23 adversarial tests: verifier (fabricated, elided, cross-hunk, removed-line and symlinked quotes), workspace trust (symlinked path, dirty tree, wrong remote, bad commit), run lock and screening ids. | | `npm run validate:manifests` | Semantic check of the plugin manifests and both skills' frontmatter. | +## 2d. Agent trajectories + +Every run of every agent is recorded as JSONL and rendered to Markdown next to it (`npm run trajectory -- `). +A rendered trajectory reads top to bottom as: **Agent instructions** (the judge's system prompt with the rubric, the +task prompt, and the full instructions of each subagent it may dispatch) → **Execution** (every tool call with its +input, every tool result or error, each probe's report, the judge's own spot-checks) → **Result** (the JSON verdict, +cost and turns) → **Verification** (what the deterministic verifier found; if it failed, the feedback sent back and +the corrective turn). The representative ones, one per agent and one per kind of feedback: + +| Agent / situation | Trajectory | What to look at | +|---|---|---| +| **Baseline** — one prompt, no tools | [`trajectories/baseline/astropy__astropy-12544.md`](trajectories/baseline/astropy__astropy-12544.md) | Instructions and task prompt, then the verdict straight away. It flags the task (humans: flag, `false_negative` 3) but also scores the issue 2 where humans gave 0 — the right decision for half the right reasons; every later configuration keeps the flag and the same over-score on the issue axis. | +| **Single agent with repository tools** (v1) | [`trajectories/v1-context/astropy__astropy-12544.md`](trajectories/v1-context/astropy__astropy-12544.md) | The same prompt plus `Read`/`Grep`/`Glob`: what it reads, and how reading the code made it *more* lenient (§5, Iteration 1). | +| **Judge + `spec-probe` + `test-probe`** (v3, the final default) | [`trajectories/v3-verify/astropy__astropy-12544.md`](trajectories/v3-verify/astropy__astropy-12544.md) | Instructions for all three agents (`### Subagent spec-probe`, `### Subagent test-probe`); the judge dispatching both in parallel; each probe's `Read`/`Grep` calls and the file contents that came back; the two probe reports with evidence; the judge re-opening the cited lines itself ("Now spot-checking the strongest claims myself"); the final verdict; `✅ Verification passed`. | +| **Verifier feedback and a corrective turn** | [`trajectories/v4-calibrated/scikit-learn__scikit-learn-11574.md`](trajectories/v4-calibrated/scikit-learn__scikit-learn-11574.md) · [`trajectories/v5-cheap-probes/sphinx-doc__sphinx-8284.md`](trajectories/v5-cheap-probes/sphinx-doc__sphinx-8284.md) | `⛔ Verification failed (attempt 1)`: the verifier names the evidence item whose quote is not in the test patch, the exact feedback text sent to the judge, the judge's re-check with `Grep`, and the corrected verdict that passes. These are the only two corrective turns in 300 evaluation runs — the announcement that quotes will be checked did the work (§5, Iteration 3). | +| **A tool refusing the agent** | same `sphinx-doc__sphinx-8284.md` | `❌ error` entries: both probes, then the judge, asked to `Read` paths that do not exist (a mistyped directory, then an absolute `/repo/…` path); the tool answered `File does not exist` with the correct working directory, and each agent corrected its path on the next call. | +| **The strict-reviewer framing** (v6) | [`trajectories/v6-target-aware/astropy__astropy-12544.md`](trajectories/v6-target-aware/astropy__astropy-12544.md) | The same case under the instructions that name the reference label as a maximum over three annotators. | +| **A real screening from the skill** | [`examples/psf__requests-2317/trajectory.md`](examples/psf__requests-2317/trajectory.md) · [`examples/django__django-11099/trajectory.md`](examples/django__django-11099/trajectory.md) | Produced by `/fairtask` on a task outside the evaluation set: no human label in the header, the calibration block the v5 configuration adds, the verdict the skill reported in §2b. | + +**Human checkpoints.** The pipeline has three, all deliberate: (1) `task-from-pr.sh` fails closed when it cannot +confirm the issue or the graded tests, and the skill reads the task back for a one-line confirmation before spending +money; (2) a verdict marked `verified: false` after the corrective turns is refused — `screen` exits non-zero and the +skill says so, rather than reporting unverified evidence; (3) every FLAG is a recommendation with cited lines, and a +human adjudicates it. Nothing is dropped from a task set on the agent's word. + +All 300 evaluation trajectories (10 runs × 30 cases) are under `trajectories//`, and every one of them embeds +the exact instructions its agents ran with, so a prompt change in `src/variants/` or `src/lib/rubric.ts` can be +matched to the runs that used it. + --- ## 3. Evaluation design diff --git a/REPRODUCE.md b/REPRODUCE.md index 4f1b708..48497a7 100644 --- a/REPRODUCE.md +++ b/REPRODUCE.md @@ -1,6 +1,6 @@ # Reproduction guide -Everything below was run on macOS 15 (Apple Silicon) with the versions listed; nothing is platform-specific except +Written for a clean machine: nothing is assumed beyond the requirements in §0. Everything below was run on macOS 15 (Apple Silicon) with the versions listed; nothing is platform-specific except that the Claude Code CLI must be installable. ## 0. Requirements @@ -59,12 +59,27 @@ npm run data:workspaces # shallow-clones 30 repos into workspaces//`. -From inside an agent instead of a shell: `npx skills add mnkprs/fairtask`, then `/fairtask `. +What to expect on the terminal (this is the committed run `examples/psf__requests-2317/`; the head lines name the +task, the variant, the model and the cloned workspace, then the verdict): + +``` +fairtask · psf__requests-2317 · v5-cheap-probes · claude-opus-5 +repository psf/requests @ 091991be0d (cloned) → workspaces/psf__requests-2317/repo +──────────────────────────────────────────────────────────────────────── +USABLE underspecified=0 false_negative=1 confidence=4/5 (evidence verified) +… +$0.49 · 195s · 8 turns +written: screenings/psf__requests-2317/verdict.json (trajectory: screenings/psf__requests-2317/trajectory.jsonl; …) +``` + +A verdict whose evidence still fails verification after two corrective turns is not written as a result: `screen` +prints `NO VERDICT` and exits 1. From inside an agent instead of a shell: `npx skills add mnkprs/fairtask`, then +`/fairtask ` (README §2b). ## 4. Run the baseline and the solution over the evaluation set @@ -114,7 +129,23 @@ scripts/finalize-report.py v3-verify baseline baseline-rerun v1-context v2-speci The scorer compares every verdict with the human labels and prints the comparison table used in the README; it also writes `results//summary.json`. The primary metric is **decision accuracy**: does the system's -usable/flag decision match the humans' (usable ⇔ both scores ≤ 1)? +usable/flag decision match the humans' (usable ⇔ both scores ≤ 1)? The first lines of `npm run score -- baseline v3-verify` +on the committed results: + +``` + baseline v3-verify +-------------------------------------------------------------- +instances scored / expected 30/30 30/30 +verdicts failing verification 0 0 +PRIMARY decision accuracy 67% 67% +balanced accuracy 65% 63% +Cohen's kappa vs humans 0.29 0.25 +flag precision / recall 78%/70% 75%/75% +``` + +A reproduction run lands within ±2 cases of these (the baseline repeated at 67%/67%; agent variants at 60–70% +across eight runs, see README §4). Representative trajectories, including the two runs where the verifier sent +feedback and the judge corrected its evidence, are listed in README §2d. ## 6. Expected runtime and cost