Tell agents not to ask clarifying questions

[leggetter]

Closes #57.

The change

One sentence added to basePromptFor(), borrowed verbatim from clerk/clerk-evals:

Do not ask clarifying questions. Complete the task with the information provided.

Why

This benchmark is single-turn. A question gets no reply and the agent is scored on whatever state it left, usually nothing. Four cells in the stored runs ended that way, against a skills delta measured at two cells in twenty-four — the artefact was larger than the signal sitting next to it.

Clerk is the closest architectural match: same two agent CLIs, same sandbox model, and it prepends that exact line to every eval prompt on its agentic path. Borrowed precedent is worth more than a marginally better sentence of our own.

What this is not

It is not a claim that asking is wrong. τ-bench's airline policy requires an agent to "list the action details and obtain explicit user confirmation (yes) to proceed" before any action that updates the booking database. The behaviour suppressed here is a pass requirement there, because that harness has a user to answer and ours does not.

So the ASKED_AND_STOPPED detector from #58 stays. The instruction removes the confound; the detector says whether it worked. Nobody else in the survey does both — Clerk instructs without measuring, upstream does neither.

The full option space is on #57: five different answers across six benchmarks.

Comparability

Everything measured before this commit was measured without the line. Results either side are not comparable, the same rule as a scenario or CLI change and for the same reason. EVALS_PUBLISH is set to false so Monday's cron and the 1 September full matrix cannot publish a snapshot that is half one treatment and half the other — this compounding with #60 is exactly the failure mode that needed heading off.

Also in this PR

AGENTS.md gains the reasoning under Conventions, plus two rules worth having:

• The base prompt is a treatment, not neutral scaffolding — it is the one string every cell shares, so a word added to it moves every number at once.
• Prefer more attempts to more instruction. --runs 3 stops at the first pass, costs ~1.11×, and removes a stopped run's power to decide a cell without touching the prompt.

The cost table is corrected. It was denominated in fifteen scenarios and there are nineteen, so a full matrix is about $81, not $64. The measured figures are left as measured with the scaling stated separately, because conflating the two is how the judge once came to be reported at twenty-eight times its real cost.

Multi-attempt costing is now documented: 1 + 0.114k where k is what a failed attempt costs relative to a typical one. k is unknown — only 4 of 114 published rows carry usage.costUsd and all four passed, so there is no failed-cell cost data. See #6. Budget a three-attempt matrix at $90–$110.

Verification

pnpm typecheck clean, pnpm -r test green (142 tests), formatted.

Not yet measured: whether the instruction actually changes behaviour. That is the next step — re-run the four known ask-and-stop cells and check the detector.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Nt2Zgjw7STjrnFXYKRRVAA