From abb2e3a2b56e202aff7a43eeca33f0944b0e97f4 Mon Sep 17 00:00:00 2001 From: Phil Leggetter Date: Thu, 27 Aug 2026 12:59:00 +0100 Subject: [PATCH] Tell agents not to ask clarifying questions MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit This benchmark is single-turn, so a question gets no reply and the agent is scored on whatever state it left — usually nothing. Four cells in the stored runs ended that way, against a skills delta measured at two cells in twenty-four. The artefact was larger than the signal beside it. The line is borrowed verbatim from clerk/clerk-evals, the closest architectural match to this benchmark: same two agent CLIs, same sandbox, and it prepends that exact sentence to every eval prompt on its agentic path. Precedent is worth more here than a marginally better sentence. It is not a claim that asking is wrong. tau-bench's airline policy *requires* an agent to obtain explicit user confirmation before any action that updates the booking database — the behaviour suppressed here is a pass requirement there, because that harness has a user to answer and ours does not. So the triage detector stays: the instruction removes the confound and the detector says whether it worked. Clerk instructs without measuring; upstream does neither. Everything measured before this commit was measured without the line, so results either side are not comparable. AGENTS.md also gains the reasoning, the rule that a cell tripping ASKED_AND_STOPPED must never be published as a capability failure, and a correction to the cost table: it is denominated in fifteen scenarios and there are nineteen, so a full matrix is about $81 rather than $64. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01Nt2Zgjw7STjrnFXYKRRVAA --- AGENTS.md | 59 +++++++++++++++++++++++++++++- apps/framework/harness/run-eval.ts | 29 ++++++++++++++- 2 files changed, 86 insertions(+), 2 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 2382e0f..1613dcb 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -108,7 +108,7 @@ as six agent failures because of it. Runs bill per token and the numbers matter, because cadence is chosen against them and the choice is otherwise re-derived from scratch every time. -Measured per full pass of fifteen scenarios: +Measured per full pass of **fifteen** scenarios: | | | |---|---| @@ -118,10 +118,31 @@ Measured per full pass of fifteen scenarios: | LLM judge (`gpt-5.5`) | about $0.10 | | Full matrix, six experiments | about $64 | +**There are nineteen benchmark scenarios now, so scale those figures by about +1.27: roughly $81 a full matrix.** The table is left at its measured denominator +rather than rewritten, because the figures were measured and the scaling is +arithmetic, and conflating the two is how the judge came to be reported at +twenty-eight times its real cost. Re-measure rather than re-scale if a decision +turns on it. + Weekly frontier plus weak pair is roughly $185 a month. Everything weekly would be $279 against a $200 budget, and the `-no-skills` twins are what gets cut to monthly, because their delta moves slowly. +**Multiple attempts are far cheaper than they look, because `--runs` stops at the +first pass.** `STOP_ON_PASS` is the default, so a retry is only paid for on a cell +that failed. At the current pass rates the expected attempts per cell at +`--runs 3` is **1.11**, not 3 — and it is unevenly distributed, 1.06 for the +frontier arms against 1.25 for the weak model, which is also where the signal is. + +The true multiplier is somewhat higher, because the attempts that repeat are by +construction the failed ones and a failing cell tends to run to the timeout. It +is `1 + 0.114k` where `k` is what a failed attempt costs relative to a typical +one — 1.11× at k=1, 1.34× at k=3. **`k` is not known**: only 4 of 114 published +rows carry `usage.costUsd` and all four passed, so there is no failed-cell cost +data to derive it from. See #6. Budget a three-attempt matrix at $90–$110 and +treat anything more precise as a residual rather than a measurement. + **Failed runs cost the same as successful ones.** On 13 August the OpenAI credit balance reached zero mid-run; thirty-seven Codex jobs then started, failed and were paid for in wall-clock time, and a further twenty-two runs from earlier @@ -612,6 +633,42 @@ real catches, so a switch is safe; it is just not worth making, because these checks are mostly negatives guarding against invented capabilities and the saving is trivial. +**The base prompt is a treatment, not neutral scaffolding.** It is the one string +every cell in every experiment shares, so a word added to it moves every number +at once and is indistinguishable in a snapshot from the thing being measured. +Change it the way a scenario or a CLI pin is changed: deliberately, in its own +commit, and with the release saying results either side are not comparable. + +It carries **"Do not ask clarifying questions. Complete the task with the +information provided."** since 27 August, borrowed verbatim from +[clerk/clerk-evals](https://github.com/clerk/clerk-evals) — the closest +architectural match to this benchmark, running the same two agent CLIs, which +prepends that line to every eval prompt on its agentic path. + +The reason is that this benchmark is single-turn. A question gets no reply and +the agent is scored on whatever state it left, usually nothing. Four cells in the +stored runs ended that way, against a skills delta of two cells in twenty-four: +the artefact was larger than the signal beside it. + +**An agent that stops to ask is not failing at anything, and must never be +published as a capability failure.** τ-bench's airline policy *requires* an agent +to "list the action details and obtain explicit user confirmation (yes) to +proceed" before any action that updates the booking database. The behaviour this +prompt suppresses is a pass requirement there, because that harness has a user to +answer and ours does not. `triage` keeps flagging it (`ASKED_AND_STOPPED`) for +exactly this reason: the instruction removes the confound, and the detector says +whether it worked. Nobody else in the survey does both — Clerk instructs without +measuring, upstream does neither. See hookdeck/evals#57 for the five answers four +other benchmarks give. + +**Prefer more attempts to more instruction.** `--runs 3` stops at the first pass, +so it costs about 1.11× and it removes a stopped run's power to decide a cell +without touching the prompt at all. It is the cheapest correction available and +should be reached for before anything is added to the base prompt. What cannot be +copied is upstream's other route: Vercel and Convex avoid clarifying questions by +writing prompts as requirement lists, and this repo has measured that adding an +instruction to build *suppresses the very failure a scenario exists to catch*. + **Classify what gates a scenario, and read failures along it.** Every scenario carries `gated_by` in its frontmatter: `discovery`, `judgement` or `mixed`. The test is one question — **if we improved our docs and skills, could this cell go diff --git a/apps/framework/harness/run-eval.ts b/apps/framework/harness/run-eval.ts index bca1f3f..158b973 100644 --- a/apps/framework/harness/run-eval.ts +++ b/apps/framework/harness/run-eval.ts @@ -335,12 +335,39 @@ function copyWithheldTests(ev: EvalManifest, workspace: string) { } } +/** + * The base prompt is a treatment, not neutral scaffolding. + * + * The last sentence was added on 27 August, borrowed verbatim from + * [clerk/clerk-evals](https://github.com/clerk/clerk-evals), which prepends the + * same line to every eval prompt on its agentic path — the closest architectural + * match to ours, running the same two agent CLIs. + * + * It exists because this benchmark is single-turn: a question gets no reply, and + * the agent is scored on whatever state it left, usually nothing. Four cells in + * the stored runs ended that way, against a skills delta measured at two cells in + * twenty-four, so the noise was larger than the signal it sat next to. + * + * **This is not a claim that asking is wrong.** τ-bench's airline policy requires + * an agent to "list the action details and obtain explicit user confirmation + * (yes) to proceed" before any action that updates the booking database — the + * behaviour penalised here is a *pass requirement* there, because that harness + * has a user to answer. Ours does not. So the line is a property of our + * instrument, and a cell that trips `ASKED_AND_STOPPED` must never be published + * as a capability failure. See the survey on hookdeck/evals#57 for the full + * option space; four other benchmarks answer this four different ways. + * + * Everything measured before this date was measured without it, so results + * either side are not comparable — the same rule as a scenario or CLI change, + * and for the same reason. + */ function basePromptFor(_mode: EvalMode): string { return ( 'You are an agent working on a real Hookdeck project, as a developer would. ' + 'Use the provided tools to inspect and change the project. ' + 'When you are done, end your turn with a short summary of what you did ' + - '(or for investigation tasks, your findings).' + '(or for investigation tasks, your findings). ' + + 'Do not ask clarifying questions. Complete the task with the information provided.' ); }