An agent that stops to ask a question scores zero, and we do not track how often

[leggetter]

What

The benchmark is single-turn. An agent that ends its turn with a question gets no reply — it is scored on whatever state it left, which for a clarifying question is usually nothing.

Measured on 25 August, across a 36-cell run of six scenarios against codex-gpt-5.4-mini, three passes per arm:

arm	failures	of which the agent asked and stopped
+skills	5	3
-no-skills	7	0

What those three look like:

"Confirm that this is the right project, and I'll inspect the current connection transform and patch it…"

"I need you to confirm that Automated Testing / evals-ci is the correct Hookdeck project before I inspect connections, deliveries, or retries."

"The remaining setup item is the ElevenLabs webhook secret… once you have that secret, it needs to be set in both Hookdeck source auth and ELEVENLABS_WEBHOOK_SECRET."

None is an agent that could not do the task. Each is an agent checking before acting on a customer's project.

Why it matters more than three cells

It is concentrated in the +skills arm — all three of them. That is a plausible mechanism rather than noise: the skills tell an agent to verify its context, and a weaker model follows the instruction literally, checks with the operator, and stops. If that holds up, loaded skills make an agent more likely to pause, and our scoring turns that into a lower score.

It also contaminates the headline it appears in. The same run's majority put skills at +2 for this model. If three of the five skills-arm failures are caution rather than incapability, the real effect is more positive than +2 — by an amount that cannot be computed, because "would it have succeeded had it not asked?" is not in the data.

What upstream does

Nothing. supabase/evals has the same single-turn design, no allowed questions, no canned answers, and no multi-turn support in its agent driver. Their system prompt is "Use the provided tools to inspect and modify the project. When you are done, end your turn with a short summary." Ours is the same shape, inherited.

So this is structural to both benchmarks, not something upstream solved.

Options, deliberately not taken yet

1. Declare it autonomous — add "you will not receive a reply; proceed on reasonable assumptions" to the base prompt. Removes the failure mode, and changes agent behaviour across every scenario. AGENTS.md already records that adding an instruction can suppress the very failure a scenario exists to catch, so this is not free.
2. Canned answers per scenario — closest to a real support workflow, and an agent asking an unanticipated question still stalls.
3. Track it. Costs nothing and keeps the data honest.

Doing (3) now: triage flags it, so the rate is visible in every run rather than noticed anecdotally. Three cells is enough to notice and not enough to justify rewriting every prompt — and adding "don't ask" today would destroy the ability to measure how often it would have happened.

Worth revisiting once there is a rate across a full matrix rather than one model on six scenarios.

[leggetter]

Wider than filed, and the skills correlation does not hold up.

The detector, run across every stored result rather than the six-scenario run this issue was written from, finds the same behaviour in both Sonnet arms and two further scenarios:

• resolve-002 x claude-code-sonnet-5 — "This key can only access this one project (it's project-scoped, so I can't list others to compare). Since you haven't named a specific org/project, can you confirm this is the right one before I look for failed events and retry them?"
• outpost-003 x claude-code-sonnet-5-no-skills — "Say go-ahead on Why do our skills make the weak model worse? #2 and I'll apply it immediately; Revisit scenario and experiment selection: are we testing the right things, and do skills earn their place? #1 needs to happen in the dashboard since there's no API for it yet."

So it is not concentrated in the +skills arm; my reading was over-fitted to twelve failures on one model. The -no-skills example above is a baseline cell.

Two things worth keeping from those transcripts:

The reasoning behind the pause is usually correct. The first agent worked out that its key is project-scoped and therefore that it cannot confirm which project it is in, and declined to retry events in a project it was not sure about. The second had derived the right answer including that part of the configuration is dashboard-only — the conclusion of #34 — and stopped for approval before changing shared deployment config. Both were right; both scored zero.

It is not rare and it is not one model. Five instances now, across three models and five scenarios, all counted as capability failures.

That strengthens the case for tracking before deciding, and weakens the case for "skills make agents more cautious" as a mechanism. The rate across a full matrix is the number worth having.

[leggetter]

What comparable benchmarks do, 27 August

Checked before deciding, per the point that this is a solved problem elsewhere.

benchmark	architecture	does it address an agent asking questions?
supabase/evals — our upstream	sandboxed tool-using agent, single-turn, step-capped	No. Nothing in the base prompt, no canned answers, no detection anywhere in the repo.
clerk/clerk-evals	two paths: code-generation, and agentic via the claude-code and codex CLIs	Yes, explicitly, on both paths.
Terminal-Bench	agent emits shell commands into a tmux session; harness feeds terminal output back	Structurally cannot arise — there is no channel to ask through. Prompts say nothing about it.
SWE-agent	agentic, tool-using	No explicit instruction found in the default config.

Clerk is the close comparable, and it instructs

They run the same two agent CLIs we do, and the instruction appears in both paths:

src/runners/shared.ts — the system prompt for every evaluation:

Do not ask clarifying questions. Complete the task with the information provided.

src/runners/agents/shared.ts — prepended to every eval prompt on the agentic path:

export async function buildAgentPrompt(evalPath: string): Promise<string> {
  const evalPrompt = await loadPrompt(evalPath)
  return `Do not ask clarifying questions. Complete the task with the information provided.\n\n---\n\n${evalPrompt}`
}

They also pass --dangerously-skip-permissions to Claude Code. Their AGENTS.md does not record why — the instruction is applied, not argued for, so there is no reasoning to borrow, only the precedent.

What this changes about the decision

The objection recorded in this issue was that declaring the benchmark autonomous "would change behaviour everywhere and destroy the ability to measure how often this happens". The first half stands. The second half no longer does, because #58 shipped a detector: we can instruct and keep counting.

Nobody else counts it. Clerk instructs and does not measure whether the instruction works; upstream neither instructs nor measures. Doing both is available to us and is not available to them, which makes it the option with something to learn in it.

Against that, honestly: instructing breaks comparability with every baseline measured before the change. That cost is lower right now than it will be later, because milestone 2 is a fresh three-attempt measurement of both arms and does not lean on the old numbers.

Recommendation

Adopt Clerk's line — or close to it — in basePromptFor(), keep the ASKED_AND_STOPPED detector, and report the count alongside the milestone-2 result. If agents still stop to ask after being told not to, that is a finding about instruction-following worth having; if they stop doing it, the confound is gone and we know it is gone rather than assuming.

Not doing this yet — recording the evidence first, as agreed.

[leggetter]

Wider survey, 27 August

Six benchmarks, five different answers. Grouped by the answer rather than by project, because the option space is the useful output.

benchmark	architecture	answer
supabase/evals	sandboxed tool-using agent, single-turn, step-capped	A — say nothing
clerk/clerk-evals	claude-code + codex CLIs, and a codegen path	B — instruct the agent not to ask
τ-bench / τ²/τ³	tool-agent-user loop	C — answer the question
Terminal-Bench, METR task standard	shell/tmux loop; env + instruction + scorer	D — remove the channel
vercel-labs/agent-eval, get-convex/convex-evals	agent CLIs in a sandbox; codegen	E — make the question unnecessary, absorb the rest

---

A. Say nothing — supabase/evals

basePromptFor() is three sentences about using the tools and ending with a summary. No instruction about questions, no canned answers, no detection anywhere in the repo. Our prompt is a retarget of theirs, so our silence is inherited rather than chosen. Worth being explicit about: we did not decide this.

B. Instruct — Clerk

The only close architectural match that addresses it, and it does so on both paths.

src/runners/shared.ts, system prompt for every evaluation:

Do not ask clarifying questions. Complete the task with the information provided.

src/runners/agents/shared.ts, prepended to every eval prompt on the agentic path:

export async function buildAgentPrompt(evalPath: string): Promise<string> {
  const evalPrompt = await loadPrompt(evalPath)
  return `Do not ask clarifying questions. Complete the task with the information provided.\n\n---\n\n${evalPrompt}`
}

Claude Code is spawned with --print --dangerously-skip-permissions. Their AGENTS.md does not record why, so there is precedent but no reasoning to borrow.

C. Answer it — τ-bench

The opposite design, and the one that should give us most pause.

An LLM plays the user. data/tau2/user_simulator/simulation_guidelines.md:

• Never make up or hallucinate information not provided in the scenario instructions. Information that is not provided in the scenario instructions should be considered unknown or unavailable.
• Disclose information progressively. Wait for the agent to ask for specific information before providing it.

Asking is not merely permitted — it is required, because the simulator withholds until asked. Three terminal tokens end an episode: ###STOP### (goal satisfied), ###TRANSFER###, and ###OUT-OF-SCOPE### for when the scenario cannot answer. That last one is the piece worth stealing conceptually: an unanswerable question is a recorded outcome, not a silent zero.

And the sharpest detail. τ-bench's airline agent policy:

Before taking any actions that update the booking database (booking, modifying flights, editing baggage, upgrading cabin class, or updating passenger information), you must list the action details and obtain explicit user confirmation (yes) to proceed.

The behaviour we score as zero is a pass requirement there. An agent that stops to confirm before mutating a customer's data passes τ-bench and fails us, and the difference is our harness, not the agent. That does not mean we are wrong — a single-turn benchmark has to score something — but it does mean "asked and stopped" should never be published as a capability failure.

D. Remove the channel — Terminal-Bench, METR

Terminal-Bench's agent emits shell commands into a tmux session and receives terminal output. There is no question-shaped action, so the prompts say nothing about it and do not need to.

METR's task standard defines a task as an environment, an instruction string, and a scorer — no user. Its framing is "evaluate language model agents for autonomous capabilities". Human interaction exists but as an explicitly opt-in platform feature: Vivaria's viv run <task> -i / --intervention, which enables human input on agent actions "if the agent code supports this by calling the rate_options or get_input functions". Off by default; scored runs are non-intervention.

E. Make it unnecessary — Vercel, Convex

Neither instructs. Both write spec-shaped prompts — Vercel's PROMPT.md is a task line plus a Requirements: list; Convex's system prompt is one sentence and the task description carries explicit file-path requirements. A requirements list gives an agent nothing to ask about.

Vercel additionally absorbs the variance rather than legislating it:

runs: 10,          // How many times to run each eval (default: 1)
earlyExit: false,  // Stop after first success? (default: true)

With earlyExit and a cached result defined as "at least one passing run", scoring is effectively pass@k. One run that stopped to ask does not decide the cell. That is a third option for us that touches neither the prompt nor the scorer.

The catch for us: we cannot adopt the spec-shaped prompt. AGENTS.md is deliberate that scenarios are ticket-shaped so an agent "can finish confidently and be wrong", and that a scenario guarding a hallucination "asks the question and nothing else" — adding a requirements list is documented there as suppressing the very failure the scenario exists to catch. Vercel and Convex avoid this problem by measuring something we have deliberately chosen not to measure.

---

What I take from it

1. Clerk remains the closest precedent and the recommendation stands. Same agent CLIs, same sandbox, and they instruct on the agentic path.
2. Nobody counts it. Clerk instructs without measuring whether the instruction works. We have the triage: flag an agent that stopped to ask a question #58 detector, so instruct-and-count is available to us and to nobody else in this list.
3. Pass@k is a real alternative and is nearly free for us: milestone 2 is already planned at three attempts per arm. Even without a prompt change, three attempts stop one stopped run from deciding a cell. Worth doing as well as instructing, not instead.
4. Never publish "asked and stopped" as a capability failure. τ-bench requires the same behaviour we penalise. gated_by should treat it as neither discovery nor judgement — it is an artefact of a single-turn harness.

Two findings for other issues

#60 (a snapshot is not a run). Vercel solved the carry-forward problem properly. A SHA-256 fingerprint per (eval, config) pair covers eval files plus result-affecting config; a separate contentFingerprint covers eval files only, so "the eval changed" is distinguishable from "a config field changed". agent-eval refingerprint carries a benign config change forward only where content is unchanged, and a reuseCompatibilityFingerprint marks runtime boundaries that are never carried forward. Our --merge has no invalidation rule at all.

τ³ handles the same problem from the other end: its v1.0.1 release states that "results produced with tau2-bench < 1.0.1 are not comparable with >= 1.0.1", re-grades affected leaderboard submissions, re-scores stored trajectories with tau2 evaluate-trajs --fresh-tasks, and pins a pre-v1.0.1 tag to reproduce old behaviour. Re-scoring stored trajectories is the clean fix for carrying rows forward, and it needs #21.

#61 (the arms read documentation differently). Vercel models exactly this as a controlled treatment. webResearch is opt-in and off by default, gating Claude Code's WebSearch/WebFetch, Codex live search and OpenCode's web tools. Their framing is the distinction #61 needs:

webResearch remains opt-in. […] Whether an agent chooses to use an available research tool remains part of the measured behavior.

Availability is controlled; choice is measured. They also refuse to run rather than let a treatment drift silently — the fx adapter "requires webResearch: true" and Agent Eval "rejects non-research fx runs instead of silently changing the treatment". And disableBundledSkills controls agent-shipped skills while leaving project skills, which is our BASELINE COMPROMISED problem solved at the harness level rather than flagged after the fact.

[leggetter]

Measured, 27 August — the instruction works

Decision implemented in #62; this is the measurement promised alongside it. Re-ran the four cells that had tripped ASKED_AND_STOPPED under the new base prompt.

The detector goes 4 → 0.

cell	before	after
resolve-002 × claude-code-sonnet-5	asked and stopped	pass (2/2)
outpost-003 × claude-code-sonnet-5-no-skills	asked and stopped	pass (4/4)
transform-001 × codex-gpt-5.4-mini	asked and stopped	pass (3/3)
verification-002 × codex-gpt-5.4-mini	asked and stopped	fail (3/5, 68 tool calls, clean stop)

The survivor is a genuine failure now: it created the ElevenLabs source and passed both signature checks on it, then failed the two handler-side Hookdeck checks. Its report ends by offering more work rather than blocking on a question. About $3 for the whole thing.

Read it as one measurement, not four. The 4 → 0 is the direct result — that behaviour is what the instruction targets. The three flips to passing are consistent with it and are not evidence of it: failing cells, one attempt each, no control arm. Calling it three scenarios bought would be Loop 2's mistake at a smaller scale.

This is what the survey said nobody else does. Clerk instructs and never checks; upstream neither instructs nor checks. We now know the instruction changes behaviour on the cells it was aimed at, rather than assuming it.

Recorded in AGENTS.md via #63. Not a loop — it changed our instrument, not the product, docs or skills.