triage: flag an agent that stopped to ask a question

[leggetter]

The benchmark is single-turn, so an agent that ends its turn with a question gets no reply and is scored on whatever state it left — usually nothing. That is not an agent failing the task; it is an agent checking before it touches a customer's project.

Five instances across three models and five scenarios, every one counted as a capability failure.

The reasoning behind the pause is usually correct

This key can only access this one project (it's project-scoped, so I can't list others to compare). Since you haven't named a specific org/project, can you confirm this is the right one before I look for failed events and retry them?

Say go-ahead on #2 and I'll apply it immediately; #1 needs to happen in the dashboard since there's no API for it yet.

The first worked out that its key is project-scoped and therefore that it cannot verify which project it is in, and declined to retry events in a project it was unsure about. The second had derived the right answer including that part of the configuration is dashboard-only — which is the conclusion of #34 — and stopped for approval before changing shared deployment config.

Both were right. Both scored zero.

Flagged, not fixed

Declaring the benchmark autonomous in the base prompt would remove the failure mode and change behaviour across every scenario — and AGENTS.md already records that adding an instruction can suppress the very failure a scenario exists to catch. It would also destroy the ability to measure how often this happens.

supabase/evals has the same single-turn design: no allowed questions, no canned answers, no multi-turn support in its agent driver. Staying comparable matters, because our floor-versus-discrimination reasoning is calibrated against their ratios.

So: make the rate visible in every run, then decide with a number rather than an impression. #57.

It corrected the issue it came from

I filed #57 saying all three known cases were in the +skills arm, and read that as a mechanism — skills tell an agent to verify its context, weak model follows literally. Running the detector over every stored result found two more, one of them a baseline cell. The correlation does not hold; my reading was over-fitted to twelve failures on one model.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Nt2Zgjw7STjrnFXYKRRVAA

[leggetter]

Corrected before merge: the comment on ASKED_AND_STOPPED claimed all three cells were in the +skills arm and that the baseline never stopped to ask. Running the detector against the stored runs contradicts it — four of twelve failures, three in +skills arms and one in a baseline (outpost-003 × claude-code-sonnet-5-no-skills).

The original claim came from a hand-read of a subset before this detector existed, which is the thing the detector is for. Recorded in the comment rather than quietly deleted, because the wrong version was doing work: an effect confined to one arm reads as caused by that arm, and a skills-tells-agents-to-verify story was available to explain it. It is not confined to one arm.

The conclusion is unchanged — four cells is enough to move a delta measured at two in twenty-four, which is why #57 wants deciding before the milestone-2 run.