From 1ad661725adf38ffaf4a618e65dd24034a7fe4f0 Mon Sep 17 00:00:00 2001 From: Phil Leggetter Date: Thu, 27 Aug 2026 13:30:15 +0100 Subject: [PATCH] Record what the do-not-ask instruction actually bought MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Four cells had tripped ASKED_AND_STOPPED. Re-run under the new base prompt, none of them does: three now pass, and the fourth made 68 tool calls, stopped cleanly and failed 3/5 on the handler-side signature checks — a capability failure on its merits rather than an agent waiting for a person. About $3. Recorded with the caveat that matters more than the number. The detector going 4->0 is the direct result, because that is what the instruction targets. The three flips to passing are consistent with it and are not evidence of it: failing cells, re-run once, no control arm. Reading them as three scenarios bought would be Loop 2's mistake in miniature. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01Nt2Zgjw7STjrnFXYKRRVAA --- AGENTS.md | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/AGENTS.md b/AGENTS.md index 1613dcb..b005a5c 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -661,6 +661,19 @@ whether it worked. Nobody else in the survey does both — Clerk instructs witho measuring, upstream does neither. See hookdeck/evals#57 for the five answers four other benchmarks give. +**Measured, same day, on the four cells that had tripped the detector: it went to +zero.** Three now pass. The fourth (`verification-002` on the weak model) made 68 +tool calls, stopped cleanly, scored 3/5 and failed on the handler-side signature +checks — a capability failure on its merits rather than an agent waiting for a +person. About $3 to find out. + +Read that as one measurement and not four: the detector going 4→0 is what the +instruction targets and is the direct result, while the three flips to passing are +consistent with it and are *not* evidence of it. These were failing cells re-run +once with no control arm, and a failing cell that is re-run can flip on its own. +Claiming the instruction bought three scenarios would be the same mistake as the +9–0 skills result in Loop 2, at a smaller scale. + **Prefer more attempts to more instruction.** `--runs 3` stops at the first pass, so it costs about 1.11× and it removes a stopped run's power to decide a cell without touching the prompt at all. It is the cheapest correction available and