From 71f5569484b863080ce057e3bdb82cdde10bbfe1 Mon Sep 17 00:00:00 2001 From: Phil Leggetter Date: Sat, 29 Aug 2026 10:31:01 +0100 Subject: [PATCH] Record what the published score counts, and what checks are for MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two decisions taken on 29 August that live nowhere in this repository, which is the repository that owns the definition. The website is a consumer of these numbers; it should not be the only place the rule is written down. The published score is scenarios completed. One row is one scenario and its `passed` is the AND of every check in it — what the harness records, what results/latest.json publishes, what a release quotes, and what supabase/evals does. The alternative was tried and reverted on the website side. Counting checks gives partial credit for partial work, which sounds fairer and collides with how our scorers are written: a scorer stops as soon as there is nothing left to check, so a total failure returns 0/1 while a near-miss returns 4/5. Failing worse became cheaper than failing partially. Measured on the 25 August snapshot that ranked the deliberately weaker model above a frontier agent, 97% to 95%, where by scenarios they tie at 89%. The consequence for anyone writing a scorer is the part worth keeping: a check list is not comparable between runs of the same scenario, so nothing should compute a rate from it or compare its length across cells. The early return itself is correct — a scorer that cannot find a tenant has nothing to say about delivery — so the fix belonged at the consumer. Fixed in https://github.com/hookdeck/website/pull/780. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01MQzUoMAwEBJWpEGVvVzSjK --- AGENTS.md | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+) diff --git a/AGENTS.md b/AGENTS.md index b005a5c..48baca6 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -682,6 +682,31 @@ copied is upstream's other route: Vercel and Convex avoid clarifying questions b writing prompts as requirement lists, and this repo has measured that adding an instruction to build *suppresses the very failure a scenario exists to catch*. +**The published score is scenarios completed, and checks are evidence rather +than a score.** One row is one scenario — a ticket an agent was given — and its +`passed` is the AND of every check in it. That is what the harness records, what +`results/latest.json` publishes, what a release quotes and what the website +prints. Upstream is the same: supabase/evals computes `passed / results.length` +and never aggregates checks into a number. + +Decided against the alternative on 29 August, after the website briefly shipped +check-level percentages to give partial credit for partial work. The idea is +reasonable and it does not survive contact with scorers that short-circuit: +**a scorer stops as soon as there is nothing left to check, so a run that fails +at the first hurdle returns `0/1` where a near-miss returns `4/5`.** Failing +worse becomes cheaper than failing partially, and each agent's denominator ends +up set by its own failures — 59, 62 and 65 checks across six arms of the same +nineteen scenarios. On the 25 August snapshot it put the deliberately weaker +model at 97% against a frontier agent's 95%, where by scenarios they tie at 89%, +and the page ranked them in that order. Check counts also become weights: they +run from one to five per scenario, set by how each scorer happens to be written. + +Two things follow for scorer authors. A check list is **not** comparable between +runs of the same scenario, so never compute a rate from it or compare its length +across cells. And an early return is still the right shape — a scorer that +cannot find a tenant has nothing to say about delivery — so the fix is at the +consumer, not here. + **Classify what gates a scenario, and read failures along it.** Every scenario carries `gated_by` in its frontmatter: `discovery`, `judgement` or `mixed`. The test is one question — **if we improved our docs and skills, could this cell go