Record what the published score counts, and what checks are for - #71
Merged
Conversation
Two decisions taken on 29 August that live nowhere in this repository, which is the repository that owns the definition. The website is a consumer of these numbers; it should not be the only place the rule is written down. The published score is scenarios completed. One row is one scenario and its `passed` is the AND of every check in it — what the harness records, what results/latest.json publishes, what a release quotes, and what supabase/evals does. The alternative was tried and reverted on the website side. Counting checks gives partial credit for partial work, which sounds fairer and collides with how our scorers are written: a scorer stops as soon as there is nothing left to check, so a total failure returns 0/1 while a near-miss returns 4/5. Failing worse became cheaper than failing partially. Measured on the 25 August snapshot that ranked the deliberately weaker model above a frontier agent, 97% to 95%, where by scenarios they tie at 89%. The consequence for anyone writing a scorer is the part worth keeping: a check list is not comparable between runs of the same scenario, so nothing should compute a rate from it or compare its length across cells. The early return itself is correct — a scorer that cannot find a tenant has nothing to say about delivery — so the fix belonged at the consumer. Fixed in hookdeck/website#780. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MQzUoMAwEBJWpEGVvVzSjK
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A decision record, no behaviour change. This repository owns the definition of the number; the website is a consumer of it, and until now the rule was written down only there.
What is recorded
The published score is scenarios completed. One row is one scenario, and its
passedis the AND of every check in it — what the harness records, whatresults/latest.jsonpublishes, what a release quotes, and what supabase/evals does (passed / results.length, never an aggregate over checks).Checks are evidence behind a verdict, not a score.
Why, because the alternative was tried
The website briefly shipped check-level percentages, to give partial credit for partial work. Reasonable idea; it collides with how our scorers are written.
A scorer stops as soon as there is nothing left to check. A run that fails at the first hurdle returns
0/1; a near-miss returns4/5:Failing worse is cheaper than failing partially, and each agent's denominator ends up set by its own failures — 59, 62 and 65 checks across six arms of the same nineteen scenarios. On the 25 August snapshot:
They tie by scenarios. By checks the deliberately weaker model reads higher, and the published page ranked it above a frontier agent on its default view.
The part that matters for scorer authors
A check list is not comparable between runs of the same scenario, so nothing should compute a rate from it or compare its length across cells.
The early return itself is correct — a scorer that cannot find a tenant has nothing honest to say about delivery — so the fix belonged at the consumer, not here. That is hookdeck/website#780.
🤖 Generated with Claude Code
https://claude.ai/code/session_01MQzUoMAwEBJWpEGVvVzSjK