Report on the scenario criteria we wrote down and never checked - #72
Open
leggetter wants to merge 1 commit into
Open
Report on the scenario criteria we wrote down and never checked#72leggetter wants to merge 1 commit into
leggetter wants to merge 1 commit into
Conversation
CONTRIBUTING.md has asked for two things since before the first scenario existed: a `motivation:` citing evidence that developers actually hit this, and at least one agent failing a scenario before it joins the published benchmark. Neither has ever been checked. Measured today: no motivation carries a citation, and five scenarios are in the published suite that no agent has ever failed — one of them across twelve measurements. That is the same lesson triage.ts exists for, arrived at a second time: an exhortation loses to a scoreboard, because the scoreboard is right there and the convention is in a file nobody opens while writing a scenario. `scenario-criteria` prints, per benchmark scenario, whether the motivation carries a citation, how many distinct measurements it has, and how many of those failed. It also prints coverage by product and stage, so the gap is visible before the next scenario is chosen rather than after: event-gateway 14 against outpost 5 and console 0, build 13 against resolve 4 and investigate 2. It reports rather than gates. Retro-fitting citations onto scenarios whose origin nobody can now reconstruct is not work a script should force, and a red build for unreachable history teaches people to disable the check. Two details worth knowing. What counts as a citation is deliberately loose, because this repository is public and CONTRIBUTING.md requires motivations that carry evidence without disclosing it — "Support ticket, June 2026" has to pass, so it cannot demand a link. What it rejects is the shape every current motivation has: an assertion about what users probably do. And measurements are deduplicated by `sourcePath` and `ranAt` together: `sourcePath` alone is stable per (scenario, experiment), so deduplicating on it collapses every measurement a pair has ever had into one, which is how #60's republished rows looked like repeat evidence. CONTRIBUTING.md also gains a "where scenarios come from" list, ordered by how well each source has actually worked — transcripts first, a coverage gap last, because a scenario written to fill a cell in a matrix has nothing behind it saying anyone gets this wrong. That is how four Outpost scenarios came to be passed by every agent. Also retargets what was still upstream's: the CI section described labels this repository does not have and linked to supabase/evals' workflow, and the seed descriptions named Supabase's project layout. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MQzUoMAwEBJWpEGVvVzSjK
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Groundwork for #47, and it turns out #47 is not an Outpost problem.
The gap
CONTRIBUTING.md already asks for both of the things upstream asks for:
motivation:citing evidence that developers actually hit this;Neither has ever been checked. Measured today:
Three of the five are Event Gateway scenarios. #47 framed this as an Outpost problem; it is a suite-wide one, and it was invisible because nothing counted.
What this adds
pnpm --filter @hookdeck-evals/framework scenario-criteriareports per scenario: citation, measurements, failures. Plus coverage, so a gap is visible before the next scenario is chosen rather than after —event-gateway 14, outpost 5andbuild 13, resolve 4, investigate 2.It reports; it does not gate. Retro-fitting citations onto scenarios whose origin nobody can reconstruct is not work a script should force, and a red build for unreachable history teaches people to disable the check.
Two implementation details worth review:
sourcePathandranAt.sourcePathis stable per (scenario, experiment), so deduplicating on it alone collapses every measurement a pair has ever had into one — the same trap as A published snapshot mixes execution dates, so counting snapshots overcounts measurements #60, where republished rows read as repeat evidence.CONTRIBUTING.md
Gains a "where scenarios come from" list, ordered by how well each source has actually worked here: transcripts first (every product finding came from one), then support tickets, then our own troubleshooting pages, then GitHub issues, and a coverage gap last — a scenario written to fill a cell in a matrix has nothing behind it saying anyone gets this wrong, which is how four Outpost scenarios came to be passed by every agent.
Also retargets what was still upstream's: the CI section described PR labels this repository does not have and linked to supabase/evals' workflow, and the seed descriptions named Supabase's project layout.
🤖 Generated with Claude Code
https://claude.ai/code/session_01MQzUoMAwEBJWpEGVvVzSjK