From 82ffa12a72212b722be6cc62db6e908a9dbcdcc1 Mon Sep 17 00:00:00 2001 From: Phil Leggetter Date: Sat, 29 Aug 2026 12:00:01 +0100 Subject: [PATCH] Report on the scenario criteria we wrote down and never checked MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit CONTRIBUTING.md has asked for two things since before the first scenario existed: a `motivation:` citing evidence that developers actually hit this, and at least one agent failing a scenario before it joins the published benchmark. Neither has ever been checked. Measured today: no motivation carries a citation, and five scenarios are in the published suite that no agent has ever failed — one of them across twelve measurements. That is the same lesson triage.ts exists for, arrived at a second time: an exhortation loses to a scoreboard, because the scoreboard is right there and the convention is in a file nobody opens while writing a scenario. `scenario-criteria` prints, per benchmark scenario, whether the motivation carries a citation, how many distinct measurements it has, and how many of those failed. It also prints coverage by product and stage, so the gap is visible before the next scenario is chosen rather than after: event-gateway 14 against outpost 5 and console 0, build 13 against resolve 4 and investigate 2. It reports rather than gates. Retro-fitting citations onto scenarios whose origin nobody can now reconstruct is not work a script should force, and a red build for unreachable history teaches people to disable the check. Two details worth knowing. What counts as a citation is deliberately loose, because this repository is public and CONTRIBUTING.md requires motivations that carry evidence without disclosing it — "Support ticket, June 2026" has to pass, so it cannot demand a link. What it rejects is the shape every current motivation has: an assertion about what users probably do. And measurements are deduplicated by `sourcePath` and `ranAt` together: `sourcePath` alone is stable per (scenario, experiment), so deduplicating on it collapses every measurement a pair has ever had into one, which is how #60's republished rows looked like repeat evidence. CONTRIBUTING.md also gains a "where scenarios come from" list, ordered by how well each source has actually worked — transcripts first, a coverage gap last, because a scenario written to fill a cell in a matrix has nothing behind it saying anyone gets this wrong. That is how four Outpost scenarios came to be passed by every agent. Also retargets what was still upstream's: the CI section described labels this repository does not have and linked to supabase/evals' workflow, and the seed descriptions named Supabase's project layout. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01MQzUoMAwEBJWpEGVvVzSjK --- AGENTS.md | 15 ++ CONTRIBUTING.md | 58 +++++- apps/framework/package.json | 3 +- apps/framework/scripts/scenario-criteria.ts | 189 ++++++++++++++++++++ 4 files changed, 255 insertions(+), 10 deletions(-) create mode 100644 apps/framework/scripts/scenario-criteria.ts diff --git a/AGENTS.md b/AGENTS.md index dba108c..c2dc45c 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -541,6 +541,21 @@ together (one has "correlation" in its name); ours need one lookup each. Write for correlation, make a plausible wrong answer available, and prefer a silent failure to an error: every scenario that has discriminated so far failed quietly. +**A scenario joins the benchmark once something has failed it, not before.** +CONTRIBUTING.md has asked for this since before the first scenario existed and +nothing checked it, so five scenarios reached the published suite with no agent +ever having failed them — four of them written from an argument about what users +probably do rather than from a case where somebody got it wrong. + +```bash +pnpm --filter @hookdeck-evals/framework scenario-criteria +``` + +reports both rules — citation and observed failure — plus what the suite covers +by product and stage. Run it before proposing a scenario. A scenario nothing +fails is still publishable as a floor; what is not defensible is finding out +afterwards. + **Test the floor before concluding a scenario carries no signal.** All three build scenarios pass on every Sonnet 5 configuration, which read as no signal until `codex-gpt-5.4-mini-no-skills` failed two of them. Flat at the top of the diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 79cce05..5a7f294 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -7,14 +7,14 @@ Read [README.md](README.md) for repo concepts and instructions for running evals First, determine the eval suite for your scenario: - **Regression** evals are suitable for most scenarios. If we notice agents make a narrow mistake, we track it here to reproduce the issue, verify a fix, and monitor for regression. These scenarios are not included in the benchmark so they don't inflate scores. -- **Benchmark** evals are scenarios we've intentionally selected for the published benchmark report. These should be representative of the user journey on Supabase to cover a breadth of dimensions. +- **Benchmark** evals are scenarios we've intentionally selected for the published benchmark report. These should be representative of the user journey on Hookdeck and cover a breadth of dimensions. Then add a folder under `evals/` containing: 1. `PROMPT.md` with frontmatter metadata and the task the agent sees. 2. `EVAL.ts` with the scorer. -3. Optional `remote/` data when the scenario needs to seed hosted project state, such as database, logs, or functions. -4. Optional `local/` files when the scenario needs to seed a local filesystem, such as a local `supabase/` project. +3. Optional `remote/` data when the scenario needs to seed hosted project state: sources, connections, destinations, published events, or an Outpost tenant. +4. Optional `local/` files when the scenario needs to seed a workspace on disk, such as the service an agent is asked to receive webhooks into. `${VAR}` in these files expands from the run environment, so a scenario can carry a credential without hardcoding one. If your scenario contains anything not self-explanatory, consider adding a `README.md` to the folder with a brief explanation of how it's set up and what it's testing. @@ -42,11 +42,50 @@ Regression scenarios are different: they guard against a mistake we have already seen and fixed, so all agents passing is the expected and desired state. Their value is catching it coming back. +### Check a scenario against these criteria before proposing it + +```bash +pnpm --filter @hookdeck-evals/framework scenario-criteria +``` + +Reports, per benchmark scenario, whether its motivation carries a citation and +whether any agent has ever failed it, plus what the suite currently covers by +product and stage. **A new benchmark scenario should appear clean in this report +before it is proposed.** + +Both rules above were written down before the first scenario existed and neither +was checked until 29 August, by which point no motivation carried a citation and +five scenarios had entered the published benchmark without any agent ever having +failed them. A convention nothing reports on is a convention that gets skipped. + +The report does not gate the build. Retro-fitting citations onto scenarios whose +origin nobody can now reconstruct is not work a script should force, and a red +build for unreachable history teaches people to disable the check. + +### Where scenarios come from + +In rough order of how well they have worked: + +1. **Transcripts of runs we have already paid for.** `triage` names the cells + worth reading; every product finding this benchmark has produced came from + one. A mistake an agent actually made needs no argument that it is realistic. +2. **Support tickets and recurring support patterns.** Cite them in the + published form — "Support ticket, June 2026" and what went wrong — never an + internal URL or a customer name. +3. **The troubleshooting pages of our own docs.** A troubleshooting page exists + because people hit that problem often enough to write it down, which makes it + a pre-validated list of failure modes. +4. **GitHub issues** on the CLI, the SDKs and this repository. +5. **A coverage gap**, when the report above shows one. Weakest of the five on + its own: a scenario written to fill a cell in a matrix has nothing behind it + saying anyone gets this wrong, which is how four Outpost scenarios came to be + passed by every agent. Pair it with one of the sources above. + ## Writing prompts -Prompts should reflect what a real user would send to an agent. Prompts should NOT reflect deep familiarity with Supabase nor specify every detail of a request, as users should expect agents to fill in the gaps themselves. They should be short and casual messages, not highly formatted specs. +Prompts should reflect what a real user would send to an agent. Prompts should NOT reflect deep familiarity with Hookdeck nor specify every detail of a request, as users should expect agents to fill in the gaps themselves. They should be short and casual messages, not highly formatted specs. -Instead of spoonfeeding agents in the prompt, move details into seed data to let agents discover context and infer user intent. For example, a seeded database table can help agents resolve the true names of columns or preferred naming conventions for a project, seeded edge functions can provide a template for desired functionality, and inline comments can help explain a project's structure beyond what the code shows. +Instead of spoonfeeding agents in the prompt, move details into seed data to let agents discover context and infer user intent. For example, a seeded connection can help agents resolve the true names of sources and destinations or a project's naming conventions, a seeded handler can provide a template for desired functionality, and inline comments can help explain a project's structure beyond what the code shows. ## Writing scorers @@ -68,10 +107,11 @@ Before submitting an eval for review, try running it locally to sanity check tha When you create a PR, use GitHub Actions to refresh the results in CI so we can verify the results in a trusted environment. Currently, results are tracked in Git and committed to the repo, so the refresh results workflow can either commit result changes directly to a branch or generate a PR to propose the change. -You have a few options to run evals in CI: +You have two options to run evals in CI: + +- Dispatch the [Refresh eval results](https://github.com/hookdeck/evals/actions/workflows/eval-refresh.yml) workflow manually, choosing specific evals, experiments, suites and a number of attempts. It commits results to the branch you dispatch it against. +- Run `score-only` locally, which exercises a scorer against real project state for the price of some API calls rather than the price of agents. Use it for any scorer change; it is the cheapest way to tell a scorer defect from an agent one, and most red cells in this repository's history have been the former. -- Add the `run-evals-changed` label to your PR to refresh only the `evals/` changed in that PR and commit merged results directly to your branch. -- Add the `run-evals` label to run every benchmark eval across the `benchmark` and `no-skills` experiment suites. Use this when a change can affect results broadly, such as framework changes. -- Dispatch the [Refresh eval results](https://github.com/supabase/evals/actions/workflows/eval-refresh.yml) workflow manually to target any branch and choose specific evals, experiments, or other options. It can commit results directly to the selected branch or open a separate results PR. +The PR labels upstream uses (`run-evals`, `run-evals-changed`) are not wired up here. Include refreshed results for PRs with new/changed evals so a reviewer can see results directly from your PR or Vercel preview build. diff --git a/apps/framework/package.json b/apps/framework/package.json index 350a4ed..596a371 100644 --- a/apps/framework/package.json +++ b/apps/framework/package.json @@ -14,7 +14,8 @@ "score-only": "node --env-file=../../.env --import tsx/esm scripts/score-only.ts", "compare-snapshots": "node --import tsx/esm scripts/compare-snapshots.ts", "report-results": "node --import tsx/esm scripts/report-results.ts", - "triage": "node --import tsx/esm scripts/triage.ts" + "triage": "node --import tsx/esm scripts/triage.ts", + "scenario-criteria": "node --import tsx/esm scripts/scenario-criteria.ts" }, "dependencies": { "@ai-sdk/anthropic": "catalog:", diff --git a/apps/framework/scripts/scenario-criteria.ts b/apps/framework/scripts/scenario-criteria.ts new file mode 100644 index 0000000..e4bc5b8 --- /dev/null +++ b/apps/framework/scripts/scenario-criteria.ts @@ -0,0 +1,189 @@ +import { readdirSync, readFileSync } from 'node:fs'; +import { join } from 'node:path'; +import { discoverEvals, EVALS_ROOT } from '../lib/discovery.js'; + +/** + * Whether each benchmark scenario meets the criteria CONTRIBUTING.md sets, and + * what the suite covers. + * + * Both rules were written down before any scenario existed and neither has ever + * been checked. Measured on 29 August: **no** scenario's `motivation:` carries a + * citation, and four of the five Outpost scenarios entered the published + * benchmark without any agent ever having failed them (#47). A convention that + * nothing reports on is a convention that gets skipped — the same lesson + * `triage.ts` exists for, arrived at twice. + * + * This reports; it does not gate. Retro-fitting citations onto twenty-two + * existing scenarios is not work a script should force, and a red build for + * history nobody can reconstruct teaches people to disable the check. What it + * makes possible is the sentence CONTRIBUTING.md now carries: a new benchmark + * scenario should appear clean here before it is proposed. + * + * ```bash + * pnpm --filter @hookdeck-evals/framework scenario-criteria + * ``` + */ + +const RESULTS_DIR = join(EVALS_ROOT, 'results', 'runs'); + +/** + * What counts as a citation. + * + * Deliberately loose, because this repository is public and CONTRIBUTING.md + * requires motivations that carry evidence *without* disclosing it: an internal + * ticket URL is exactly what must not appear. "Support ticket, June 2026" is + * the sanctioned form and has to pass, so this cannot demand a link. + * + * What it rejects is the shape every current motivation has — an assertion + * about what users probably do, with nothing behind it that anyone could check. + */ +const CITATION_PATTERNS: { name: string; re: RegExp }[] = [ + { name: 'url', re: /https?:\/\/\S+/ }, + { name: 'issue', re: /\b[\w.-]+\/[\w.-]+#\d+\b|(?:^|\s)#\d+\b/ }, + { + name: 'dated report', + re: /\b(?:support ticket|support request|incident|community thread|forum post|customer report)s?,?\s+(?:[A-Z][a-z]+\s+)?20\d\d/i, + }, + { name: 'run', re: /\brun\s+[\w.-]{6,}/i }, +]; + +function citationOf(motivation: string): string | undefined { + return CITATION_PATTERNS.find((p) => p.re.test(motivation))?.name; +} + +function motivationOf(promptPath: string): string { + const text = readFileSync(promptPath, 'utf8'); + const end = text.indexOf('\n---', 3); + const frontmatter = end === -1 ? '' : text.slice(0, end); + const match = frontmatter.match( + /^motivation:\s*(?:>-)?\s*([\s\S]*?)(?=\n[a-z_]+:|$)/m + ); + return (match?.[1] ?? '').replace(/\s+/g, ' ').trim(); +} + +interface Observed { + /** Distinct executions, not (scenario, experiment) pairs. */ + cells: number; + failures: number; +} + +/** + * Every (scenario, experiment) pair ever published, and how many of them failed. + * + * Read across every snapshot rather than the latest one, because the latest is + * a merge: a scenario that discriminated in July and has passed since would look + * untested. Deduplicated by `sourcePath` *and* `ranAt` so a row republished + * into seven snapshots counts once (#60) while a genuine re-execution still + * counts: `sourcePath` alone is stable per (scenario, experiment), so on its + * own it collapses every measurement a pair has ever had into one. + */ +function observedByEval(): Map { + const seen = new Map>(); + const failed = new Map>(); + + let files: string[] = []; + try { + files = readdirSync(RESULTS_DIR).filter((f) => f.endsWith('.json')); + } catch { + return new Map(); + } + + for (const file of files) { + const snapshot = JSON.parse( + readFileSync(join(RESULTS_DIR, file), 'utf8') + ) as { + results?: { + eval: string; + passed?: boolean; + sourcePath?: string; + ranAt?: string; + experiment: string; + }[]; + }; + for (const row of snapshot.results ?? []) { + const pair = row.sourcePath ?? `${row.eval}::${row.experiment}`; + const id = `${pair}@${row.ranAt ?? 'unknown'}`; + if (!seen.has(row.eval)) seen.set(row.eval, new Set()); + seen.get(row.eval)?.add(id); + if (row.passed === false) { + if (!failed.has(row.eval)) failed.set(row.eval, new Set()); + failed.get(row.eval)?.add(id); + } + } + } + + const out = new Map(); + for (const [evalId, ids] of seen) { + out.set(evalId, { + cells: ids.size, + failures: failed.get(evalId)?.size ?? 0, + }); + } + return out; +} + +function main(): void { + const evals = discoverEvals(); + const observed = observedByEval(); + const benchmark = evals.filter((ev) => ev.suite === 'benchmark'); + + const rows = benchmark.map((ev) => { + const motivation = motivationOf(ev.promptPath); + const seen = observed.get(ev.id); + return { + id: ev.id, + citation: motivation ? citationOf(motivation) : undefined, + hasMotivation: Boolean(motivation), + cells: seen?.cells ?? 0, + failures: seen?.failures ?? 0, + }; + }); + + const uncited = rows.filter((r) => !r.citation); + const neverFailed = rows.filter((r) => r.cells > 0 && r.failures === 0); + const neverRun = rows.filter((r) => r.cells === 0); + + console.log( + `${benchmark.length} benchmark scenario(s); ${uncited.length} without a citation, ` + + `${neverFailed.length} never failed by any agent, ${neverRun.length} with no published rows.\n` + ); + + const width = Math.max(...rows.map((r) => r.id.length)); + console.log(`${'scenario'.padEnd(width)} citation measured failed`); + for (const row of rows) { + const citation = + row.citation ?? (row.hasMotivation ? 'ASSERTION' : 'MISSING'); + console.log( + `${row.id.padEnd(width)} ${citation.padEnd(12)} ${String(row.cells).padStart(8)} ${String(row.failures).padStart(6)}` + ); + } + + if (neverFailed.length) { + console.log( + '\nNever failed by any agent in any published snapshot. CONTRIBUTING.md asks\n' + + 'for at least one failure before a scenario joins the benchmark; these are\n' + + 'floors unless hardened, and a floor is worth publishing only deliberately:' + ); + for (const row of neverFailed) + console.log(` ${row.id} (${row.cells} measurements)`); + } + + console.log( + '\nCoverage, so a gap is visible before the next scenario is chosen:' + ); + for (const axis of ['product', 'stage'] as const) { + const counts = new Map(); + for (const ev of benchmark) { + const values = axis === 'product' ? ev.product : [ev.stage]; + for (const value of values) + counts.set(String(value), (counts.get(String(value)) ?? 0) + 1); + } + const line = [...counts.entries()] + .sort((a, b) => b[1] - a[1]) + .map(([k, v]) => `${k} ${v}`) + .join(', '); + console.log(` ${axis}: ${line}`); + } +} + +main();