diff --git a/AGENTS.md b/AGENTS.md index dba108c..c2dc45c 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -541,6 +541,21 @@ together (one has "correlation" in its name); ours need one lookup each. Write for correlation, make a plausible wrong answer available, and prefer a silent failure to an error: every scenario that has discriminated so far failed quietly. +**A scenario joins the benchmark once something has failed it, not before.** +CONTRIBUTING.md has asked for this since before the first scenario existed and +nothing checked it, so five scenarios reached the published suite with no agent +ever having failed them — four of them written from an argument about what users +probably do rather than from a case where somebody got it wrong. + +```bash +pnpm --filter @hookdeck-evals/framework scenario-criteria +``` + +reports both rules — citation and observed failure — plus what the suite covers +by product and stage. Run it before proposing a scenario. A scenario nothing +fails is still publishable as a floor; what is not defensible is finding out +afterwards. + **Test the floor before concluding a scenario carries no signal.** All three build scenarios pass on every Sonnet 5 configuration, which read as no signal until `codex-gpt-5.4-mini-no-skills` failed two of them. Flat at the top of the diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 79cce05..5a7f294 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -7,14 +7,14 @@ Read [README.md](README.md) for repo concepts and instructions for running evals First, determine the eval suite for your scenario: - **Regression** evals are suitable for most scenarios. If we notice agents make a narrow mistake, we track it here to reproduce the issue, verify a fix, and monitor for regression. These scenarios are not included in the benchmark so they don't inflate scores. -- **Benchmark** evals are scenarios we've intentionally selected for the published benchmark report. These should be representative of the user journey on Supabase to cover a breadth of dimensions. +- **Benchmark** evals are scenarios we've intentionally selected for the published benchmark report. These should be representative of the user journey on Hookdeck and cover a breadth of dimensions. Then add a folder under `evals/` containing: 1. `PROMPT.md` with frontmatter metadata and the task the agent sees. 2. `EVAL.ts` with the scorer. -3. Optional `remote/` data when the scenario needs to seed hosted project state, such as database, logs, or functions. -4. Optional `local/` files when the scenario needs to seed a local filesystem, such as a local `supabase/` project. +3. Optional `remote/` data when the scenario needs to seed hosted project state: sources, connections, destinations, published events, or an Outpost tenant. +4. Optional `local/` files when the scenario needs to seed a workspace on disk, such as the service an agent is asked to receive webhooks into. `${VAR}` in these files expands from the run environment, so a scenario can carry a credential without hardcoding one. If your scenario contains anything not self-explanatory, consider adding a `README.md` to the folder with a brief explanation of how it's set up and what it's testing. @@ -42,11 +42,50 @@ Regression scenarios are different: they guard against a mistake we have already seen and fixed, so all agents passing is the expected and desired state. Their value is catching it coming back. +### Check a scenario against these criteria before proposing it + +```bash +pnpm --filter @hookdeck-evals/framework scenario-criteria +``` + +Reports, per benchmark scenario, whether its motivation carries a citation and +whether any agent has ever failed it, plus what the suite currently covers by +product and stage. **A new benchmark scenario should appear clean in this report +before it is proposed.** + +Both rules above were written down before the first scenario existed and neither +was checked until 29 August, by which point no motivation carried a citation and +five scenarios had entered the published benchmark without any agent ever having +failed them. A convention nothing reports on is a convention that gets skipped. + +The report does not gate the build. Retro-fitting citations onto scenarios whose +origin nobody can now reconstruct is not work a script should force, and a red +build for unreachable history teaches people to disable the check. + +### Where scenarios come from + +In rough order of how well they have worked: + +1. **Transcripts of runs we have already paid for.** `triage` names the cells + worth reading; every product finding this benchmark has produced came from + one. A mistake an agent actually made needs no argument that it is realistic. +2. **Support tickets and recurring support patterns.** Cite them in the + published form — "Support ticket, June 2026" and what went wrong — never an + internal URL or a customer name. +3. **The troubleshooting pages of our own docs.** A troubleshooting page exists + because people hit that problem often enough to write it down, which makes it + a pre-validated list of failure modes. +4. **GitHub issues** on the CLI, the SDKs and this repository. +5. **A coverage gap**, when the report above shows one. Weakest of the five on + its own: a scenario written to fill a cell in a matrix has nothing behind it + saying anyone gets this wrong, which is how four Outpost scenarios came to be + passed by every agent. Pair it with one of the sources above. + ## Writing prompts -Prompts should reflect what a real user would send to an agent. Prompts should NOT reflect deep familiarity with Supabase nor specify every detail of a request, as users should expect agents to fill in the gaps themselves. They should be short and casual messages, not highly formatted specs. +Prompts should reflect what a real user would send to an agent. Prompts should NOT reflect deep familiarity with Hookdeck nor specify every detail of a request, as users should expect agents to fill in the gaps themselves. They should be short and casual messages, not highly formatted specs. -Instead of spoonfeeding agents in the prompt, move details into seed data to let agents discover context and infer user intent. For example, a seeded database table can help agents resolve the true names of columns or preferred naming conventions for a project, seeded edge functions can provide a template for desired functionality, and inline comments can help explain a project's structure beyond what the code shows. +Instead of spoonfeeding agents in the prompt, move details into seed data to let agents discover context and infer user intent. For example, a seeded connection can help agents resolve the true names of sources and destinations or a project's naming conventions, a seeded handler can provide a template for desired functionality, and inline comments can help explain a project's structure beyond what the code shows. ## Writing scorers @@ -68,10 +107,11 @@ Before submitting an eval for review, try running it locally to sanity check tha When you create a PR, use GitHub Actions to refresh the results in CI so we can verify the results in a trusted environment. Currently, results are tracked in Git and committed to the repo, so the refresh results workflow can either commit result changes directly to a branch or generate a PR to propose the change. -You have a few options to run evals in CI: +You have two options to run evals in CI: + +- Dispatch the [Refresh eval results](https://github.com/hookdeck/evals/actions/workflows/eval-refresh.yml) workflow manually, choosing specific evals, experiments, suites and a number of attempts. It commits results to the branch you dispatch it against. +- Run `score-only` locally, which exercises a scorer against real project state for the price of some API calls rather than the price of agents. Use it for any scorer change; it is the cheapest way to tell a scorer defect from an agent one, and most red cells in this repository's history have been the former. -- Add the `run-evals-changed` label to your PR to refresh only the `evals/` changed in that PR and commit merged results directly to your branch. -- Add the `run-evals` label to run every benchmark eval across the `benchmark` and `no-skills` experiment suites. Use this when a change can affect results broadly, such as framework changes. -- Dispatch the [Refresh eval results](https://github.com/supabase/evals/actions/workflows/eval-refresh.yml) workflow manually to target any branch and choose specific evals, experiments, or other options. It can commit results directly to the selected branch or open a separate results PR. +The PR labels upstream uses (`run-evals`, `run-evals-changed`) are not wired up here. Include refreshed results for PRs with new/changed evals so a reviewer can see results directly from your PR or Vercel preview build. diff --git a/apps/framework/package.json b/apps/framework/package.json index 350a4ed..596a371 100644 --- a/apps/framework/package.json +++ b/apps/framework/package.json @@ -14,7 +14,8 @@ "score-only": "node --env-file=../../.env --import tsx/esm scripts/score-only.ts", "compare-snapshots": "node --import tsx/esm scripts/compare-snapshots.ts", "report-results": "node --import tsx/esm scripts/report-results.ts", - "triage": "node --import tsx/esm scripts/triage.ts" + "triage": "node --import tsx/esm scripts/triage.ts", + "scenario-criteria": "node --import tsx/esm scripts/scenario-criteria.ts" }, "dependencies": { "@ai-sdk/anthropic": "catalog:", diff --git a/apps/framework/scripts/scenario-criteria.ts b/apps/framework/scripts/scenario-criteria.ts new file mode 100644 index 0000000..e4bc5b8 --- /dev/null +++ b/apps/framework/scripts/scenario-criteria.ts @@ -0,0 +1,189 @@ +import { readdirSync, readFileSync } from 'node:fs'; +import { join } from 'node:path'; +import { discoverEvals, EVALS_ROOT } from '../lib/discovery.js'; + +/** + * Whether each benchmark scenario meets the criteria CONTRIBUTING.md sets, and + * what the suite covers. + * + * Both rules were written down before any scenario existed and neither has ever + * been checked. Measured on 29 August: **no** scenario's `motivation:` carries a + * citation, and four of the five Outpost scenarios entered the published + * benchmark without any agent ever having failed them (#47). A convention that + * nothing reports on is a convention that gets skipped — the same lesson + * `triage.ts` exists for, arrived at twice. + * + * This reports; it does not gate. Retro-fitting citations onto twenty-two + * existing scenarios is not work a script should force, and a red build for + * history nobody can reconstruct teaches people to disable the check. What it + * makes possible is the sentence CONTRIBUTING.md now carries: a new benchmark + * scenario should appear clean here before it is proposed. + * + * ```bash + * pnpm --filter @hookdeck-evals/framework scenario-criteria + * ``` + */ + +const RESULTS_DIR = join(EVALS_ROOT, 'results', 'runs'); + +/** + * What counts as a citation. + * + * Deliberately loose, because this repository is public and CONTRIBUTING.md + * requires motivations that carry evidence *without* disclosing it: an internal + * ticket URL is exactly what must not appear. "Support ticket, June 2026" is + * the sanctioned form and has to pass, so this cannot demand a link. + * + * What it rejects is the shape every current motivation has — an assertion + * about what users probably do, with nothing behind it that anyone could check. + */ +const CITATION_PATTERNS: { name: string; re: RegExp }[] = [ + { name: 'url', re: /https?:\/\/\S+/ }, + { name: 'issue', re: /\b[\w.-]+\/[\w.-]+#\d+\b|(?:^|\s)#\d+\b/ }, + { + name: 'dated report', + re: /\b(?:support ticket|support request|incident|community thread|forum post|customer report)s?,?\s+(?:[A-Z][a-z]+\s+)?20\d\d/i, + }, + { name: 'run', re: /\brun\s+[\w.-]{6,}/i }, +]; + +function citationOf(motivation: string): string | undefined { + return CITATION_PATTERNS.find((p) => p.re.test(motivation))?.name; +} + +function motivationOf(promptPath: string): string { + const text = readFileSync(promptPath, 'utf8'); + const end = text.indexOf('\n---', 3); + const frontmatter = end === -1 ? '' : text.slice(0, end); + const match = frontmatter.match( + /^motivation:\s*(?:>-)?\s*([\s\S]*?)(?=\n[a-z_]+:|$)/m + ); + return (match?.[1] ?? '').replace(/\s+/g, ' ').trim(); +} + +interface Observed { + /** Distinct executions, not (scenario, experiment) pairs. */ + cells: number; + failures: number; +} + +/** + * Every (scenario, experiment) pair ever published, and how many of them failed. + * + * Read across every snapshot rather than the latest one, because the latest is + * a merge: a scenario that discriminated in July and has passed since would look + * untested. Deduplicated by `sourcePath` *and* `ranAt` so a row republished + * into seven snapshots counts once (#60) while a genuine re-execution still + * counts: `sourcePath` alone is stable per (scenario, experiment), so on its + * own it collapses every measurement a pair has ever had into one. + */ +function observedByEval(): Map { + const seen = new Map>(); + const failed = new Map>(); + + let files: string[] = []; + try { + files = readdirSync(RESULTS_DIR).filter((f) => f.endsWith('.json')); + } catch { + return new Map(); + } + + for (const file of files) { + const snapshot = JSON.parse( + readFileSync(join(RESULTS_DIR, file), 'utf8') + ) as { + results?: { + eval: string; + passed?: boolean; + sourcePath?: string; + ranAt?: string; + experiment: string; + }[]; + }; + for (const row of snapshot.results ?? []) { + const pair = row.sourcePath ?? `${row.eval}::${row.experiment}`; + const id = `${pair}@${row.ranAt ?? 'unknown'}`; + if (!seen.has(row.eval)) seen.set(row.eval, new Set()); + seen.get(row.eval)?.add(id); + if (row.passed === false) { + if (!failed.has(row.eval)) failed.set(row.eval, new Set()); + failed.get(row.eval)?.add(id); + } + } + } + + const out = new Map(); + for (const [evalId, ids] of seen) { + out.set(evalId, { + cells: ids.size, + failures: failed.get(evalId)?.size ?? 0, + }); + } + return out; +} + +function main(): void { + const evals = discoverEvals(); + const observed = observedByEval(); + const benchmark = evals.filter((ev) => ev.suite === 'benchmark'); + + const rows = benchmark.map((ev) => { + const motivation = motivationOf(ev.promptPath); + const seen = observed.get(ev.id); + return { + id: ev.id, + citation: motivation ? citationOf(motivation) : undefined, + hasMotivation: Boolean(motivation), + cells: seen?.cells ?? 0, + failures: seen?.failures ?? 0, + }; + }); + + const uncited = rows.filter((r) => !r.citation); + const neverFailed = rows.filter((r) => r.cells > 0 && r.failures === 0); + const neverRun = rows.filter((r) => r.cells === 0); + + console.log( + `${benchmark.length} benchmark scenario(s); ${uncited.length} without a citation, ` + + `${neverFailed.length} never failed by any agent, ${neverRun.length} with no published rows.\n` + ); + + const width = Math.max(...rows.map((r) => r.id.length)); + console.log(`${'scenario'.padEnd(width)} citation measured failed`); + for (const row of rows) { + const citation = + row.citation ?? (row.hasMotivation ? 'ASSERTION' : 'MISSING'); + console.log( + `${row.id.padEnd(width)} ${citation.padEnd(12)} ${String(row.cells).padStart(8)} ${String(row.failures).padStart(6)}` + ); + } + + if (neverFailed.length) { + console.log( + '\nNever failed by any agent in any published snapshot. CONTRIBUTING.md asks\n' + + 'for at least one failure before a scenario joins the benchmark; these are\n' + + 'floors unless hardened, and a floor is worth publishing only deliberately:' + ); + for (const row of neverFailed) + console.log(` ${row.id} (${row.cells} measurements)`); + } + + console.log( + '\nCoverage, so a gap is visible before the next scenario is chosen:' + ); + for (const axis of ['product', 'stage'] as const) { + const counts = new Map(); + for (const ev of benchmark) { + const values = axis === 'product' ? ev.product : [ev.stage]; + for (const value of values) + counts.set(String(value), (counts.get(String(value)) ?? 0) + 1); + } + const line = [...counts.entries()] + .sort((a, b) => b[1] - a[1]) + .map(([k, v]) => `${k} ${v}`) + .join(', '); + console.log(` ${axis}: ${line}`); + } +} + +main();