Report on the scenario criteria we wrote down and never checked

AGENTS.md

@@ -541,6 +541,21 @@ together (one has "correlation" in its name); ours need one lookup each. Write
 for correlation, make a plausible wrong answer available, and prefer a silent
 failure to an error: every scenario that has discriminated so far failed quietly.
 
+**A scenario joins the benchmark once something has failed it, not before.**
+CONTRIBUTING.md has asked for this since before the first scenario existed and
+nothing checked it, so five scenarios reached the published suite with no agent
+ever having failed them — four of them written from an argument about what users
+probably do rather than from a case where somebody got it wrong.
+
+```bash
+pnpm --filter @hookdeck-evals/framework scenario-criteria
+```
+
+reports both rules — citation and observed failure — plus what the suite covers
+by product and stage. Run it before proposing a scenario. A scenario nothing
+fails is still publishable as a floor; what is not defensible is finding out
+afterwards.
+
 **Test the floor before concluding a scenario carries no signal.** All three
 build scenarios pass on every Sonnet 5 configuration, which read as no signal
 until `codex-gpt-5.4-mini-no-skills` failed two of them. Flat at the top of the

CONTRIBUTING.md

@@ -7,14 +7,14 @@ Read [README.md](README.md) for repo concepts and instructions for running evals
 First, determine the eval suite for your scenario:
 
 - **Regression** evals are suitable for most scenarios. If we notice agents make a narrow mistake, we track it here to reproduce the issue, verify a fix, and monitor for regression. These scenarios are not included in the benchmark so they don't inflate scores.
-- **Benchmark** evals are scenarios we've intentionally selected for the published benchmark report. These should be representative of the user journey on Supabase to cover a breadth of dimensions.
+- **Benchmark** evals are scenarios we've intentionally selected for the published benchmark report. These should be representative of the user journey on Hookdeck and cover a breadth of dimensions.
 
 Then add a folder under `evals/` containing:
 
 1. `PROMPT.md` with frontmatter metadata and the task the agent sees.
 2. `EVAL.ts` with the scorer.
-3. Optional `remote/` data when the scenario needs to seed hosted project state, such as database, logs, or functions.
-4. Optional `local/` files when the scenario needs to seed a local filesystem, such as a local `supabase/` project.
+3. Optional `remote/` data when the scenario needs to seed hosted project state: sources, connections, destinations, published events, or an Outpost tenant.
+4. Optional `local/` files when the scenario needs to seed a workspace on disk, such as the service an agent is asked to receive webhooks into. `${VAR}` in these files expands from the run environment, so a scenario can carry a credential without hardcoding one.
 
 If your scenario contains anything not self-explanatory, consider adding a `README.md` to the folder with a brief explanation of how it's set up and what it's testing.
 
@@ -42,11 +42,50 @@ Regression scenarios are different: they guard against a mistake we have already
 seen and fixed, so all agents passing is the expected and desired state. Their
 value is catching it coming back.
 
+### Check a scenario against these criteria before proposing it
+
+```bash
+pnpm --filter @hookdeck-evals/framework scenario-criteria
+```
+
+Reports, per benchmark scenario, whether its motivation carries a citation and
+whether any agent has ever failed it, plus what the suite currently covers by
+product and stage. **A new benchmark scenario should appear clean in this report
+before it is proposed.**
+
+Both rules above were written down before the first scenario existed and neither
+was checked until 29 August, by which point no motivation carried a citation and
+five scenarios had entered the published benchmark without any agent ever having
+failed them. A convention nothing reports on is a convention that gets skipped.
+
+The report does not gate the build. Retro-fitting citations onto scenarios whose
+origin nobody can now reconstruct is not work a script should force, and a red
+build for unreachable history teaches people to disable the check.
+
+### Where scenarios come from
+
+In rough order of how well they have worked:
+
+1. **Transcripts of runs we have already paid for.** `triage` names the cells
+   worth reading; every product finding this benchmark has produced came from
+   one. A mistake an agent actually made needs no argument that it is realistic.
+2. **Support tickets and recurring support patterns.** Cite them in the
+   published form — "Support ticket, June 2026" and what went wrong — never an
+   internal URL or a customer name.
+3. **The troubleshooting pages of our own docs.** A troubleshooting page exists
+   because people hit that problem often enough to write it down, which makes it
+   a pre-validated list of failure modes.
+4. **GitHub issues** on the CLI, the SDKs and this repository.
+5. **A coverage gap**, when the report above shows one. Weakest of the five on
+   its own: a scenario written to fill a cell in a matrix has nothing behind it
+   saying anyone gets this wrong, which is how four Outpost scenarios came to be
+   passed by every agent. Pair it with one of the sources above.
+
 ## Writing prompts
 
-Prompts should reflect what a real user would send to an agent. Prompts should NOT reflect deep familiarity with Supabase nor specify every detail of a request, as users should expect agents to fill in the gaps themselves. They should be short and casual messages, not highly formatted specs.
+Prompts should reflect what a real user would send to an agent. Prompts should NOT reflect deep familiarity with Hookdeck nor specify every detail of a request, as users should expect agents to fill in the gaps themselves. They should be short and casual messages, not highly formatted specs.
 
-Instead of spoonfeeding agents in the prompt, move details into seed data to let agents discover context and infer user intent. For example, a seeded database table can help agents resolve the true names of columns or preferred naming conventions for a project, seeded edge functions can provide a template for desired functionality, and inline comments can help explain a project's structure beyond what the code shows.
+Instead of spoonfeeding agents in the prompt, move details into seed data to let agents discover context and infer user intent. For example, a seeded connection can help agents resolve the true names of sources and destinations or a project's naming conventions, a seeded handler can provide a template for desired functionality, and inline comments can help explain a project's structure beyond what the code shows.
 
 ## Writing scorers
 
@@ -68,10 +107,11 @@ Before submitting an eval for review, try running it locally to sanity check tha
 
 When you create a PR, use GitHub Actions to refresh the results in CI so we can verify the results in a trusted environment. Currently, results are tracked in Git and committed to the repo, so the refresh results workflow can either commit result changes directly to a branch or generate a PR to propose the change.
 
-You have a few options to run evals in CI:
+You have two options to run evals in CI:
+
+- Dispatch the [Refresh eval results](https://github.com/hookdeck/evals/actions/workflows/eval-refresh.yml) workflow manually, choosing specific evals, experiments, suites and a number of attempts. It commits results to the branch you dispatch it against.
+- Run `score-only` locally, which exercises a scorer against real project state for the price of some API calls rather than the price of agents. Use it for any scorer change; it is the cheapest way to tell a scorer defect from an agent one, and most red cells in this repository's history have been the former.
 
-- Add the `run-evals-changed` label to your PR to refresh only the `evals/` changed in that PR and commit merged results directly to your branch.
-- Add the `run-evals` label to run every benchmark eval across the `benchmark` and `no-skills` experiment suites. Use this when a change can affect results broadly, such as framework changes.
-- Dispatch the [Refresh eval results](https://github.com/supabase/evals/actions/workflows/eval-refresh.yml) workflow manually to target any branch and choose specific evals, experiments, or other options. It can commit results directly to the selected branch or open a separate results PR.
+The PR labels upstream uses (`run-evals`, `run-evals-changed`) are not wired up here.
 
 Include refreshed results for PRs with new/changed evals so a reviewer can see results directly from your PR or Vercel preview build.

apps/framework/package.json

@@ -14,7 +14,8 @@
     "score-only": "node --env-file=../../.env --import tsx/esm scripts/score-only.ts",
     "compare-snapshots": "node --import tsx/esm scripts/compare-snapshots.ts",
     "report-results": "node --import tsx/esm scripts/report-results.ts",
-    "triage": "node --import tsx/esm scripts/triage.ts"
+    "triage": "node --import tsx/esm scripts/triage.ts",
+    "scenario-criteria": "node --import tsx/esm scripts/scenario-criteria.ts"
   },
   "dependencies": {
     "@ai-sdk/anthropic": "catalog:",

apps/framework/scripts/scenario-criteria.ts

@@ -0,0 +1,189 @@
+import { readdirSync, readFileSync } from 'node:fs';
+import { join } from 'node:path';
+import { discoverEvals, EVALS_ROOT } from '../lib/discovery.js';
+
+/**
+ * Whether each benchmark scenario meets the criteria CONTRIBUTING.md sets, and
+ * what the suite covers.
+ *
+ * Both rules were written down before any scenario existed and neither has ever
+ * been checked. Measured on 29 August: **no** scenario's `motivation:` carries a
+ * citation, and four of the five Outpost scenarios entered the published
+ * benchmark without any agent ever having failed them (#47). A convention that
+ * nothing reports on is a convention that gets skipped — the same lesson
+ * `triage.ts` exists for, arrived at twice.
+ *
+ * This reports; it does not gate. Retro-fitting citations onto twenty-two
+ * existing scenarios is not work a script should force, and a red build for
+ * history nobody can reconstruct teaches people to disable the check. What it
+ * makes possible is the sentence CONTRIBUTING.md now carries: a new benchmark
+ * scenario should appear clean here before it is proposed.
+ *
+ * ```bash
+ * pnpm --filter @hookdeck-evals/framework scenario-criteria
+ * ```
+ */
+
+const RESULTS_DIR = join(EVALS_ROOT, 'results', 'runs');
+
+/**
+ * What counts as a citation.
+ *
+ * Deliberately loose, because this repository is public and CONTRIBUTING.md
+ * requires motivations that carry evidence *without* disclosing it: an internal
+ * ticket URL is exactly what must not appear. "Support ticket, June 2026" is
+ * the sanctioned form and has to pass, so this cannot demand a link.
+ *
+ * What it rejects is the shape every current motivation has — an assertion
+ * about what users probably do, with nothing behind it that anyone could check.
+ */
+const CITATION_PATTERNS: { name: string; re: RegExp }[] = [
+  { name: 'url', re: /https?:\/\/\S+/ },
+  { name: 'issue', re: /\b[\w.-]+\/[\w.-]+#\d+\b|(?:^|\s)#\d+\b/ },
+  {
+    name: 'dated report',
+    re: /\b(?:support ticket|support request|incident|community thread|forum post|customer report)s?,?\s+(?:[A-Z][a-z]+\s+)?20\d\d/i,
+  },
+  { name: 'run', re: /\brun\s+[\w.-]{6,}/i },
+];
+
+function citationOf(motivation: string): string | undefined {
+  return CITATION_PATTERNS.find((p) => p.re.test(motivation))?.name;
+}
+
+function motivationOf(promptPath: string): string {
+  const text = readFileSync(promptPath, 'utf8');
+  const end = text.indexOf('\n---', 3);
+  const frontmatter = end === -1 ? '' : text.slice(0, end);
+  const match = frontmatter.match(
+    /^motivation:\s*(?:>-)?\s*([\s\S]*?)(?=\n[a-z_]+:|$)/m
+  );
+  return (match?.[1] ?? '').replace(/\s+/g, ' ').trim();
+}
+
+interface Observed {
+  /** Distinct executions, not (scenario, experiment) pairs. */
+  cells: number;
+  failures: number;
+}
+
+/**
+ * Every (scenario, experiment) pair ever published, and how many of them failed.
+ *
+ * Read across every snapshot rather than the latest one, because the latest is
+ * a merge: a scenario that discriminated in July and has passed since would look
+ * untested. Deduplicated by `sourcePath` *and* `ranAt` so a row republished
+ * into seven snapshots counts once (#60) while a genuine re-execution still
+ * counts: `sourcePath` alone is stable per (scenario, experiment), so on its
+ * own it collapses every measurement a pair has ever had into one.
+ */
+function observedByEval(): Map<string, Observed> {
+  const seen = new Map<string, Set<string>>();
+  const failed = new Map<string, Set<string>>();
+
+  let files: string[] = [];
+  try {
+    files = readdirSync(RESULTS_DIR).filter((f) => f.endsWith('.json'));
+  } catch {
+    return new Map();
+  }
+
+  for (const file of files) {
+    const snapshot = JSON.parse(
+      readFileSync(join(RESULTS_DIR, file), 'utf8')
+    ) as {
+      results?: {
+        eval: string;
+        passed?: boolean;
+        sourcePath?: string;
+        ranAt?: string;
+        experiment: string;
+      }[];
+    };
+    for (const row of snapshot.results ?? []) {
+      const pair = row.sourcePath ?? `${row.eval}::${row.experiment}`;
+      const id = `${pair}@${row.ranAt ?? 'unknown'}`;
+      if (!seen.has(row.eval)) seen.set(row.eval, new Set());
+      seen.get(row.eval)?.add(id);
+      if (row.passed === false) {
+        if (!failed.has(row.eval)) failed.set(row.eval, new Set());
+        failed.get(row.eval)?.add(id);
+      }
+    }
+  }
+
+  const out = new Map<string, Observed>();
+  for (const [evalId, ids] of seen) {
+    out.set(evalId, {
+      cells: ids.size,
+      failures: failed.get(evalId)?.size ?? 0,
+    });
+  }
+  return out;
+}
+
+function main(): void {
+  const evals = discoverEvals();
+  const observed = observedByEval();
+  const benchmark = evals.filter((ev) => ev.suite === 'benchmark');
+
+  const rows = benchmark.map((ev) => {
+    const motivation = motivationOf(ev.promptPath);
+    const seen = observed.get(ev.id);
+    return {
+      id: ev.id,
+      citation: motivation ? citationOf(motivation) : undefined,
+      hasMotivation: Boolean(motivation),
+      cells: seen?.cells ?? 0,
+      failures: seen?.failures ?? 0,
+    };
+  });
+
+  const uncited = rows.filter((r) => !r.citation);
+  const neverFailed = rows.filter((r) => r.cells > 0 && r.failures === 0);
+  const neverRun = rows.filter((r) => r.cells === 0);
+
+  console.log(
+    `${benchmark.length} benchmark scenario(s); ${uncited.length} without a citation, ` +
+      `${neverFailed.length} never failed by any agent, ${neverRun.length} with no published rows.\n`
+  );
+
+  const width = Math.max(...rows.map((r) => r.id.length));
+  console.log(`${'scenario'.padEnd(width)}  citation      measured  failed`);
+  for (const row of rows) {
+    const citation =
+      row.citation ?? (row.hasMotivation ? 'ASSERTION' : 'MISSING');
+    console.log(
+      `${row.id.padEnd(width)}  ${citation.padEnd(12)}  ${String(row.cells).padStart(8)}  ${String(row.failures).padStart(6)}`
+    );
+  }
+
+  if (neverFailed.length) {
+    console.log(
+      '\nNever failed by any agent in any published snapshot. CONTRIBUTING.md asks\n' +
+        'for at least one failure before a scenario joins the benchmark; these are\n' +
+        'floors unless hardened, and a floor is worth publishing only deliberately:'
+    );
+    for (const row of neverFailed)
+      console.log(`  ${row.id} (${row.cells} measurements)`);
+  }
+
+  console.log(
+    '\nCoverage, so a gap is visible before the next scenario is chosen:'
+  );
+  for (const axis of ['product', 'stage'] as const) {
+    const counts = new Map<string, number>();
+    for (const ev of benchmark) {
+      const values = axis === 'product' ? ev.product : [ev.stage];
+      for (const value of values)
+        counts.set(String(value), (counts.get(String(value)) ?? 0) + 1);
+    }
+    const line = [...counts.entries()]
+      .sort((a, b) => b[1] - a[1])
+      .map(([k, v]) => `${k} ${v}`)
+      .join(', ');
+    console.log(`  ${axis}: ${line}`);
+  }
+}
+
+main();