Publish the first adjudicated real-history accuracy row (2026-W26)

README.md

@@ -22,13 +22,19 @@ Local-first and static by default — no agent execution, tool calls, LLM calls,
 <!-- Canonical tagline: The deterministic merge gate for AI-generated agent capability changes. -->
 
 > [!IMPORTANT]
-> **Status: pre-1.0 (beta).** The decision engine is deterministic and stable,
-> but Shipgate's real-world detection accuracy is still being validated against
-> a labeled corpus of agent PRs — no precision/recall numbers are published yet.
-> On heavily dynamic tool surfaces (factory-built toolsets, config-bound
-> allowlists, runtime-assembled tools), Shipgate deliberately returns
-> `insufficient_evidence` rather than guess. Treat it as an advisory gate while
-> that accuracy work is in progress — see [ROADMAP.md](ROADMAP.md).
+> **Status: pre-1.0 (beta).** The decision engine is deterministic and stable.
+> First real-history accuracy numbers (small n, published in full in
+> [`benchmark/miner/README.md`](benchmark/miner/README.md)): across 361 merged
+> PRs mined from 9 real agent repos, 336 (93%) organically skip the trigger; of the
+> 10 PRs the gate engaged on the 2026-W26 toolkit corpus, it **never wrongly
+> passed** an authority-bearing change (2/2 held for a human, zero benign
+> escalations) — but it also never cleanly passed a safe one: 4/8 safe PRs
+> returned `insufficient_evidence` (the dynamic-toolkit gap, the active fix)
+> and 4/8 hit a since-chipped scan crash. Labels are AI-adjudicated
+> (disagreement 0/10), pending human spot-check. On heavily dynamic tool
+> surfaces Shipgate deliberately returns `insufficient_evidence` rather than
+> guess. Treat it as an advisory gate while this work closes — see
+> [ROADMAP.md](ROADMAP.md).
 
 ## 60 seconds: watch it block two PRs
 

benchmark/miner/README.md

@@ -155,6 +155,39 @@ trigger-skips) and the real-history **extraction-coverage** (`insufficient_evide
   surface `must_block` positives — those still must come from the
   constructed-adversarial stratum.
 
+### 2026-W26 labels + score — first adjudicated real-history accuracy row
+
+The W26 worksheet (10 rows: 6 evaluated + 4 `scan_failed`) is now labeled and
+committed as [`results/2026-W26-mined.labels.csv`](results/2026-W26-mined.labels.csv).
+
+- **Method disclosure.** Two **independent AI labelers** (separate contexts,
+  no coordination; each fetched the real PR diffs) filled the worksheet per
+  [LABELING.md](LABELING.md); a third pass adjudicated.
+  **Disagreement rate: 0/10.** Both labelers independently flagged the same
+  two rows as `needs_human` with the same reasoning (stripe/ai#338: a new
+  auto-synced skill directing agents to install the Stripe CLI and load
+  further skills; stripe/ai#312: the skill-sync supply chain rewired to an
+  unauthenticated source with a daily cron and a dropped API-key
+  requirement). Treat these as AI-generated labels pending human spot-check —
+  the run notes exist so that caveat travels with the numbers.
+- **Ground truth:** 8 `safe_to_merge`, 2 `needs_human`, 0 `must_block`
+  (consistent with the W24–W26 base-rate finding above).
+- **Score** (`python -m benchmark.miner score`):
+  `needs_human_caught` **1.0** (2/2 — both authority-bearing changes held for
+  a human), `benign_escalation_rate` **0.0** (no safe PR was
+  blocked/review-routed), `ie_rate_on_safe` **0.5** (4/8 safe PRs returned
+  `insufficient_evidence`), and the remaining 4/8 safe PRs are `unscored`
+  (the goose `Duplicate action_surface action_id` crash above).
+  `must_block_caught`/`blocked_recall` are **null** on this corpus — real
+  history contributes no `must_block` rows; that proof stays with the
+  constructed-adversarial stratum.
+- **Reading:** on real agent-toolkit history the gate currently **never
+  wrongly passes** and **never cleanly passes** — every safe PR it engaged
+  ended in abstention or a crash. The abstentions are the config-bound /
+  dynamic-toolkit gap (`docs/engineering/config-bound-capability-detection.md`);
+  the crashes are the chipped fail-soft bug. Both are the active fixes this
+  row exists to measure against.
+
 ### 2026-W25 findings — diminishing returns from framework-core breadth
 
 - **The base rate of capability-changing merged PRs is low, and now quantified.**

benchmark/miner/results/2026-W26-mined.labels.csv

@@ -0,0 +1,11 @@
+pr_url,label,rationale
+https://github.com/stripe/ai/pull/400,safe_to_merge,"Pure content refresh of existing SKILL.md copies: bumps the recommended Stripe API version string 2026-03-25.dahlia to 2026-04-22.dahlia everywhere; no new instructions, tools, or scopes."
+https://github.com/stripe/ai/pull/353,safe_to_merge,"Adds a security best-practices reference (references/security.md: RAKs, key handling, webhook signature verification) plus a routing-table row to the existing stripe-best-practices skill; guidance-only content that tightens rather than expands what the agent is told to do."
+https://github.com/stripe/ai/pull/338,needs_human,"Adds a brand-new skill (stripe-projects) instructing coding agents to install the Stripe CLI (brew install + stripe plugin install projects), run stripe projects init, and prefer the additional local skills that init creates; a new instruction surface directing software installation and dynamic skill loading is plausibly fine (Stripe's own CLI) but not self-evidently safe."
+https://github.com/stripe/ai/pull/336,safe_to_merge,"Pure content refresh of existing SKILL.md copies: API version string 2026-02-25.clover to 2026-03-25.dahlia throughout; no new instructions, tools, or scopes."
+https://github.com/stripe/ai/pull/332,safe_to_merge,"Doc refresh of existing upgrade-stripe skill: adds Dahlia to the release-name list and updates the Stripe.js script URL clover to dahlia; version-name text only, no authority change."
+https://github.com/stripe/ai/pull/312,needs_human,"Rewires the agent-skill supply chain: sync source moves from authenticated mcp.stripe.com to unauthenticated curl of docs.stripe.com/.well-known/skills, enables the daily cron, and the remote manifest now dictates which files sync.js writes (unsanitized path.join of manifest-supplied names, URL interpolated into execSync curl); output stays PR-gated so not a silent grant, but a provenance change to the instruction trust root warrants a human."
+https://github.com/aaif-goose/goose/pull/9798,safe_to_merge,"Adds an opt-in (goose.includeLastMessageSnippet meta flag) bounded single-line snippet of the last user-visible message to ACP listSessions; excludes tool/thinking/agent-only content, capped length, well-tested; metadata exposure to a local client that can already open sessions, no new agent action."
+https://github.com/aaif-goose/goose/pull/9717,safe_to_merge,"Consolidation that shrinks surface: Desktop switches to the existing ACP search-session path and the goose-server HTTP routes (/sessions list, search, insights, delete, export, import) plus chat_history_search are deleted (net -1158 lines); removes capability rather than adding it."
+https://github.com/aaif-goose/goose/pull/9684,safe_to_merge,Automated release chore: version bump in Cargo.toml/package.json plus regenerated canonical model/provider catalog JSON (new upstream model entries and provider metadata that only activate with user-supplied API keys); no new agent tools or scopes.
+https://github.com/aaif-goose/goose/pull/9637,safe_to_merge,"Developer eval tooling only: rewrites the compare_tasks SKILL.md doc (cache paths, cmd.py usage) and adds two harbor analysis recipe YAMLs that mount only the builtin developer extension for locally-run benchmark failure analysis; no change to shipped agent authority."