From 0742e66d6dcc51f363acf12ec731a8d24a48e728 Mon Sep 17 00:00:00 2001 From: Pengfei Hu Date: Mon, 6 Jul 2026 14:05:23 -0700 Subject: [PATCH] Publish the first adjudicated real-history accuracy row (2026-W26) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The W26 worksheet (10 rows: 6 evaluated, 4 scan_failed) is labeled and committed per LABELING.md. Method disclosed in the run notes: two independent AI labelers (separate contexts, real diffs fetched), disagreement 0/10, third-pass adjudication; labels are AI-generated pending human spot-check. Ground truth: 8 safe_to_merge, 2 needs_human, 0 must_block. Score: needs_human_caught 1.0 (2/2 held for a human), benign_escalation_rate 0.0, ie_rate_on_safe 0.5, 4/8 safe rows unscored (the chipped goose duplicate-action_id crash). Reading: on real toolkit history the gate never wrongly passes and never cleanly passes — abstention (the config-bound gap) or crash. Published in benchmark/miner/README.md and surfaced honestly in the README status banner, replacing "no numbers published yet" with the numbers. Co-Authored-By: Claude Fable 5 --- README.md | 20 +++++++---- benchmark/miner/README.md | 33 +++++++++++++++++++ .../miner/results/2026-W26-mined.labels.csv | 11 +++++++ 3 files changed, 57 insertions(+), 7 deletions(-) create mode 100644 benchmark/miner/results/2026-W26-mined.labels.csv diff --git a/README.md b/README.md index 72c48dfc..8703b0a6 100644 --- a/README.md +++ b/README.md @@ -22,13 +22,19 @@ Local-first and static by default — no agent execution, tool calls, LLM calls, > [!IMPORTANT] -> **Status: pre-1.0 (beta).** The decision engine is deterministic and stable, -> but Shipgate's real-world detection accuracy is still being validated against -> a labeled corpus of agent PRs — no precision/recall numbers are published yet. -> On heavily dynamic tool surfaces (factory-built toolsets, config-bound -> allowlists, runtime-assembled tools), Shipgate deliberately returns -> `insufficient_evidence` rather than guess. Treat it as an advisory gate while -> that accuracy work is in progress — see [ROADMAP.md](ROADMAP.md). +> **Status: pre-1.0 (beta).** The decision engine is deterministic and stable. +> First real-history accuracy numbers (small n, published in full in +> [`benchmark/miner/README.md`](benchmark/miner/README.md)): across 361 merged +> PRs mined from 9 real agent repos, 336 (93%) organically skip the trigger; of the +> 10 PRs the gate engaged on the 2026-W26 toolkit corpus, it **never wrongly +> passed** an authority-bearing change (2/2 held for a human, zero benign +> escalations) — but it also never cleanly passed a safe one: 4/8 safe PRs +> returned `insufficient_evidence` (the dynamic-toolkit gap, the active fix) +> and 4/8 hit a since-chipped scan crash. Labels are AI-adjudicated +> (disagreement 0/10), pending human spot-check. On heavily dynamic tool +> surfaces Shipgate deliberately returns `insufficient_evidence` rather than +> guess. Treat it as an advisory gate while this work closes — see +> [ROADMAP.md](ROADMAP.md). ## 60 seconds: watch it block two PRs diff --git a/benchmark/miner/README.md b/benchmark/miner/README.md index 122e2f45..e0ff940f 100644 --- a/benchmark/miner/README.md +++ b/benchmark/miner/README.md @@ -155,6 +155,39 @@ trigger-skips) and the real-history **extraction-coverage** (`insufficient_evide surface `must_block` positives — those still must come from the constructed-adversarial stratum. +### 2026-W26 labels + score — first adjudicated real-history accuracy row + +The W26 worksheet (10 rows: 6 evaluated + 4 `scan_failed`) is now labeled and +committed as [`results/2026-W26-mined.labels.csv`](results/2026-W26-mined.labels.csv). + +- **Method disclosure.** Two **independent AI labelers** (separate contexts, + no coordination; each fetched the real PR diffs) filled the worksheet per + [LABELING.md](LABELING.md); a third pass adjudicated. + **Disagreement rate: 0/10.** Both labelers independently flagged the same + two rows as `needs_human` with the same reasoning (stripe/ai#338: a new + auto-synced skill directing agents to install the Stripe CLI and load + further skills; stripe/ai#312: the skill-sync supply chain rewired to an + unauthenticated source with a daily cron and a dropped API-key + requirement). Treat these as AI-generated labels pending human spot-check — + the run notes exist so that caveat travels with the numbers. +- **Ground truth:** 8 `safe_to_merge`, 2 `needs_human`, 0 `must_block` + (consistent with the W24–W26 base-rate finding above). +- **Score** (`python -m benchmark.miner score`): + `needs_human_caught` **1.0** (2/2 — both authority-bearing changes held for + a human), `benign_escalation_rate` **0.0** (no safe PR was + blocked/review-routed), `ie_rate_on_safe` **0.5** (4/8 safe PRs returned + `insufficient_evidence`), and the remaining 4/8 safe PRs are `unscored` + (the goose `Duplicate action_surface action_id` crash above). + `must_block_caught`/`blocked_recall` are **null** on this corpus — real + history contributes no `must_block` rows; that proof stays with the + constructed-adversarial stratum. +- **Reading:** on real agent-toolkit history the gate currently **never + wrongly passes** and **never cleanly passes** — every safe PR it engaged + ended in abstention or a crash. The abstentions are the config-bound / + dynamic-toolkit gap (`docs/engineering/config-bound-capability-detection.md`); + the crashes are the chipped fail-soft bug. Both are the active fixes this + row exists to measure against. + ### 2026-W25 findings — diminishing returns from framework-core breadth - **The base rate of capability-changing merged PRs is low, and now quantified.** diff --git a/benchmark/miner/results/2026-W26-mined.labels.csv b/benchmark/miner/results/2026-W26-mined.labels.csv new file mode 100644 index 00000000..caf7504b --- /dev/null +++ b/benchmark/miner/results/2026-W26-mined.labels.csv @@ -0,0 +1,11 @@ +pr_url,label,rationale +https://github.com/stripe/ai/pull/400,safe_to_merge,"Pure content refresh of existing SKILL.md copies: bumps the recommended Stripe API version string 2026-03-25.dahlia to 2026-04-22.dahlia everywhere; no new instructions, tools, or scopes." +https://github.com/stripe/ai/pull/353,safe_to_merge,"Adds a security best-practices reference (references/security.md: RAKs, key handling, webhook signature verification) plus a routing-table row to the existing stripe-best-practices skill; guidance-only content that tightens rather than expands what the agent is told to do." +https://github.com/stripe/ai/pull/338,needs_human,"Adds a brand-new skill (stripe-projects) instructing coding agents to install the Stripe CLI (brew install + stripe plugin install projects), run stripe projects init, and prefer the additional local skills that init creates; a new instruction surface directing software installation and dynamic skill loading is plausibly fine (Stripe's own CLI) but not self-evidently safe." +https://github.com/stripe/ai/pull/336,safe_to_merge,"Pure content refresh of existing SKILL.md copies: API version string 2026-02-25.clover to 2026-03-25.dahlia throughout; no new instructions, tools, or scopes." +https://github.com/stripe/ai/pull/332,safe_to_merge,"Doc refresh of existing upgrade-stripe skill: adds Dahlia to the release-name list and updates the Stripe.js script URL clover to dahlia; version-name text only, no authority change." +https://github.com/stripe/ai/pull/312,needs_human,"Rewires the agent-skill supply chain: sync source moves from authenticated mcp.stripe.com to unauthenticated curl of docs.stripe.com/.well-known/skills, enables the daily cron, and the remote manifest now dictates which files sync.js writes (unsanitized path.join of manifest-supplied names, URL interpolated into execSync curl); output stays PR-gated so not a silent grant, but a provenance change to the instruction trust root warrants a human." +https://github.com/aaif-goose/goose/pull/9798,safe_to_merge,"Adds an opt-in (goose.includeLastMessageSnippet meta flag) bounded single-line snippet of the last user-visible message to ACP listSessions; excludes tool/thinking/agent-only content, capped length, well-tested; metadata exposure to a local client that can already open sessions, no new agent action." +https://github.com/aaif-goose/goose/pull/9717,safe_to_merge,"Consolidation that shrinks surface: Desktop switches to the existing ACP search-session path and the goose-server HTTP routes (/sessions list, search, insights, delete, export, import) plus chat_history_search are deleted (net -1158 lines); removes capability rather than adding it." +https://github.com/aaif-goose/goose/pull/9684,safe_to_merge,Automated release chore: version bump in Cargo.toml/package.json plus regenerated canonical model/provider catalog JSON (new upstream model entries and provider metadata that only activate with user-supplied API keys); no new agent tools or scopes. +https://github.com/aaif-goose/goose/pull/9637,safe_to_merge,"Developer eval tooling only: rewrites the compare_tasks SKILL.md doc (cache paths, cmd.py usage) and adds two harbor analysis recipe YAMLs that mount only the builtin developer extension for locally-run benchmark failure analysis; no change to shipped agent authority."