Publish the first adjudicated real-history accuracy row (2026-W26)#255
Merged
Merged
Conversation
pengfei-threemoonslab
force-pushed
the
check-verify-required
branch
from
65ab41e to
7cbf0be
Compare
pengfei-threemoonslab
force-pushed
the
w26-measurement
branch
from
48c3e59 to
7c5b41b
Compare
The W26 worksheet (10 rows: 6 evaluated, 4 scan_failed) is labeled and committed per LABELING.md. Method disclosed in the run notes: two independent AI labelers (separate contexts, real diffs fetched), disagreement 0/10, third-pass adjudication; labels are AI-generated pending human spot-check. Ground truth: 8 safe_to_merge, 2 needs_human, 0 must_block. Score: needs_human_caught 1.0 (2/2 held for a human), benign_escalation_rate 0.0, ie_rate_on_safe 0.5, 4/8 safe rows unscored (the chipped goose duplicate-action_id crash). Reading: on real toolkit history the gate never wrongly passes and never cleanly passes — abstention (the config-bound gap) or crash. Published in benchmark/miner/README.md and surfaced honestly in the README status banner, replacing "no numbers published yet" with the numbers. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
pengfei-threemoonslab
force-pushed
the
w26-measurement
branch
from
7c5b41b to
0742e66
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #254 (merge that first; GitHub retargeting note: re-base this to main after #254 lands if the branch isn't auto-deleted).
Roadmap item 1 (阻塞级): the W26 mined corpus (120 real merged PRs: stripe/agent-toolkit, block/goose, pydantic/pydantic-ai) is now labeled, scored, and published — ugly numbers included, per the "难看也发" instruction.
What the numbers say
safe_to_merge, 2needs_human, 0must_block.needs_human_caught1.0 — both authority-bearing changes (a new auto-synced skill directing CLI installs; a skill-sync supply-chain rewiring with a dropped API-key requirement) were held for a human.benign_escalation_rate0.0.ie_rate_on_safe0.5; the other 4 safe rows crashed. The gate never wrongly passes and never cleanly passes on this corpus — abstention (config-bound gap) or crash, which is precisely what the in-flight dynamic-toolkit work must move.Method disclosure (in the run notes, so the caveat travels with the numbers)
Two independent AI labelers (separate contexts, no coordination, real diffs fetched) per
LABELING.md; disagreement 0/10 — both independently flagged the same two rows asneeds_humanwith the same reasoning; third-pass adjudication. Labels are AI-generated pending human spot-check.Where it's published
benchmark/miner/results/2026-W26-mined.labels.csv(adjudicated, per protocol: only the adjudicated file is committed).benchmark/miner/README.md§ "2026-W26 labels + score" (full confusion matrix, method, reading).test_public_surface_contract+test_miner_corpusgreen.🤖 Generated with Claude Code