diff --git a/.github/workflows/validate.yml b/.github/workflows/validate.yml
index 6f21950..0cda189 100644
--- a/.github/workflows/validate.yml
+++ b/.github/workflows/validate.yml
@@ -22,7 +22,7 @@ jobs:
run: skills-ref validate ./practical-coding
- name: Run benchmark harness tests
working-directory: practical-coding
- run: python -m unittest benchmarks.test_benchmarks benchmarks.test_stability benchmarks.test_catalog
+ run: python -m unittest benchmarks.test_benchmarks benchmarks.test_stability benchmarks.test_catalog benchmarks.test_ladder_analysis benchmarks.test_progressive_validation
- name: Check Codex default_prompt references the skill as $skill-name
run: grep -qF '$practical-coding' practical-coding/agents/openai.yaml
- name: Ensure legacy local graph runtime is not reintroduced
diff --git a/AGENTS.md b/AGENTS.md
index a008d0c..c2a52b3 100644
--- a/AGENTS.md
+++ b/AGENTS.md
@@ -1,23 +1,33 @@
# Practical Coding
-This repository is an Agent Skill. If you are a coding agent reading this file from a checkout, apply the skill as follows.
+This repository is an Agent Skill. Apply [`SKILL.md`](SKILL.md) when working from this checkout.
-Read [`SKILL.md`](SKILL.md) for the route-agnostic shortest-path Core, the three-branch Event Router, and the Retrieval Policy. The Core always applies. Ordinary well-specified work stays Direct with no reasoning reference and no worker.
+## Runtime model
-The Event Router handles only unresolved blockers that change how the task must be reasoned about:
+1. Apply the Core and stay Direct unless one present unresolved event matches the Router.
+2. Route only Debugging, Decision, or Implementation; load at most one reasoning reference for the current event.
+3. Complete routing before diagnostic, decision-research, or change-mapping source work. The selected reference is the next read.
+4. Keep retrieval orthogonal. Unknown paths, callers, consumers, and data flow are retrieval questions, not Implementation events.
+5. Contract to the smallest affected surface as soon as the cause, choice, contract, invariant, or evidence boundary is established.
-| Trigger | Reasoning module |
+## Event Router
+
+| Present unresolved event | Reference |
|---|---|
-| An observed failure, regression, or incorrect behavior still lacks an evidenced cause | [`references/debugging.md`](references/debugging.md) |
-| A material unresolved user-owned choice about architecture, whether or which external dependency/implementation to adopt, APIs, data models, or compatibility would change the next action | [`references/decision.md`](references/decision.md) |
-| An unknown contract/invariant, an unresolved material risk boundary (security/permissions, irreversible side effects, persistence/migration, concurrency/transactions, compatibility), or insufficient evidence for a risky material claim blocks safe execution | [`references/implementation.md`](references/implementation.md) |
+| Observed failure still lacks an evidenced cause | [`references/debugging.md`](references/debugging.md) |
+| Material user-owned implementation choice changes the next action | [`references/decision.md`](references/decision.md) |
+| Unknown contract/invariant, coordinated guarantee, material risk boundary, or evidence plan blocks safe execution | [`references/implementation.md`](references/implementation.md) |
+
+A known target and settled behavior/boundary/check stay Direct even when risk nouns are present. A read-only mapping request is Direct plus Retrieval.
+
+Requirements interviewing is explicit-only through [`references/manual/clarification.md`](references/manual/clarification.md).
-Load exactly one first-match reasoning module in addition to the Core. A choice already settled by the request or repository is input, not a Decision event. A security, persistence, migration, concurrency, or compatibility noun is not itself an Implementation event when the governing boundary, affected surface, and sufficient check are already established. Do not treat file count, task nouns, search needs, or the existence of another library as routing evidence. If a different blocker appears later, reassess it without accumulating another reasoning reference in the root; use the Core when sufficient or isolate substantial follow-up work when the saved context exceeds handoff cost.
+## Retrieval
-Navigation is not a fourth Event Router branch. Code retrieval follows the cheapest sufficient available path: known source first, then bounded/ranked source discovery, then an already-available structural index only when relationship queries materially reduce exploration, followed by current-source verification for material claims. Routine targeted lookup needs no Navigation reference.
+Use known source, then bounded/ranked search, then an already-available structural capability when it materially reduces relationship discovery. Use exhaustive coverage or external authoritative sources only when the claim requires them. Source remains authoritative.
-Read [`references/navigation.md`](references/navigation.md) only when broad retrieval itself is substantial enough to need the detailed procedure. Host-native ranked search, FFF-style retrieval, and `DeusData/codebase-memory-mcp` are optional capabilities, not project requirements. Use them only when already available; otherwise fall back to ordinary source search without installing tooling or changing project configuration solely for retrieval. An already-integrated structural backend may maintain or refresh its own index as part of normal use.
+Read [`references/navigation.md`](references/navigation.md) only for substantial retrieval. Missing graph/ranked capabilities fall back without installing or persisting tooling solely for retrieval.
-For a substantial triggered event, prefer an isolated no-history worker only when its context savings exceed handoff cost; otherwise load the one selected reasoning reference in the root agent. Keep the root to the Core plus at most one loaded reasoning reference for the task. If broad mapping becomes expensive while another reasoning reference is already resident, prefer a read-only Navigation worker rather than loading a second large reference into the root.
+## Evolution
-The root agent owns user intent, authorization, repository state, routing, integration, and the final completion claim. A worker reads [`references/delegation.md`](references/delegation.md) plus exactly one assigned reference and returns a compact capsule. Decision, Debugging, and Navigation workers are read-only. An Implementation worker may write only when its assignment explicitly includes implementation, must have a bounded scope, and must be the sole writer there. Treat a capsule as stale after relevant repository changes.
+`evolution/` is maintainer knowledge and must not enter ordinary runtime context. During Skill maintenance, record mechanisms and failed changes there before modifying another runtime rule. Iterations use n=1; only a frozen release candidate receives the complete n=3 matrix.
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
index 295ab75..4e12bd6 100644
--- a/CONTRIBUTING.md
+++ b/CONTRIBUTING.md
@@ -1,40 +1,43 @@
# Contributing
-Contributions should preserve Practical Coding as one compact Skill with a small route-agnostic Core, three independently loadable reasoning modules, and a retrieval policy that does not become a permanent prompt tax.
+Practical Coding optimizes for the smallest quality-qualified runtime contract, not the most complete-looking workflow.
-- Keep `SKILL.md` as a small shortest-path Core plus Event Router and compact Retrieval Policy. Do not add routing intensity modes unless a mature cross-agent mechanism materially improves evidence.
-- The Event Router owns only unresolved reasoning blockers: Debugging, Decision, and Implementation. Navigation is retrieval, not a fourth reasoning route.
-- Do not retune Core or module wording from a single failed benchmark cell, and do not add case-specific bans named after benchmark tasks.
-- Keep the Direct Path real: simple, well-specified work must not require a reference, broad repository scan, or worker.
-- Routine targeted source lookup must not require `references/navigation.md`. Load that reference only when broad retrieval itself is substantial enough to justify the prompt cost.
-- Preserve context isolation. The root should normally carry the Core plus at most one reasoning reference. If broad mapping becomes expensive while another reference is resident, prefer a read-only Navigation worker when saved context exceeds handoff cost.
-- Workers read `references/delegation.md` plus exactly one assigned reference and return compact evidence capsules, not transcripts.
-- Keep shared-workspace workers read-only by default. An Implementation worker may write only when its assignment explicitly includes implementation, with one bounded writer scope.
-- Prefer strengthening an existing module or retrieval primitive over adding another module. Verification remains part of Core or Implementation; do not create a mandatory Verification route.
-- Do not introduce mandatory plans, execution documents, Git workflows, tests, reviews, documentation, or tool-specific ceremony as universal gates.
-- Preserve reuse-before-invention, mature-implementation-first, risk-proportional verification, evidence-driven debugging, and resistance to speculative code and defensive bloat.
-- Avoid new scripts, dependencies, configuration, generated project files, or persistent services unless they solve a demonstrated project need rather than merely making retrieval possible.
+## Runtime boundaries
-## Retrieval backends
+- Keep `SKILL.md` compact and route-agnostic outside the Event Router.
+- Route only a present unresolved Debugging, Decision, or Implementation event.
+- Load at most one reasoning reference for the current event.
+- Keep source/context retrieval independent from reasoning selection.
+- Requirements interviewing remains explicit-only.
+- Do not introduce mandatory plans, reviews, tests, documents, Git workflows, workers, or lifecycle ceremony.
-Practical Coding manages retrieval cost, not ownership of a particular search engine.
+Unknown locations, callers, consumers, or relationships are retrieval questions. They become Implementation only when a requested coordinated change has an unresolved governing contract or material risk/evidence boundary.
-1. Prefer already-known source and narrow direct reads.
-2. Prefer an already-available bounded/ranked source-search primitive over unbounded output. Host-native ranked search and FFF-style retrieval are examples, not requirements.
-3. Prefer an already-available structural index only for relationship-heavy questions where it materially reduces repeated source exploration. `DeusData/codebase-memory-mcp` is one mature example.
-4. If a stronger capability is unavailable, fall back to ordinary source search without changing repository configuration or installing/persisting tooling solely for retrieval.
-5. Material conclusions must still be checked against current source.
+## Evolution before wording
-Do not reimplement mature retrieval engines inside Practical Coding merely to avoid an optional external capability. Conversely, do not turn an optional capability into a hard dependency or automatic installation side effect.
+Runtime agents do not read `evolution/`. Skill maintenance must:
-## Mature implementation first
+1. record benchmark or real-project evidence;
+2. consolidate repeated mechanisms under `evolution/wiki/`;
+3. freeze the hypothesis and validation before changing runtime rules;
+4. preserve rejected changes under `evolution/rejected/`.
+
+Do not add benchmark case nouns to runtime wording. A module must have an observable pre-load trigger and stable quality-qualified lift over the smaller parent; otherwise tighten, merge, or remove it.
+
+## Benchmark discipline
+
+- Quality, safety, and build/reachability precede routing and cost.
+- Iteration runs use n=1. Run n=3 only for a frozen candidate believed ready to release.
+- Current-only runs may compare against prior published reports offline, but are not paired ranking evidence.
+- Keep deterministic prompts and oracles contract-consistent; do not reward behavior the prompt forbids.
+- Add a test when a newly discovered mechanism or scorer invariant would otherwise regress.
-For any non-trivial capability with credible prior art:
+Public regression covers Delivery, Debug, Decision, Router, and Native Behavior. Real-repository held-out coverage validates delivered evidence, zero spontaneous requirements interviewing, event selection, and retrieval scope.
-1. Inspect maintained mature implementations first.
-2. Prefer supported public integration surfaces — API, CLI, protocol, package, library, binary, or host-native tool — over copying internals or rebuilding the subsystem.
-3. Verify fit, maintenance state, known issues, release activity, operational constraints, and license.
-4. Add local code only for concrete gaps or confirmed upstream defects.
-5. Keep local patches narrow, attributable, and removable when upstream fixes the issue.
+## Retrieval capabilities
+
+Known source → bounded/ranked search → structural capability when useful → bounded exhaustive or authoritative external evidence only when required. FFF-style search, ordinary search, LSP/AST, and Codebase Memory are optional capabilities. Verify material claims against current source and disclose coverage gaps.
+
+## Mature implementation first
-A change is moving in the wrong direction if a trivial local edit must load or execute more process after the change than before it, if retrieval dumps more irrelevant context into the model, or if Practical Coding starts maintaining a weaker duplicate of a mature subsystem.
+For a non-trivial new capability, inspect maintained prior art, extract the smallest fitting mechanism, verify maintenance/license/API fit, and keep the result removable. Do not copy an entire expert workflow into the Core.
diff --git a/README.md b/README.md
index 29fda4d..18f5813 100644
--- a/README.md
+++ b/README.md
@@ -1,249 +1,74 @@
# Practical Coding
-
- 
- 
- 
- 
-
+Practical Coding is an Agent Skill for producing the smallest reliable coding change without turning every task into a heavyweight workflow.
-
- English · 简体中文
-
+It uses one compact Core, three evidence-triggered reasoning modules, and an orthogonal retrieval policy:
-> ## The right amount of engineering and context for every coding task.
->
-> **Simple work stays direct. Unknown bugs get root-cause debugging. Risky changes get rigor. Code retrieval stops at the cheapest sufficient path.**
-
-Practical Coding is a lean Agent Skill for coding assistants. It controls two costs independently:
-
-1. **Reasoning cost:** only unresolved blockers may load Debugging, Decision, or Implementation.
-2. **Context cost:** code discovery progresses from known source to bounded/ranked search to structural indexes only when each stronger rung is actually needed.
-
-```bash
-npx skills@latest add Hubujiu/practical-coding
-```
-
-## What changes with v1.2
-
-Navigation is no longer a fourth Event Router branch. It is a retrieval policy shared by Direct work and every routed event.
-
-| Situation | Practical Coding behavior |
-|---|---|
-| Rename, CSS tweak, known local edit | **Direct Path** — Core only |
-| Observed bug with unknown cause | Core + **Debugging** |
-| Material unresolved architecture/API/dependency choice | Core + **Decision** |
-| Unknown contract or unresolved migration, permission, persistence, concurrency, compatibility, or other material risk boundary blocks safe work | Core + **Implementation** |
-| Need to locate code | Use the cheapest sufficient retrieval capability; no reasoning route is selected merely because search is needed |
-| Broad relationship-heavy mapping | Prefer an already-available structural index when it saves exploration; otherwise fall back to bounded source search |
-
-The main invariant is now:
-
-> **Core + at most one reasoning module; retrieval is orthogonal and capability-based.** Risk-related nouns do not trigger Implementation after the governing boundary, affected surface, and sufficient check are already established.
-
-Legacy `.practical-coding.yaml` files from v1.1 are no longer read by the Skill and can be removed. Retrieval capability is discovered from the current host/environment instead of stored as a project preference.
-
----
-
-## Architecture
-
-```mermaid
-flowchart TB
- T[User coding task] --> C[Always-On Core]
- C --> E{Present unresolved reasoning blocker?}
- E -->|No| D[Direct Path]
- E -->|Observed failure lacks cause| G[Debugging]
- E -->|Material user-owned choice| A[Decision]
- E -->|Unknown contract / risk boundary| I[Implementation]
-
- D --> R{Need more code context?}
- G --> R
- A --> R
- I --> R
-
- R -->|No| V[Cheapest focused verification]
- R -->|Known path/symbol| K[Targeted read]
- R -->|Location unknown| S[Bounded / ranked source search]
- R -->|Structural relationship needed| X[Available structural index]
+```text
+Core / Direct
+├─ unresolved observed failure → Debugging
+├─ unresolved material implementation choice → Decision
+└─ unresolved contract, invariant, or risk boundary → Implementation
- K --> V
- S --> V
- X --> Q[Verify material claims in current source]
- Q --> V
- V --> O[Evidence-based completion]
+Retrieval (independent):
+known target → bounded/ranked search → structural or authoritative evidence → bounded exhaustive coverage
```
-### Always-On Core
+## Runtime contract
-The resident `SKILL.md` stays short and route-agnostic:
+The Core applies to every task:
- define the smallest observable success;
-- stop at the first implementation rung that works;
-- reuse established project primitives and contracts;
-- add no speculative abstractions, options, wrappers, configuration, or scaffolding;
-- make the smallest coherent reachable change;
-- prefer deletion and boring code;
-- add tests, fallback, validation, comments, or documentation only when a current requirement, contract, project rule, or necessary verification requires them;
-- run the cheapest focused check once;
-- claim only what fresh evidence supports.
-
-### Three reasoning modules
-
-| Module | Trigger | Purpose |
-|---|---|---|
-| [`debugging.md`](references/debugging.md) | An observed failure still lacks an evidenced cause | Reproduce → earliest broken state → supported cause → root-cause fix |
-| [`decision.md`](references/decision.md) | A material user-owned choice remains open and changes the next action | Resolve the smallest real decision frontier |
-| [`implementation.md`](references/implementation.md) | Safe execution is blocked by an unknown contract/invariant, material risk boundary, or insufficient evidence for a risky claim | Map the boundary, preserve guarantees, and choose sufficient evidence |
-
-The Event Router chooses only among these three. File count, task nouns, search needs, or the mere existence of another library do not select a reasoning module.
-
----
-
-## Retrieval: context selection instead of another workflow
-
-Retrieval answers a different question from the Event Router:
-
-> **What is the cheapest way to obtain the code context needed for the current task?**
-
-The ladder is deliberately progressive:
-
-1. **Current context / known target** → read only the known source.
-2. **Unknown location** → use an already-available bounded or ranked source-search primitive.
-3. **No ranked primitive** → fall back to ordinary filename, text, and symbol search such as host search, `rg`, `grep`, or `find`.
-4. **Relationship-heavy question** → use an already-available structural index only when it materially reduces repeated exploration.
-5. **Material conclusion** → verify against current source; source is authoritative.
-
-Stop at the first sufficient rung.
-
-### FFF-style retrieval and Codebase Memory are complementary
+- reuse established project primitives;
+- add no speculative abstractions, dependencies, configuration, validation, tests, or documentation;
+- preserve unrelated behavior and user changes;
+- verify with the cheapest check that can falsify the material claim.
-| Capability | Best at | Role in Practical Coding |
-|---|---|---|
-| Host-native / FFF-style ranked retrieval | Finding likely files and text candidates with bounded output and ranking signals | Cheap candidate discovery when already available |
-| Ordinary `rg` / filename / symbol search | Exact text, names, small repositories, universal fallback | Zero-special-backend fallback |
-| [`DeusData/codebase-memory-mcp`](https://github.com/DeusData/codebase-memory-mcp) or another structural index | Callers, callees, imports, implementations, dependency edges, cross-file flow | Optional structural retrieval when already available |
+If no unresolved Event Router condition matches, stay Direct. A risk-related noun, multiple files, unknown paths, or caller discovery does not itself justify a reasoning module.
-Practical Coding does **not** require `@ff-labs/pi-fff`, FFF, Codebase Memory, `.practical-coding.yaml`, or any persistent graph service. It also does not automatically install retrieval tooling merely because a stronger backend would be convenient. Missing capabilities degrade to the next available rung.
+When an event is present, load exactly one reference:
-`references/navigation.md` contains the detailed broad-retrieval procedure. Routine targeted lookup does not load it.
+- [`references/debugging.md`](references/debugging.md) — an observed failure still lacks an evidenced cause;
+- [`references/decision.md`](references/decision.md) — a material user-owned implementation choice remains open;
+- [`references/implementation.md`](references/implementation.md) — safe execution is blocked by an unresolved contract, coordinated invariant, material risk boundary, or evidence plan.
----
+Requirements interviewing and `grill-me` behavior are explicit-only through [`references/manual/clarification.md`](references/manual/clarification.md). One unavoidable blocking question in an ordinary task is normal interaction, not an interview mode.
-## Context isolation
+## Retrieval policy
-A textual instruction such as “return to Direct” cannot remove a reference that is already in the model context. Practical Coding therefore treats context isolation as a real resource decision:
+Retrieval is separate from reasoning. Use the cheapest available capability that supplies enough current context:
-- Direct work and small routed events use no worker.
-- The root should normally carry the Core plus at most one reasoning reference.
-- Routine source search uses host tools directly without loading Navigation.
-- If broad mapping becomes expensive while Debugging, Decision, or Implementation is already resident, a read-only Navigation worker is preferred only when the context saved exceeds handoff cost.
-- Workers return compact evidence capsules, not raw search transcripts or graph dumps.
+1. read a known path or symbol;
+2. use bounded/ranked filename, text, or symbol search;
+3. use an already-available structural index for relationship questions when it saves work;
+4. use bounded exhaustive coverage only for explicit exhaustive claims, and authoritative external sources only for contracts the repository cannot establish;
+5. verify material conclusions against current source.
-This is how progressive disclosure remains a context optimization rather than only a file-organization convention.
+[`references/navigation.md`](references/navigation.md) is the optional detailed procedure for substantial retrieval. Codebase Memory, LSP/AST, ranked search, and ordinary search are capabilities, not required dependencies.
----
+## Evolution discipline
-## Why not just install Ponytail + Superpowers together?
+Runtime agents do not read `evolution/`. Maintainers record experiences, consolidate repeated mechanisms, freeze experiments before changing runtime rules, and preserve rejected changes.
-Practical Coding is influenced by both projects, but its differentiator is the control policy.
+The rejected E/R depth and specialist-leaf experiment is retained under [`evolution/rejected/`](evolution/rejected/) with its n=3 evidence in [`benchmarks/results/progressive-tree/`](benchmarks/results/progressive-tree/). The replacement event-router experiment is documented in [`evolution/experiments/event-router-restoration.md`](evolution/experiments/event-router-restoration.md).
-| Question | Ponytail + Superpowers | Practical Coding |
-|---|---|---|
-| Tiny obvious edit | Two broad philosophies remain available to the host/model | **Core only** |
-| Unknown bug | Multiple applicable process rules may coexist | **Debugging only** |
-| High-risk change | Rigor exists, but selection belongs to separate systems | **Implementation only when the risk boundary is unresolved** |
-| Code discovery | Depends on host/tool behavior | **Explicit cheapest-sufficient retrieval ladder** |
-| Context footprint | Independent systems may accumulate | **Core + at most one reasoning reference; broad retrieval isolated only when worth it** |
+The accepted v1.5 release evidence is published under [`benchmarks/results/v1.5/`](benchmarks/results/v1.5/). Its frozen current-only n=3 matrix had zero indeterminate cells: Delivery 54/54, Debug 40/42, Decision 29/30, Native Behavior 52/54, and 61/66 held-out quality cells across 22 real tasks. Event reasoning was 113/114; after correcting three retrieval expectations that contradicted the current structural-mapping contract, the public Router result was 107/114 (reasoning 113/114, retrieval 108/114). These are non-paired release results; they do not claim superiority over other skills.
-Practical Coding is therefore not `ponytail.md + superpowers.md`. It is an adaptive policy for deciding **how much engineering reasoning and how much repository context are worth paying for now**.
+## Validation
----
-
-## Benchmark evidence
-
-The final v1.2 evidence is published under [`benchmarks/results/v1.2/`](benchmarks/results/v1.2/): reasoning classification passed 114/114, independent Retrieval classification passed 106/114, Native Behavior passed 54/54, and the Practical-only Delivery/Decision/Debug regression passed 75/75. The v1.1 five-route results remain historical evidence and are not score-comparable with the v1.2 two-dimensional Router schema.
-
-The published v1.1 results remain:
-
-| Suite | Practical v1.1 |
-|---|---:|
-| Delivery | **100% (27/27)** |
-| Decision | **100% (18/18)** |
-| Debug | **96.7% (29/30)** |
-| Router | **100% (114/114)** |
-| Native behavior | **100% (54/54)** |
-| Applicable total | **99.6% (242/243)** |
-
-See the [v1.1 data](benchmarks/results/v1.1/README.md), [Chinese report](benchmarks/results/v1.1/REPORT_ZH.md), and [reproduction guide](benchmarks/REPRODUCING.md). A fresh v1.2 run is required before publishing new comparative claims.
-
----
-
-## Installation
-
-Recommended:
-
-```bash
-npx skills@latest add Hubujiu/practical-coding
-```
-
-Claude Code:
-
-```bash
-git clone https://github.com/Hubujiu/practical-coding.git ~/.claude/skills/practical-coding
-```
-
-Cursor / Codex / Copilot CLI / Gemini CLI / Antigravity / Goose on macOS/Linux:
-
-```bash
-git clone https://github.com/Hubujiu/practical-coding.git ~/.agents/skills/practical-coding
-```
-
-Windows PowerShell:
+Public regression and real-repository held-out validation use `gpt-5.6-luna` at medium reasoning. Iteration runs use `n=1`; release claims require the complete current-only matrix at `n=3`.
```powershell
-git clone https://github.com/Hubujiu/practical-coding.git "$env:USERPROFILE\.agents\skills\practical-coding"
-```
+pwsh -NoProfile -File benchmarks/run.ps1 -SelfTest
+pwsh -NoProfile -File benchmarks/run.ps1 -ProgressiveSelfTest
-Project-local:
+python benchmarks/run_catalog.py --profile full --runs 3 --workers 3 `
+ --arm practical-current --arm practical-native --output benchmark-results/public-final
-```bash
-git clone https://github.com/Hubujiu/practical-coding.git .github/skills/practical-coding
+python benchmarks/progressive_validation.py --phase all --current-only --runs 3 --workers 3 `
+ --output benchmark-results/heldout-final
```
----
-
-## Repository structure
-
-```text
-practical-coding/
-├── SKILL.md
-├── AGENTS.md
-├── README.md
-├── README_zh.md
-├── references/
-│ ├── debugging.md
-│ ├── decision.md
-│ ├── implementation.md
-│ ├── navigation.md
-│ └── delegation.md
-├── benchmarks/
-├── examples/
-├── agents/
-└── docs/evaluations/
-```
-
-## Inspirations
-
-- [DietrichGebert/ponytail](https://github.com/DietrichGebert/ponytail): YAGNI, native/stdlib-first thinking, deletion over addition.
-- [obra/superpowers](https://github.com/obra/superpowers): systematic debugging, engineering rigor, verification, isolation.
-- [mattpocock/skills](https://github.com/mattpocock/skills) / [Agent Skills Spec](https://agentskills.io): progressive disclosure and composable Skill structure.
-- [dmtrKovalenko/fff](https://github.com/dmtrKovalenko/fff): bounded/ranked code retrieval ideas such as frecency-aware candidate discovery.
-- [DeusData/codebase-memory-mcp](https://github.com/DeusData/codebase-memory-mcp): structural code intelligence and graph-backed relationship queries.
-
-The differentiator is not ownership of those ideas. It is the policy that decides **when each capability is worth its implementation, retrieval, and context cost**.
-
-## Contributing
-
-If a real coding task exposes over-engineering, a missed escalation, noisy retrieval, unnecessary context loading, or unsafe simplification, open the smallest reproducible issue or PR. See [CONTRIBUTING.md](CONTRIBUTING.md).
+Historical published evidence remains version-specific and non-paired unless its arms are rerun in one frozen matrix.
-MIT License. See [THIRD_PARTY_NOTICES.md](THIRD_PARTY_NOTICES.md) for applicable upstream attribution.
+MIT License. See `THIRD_PARTY_NOTICES.md` for attribution.
diff --git a/README_zh.md b/README_zh.md
index 4b06743..e53b50a 100644
--- a/README_zh.md
+++ b/README_zh.md
@@ -1,249 +1,74 @@
# Practical Coding
-
-
-
-
-
-
+Practical Coding 是一个 Agent Skill：目标是交付最小、可靠的代码修改，同时避免把所有任务都变成重量级流程。
-
- English · 简体中文
-
+运行时只有一个 Core、三个由证据触发的推理模块，以及一条独立的检索策略：
-> ## 每个编码任务，只支付它真正需要的工程强度和上下文成本。
->
-> **简单工作保持 Direct；未知 Bug 才进入根因调试；高风险修改才增加严谨度；代码检索在第一个足够的层级停止。**
-
-Practical Coding 是一个轻量的编码 Agent Skill。它把两种成本分开控制：
-
-1. **推理成本：** 只有真正未解决的 blocker 才允许加载 Debugging、Decision 或 Implementation。
-2. **上下文成本：** 代码检索从已知源码开始，按需升级到 bounded/ranked search，再到结构化索引；只有前一级不足时才继续。
-
-```bash
-npx skills@latest add Hubujiu/practical-coding
-```
-
-## v1.2 的核心变化
-
-Navigation 不再是 Event Router 的第四条互斥分支，而变成 Direct 和所有 routed event 都可使用的 **Retrieval Policy**。
-
-| 当前情况 | Practical Coding 行为 |
-|---|---|
-| 改名、CSS、已知局部修改 | **Direct Path**：只用 Core |
-| 已观察到 Bug，但根因未知 | Core + **Debugging** |
-| 架构/API/依赖等实质选择仍未确定 | Core + **Decision** |
-| 未知契约或尚未解决的安全、迁移、权限、持久化、并发、兼容性等重要边界阻塞安全执行 | Core + **Implementation** |
-| 只是需要找到相关代码 | 走最便宜的充分检索路径；“需要搜索”本身不会选择 reasoning module |
-| 需要大范围调用链/依赖关系映射 | 已有结构化索引能明显减少探索时才使用；没有就直接回退 bounded source search |
-
-新的核心不变量：
-
-> **Core + 最多一个 reasoning module；Retrieval 与 Event Router 正交。** 当 governing boundary、affected surface 和 sufficient check 已经确定时，仅仅出现安全、持久化、迁移、并发或兼容性名词并不会触发 Implementation。
-
-v1.1 遗留的 `.practical-coding.yaml` 不再被 Skill 读取，可以直接删除。Retrieval 能力改为根据当前宿主/环境中已经存在的工具动态选择，而不是保存为项目级偏好。
-
----
-
-## 架构
-
-```mermaid
-flowchart TB
- T[用户编码任务] --> C[Always-On Core]
- C --> E{是否存在当前未解决的推理 blocker?}
- E -->|否| D[Direct Path]
- E -->|故障仍无根因| G[Debugging]
- E -->|存在实质未决选择| A[Decision]
- E -->|未知契约 / 风险边界| I[Implementation]
-
- D --> R{是否还需要更多代码上下文?}
- G --> R
- A --> R
- I --> R
-
- R -->|不需要| V[最便宜的聚焦验证]
- R -->|已知路径/符号| K[Targeted read]
- R -->|位置未知| S[Bounded / ranked source search]
- R -->|需要结构关系| X[已有 structural index]
+```text
+Core / Direct
+├─ 已观察失败但原因未证实 → Debugging
+├─ 会改变实现方向的重大选择尚未解决 → Decision
+└─ 契约、不变量、风险边界或证据计划未解决 → Implementation
- K --> V
- S --> V
- X --> Q[重要结论回到当前源码验证]
- Q --> V
- V --> O[只基于新证据交付]
+检索独立：
+已知目标 → 有界/排序搜索 → 结构或权威证据 → 有界穷举覆盖
```
-### Always-On Core
-
-常驻 `SKILL.md` 继续只保留所有编码任务都适用的最小规则：
-
-- 先定义最小可观察成功条件；
-- 实现上在第一个能工作的阶梯停止；
-- 复用已有 primitive、API 和 contract；
-- 不增加推测性的抽象、配置、wrapper、alias 或脚手架；
-- 只做最小 coherent reachable change；
-- 删除优先，普通代码优先；
-- validation、fallback、retry、测试、注释、文档只在真实需求、既有 contract、项目规则或必要验证要求时添加；
-- 最终只跑一次最便宜、最聚焦的检查；
-- 只声明最新证据真正支持的内容。
-
-### 三个 reasoning module
-
-| 模块 | 触发条件 | 目的 |
-|---|---|---|
-| [`debugging.md`](references/debugging.md) | 已观察故障仍缺少证据化根因 | 复现 → 最早错误状态 → 支持的根因 → 根因修复 |
-| [`decision.md`](references/decision.md) | 一个由用户决定的实质选择仍未解决，并会改变下一步 | 收敛最小真实 decision frontier |
-| [`implementation.md`](references/implementation.md) | 安全执行被未知 contract/invariant、重要风险边界或不足以支撑高风险结论的证据阻塞 | 映射边界、保留保证并确定充分证据 |
-
-Event Router 只在这三个模块之间选择。文件数量、任务名、需要检索代码、或者存在另一个 library，都不是 reasoning route 的触发条件。
-
----
-
-## Retrieval：上下文筛选，而不是另一套 workflow
-
-Retrieval 回答的是和 Event Router 不同的问题：
-
-> **当前任务需要的代码上下文，怎样以最低成本获得？**
-
-检索阶梯：
-
-1. **当前上下文 / 已知目标** → 直接读取目标源码。
-2. **不知道位置** → 优先使用宿主已经提供的 bounded/ranked search。
-3. **没有 ranked primitive** → 回退普通 filename / text / symbol search，例如宿主搜索、`rg`、`grep`、`find`。
-4. **问题主要是结构关系** → 只有已有 structural index 能显著减少重复探索时才使用。
-5. **重要结论** → 回到当前源码验证，源码始终是权威来源。
+## 运行时契约
-在第一个足够的层级停止。
+Core 始终适用：
-### FFF 式检索与 Codebase Memory 是互补关系
+- 先定义最小可观察成功；
+- 复用项目已经存在的 primitive；
+- 不添加推测性的抽象、依赖、配置、验证、测试或文档；
+- 保留无关行为和用户已有修改；
+- 用能证伪关键结论的最便宜检查验证。
-| 能力 | 最擅长 | 在 Practical Coding 中的角色 |
-|---|---|---|
-| 宿主原生 / FFF 式 ranked retrieval | 用有限输出和排序信号找到最可能相关的文件、文本候选 | 已经可用时作为低成本候选发现 |
-| 普通 `rg` / filename / symbol search | 精确文本、名称、小仓库、通用场景 | 零特殊后端的 fallback |
-| [`DeusData/codebase-memory-mcp`](https://github.com/DeusData/codebase-memory-mcp) 或其它 structural index | callers、callees、imports、implementations、依赖边、跨文件 flow | 已经可用且结构问题值得时使用 |
+没有 Event Router 条件时保持 Direct。风险名词、文件数量、路径未知或需要找 caller，本身都不是推理升级理由。
-Practical Coding **不要求** `@ff-labs/pi-fff`、FFF、Codebase Memory、`.practical-coding.yaml` 或任何常驻图谱服务，也不会仅仅因为“更强的后端可能方便”就自动安装检索工具。能力不存在就无损降级到下一层。
+存在未解决事件时只加载一个 reference：
-`references/navigation.md` 保存更详细的大范围检索流程。普通 targeted lookup 不需要加载它。
+- [`references/debugging.md`](references/debugging.md)：已观察失败仍没有证据化原因；
+- [`references/decision.md`](references/decision.md)：会改变实现方向的重大用户选择尚未解决；
+- [`references/implementation.md`](references/implementation.md)：安全执行被未知契约、协同不变量、重大风险边界或证据计划阻塞。
----
+需求采访和 `grill-me` 只能由用户显式激活 [`references/manual/clarification.md`](references/manual/clarification.md)。普通任务里一个不可避免的阻塞问题不算进入采访模式。
-## 上下文隔离
+## 检索策略
-“return to Direct” 这样的文字无法把已经读进模型上下文的 reference 真正移除，因此 v1.2 把隔离当成真实资源问题处理：
+检索与推理正交，始终使用能提供充分当前上下文的最便宜能力：
-- Direct 和小型 routed event 不使用 worker；
-- Root 通常只携带 Core + 最多一个 reasoning reference；
-- 普通源码搜索直接使用宿主工具，不加载 Navigation；
-- 如果 Debugging / Decision / Implementation 已经驻留，而大范围 mapping 会产生明显上下文噪声，只有隔离收益大于 handoff 成本时才派只读 Navigation worker；
-- worker 返回 compact evidence capsule，而不是 raw grep、搜索日志或 graph dump。
+1. 读取已知路径或 symbol；
+2. 使用有界/排序的文件名、文本或 symbol 搜索；
+3. 关系问题在确实节省探索成本时使用已经可用的结构索引；
+4. 只有明确穷举结论才做有界覆盖，仓库无法建立的外部契约才查询权威来源；
+5. 重要结论必须回到当前源码验证。
-这样 Progressive Disclosure 才真正是在节省上下文，而不只是把同一份大提示词拆成多个文件。
+[`references/navigation.md`](references/navigation.md) 只用于较重的检索过程。Codebase Memory、LSP/AST、排序搜索和普通搜索都是可选能力，不是依赖。
----
+## 演化纪律
-## 为什么不直接同时安装 Ponytail + Superpowers？
+普通运行时不读取 `evolution/`。维护阶段才记录体验、合并重复机制、先冻结实验再修改运行时规则，并保留失败改进。
-Practical Coding 的差异不在于“拥有更多规则”，而在控制策略。
+被拒绝的 E/R 深度与专家叶子实验保存在 [`evolution/rejected/`](evolution/rejected/)，其 n=3 证据位于 [`benchmarks/results/progressive-tree/`](benchmarks/results/progressive-tree/)。替代实验记录在 [`evolution/experiments/event-router-restoration.md`](evolution/experiments/event-router-restoration.md)。
-| 问题 | Ponytail + Superpowers | Practical Coding |
-|---|---|---|
-| 很小且明确的修改 | 两套宽泛哲学仍交给宿主/模型协调 | **只用 Core** |
-| 未知 Bug | 多套流程规则可能同时相关 | **只加载 Debugging** |
-| 高风险改动 | 有严谨能力，但由不同系统各自触发 | **只有风险边界未解决才加载 Implementation** |
-| 代码检索 | 依赖宿主自己的工具行为 | **显式 cheapest-sufficient retrieval ladder** |
-| 上下文成本 | 独立系统可能累计 | **Core + 最多一个 reasoning reference；昂贵检索只在值得时隔离** |
+已接受的 v1.5 发布证据位于 [`benchmarks/results/v1.5/`](benchmarks/results/v1.5/)。冻结的 current-only n=3 矩阵没有 indeterminate：Delivery 54/54、Debug 40/42、Decision 29/30、Native Behavior 52/54，22 个真实任务的 held-out 质量为 61/66。事件推理为 113/114；修正 3 个与当前“结构关系映射”合同矛盾的 Retrieval 期望后，公共 Router 为 107/114（reasoning 113/114、retrieval 108/114）。这些是非配对发布结果，不用于宣称优于其他 Skill。
-所以 Practical Coding 不是 `ponytail.md + superpowers.md`，而是在决定：**此刻值得支付多少工程推理成本，以及多少代码库上下文成本。**
+## 验证
----
-
-## Benchmark 证据
-
-最终 v1.2 证据已发布到 [`benchmarks/results/v1.2/`](benchmarks/results/v1.2/)：reasoning 分类 114/114、独立 Retrieval 分类 106/114、Native Behavior 54/54，Practical-only 的 Delivery/Decision/Debug 回归 75/75。v1.1 五路由结果继续作为历史证据保留，但不能与 v1.2 双维 Router schema 直接比较分数。
-
-当前公开的 v1.1 结果仍为：
-
-| Suite | Practical v1.1 |
-|---|---:|
-| Delivery | **100%（27/27）** |
-| Decision | **100%（18/18）** |
-| Debug | **96.7%（29/30）** |
-| Router | **100%（114/114）** |
-| Native behavior | **100%（54/54）** |
-| 适用总计 | **99.6%（242/243）** |
-
-查看 [v1.1 数据](benchmarks/results/v1.1/README.md)、[中文报告](benchmarks/results/v1.1/REPORT_ZH.md) 和 [复现指南](benchmarks/REPRODUCING.md)。在发布新的对比结论之前，需要重新跑 v1.2。
-
----
-
-## 安装
-
-推荐：
-
-```bash
-npx skills@latest add Hubujiu/practical-coding
-```
-
-Claude Code：
-
-```bash
-git clone https://github.com/Hubujiu/practical-coding.git ~/.claude/skills/practical-coding
-```
-
-Cursor / Codex / Copilot CLI / Gemini CLI / Antigravity / Goose（macOS/Linux）：
-
-```bash
-git clone https://github.com/Hubujiu/practical-coding.git ~/.agents/skills/practical-coding
-```
-
-Windows PowerShell：
+公共回归与真实仓库 held-out 使用 `gpt-5.6-luna`、medium reasoning。迭代阶段使用 `n=1`；发布结论必须完成 current-only 全矩阵 `n=3`。
```powershell
-git clone https://github.com/Hubujiu/practical-coding.git "$env:USERPROFILE\.agents\skills\practical-coding"
-```
+pwsh -NoProfile -File benchmarks/run.ps1 -SelfTest
+pwsh -NoProfile -File benchmarks/run.ps1 -ProgressiveSelfTest
-项目级安装：
+python benchmarks/run_catalog.py --profile full --runs 3 --workers 3 `
+ --arm practical-current --arm practical-native --output benchmark-results/public-final
-```bash
-git clone https://github.com/Hubujiu/practical-coding.git .github/skills/practical-coding
+python benchmarks/progressive_validation.py --phase all --current-only --runs 3 --workers 3 `
+ --output benchmark-results/heldout-final
```
----
-
-## 仓库结构
-
-```text
-practical-coding/
-├── SKILL.md
-├── AGENTS.md
-├── README.md
-├── README_zh.md
-├── references/
-│ ├── debugging.md
-│ ├── decision.md
-│ ├── implementation.md
-│ ├── navigation.md
-│ └── delegation.md
-├── benchmarks/
-├── examples/
-├── agents/
-└── docs/evaluations/
-```
-
-## 灵感来源
-
-- [DietrichGebert/ponytail](https://github.com/DietrichGebert/ponytail)：YAGNI、native/stdlib-first、删除优先。
-- [obra/superpowers](https://github.com/obra/superpowers)：系统化 debugging、工程严谨性、验证、任务隔离。
-- [mattpocock/skills](https://github.com/mattpocock/skills) / [Agent Skills Spec](https://agentskills.io)：Progressive Disclosure 和可组合 Skill 结构。
-- [dmtrKovalenko/fff](https://github.com/dmtrKovalenko/fff)：frecency 等面向 Agent 的 bounded/ranked code retrieval 思路。
-- [DeusData/codebase-memory-mcp](https://github.com/DeusData/codebase-memory-mcp)：结构化代码智能与 graph-backed relationship query。
-
-真正的差异不是“谁发明了这些思想”，而是：**什么时候值得为哪一种能力支付实现、检索和上下文成本。**
-
-## 贡献
-
-如果真实任务暴露出过度工程、漏升级、检索噪声、无意义上下文加载或不安全的极简化，欢迎提交最小可复现 issue/PR。详见 [CONTRIBUTING.md](CONTRIBUTING.md)。
+历史报告只证明生成它的版本；除非在同一冻结矩阵中重跑，否则只能做非配对参照。
-MIT License。适用的第三方致谢见 [THIRD_PARTY_NOTICES.md](THIRD_PARTY_NOTICES.md)。
+MIT License。第三方归属见 `THIRD_PARTY_NOTICES.md`。
diff --git a/SKILL.md b/SKILL.md
index 6eb1a67..66d07fc 100644
--- a/SKILL.md
+++ b/SKILL.md
@@ -1,56 +1,70 @@
---
name: practical-coding
-description: "Use for implementing, fixing, refactoring, or reviewing code with the smallest correct change; routes only unresolved debugging, architecture/choice, or risk-boundary blockers, while code retrieval uses the cheapest sufficient available capability."
+description: "Use for implementing, fixing, refactoring, or reviewing code with the smallest correct change; routes only unresolved debugging, decision, or execution-boundary blockers, while retrieval expands independently."
license: MIT
metadata:
author: Hubujiu
- version: "1.2"
+ version: "1.5"
---
# Practical Coding
-One short, route-agnostic core for every coding task. Reasoning escalates only for unresolved blockers; code retrieval escalates independently only when cheaper context selection is insufficient.
+Use the Core for every coding task. Load one reasoning reference only for a present unresolved event; expand retrieval independently.
+
+Before the first diagnostic, decision-research, or change-mapping source command, apply the Event Router. If a condition matches, its reference is the next read. Otherwise stay Direct.
## Core
-- Read the request and the code it actually touches; define the smallest observable success before editing.
-- Stop at the first rung that works: do nothing; reuse the nearest existing project primitive; use the standard library; use a native platform or environment feature; use an already-available dependency; one line; otherwise write the minimum local code.
-- Reuse established APIs and contracts instead of restating them. Prefer the thinnest adapter over an existing primitive, and build only behavior a current requirement or caller needs; names, conventions, aesthetics, and sibling richness are not requirements.
+- Read the request and touched code; define the smallest observable success.
+- Stop at the first rung that works: do nothing; reuse the nearest project primitive; standard library; platform feature; available dependency; one line; otherwise minimum local code.
+- Reuse established APIs and established contracts. Build only behavior required by a current caller or requirement; nearby richness is not a requirement.
+- When one established primitive owns a shared behavior, repair it once instead of adding caller-specific branches or modes.
- Add no speculative options, wrappers, aliases, configuration, scaffolding, helper layers, or one-implementation interfaces.
-- Make the smallest coherent reachable change. A standalone artifact needs no demo or new caller unless requested; a user-facing feature must be reachable. Keep unrelated code and existing user changes untouched.
-- Prefer deletion and boring code. Before finishing, remove each new dependency, file, option, wrapper, or comment that is unnecessary for the stated success, already-established contracts, and the chosen check.
-- Add validation, fallback, retry, documentation, comments, or tests only when required by stated behavior, an already-established project rule or contract, or necessary verification.
-- Run the cheapest focused check once after the final edit. Never repeat an unchanged check or substitute diff inspection for a required compile, type, or build gate. In an isolated workspace, routine installation of declared dependencies is allowed only as a bounded prerequisite; otherwise report missing prerequisites instead of changing the project solely for verification. If the check creates unrelated generated churn, inspect at most one diff, then leave and report it.
-- State only what fresh evidence supports; keep unrequested explanation short.
+- Make the smallest coherent reachable change. A standalone artifact needs no demo; a user-facing feature must be reachable. Preserve unrelated code and user changes.
+- Prefer deletion. Remove each new dependency, file, option, wrapper, comment, fallback, retry, test, or document not required by behavior, project contract, or verification.
+- Run the cheapest focused check once after the final edit. If no repository test exists, run one focused executable check, not a broad test runner. Prefer no-write check modes (for example Python `-B`). Never repeat an unchanged check or replace a required build gate with diff inspection. If disposable output remains, clean it once when safe; after a blocked or failed cleanup, stop and report it without another inspection or command. Install declared dependencies only as a bounded prerequisite in isolation; otherwise report the missing prerequisite.
+- State only fresh evidence. Unless requested, finish with the outcome, changed surface, check, and remaining uncertainty—no process recap.
## Direct Path
-The Core always applies. If no Event Router condition matches, proceed immediately with the Core alone. Targeted reads, filename/text/symbol searches, and use of an already-known project path are ordinary Direct work; they do not select a reasoning module.
+When no Event Router condition matches, proceed with the Core alone. Targeted reads and searches are ordinary Direct work.
## Event Router
-The Router selects only whether one additional reasoning module must be loaded; it does not own retrieval strategy or add implementation rules. Route only on a present unresolved blocker. Settled facts and choices are inputs, not events. A security, persistence, migration, concurrency, or compatibility noun is not itself a blocker: when the governing boundary, affected surface, and sufficient check are already established, stay Direct. Use this first-match ladder:
+Route only a present unresolved blocker. Settled facts and choices are inputs, not events; risk or technology nouns do not route by themselves.
+
+Use this first-match ladder:
+
+1. An observed failure, regression, incorrect behavior, or failed check still lacks an evidenced cause: read `references/debugging.md`.
+2. A material user-owned choice about architecture, dependency, implementation, API, data model, or compatibility remains unresolved and would change the next action: read `references/decision.md`.
+3. Safe execution is blocked by an unknown contract or invariant; required producers and consumers must change together but their joint contract is unknown; a material security, irreversible-effect, persistence/migration, concurrency/transaction, or compatibility boundary remains unresolved; or evidence sufficient for a risky material claim is unknown: read `references/implementation.md`.
+
+Read exactly that reference plus the Core. Resolve the blocker, then contract. Do not preload candidates. A failed check of your proposed change stays inside the active event; correct the candidate without loading Debugging. Reassess only for a different later blocker; handle a trivial one with the Core or isolate a substantial one when the saved context exceeds handoff cost.
-1. An observed failure, regression, or incorrect behavior still lacks an evidenced cause: read `references/debugging.md`. A symptom or named failing function is not a diagnosed cause.
-2. A material user-owned choice about architecture, whether or which external dependency or implementation to adopt, API, data model, or compatibility remains unresolved and would change the next action: read `references/decision.md`. A choice already specified and authorized by the user is settled input; research needed to resolve an open choice belongs inside Decision.
-3. Safe execution is blocked by an unknown contract or invariant, an unresolved material risk boundary such as security/permissions, irreversible effects, persistence/migration, concurrency/transactions, or compatibility, or insufficient evidence for a risky material claim: read `references/implementation.md`.
+Stay Direct when the cause, choice, governing boundary, affected surface, and sufficient check are already established. A named target with settled behavior remains Direct even when it concerns risk; a requested standalone artifact with no integration remains Direct. Unknown locations, file count, callers, consumers, and data flow are Retrieval questions, not Implementation events. Read-only source mapping is never an Implementation event. Choosing evidence sufficient to support a material risk or performance claim is an Implementation boundary, not a user-owned product Decision.
-Read exactly that one reasoning module in addition to the Core. Resolve the blocker, then reassess only if a different blocker appears. Reassessment does not justify accumulating another reasoning reference in the root: handle a trivial later blocker with the Core, or isolate a substantial later event in a worker when the saved context exceeds handoff cost. Do not preload modules or load candidates together to compare them. Task nouns, file count, and the mere existence of an alternative library do not select a module.
+## Explicit-only requirements interview
+
+Load `references/manual/clarification.md` only when the current instruction explicitly asks to be interviewed, grilled, or questioned before implementation. Ambiguity, importance, risk, or one unavoidable blocking question does not activate it. Decision resolves a genuinely open material choice; alternatives alone do not activate it.
## Retrieval Policy
-Retrieval is orthogonal to the Event Router. Use the cheapest available capability that can provide sufficient task-relevant code context, and stop at the first sufficient rung:
+Retrieval is orthogonal to execution. Stop at the first sufficient rung:
-1. If current context, a known path, or a known symbol is sufficient, read only that source.
-2. Otherwise use an already-available bounded or ranked source-search capability; fall back to ordinary filename, text, and symbol search.
-3. When the unresolved question is primarily structural — callers, callees, imports, implementations, dependencies, or cross-file flow — prefer an already-available structural code index when it materially reduces exploration.
-4. If a stronger retrieval capability is unavailable, fall back without changing project configuration or installing/persisting tooling solely for retrieval.
-5. Verify material conclusions against current source; source is authoritative.
+1. Read a known path or symbol directly. Do not inventory history, branches, or unrelated files, or search outside the project for an implementation unless the blocker or request requires it.
+2. Otherwise use an available bounded/ranked source search, falling back to filename, text, or symbol search.
+3. For unknown callers, dependencies, authoritative boundaries, or cross-file guarantees, prefer an available structural code index when it materially reduces exploration. Stay Bounded when known identifiers or a finite known consumer set can be located by text search; a known edit target alone is not Targeted when relationships are unknown.
+4. For bounded exhaustive repository claims, use coverage-aware discovery and disclose gaps. For external contracts, use the smallest authoritative current source.
+5. Fall back without installing retrieval tooling; verify material conclusions in current source. Use NONE when only user-owned policy is missing, and retrieve only facts needed to resolve it.
-Do not read `references/navigation.md` for routine lookups. Read it only when broad retrieval itself is substantial enough to require the detailed retrieval procedure. If a reasoning reference is already loaded, do not add Navigation to the same root context merely to search: use the short policy above, or isolate substantial broad mapping in a read-only worker when context savings exceed handoff cost.
+Routine lookup stays here; do not load `references/navigation.md`. Load it only for substantial broad structural mapping or bounded exhaustive discovery. Do not add Navigation beside a reasoning reference merely to search; use this policy or isolate the mapping when worthwhile.
## Isolation Gate
-Direct work and a single routed event in a small context use no worker. The root never reads `references/delegation.md`. Keep the root to the Core plus at most one loaded reasoning reference for the task; do not build a sequence of references that only accumulates context.
+Direct work and one routed event in small context use no worker. Keep the root at Core plus one active reasoning reference.
+
+When isolation saves more context than its handoff cost, dispatch one worker with `references/delegation.md` and one assigned reference. Navigation and Debugging workers are read-only. Decision is read-only unless the root authorizes settled implementation. Implementation writes only an assigned non-overlapping scope as sole writer. Never overlap writers or build worker pipelines.
+
+## Evolution contract
-When isolation clearly saves more context than its handoff costs, dispatch one worker and tell that worker to read `references/delegation.md` plus exactly one assigned reference. Navigation workers are read-only. Decision and Debugging workers are read-only. An Implementation worker may write only when its assignment explicitly includes implementation, must have a bounded scope, and must be the sole writer there. Never use overlapping writers or worker pipelines.
+Runtime agents do not read `evolution/`. Maintenance records benchmark and real-project receipts there, freezes experiments before changing runtime rules, and preserves rejected changes. Never add benchmark-specific nouns or keep a module for symmetry; each runtime module must earn quality-qualified net lift over its smaller parent.
diff --git a/agents/openai.yaml b/agents/openai.yaml
index 909eb85..43911f5 100644
--- a/agents/openai.yaml
+++ b/agents/openai.yaml
@@ -1,6 +1,6 @@
interface:
display_name: "Practical Coding"
- short_description: "Small correct changes with adaptive code retrieval"
- default_prompt: "Use $practical-coding. Keep well-specified work on the Direct Path, load at most one reasoning reference for an unresolved blocker, and retrieve code through the cheapest sufficient available capability."
+ short_description: "Small correct changes with event routing and bounded retrieval"
+ default_prompt: "Use $practical-coding. Apply the Core, route only a present unresolved Debugging, Decision, or Implementation event, and use the cheapest sufficient retrieval capability. Load at most one reasoning reference for the current event."
policy:
allow_implicit_invocation: true
diff --git a/benchmarks/LADDER_EVOLUTION.md b/benchmarks/LADDER_EVOLUTION.md
new file mode 100644
index 0000000..48d7b5b
--- /dev/null
+++ b/benchmarks/LADDER_EVOLUTION.md
@@ -0,0 +1,174 @@
+# Historical rejected experiment: progressive depth and capability-tree protocol
+
+This document preserves the frozen methodology used to reject the E/R depth and specialist-leaf tree. It is not the active runtime or release protocol. See [`NEXT_VALIDATION.md`](NEXT_VALIDATION.md) for the Event Router validation.
+
+This protocol evaluates whether Practical Coding chooses the **lowest quality-qualified depth and the smallest useful capability path**.
+
+The current architecture is an experiment:
+
+- execution depth: `E0 E1 E2 E3` where E1 is **Probe**;
+- retrieval depth: `R0 R1 R2 R3`;
+- E2 roots: `diagnosis`, `engineering`;
+- E3 specialist leaves: `security`, `state`, `compatibility`, `performance`, `quality`, `interface` where valid under the active root.
+
+No level or node is permanent.
+
+## 1. Quality before cost
+
+A variant is sufficient only after correctness/safety and build/reachability gates pass. Only then compare tokens, time, tool calls, LOC, and loaded references.
+
+A cheaper failure is not a win.
+
+## 2. Always keep real baselines
+
+For every candidate architecture, retain at least:
+
+1. **no-skill**;
+2. **accepted prior Practical Coding**;
+3. **candidate adaptive tree**.
+
+Use expert skills such as debugging, review, security, or design-oriented skills as informative specialist comparators on task families they actually claim to cover, not as universal baselines.
+
+This distinguishes net skill lift from merely moving work between prompt layers.
+
+## 3. Freeze before observing
+
+Freeze candidate commit, task manifest, scorer/oracle, model/harness, depth-capped bundles, capability-path ablations, and repetition count before the first result is inspected.
+
+Do not create a per-case prompt after seeing the answer.
+
+## 4. Keep execution and retrieval orthogonal
+
+The axes answer different questions and must not be inferred from the same action:
+
+- **Retrieval:** how much source/context had to be acquired before the next material decision was supported?
+- **Execution:** after relevant evidence was available, how much structured engineering reasoning was required?
+
+Source discovery alone never raises execution depth. Finding/reading callers, references, siblings, contracts, implementations, or configuration belongs to R0–R3. A task may legitimately be `E0/R1` or `E0/R2`.
+
+E1 requires a **cheap executable probe**: reproduce one behavior, exercise one path, falsify one concrete hypothesis, or run one focused check whose result determines the next action. If a case definition cannot point to such a probe, do not label retrieval activity as E1.
+
+This boundary is part of the experiment. If E1 rarely earns a distinct minimum-sufficient role after retrieval is separated, test merging/removing it rather than broadening it back into source inspection.
+
+## 5. Calibrate depth independently
+
+### Execution
+
+Run caps at `E0`, `E1`, `E2`, `E3` with retrieval permissive enough not to be the bottleneck. The first stable quality-qualified cap is the minimum sufficient execution depth.
+
+For E1-specific cases, freeze the executable probe allowed by the cap. Retrieval-only expansion must remain available independently and must not be counted as E1 behavior.
+
+### Retrieval
+
+Run caps at `R0`, `R1`, `R2`, `R3` with execution permissive enough not to be the bottleneck. R2 permits the appropriate specialized branch (structural or external); R3 permits bounded exhaustive repository discovery.
+
+External evidence is not an `R4` successor to repository search.
+
+Use at least `n=3` determinate repetitions for boundary claims. Mark unstable cells indeterminate.
+
+## 6. Calibrate tree nodes by ablation
+
+Depth alone cannot tell whether a specialist node earns its context cost.
+
+For tasks whose adaptive run selects a capability path, freeze the smallest relevant ablation set before running:
+
+```text
+parent-only
+parent + claimed leaf
+candidate adaptive tree
+```
+
+When useful, add one plausible sibling as a branch-confusion control. Do not test every leaf on every task.
+
+A leaf is justified only when, on the population it claims to cover, it produces stable net quality lift over its parent or preserves quality while materially lowering cost/routing error elsewhere.
+
+Track:
+
+- **unnecessary root load** — E0/E1 was quality-sufficient but adaptive loaded a root;
+- **unnecessary leaf load** — parent was quality-sufficient but adaptive loaded a leaf;
+- **missed root/leaf** — adaptive failed at a shallower path while the frozen deeper path succeeds;
+- **branch confusion** — the selected sibling fails or costs materially more while another pre-frozen path succeeds;
+- **path exactness** — adaptive selects the lowest quality-qualified frozen path.
+
+Do not infer leaf value from task nouns alone.
+
+## 7. Observation format
+
+`benchmarks/ladder_analysis.py` consumes aggregated JSONL after repetitions are classified.
+
+Capped depth row:
+
+```json
+{"task_id":"bug-017","axis":"execution","arm":"cap","level":"E2","qualified":true,"tokens":4200,"duration_seconds":31.2,"tool_calls":8}
+```
+
+Adaptive row with routing instrumentation:
+
+```json
+{"task_id":"bug-017","axis":"execution","arm":"adaptive","level":"E3","qualified":true,"capability_path":["diagnosis","state"],"references_loaded":["references/debugging.md","references/specialists/state.md"],"tokens":6100,"duration_seconds":45.1,"tool_calls":12}
+```
+
+The routing fields are benchmark-only instrumentation; runtime answers need not expose labels.
+
+## 8. Family-level analysis
+
+Report over/under-escalation and path behavior by task family and repository, not only globally. A boundary that looks good in aggregate can systematically fail on one mechanism.
+
+Useful families include known-target edits, retrieval-only local/structural discovery, one-probe execution uncertainty, observed-failure diagnosis, unresolved contract/invariant changes, security boundary, state/concurrency, compatibility/migration, measured performance, structural review/refactor, and material interface work.
+
+Use mechanism labels only for analysis; do not paste benchmark-specific nouns into runtime triggers.
+
+## 9. Retrieval-specific calibration
+
+Measure more than tool choice:
+
+- candidate results inspected before localization;
+- source lines/files read;
+- structural index/graph use when available;
+- pagination/coverage for exhaustive claims;
+- contraction point after localization;
+- unnecessary external lookup and unnecessary repository-wide expansion.
+
+A better retrieval path is one that reaches authoritative evidence with less irrelevant context, not one that uses a particular tool.
+
+`references/navigation.md` is evaluated as the deeper R2 Structural/R3 coverage procedure inside this axis; do not score Navigation as a third independent depth.
+
+## 10. Real-project experience
+
+Benchmark tasks are necessary but not sufficient. Record real-project successes, routing mistakes, repeated user corrections, and expensive dead ends as **experience receipts** using `evolution/EXPERIENCE_SCHEMA.md`.
+
+Do not promote one anecdote directly into `SKILL.md`. Consolidate repeated mechanisms into persistent evolution knowledge first.
+
+## 11. Evolution loop
+
+```text
+benchmark runs + real-project receipts
+ ↓
+ evolution wiki knowledge
+ ↓
+ frozen candidate hypothesis
+ ↓
+ depth caps + path ablations + baselines
+ ↓
+ held-out validation
+ ↙ ↘
+ accept reject
+ ↓ ↓
+ runtime Skill retain lesson only
+```
+
+This mirrors the useful separation from WikiSkill: raw experience, accumulated maintenance knowledge, and executable Skill wording remain distinct.
+
+## 12. Acceptance gate for this branch
+
+Before proposing merge to `main`:
+
+1. existing harness self-tests pass;
+2. no stable correctness/safety/build regression versus accepted Practical Coding and no-skill reference points;
+3. claimed depth boundaries have at least three determinate repetitions;
+4. execution/retrieval labeling demonstrates the E1 Probe vs R-depth boundary rather than conflating source inspection with execution;
+5. changed boundaries are tested on held-out tasks;
+6. new specialist leaves have parent-vs-leaf ablation evidence on their claimed families;
+7. over/under-escalation and unnecessary/missed leaf rates are reported;
+8. real-project evidence is treated as calibration input, not hidden held-out proof;
+9. no node survives only because the tree looks conceptually neat.
diff --git a/benchmarks/NEXT_VALIDATION.md b/benchmarks/NEXT_VALIDATION.md
index 7556c2a..a8f89cc 100644
--- a/benchmarks/NEXT_VALIDATION.md
+++ b/benchmarks/NEXT_VALIDATION.md
@@ -1,147 +1,45 @@
-# Next validation protocol
+# Release validation protocol — event-router restoration
-This document freezes the evidence requirements for the next Practical Coding validation cycle before new results are inspected. Its purpose is to prevent post-hoc metric selection, public-regression overfitting, and marketing claims stronger than the evidence.
+This protocol freezes the final validation for `experiment/progressive-ladders` after n=1 iteration.
-The current public release is **Practical Coding v1.1**. Its compact aggregates and evidence boundaries are in [`results/v1.1/`](results/v1.1/); the v1.0 release remains archived in [`results/v1.0/`](results/v1.0/).
+## Candidate contract
-## 1. Freeze before running
+- Core plus Direct default;
+- exactly three adaptive reasoning modules: Debugging, Decision, Implementation;
+- retrieval orthogonal and cheapest-sufficient;
+- requirements interviewing explicit-only;
+- no numeric execution/retrieval runtime depths or specialist leaves.
-Before any release-quality model run:
+## Iteration gate
-1. commit the candidate and use a clean working tree;
-2. record the exact candidate commit and benchmark manifest hashes;
-3. do not change Skill text, tasks, scorers, or acceptance thresholds after seeing partial results from the same cycle;
-4. if an instrument bug is found, invalidate the affected run, fix the instrument, document why, and rerun the complete affected matrix;
-5. preserve the complete candidate Skill bundle and comparator pins.
+Use n=1 while changing a mechanism. Save the full result, classify failures as infrastructure, scorer/oracle, stochastic, routing, or genuine capability failures, and record reusable lessons under `evolution/`. Never add case-specific nouns to runtime text.
-Documentation-only changes do not justify retuning against already-inspected public cells. New evidence should come from held-out tasks, stack/interference tests, or repeated independent failures.
+## Final gate
-## 2. Required gate order
-
-### Gate A — harness self-test
+The candidate must be committed and unchanged before both commands run:
```powershell
-pwsh -NoProfile -File benchmarks/run.ps1 -SelfTest
-```
-
-A failing self-test blocks all benchmark claims.
-
-### Gate B — public regression/current-vs-previous gate
-
-Run whenever `SKILL.md` or `references/` behavior changes:
-
-```powershell
-pwsh -NoProfile -File benchmarks/run.ps1 `
- -Profile full `
- -Runs 3 `
- -Workers 3 `
- -BaselineRef `
- -IncludeBaseline `
- -RequireStableRanking
-```
-
-Acceptance order remains correctness/safety → build/reachability → efficiency.
-
-### Gate C — private held-out evidence
-
-The strongest generalization claim requires a task set that was not consulted while editing the Skill.
-
-Minimum first held-out target:
-
-- at least 20 real coding tasks;
-- include simple/direct tasks, root-cause bugs, risky multi-file changes, and architecture/navigation tasks;
-- executable verification whenever possible;
-- seed state fails and oracle/reference state passes before model calls;
-- same fixed model/harness for `no-skill` and Practical arms;
-- at least three paired repetitions for publication-quality claims.
-
-## 3. Required combined-stack benchmark
-
-Before claiming that Practical Coding is experimentally better than installing Ponytail and Superpowers together, add this arm:
-
-```text
-no-skill
-Ponytail
-Superpowers
-Ponytail + Superpowers
-Practical Coding
-```
+python benchmarks/run_catalog.py --profile full --runs 3 --workers 3 `
+ --arm practical-current --arm practical-native `
+ --output benchmark-results/event-router-final-public
-The combined arm must install the **actual current Skills simultaneously** in the same harness rather than concatenate their text manually.
-
-Measure at minimum:
-
-- task success, safety, and build;
-- total/uncached input tokens, output tokens, reasoning tokens;
-- model time and tool calls;
-- number of Skill/reference files loaded;
-- unnecessary process/module loads;
-- missed escalations;
-- number of planning/debugging/delegation phases entered;
-- final production/test LOC;
-- whether both broad coding/process policies are invoked on simple tasks.
-
-The task set must include at least:
-
-1. trivial/local direct edits;
-2. clear multi-file but low-risk changes;
-3. unknown root-cause bugs;
-4. security/persistence/concurrency boundaries;
-5. unresolved architecture/dependency decisions.
-
-### Hypothesis being tested
-
-The architectural hypothesis is not "Ponytail is bad" or "Superpowers is bad." It is:
-
-> Two independently broad Skills may provide useful capabilities but incur duplicated routing/process context and leave their interaction to the host/model, while Practical's single event router should preserve similar specialist rigor with less unnecessary process on tasks that do not need it.
-
-This remains a hypothesis until the combined arm is measured.
-
-## 4. Routing and interference ablation
-
-To attribute any gain to adaptive routing rather than prompt wording, test:
-
-```text
-no-skill
-Core only
-Core + Decision
-Core + Debugging
-Core + Implementation
-Full Practical
-Ponytail + Superpowers
+python benchmarks/progressive_validation.py --phase all --current-only --runs 3 --workers 3 `
+ --output benchmark-results/event-router-final-heldout
```
-Record:
-
-- unnecessary module loads;
-- missed escalations;
-- references loaded and bytes/tokens injected;
-- route changes per task;
-- time/tokens before the correct route is reached;
-- worker/subagent dispatches.
-
-## 5. Statistical language
-
-`n=3` is a stability gate, not proof that small differences are statistically resolved. For small deltas use language such as `numerically ahead`, `numerically behind`, or `tied on this matrix` and report task counts separately from repeated trials.
-
-Future confidence intervals should bootstrap by task/case ID so repetitions of one task are not treated as independent tasks.
-
-## 6. Failure discipline
-
-When a failure appears:
+Required evidence:
-1. save the complete run first;
-2. classify infrastructure vs scorer/oracle defect vs stochastic behavior vs genuine Skill behavior;
-3. do not add case-specific nouns merely to turn a public cell green;
-4. prefer a general invariant only after the same mechanism appears independently;
-5. rerun the complete affected gate after a behavior change.
+- zero indeterminate cells and at least three determinate repetitions per cell;
+- no Delivery correctness/safety/build regression;
+- Debug, Decision, and Native Behavior stable enough for a release claim;
+- Router reasoning and retrieval reported separately;
+- at least 20 held-out real tasks across multiple repositories;
+- zero spontaneous requirements-interview activation;
+- held-out quality and routing failures individually classified;
+- raw machine paths excluded from published compact artifacts.
-## 7. Claim ladder
+Historical v1.2 reports may be compared offline, but this current-only cycle cannot make a paired superiority claim against v1.2, no-skill, Ponytail, or combined skill arms.
-| Evidence completed | Allowed claim |
-|---|---|
-| Public regression only | Stable / numerically competitive on the fixed public matrix |
-| + combined Ponytail/Superpowers arm | Bounded claims about integrated-stack efficiency/quality on that task population |
-| + private held-out paired run | Bounded generalization claims for the held-out population |
+## Merge gate
-Never collapse Delivery vs Ponytail, Decision vs grilling, Debug vs Superpowers, and the combined-stack comparison into a single universal score.
+Update the formal README and compact result artifacts from the final reports, run all unit/self/Skill validation, push the branch, and require PR CI success. If a genuine quality or stable reasoning regression remains, return to n=1 iteration and freeze the next mechanism change before editing runtime rules.
diff --git a/benchmarks/README.md b/benchmarks/README.md
index c6f85ed..6ad3ada 100644
--- a/benchmarks/README.md
+++ b/benchmarks/README.md
@@ -1,139 +1,46 @@
# Practical Coding benchmark chain
-This directory contains the reproducible evaluation harness for **Practical Coding v1.2**. The v1.2 runner uses a two-dimensional routing contract; committed v1.1 results remain historical evidence for the former five-way classifier.
+The active release candidate uses one Core, a Debugging/Decision/Implementation Event Router, and orthogonal retrieval. The rejected E/R depth and specialist-leaf experiment remains historical evidence under [`results/progressive-tree/`](results/progressive-tree/) and [`../evolution/rejected/`](../evolution/rejected/).
-The benchmark design intentionally avoids a single manufactured leaderboard. Each capability is compared with the most relevant specialist behavior, while Practical-owned routing suites test the integration layer that specialists do not provide by themselves.
+## Active questions
-For exact commands, pinned upstream commits, evidence boundaries, and reproduction requirements, see [`REPRODUCING.md`](REPRODUCING.md). Current compact evidence lives in [`results/v1.2/`](results/v1.2/); [`results/v1.1/`](results/v1.1/) and [`results/v1.0/`](results/v1.0/) are retained as historical evidence.
+1. Does the Skill deliver a correct, safe, reachable result?
+2. Does it load the one reasoning module required by the present unresolved event—and no module for Direct work?
+3. Does retrieval stop at the cheapest sufficient capability?
+4. Does requirements interviewing remain at zero spontaneous activation?
-## What is measured
+## Iteration versus release
-| Suite | Compared arms | What it measures |
-|---|---|---|
-| Delivery | Practical vs Ponytail | Correctness, safety, build reachability, LOC, tokens, time, tool calls |
-| Decision | Practical vs Matt Pocock `grilling` | Whether a material unresolved choice is surfaced and converged without premature implementation |
-| Debug | Practical vs Superpowers | Root-cause repair, sibling callers, delivered invariant, safety, efficiency |
-| Router | Practical vs expected two-dimensional contract | Reasoning (`NONE` plus Decision / Debugging / Implementation) and independent Retrieval (`NONE` / Targeted / Bounded / Structural) |
-| Native behavior | Practical only | Real Skill discovery, reasoning-reference isolation, and independent Retrieval/reference/backend behavior without prompt injection |
-| Navigation ablation | Source search vs optional graph backend | Whether AST/LSP graph navigation pays for itself on real repositories |
-
-The Decision and Debug comparisons are controlled project comparisons; they are not official upstream benchmark claims.
-
-## Run the harness
-
-Self-test without model calls:
+Use `n=1` while changing mechanisms or scorer contracts. Run the complete `n=3` matrices only after focused n=1 evidence supports release.
```powershell
pwsh -NoProfile -File benchmarks/run.ps1 -SelfTest
+pwsh -NoProfile -File benchmarks/run.ps1 -ProgressiveSelfTest
```
-Normal release-quality matrix:
+Current-only public matrix:
```powershell
-pwsh -NoProfile -File benchmarks/run.ps1 -Profile standard -Runs 3 -Workers 3 -RequireStableRanking
+python benchmarks/run_catalog.py --profile full --runs 1 --workers 3 `
+ --arm practical-current --arm practical-native `
+ --output benchmark-results/public-n1
```
-Complete public regression matrix:
+Current-only real-repository held-out:
```powershell
-pwsh -NoProfile -File benchmarks/run.ps1 -Profile full -Runs 3 -Workers 3 -RequireStableRanking
-```
-
-Focused examples:
-
-```powershell
-pwsh -NoProfile -File benchmarks/run.ps1 -Profile standard -Suite router -Runs 3 -RequireStableRanking
-pwsh -NoProfile -File benchmarks/run.ps1 -Profile standard -Suite debug -Runs 3 -RequireStableRanking
-pwsh -NoProfile -File benchmarks/run.ps1 -Profile smoke -Suite router -Case direct-artifact -Arm practical-current
-```
-
-Candidate before/after gate:
-
-```powershell
-pwsh -NoProfile -File benchmarks/run.ps1 `
- -Profile full `
- -Runs 3 `
- -Workers 3 `
- -BaselineRef `
- -IncludeBaseline `
- -RequireStableRanking
-```
-
-## Profiles
-
-| Profile | Delivery | Router | Decision | Debug | Native behavior | Default runs |
-|---|---:|---:|---:|---:|---:|---:|
-| `smoke` | 3 | 4 | 1 | 1 | 3 | 1 |
-| `standard` | 9 | 38 | 6 | 10 | 18 | 3 |
-| `full` | 18 | 38 | 10 | 14 | 18 | 3 |
-
-`standard` is the normal public release gate. `full` carries the broader complete public regression matrix. Router cases cover all four reasoning outputs (`NONE` plus three reasoning routes), all four Retrieval modes, and cross-products such as `NONE+STRUCTURAL` and `IMPLEMENTATION+STRUCTURAL`. Native behavior repeats Direct/Implementation boundaries without injected Skill text, adds Decision/Debug precedence cases, and scores Navigation/backend use independently from reasoning-reference selection. The expanded Debug set covers fourteen cases across parsing, normalization, tenant isolation, pagination, units, row handling, state invariants, TTL semantics, URL handling, and the upstream transfer/amount tasks. Decision grows from six to ten two-turn decisions in `full`. A stable published ranking requires at least three determinate repetitions per selected case/arm.
-
-## Acceptance order
-
-Interpret results in this order:
-
-1. correctness and safety;
-2. build/reachability;
-3. only then LOC, tokens, model time, and tool calls.
-
-A cheap failure cannot beat a correct safe result. The scorecard first applies a conservative quality gate and only computes relative efficiency after that gate is satisfied. See [`../docs/evaluations/2026-08-26-quality-gated-scorecard.md`](../docs/evaluations/2026-08-26-quality-gated-scorecard.md).
-
-The public task catalog is a **regression corpus**, not a private generalization set. Once a case has influenced Skill wording, its future score is evidence against regression, not independent evidence that the same behavior generalizes everywhere.
-
-## Why the router matters
-
-Practical Coding's main architectural claim cannot be established by comparing only against Ponytail or only against Superpowers. The project also measures whether the integration policy itself behaves as intended:
-
-- ordinary work selects no reasoning route;
-- an unresolved bug loads Debugging, not every engineering module;
-- a material choice loads Decision;
-- risky boundaries load Implementation;
-- Retrieval is classified independently as none, targeted, bounded, or structural;
-- broad structural exploration may load Navigation without becoming a fourth reasoning route;
-- only the selected reasoning reference is read;
-- workers are not a mandatory stage.
-
-The next validation cycle also includes a **Ponytail + Superpowers combined-install arm**. That test is required before claiming that Practical is experimentally superior to installing both together. See [`NEXT_VALIDATION.md`](NEXT_VALIDATION.md).
-
-## Output artifacts
-
-Normal run artifacts are written under ignored `benchmark-results/` directories and include:
-
-```text
-manifest.json
-results.json
-summary.json
-comparisons.json
-rollups.json
-rollup-comparisons.json
-scorecards.json
-report.md
-cells/
+python benchmarks/progressive_validation.py --phase all --current-only --runs 1 --workers 3 `
+ --output benchmark-results/heldout-n1
```
-The committed release directory contains only compact aggregates suitable for public review. Raw transcripts/workspaces stay local because they are large and may contain machine-specific paths.
-
-## Suites and scoring
-
-- `delivery`: Ponytail's published agentic tasks and deterministic scorer. For frontend template cases, the runner installs the pinned lockfile dependencies before the agent starts, so the agent and the runner-owned production build use the same executable type/build environment. Reports correctness, safety, production LOC, test LOC, files, tokens, time, tool calls, setup time, and optional frontend build result.
-- `router`: exact two-field classification. `REASONING` is `NONE`, `DECISION`, `DEBUGGING`, or `IMPLEMENTATION`; only the latter three are reasoning routes. `RETRIEVAL` is independently `NONE`, `TARGETED`, `BOUNDED`, or `STRUCTURAL`. A cell passes only when both fields match. The former Verification route remains folded into Implementation, while former Exploration cases now expect `REASONING=NONE; RETRIEVAL=STRUCTURAL`; results are therefore not comparable with the v1.1 five-way classifier.
-- `decision`: Practical versus Matt Pocock `grilling`. Uses a real resumed second turn and gates on frontier questions, one recommendation per question, no premature implementation, and convergence after scripted user decisions. Trade-off language is reported diagnostically but is not a declared grilling contract gate.
-- `debug`: shared-root-cause tasks scored on the repaired invariant and sibling callers. Tests/TDD process receives no bonus. Each Practical-owned Debug seed is required to fail its deterministic scorer, and a separate oracle fixture must pass it before the case is accepted into the catalog.
-- `behavior`: installs Practical Coding into an isolated native `CODEX_HOME`, does not inject its text into the prompt, and mechanically inspects command traces for `SKILL.md` discovery. Reasoning references are scored separately from Retrieval: Direct reads no reasoning reference; Decision, Debugging, and Implementation read only their expected reasoning reference; structural Retrieval may read `navigation.md` and is separately checked against the expected source/graph backend. Infrastructure, timeout, transcript-capture, missing-runtime, and build OOM failures are reported as `indeterminate`, not Skill failures. Comparisons omit pairs containing indeterminate cells instead of rewarding the unaffected arm.
-
-`total_tokens` includes cached input because that is how Codex reports turn input. The report therefore also separates cached input, uncached input, output, and reasoning tokens. `duration_seconds` is per-cell process duration; suite elapsed time is recorded separately and is not obtained by summing concurrent cell durations.
-
-## Acceptance
-
-Use repeated paired results. A candidate is not accepted merely because its prose matches a Skill contract. Require no correctness/build regression, then compare delivered code and behavior. Treat LOC, tokens, and time as secondary within equally correct artifacts. `n=1` is a smoke result, not a stable ranking.
-
-The machine-readable scorecard makes that order explicit. A comparison first has to stay within a 3 percentage-point suite pass-rate non-inferiority margin, with no lower suite build/safety rate and no case-level safety regression. Cost cannot rescue a failed quality gate. For a quality-qualified comparison, relative efficiency is the weighted geometric mean
-
-`E = exp(sum(w_i * ln(cost_comparator_i / cost_practical_i)))`
+Change `--runs 1` to `--runs 3` only for the frozen final candidate.
-over uncached input tokens (0.35), output tokens (0.15), model time (0.35), and tool calls (0.15), renormalized when a metric is unavailable. The diagnostic utility is `U = ((Q_practical + 0.01) / (Q_comparator + 0.01))^2 * E`. It is a relative sensitivity summary, not an absolute leaderboard score. The report also preserves the Pareto result, so users can see when one arm dominates and when the result is a real quality/cost trade-off. `qualified` additionally requires at least three determinate repetitions in every paired case; otherwise a quality-passing score remains `provisional`.
+## Interpretation
-A published internal stable ranking must pass `benchmarks/check_stability.py` with the default minimum `n=3`. The gate checks distinct repetition IDs, complete-run metadata, and infrastructure errors. Behavioral or build failures remain valid benchmark observations and therefore do not invalidate the sample by themselves.
+- Delivery and Debug grade delivered behavior, safety, and build evidence.
+- Decision grades compact two-turn convergence.
+- Router grades reasoning selection and retrieval separately.
+- Native Behavior verifies actual Skill discovery and module isolation.
+- Held-out tasks use frozen commits from three real repositories and mechanically grade evidence coverage, executable probes, clean workspaces, event/retrieval traces, and spontaneous requirements interviewing.
-The public catalog is a **regression suite**, not a hidden generalization test. Once a case has influenced Skill wording, its future 100% score should be treated as a ceiling check. A private held-out set is still required for the strongest generalization claims.
+Historical reports are version-specific. Offline comparison with v1.2 is non-paired unless old and new arms are rerun together in one frozen matrix.
diff --git a/benchmarks/case_catalog.py b/benchmarks/case_catalog.py
index a50c306..6cf8265 100644
--- a/benchmarks/case_catalog.py
+++ b/benchmarks/case_catalog.py
@@ -95,12 +95,12 @@
),
"implementation-sqlite-transaction-unknown": (
"IMPLEMENTATION",
- "TARGETED",
+ "STRUCTURAL",
"Implement the body of save_history() with sqlite3. The function and dependency are fixed, but atomicity, concurrent callers, restart behavior, and the authoritative transaction boundary are unknown.",
),
"implementation-one-line-irreversible": (
"IMPLEMENTATION",
- "TARGETED",
+ "STRUCTURAL",
"Flip purge_after_sync from false to true in one configuration file. The flag permanently deletes local records, and retry, rollback, idempotency, and ownership of that guarantee are not mapped.",
),
"implementation-security-local-boundary": (
@@ -419,8 +419,8 @@ def contained_or_rejected(call: Any) -> bool:
if score == "ttl-zero":
module = _load_module(workspace, "ttl.py", "debug_ttl_zero")
correct = module.session_ttl({"SESSION_TTL": "0"}) == 0
- safe = module.cache_ttl({"CACHE_TTL": "0"}) == 0 and module.cache_ttl({}) == 60
- return _result(correct, safe, "explicit zero survives shared TTL parsing")
+ safe = module.cache_ttl({"CACHE_TTL": "0"}) == 60 and module.cache_ttl({}) == 60
+ return _result(correct, safe, "session zero is preserved without changing sibling cache semantics")
if score == "null-sort":
module = _load_module(workspace, "sorting.py", "debug_null_sort")
diff --git a/benchmarks/debug_oracles.py b/benchmarks/debug_oracles.py
index 376afb0..93e2d63 100644
--- a/benchmarks/debug_oracles.py
+++ b/benchmarks/debug_oracles.py
@@ -26,7 +26,7 @@
"inventory.py": """def _debit(stock, sku, quantity):\n available = stock.get(sku, 0)\n if quantity > available:\n return False\n stock[sku] = available - quantity\n return True\n\ndef reserve(stock, sku, quantity):\n return _debit(stock, sku, quantity)\n\ndef consume(stock, sku, quantity):\n return _debit(stock, sku, quantity)\n""",
},
"trace-ttl-zero": {
- "ttl.py": """def parse_ttl(value, default):\n if value is None:\n return default\n return int(value)\n\ndef session_ttl(env):\n return parse_ttl(env.get(\"SESSION_TTL\"), 300)\n\ndef cache_ttl(env):\n return parse_ttl(env.get(\"CACHE_TTL\"), 60)\n""",
+ "ttl.py": """def parse_ttl(value, default):\n if value is None:\n return default\n return int(value) or default\n\ndef session_ttl(env):\n value = env.get(\"SESSION_TTL\")\n if value is None:\n return 300\n return int(value)\n\ndef cache_ttl(env):\n return parse_ttl(env.get(\"CACHE_TTL\"), 60)\n""",
},
"trace-null-sort": {
"sorting.py": """def name_key(value):\n return (value is None, \"\" if value is None else value.lower())\n\ndef sort_products(rows):\n return sorted(rows, key=lambda row: name_key(row.get(\"name\")))\n\ndef sort_customers(rows):\n return sorted(rows, key=lambda row: name_key(row.get(\"name\")))\n""",
diff --git a/benchmarks/ladder_analysis.py b/benchmarks/ladder_analysis.py
new file mode 100644
index 0000000..749ff8b
--- /dev/null
+++ b/benchmarks/ladder_analysis.py
@@ -0,0 +1,220 @@
+#!/usr/bin/env python3
+"""Analyze aggregated progressive-depth calibration observations.
+
+Input is JSONL with one aggregated row per task/axis/arm/level. Optional
+adaptive routing fields are summarized to support capability-tree tuning.
+See benchmarks/LADDER_EVOLUTION.md.
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+from collections import Counter, defaultdict
+from pathlib import Path
+from statistics import mean
+from typing import Any, Iterable
+
+LEVELS = {
+ "execution": ["E0", "E1", "E2", "E3"],
+ "retrieval": ["R0", "R1", "R2", "R3"],
+}
+COST_FIELDS = ("tokens", "duration_seconds", "tool_calls")
+
+
+def _rank(axis: str, level: str) -> int:
+ try:
+ return LEVELS[axis].index(level)
+ except (KeyError, ValueError) as exc:
+ raise ValueError(f"invalid level {level!r} for axis {axis!r}") from exc
+
+
+def _normalize_path(value: Any) -> tuple[str, ...] | None:
+ if value is None:
+ return None
+ if isinstance(value, str):
+ parts = [part.strip() for part in value.replace("/", ">").split(">") if part.strip()]
+ elif isinstance(value, list) and all(isinstance(part, str) for part in value):
+ parts = [part.strip() for part in value if part.strip()]
+ else:
+ raise ValueError("capability_path must be a string, string list, or null")
+ return tuple(parts) if parts else None
+
+
+def validate_record(record: dict[str, Any]) -> None:
+ required = {"task_id", "axis", "arm", "level", "qualified"}
+ missing = sorted(required - record.keys())
+ if missing:
+ raise ValueError(f"missing fields: {', '.join(missing)}")
+ axis = record["axis"]
+ if axis not in LEVELS:
+ raise ValueError(f"invalid axis: {axis!r}")
+ _rank(axis, record["level"])
+ if record["arm"] not in {"cap", "adaptive"}:
+ raise ValueError(f"invalid arm: {record['arm']!r}")
+ if not isinstance(record["qualified"], bool):
+ raise ValueError("qualified must be boolean")
+ _normalize_path(record.get("capability_path"))
+ refs = record.get("references_loaded")
+ if refs is not None and (not isinstance(refs, list) or not all(isinstance(item, str) for item in refs)):
+ raise ValueError("references_loaded must be a string list or null")
+
+
+def load_jsonl(path: Path) -> list[dict[str, Any]]:
+ records: list[dict[str, Any]] = []
+ with path.open("r", encoding="utf-8") as handle:
+ for line_number, raw in enumerate(handle, 1):
+ line = raw.strip()
+ if not line:
+ continue
+ try:
+ record = json.loads(line)
+ except json.JSONDecodeError as exc:
+ raise ValueError(f"{path}:{line_number}: invalid JSON: {exc}") from exc
+ try:
+ validate_record(record)
+ except ValueError as exc:
+ raise ValueError(f"{path}:{line_number}: {exc}") from exc
+ records.append(record)
+ return records
+
+
+def _average_costs(rows: Iterable[dict[str, Any]]) -> dict[str, float]:
+ rows = list(rows)
+ result: dict[str, float] = {}
+ for field in COST_FIELDS:
+ values = [float(row[field]) for row in rows if row.get(field) is not None]
+ if values:
+ result[field] = mean(values)
+ return result
+
+
+def analyze(records: Iterable[dict[str, Any]]) -> dict[str, Any]:
+ grouped: dict[tuple[str, str], list[dict[str, Any]]] = defaultdict(list)
+ all_records = list(records)
+ for record in all_records:
+ validate_record(record)
+ grouped[(str(record["task_id"]), str(record["axis"]))].append(record)
+
+ axis_reports: dict[str, Any] = {}
+
+ for axis, levels in LEVELS.items():
+ statuses: Counter[str] = Counter()
+ minimum_counts: Counter[str] = Counter()
+ adaptive_paths: Counter[str] = Counter()
+ qualified_adaptive_paths: Counter[str] = Counter()
+ reference_loads: Counter[str] = Counter()
+ cases: list[dict[str, Any]] = []
+
+ for (task_id, case_axis), rows in sorted(grouped.items()):
+ if case_axis != axis:
+ continue
+
+ cap_rows = [row for row in rows if row["arm"] == "cap"]
+ adaptive_rows = [row for row in rows if row["arm"] == "adaptive"]
+ passing_caps = sorted(
+ (row for row in cap_rows if row["qualified"]),
+ key=lambda row: _rank(axis, row["level"]),
+ )
+ minimum = passing_caps[0]["level"] if passing_caps else None
+ if minimum is not None:
+ minimum_counts[minimum] += 1
+
+ adaptive_path = None
+ refs: list[str] = []
+ if len(adaptive_rows) == 1:
+ path = _normalize_path(adaptive_rows[0].get("capability_path"))
+ if path:
+ adaptive_path = ">".join(path)
+ adaptive_paths[adaptive_path] += 1
+ if adaptive_rows[0]["qualified"]:
+ qualified_adaptive_paths[adaptive_path] += 1
+ refs = list(adaptive_rows[0].get("references_loaded") or [])
+ reference_loads.update(refs)
+
+ if len(adaptive_rows) != 1 or minimum is None:
+ status = "unscored"
+ adaptive_level = adaptive_rows[0]["level"] if len(adaptive_rows) == 1 else None
+ adaptive_qualified = adaptive_rows[0]["qualified"] if len(adaptive_rows) == 1 else None
+ else:
+ adaptive = adaptive_rows[0]
+ adaptive_level = adaptive["level"]
+ adaptive_qualified = adaptive["qualified"]
+ adaptive_rank = _rank(axis, adaptive_level)
+ minimum_rank = _rank(axis, minimum)
+
+ if adaptive_qualified and adaptive_rank == minimum_rank:
+ status = "exact"
+ elif adaptive_qualified and adaptive_rank > minimum_rank:
+ status = "over_escalation"
+ elif (not adaptive_qualified) and adaptive_rank < minimum_rank:
+ status = "under_escalation"
+ elif (not adaptive_qualified) and adaptive_rank >= minimum_rank:
+ status = "quality_failure"
+ else:
+ status = "inconsistent"
+
+ statuses[status] += 1
+ cases.append({
+ "task_id": task_id,
+ "minimum_sufficient": minimum,
+ "adaptive_level": adaptive_level,
+ "adaptive_qualified": adaptive_qualified,
+ "adaptive_capability_path": adaptive_path,
+ "references_loaded": refs,
+ "status": status,
+ })
+
+ scorable = sum(statuses[name] for name in ("exact", "over_escalation", "under_escalation", "quality_failure", "inconsistent"))
+ exact_or_over_under = statuses["exact"] + statuses["over_escalation"] + statuses["under_escalation"]
+ qualified_adaptive = statuses["exact"] + statuses["over_escalation"]
+
+ cost_by_level: dict[str, Any] = {}
+ for level in levels:
+ rows = [row for row in all_records if row["axis"] == axis and row["arm"] == "cap" and row["level"] == level and row["qualified"]]
+ if rows:
+ cost_by_level[level] = _average_costs(rows)
+
+ axis_reports[axis] = {
+ "tasks_seen": sum(1 for _, case_axis in grouped if case_axis == axis),
+ "scorable_tasks": scorable,
+ "status_counts": dict(sorted(statuses.items())),
+ "qualified_adaptive_rate": (qualified_adaptive / scorable) if scorable else None,
+ "overall_exact_rate": (statuses["exact"] / scorable) if scorable else None,
+ "quality_failure_rate": (statuses["quality_failure"] / scorable) if scorable else None,
+ "inconsistent_rate": (statuses["inconsistent"] / scorable) if scorable else None,
+ "over_escalation_rate": (statuses["over_escalation"] / exact_or_over_under) if exact_or_over_under else None,
+ "under_escalation_rate": (statuses["under_escalation"] / exact_or_over_under) if exact_or_over_under else None,
+ # Conditional routing rate retained for compatibility. It excludes
+ # quality failures and inconsistent rows; use overall_exact_rate
+ # for the end-to-end adaptive result.
+ "exact_rate": (statuses["exact"] / exact_or_over_under) if exact_or_over_under else None,
+ "minimum_sufficient_counts": {level: minimum_counts[level] for level in levels},
+ "levels_never_minimum": [level for level in levels if minimum_counts[level] == 0],
+ "qualified_cap_cost_by_level": cost_by_level,
+ "adaptive_capability_path_counts": dict(sorted(adaptive_paths.items())),
+ "qualified_adaptive_capability_path_counts": dict(sorted(qualified_adaptive_paths.items())),
+ "adaptive_reference_load_counts": dict(sorted(reference_loads.items())),
+ "cases": cases,
+ }
+
+ return {"axes": axis_reports}
+
+
+def main() -> int:
+ parser = argparse.ArgumentParser(description=__doc__)
+ parser.add_argument("observations", type=Path, help="aggregated JSONL observations")
+ parser.add_argument("--output", type=Path, help="optional JSON output path")
+ args = parser.parse_args()
+
+ report = analyze(load_jsonl(args.observations))
+ text = json.dumps(report, indent=2, sort_keys=True)
+ if args.output:
+ args.output.write_text(text + "\n", encoding="utf-8")
+ else:
+ print(text)
+ return 0
+
+
+if __name__ == "__main__":
+ raise SystemExit(main())
diff --git a/benchmarks/progressive_cases.py b/benchmarks/progressive_cases.py
new file mode 100644
index 0000000..3029eb9
--- /dev/null
+++ b/benchmarks/progressive_cases.py
@@ -0,0 +1,300 @@
+"""Frozen real-repository cases for adaptive runtime experiments.
+
+The cases are intentionally read-only. They exercise source retrieval, executable
+probes, debugging, implementation guarantees, and direct reporting without letting
+benchmark work mutate the source repositories.
+"""
+
+from __future__ import annotations
+
+
+REPOSITORIES = {
+ "personal-progress": {
+ "url": "https://github.com/Hubujiu/personal-progress.git",
+ "commit": "515c2e2193c3d547e04e65687da6666dc877ab61",
+ "local_name": "personal-progress",
+ },
+ "cover-atelier": {
+ "url": "https://github.com/Hubujiu/cover-atelier.git",
+ "commit": "fc3b12b3a944f45b5a1d19963e29307d95b120fb",
+ "local_name": "cover-atelier",
+ },
+ "super-agent": {
+ "url": "https://github.com/java-up-up/super-agent.git",
+ "commit": "d44edf063032a2d8797549411f11923aa4a83ec3",
+ "local_name": "super-agent",
+ },
+}
+
+
+def _case(
+ task_id: str,
+ repository: str,
+ family: str,
+ execution: str,
+ retrieval: str,
+ prompt: str,
+ required: list[list[str]],
+ *,
+ capability_path: list[str] | None = None,
+ probe_terms: list[list[str]] | None = None,
+ reasoning: str | None = None,
+ calibration: bool = False,
+) -> dict[str, object]:
+ # Active event routing is independent of the rejected capability tree.
+ # capability_path remains frozen experiment metadata only.
+ expected_reasoning = reasoning or "NONE"
+ retrieval_mode = {
+ "R0": "TARGETED",
+ "R1": "BOUNDED",
+ "R2": "STRUCTURAL",
+ "R3": "STRUCTURAL",
+ }[retrieval]
+ return {
+ "task_id": task_id,
+ "repository": repository,
+ "family": family,
+ "expected_execution": execution,
+ "expected_retrieval": retrieval,
+ "capability_path": capability_path or [],
+ "expected_reasoning": expected_reasoning,
+ "expected_retrieval_mode": retrieval_mode,
+ "prompt": prompt,
+ "required": required,
+ "probe_terms": probe_terms or [],
+ "calibration": calibration,
+ }
+
+
+CASES = [
+ _case(
+ "pp-database-names-target",
+ "personal-progress",
+ "known-target",
+ "E0",
+ "R0",
+ "Read progress-core/src/main/java/com/hubujiu/progress/core/database/PluginDatabaseNames.java only. Report the schema/role naming invariants and why long plugin IDs cannot collide. Do not edit files.",
+ [["63", "max_identifier_bytes"], ["sha-256", "digest"], ["plugin_", "plugin_role_"]],
+ calibration=True,
+ ),
+ _case(
+ "pp-management-controller-local",
+ "personal-progress",
+ "local-retrieval",
+ "E0",
+ "R1",
+ "Locate the platform controller that owns plugin management operations and identify the lifecycle service it delegates to plus the focused API test. Report paths and symbols; do not edit files.",
+ [["PluginManagementController"], ["PluginLifecycleService"], ["PlatformManagementApiTest"]],
+ calibration=True,
+ ),
+ _case(
+ "pp-lifecycle-structural",
+ "personal-progress",
+ "structural-retrieval",
+ "E0",
+ "R2",
+ "Trace the install/start/stop request path from the platform API into runtime lifecycle execution. Identify the controller, lifecycle service, operation executor, and state machine, with source paths. Report only; do not edit files.",
+ [["PluginManagementController"], ["PluginLifecycleService"], ["PluginOperationExecutor", "DefaultPluginOperationExecutor"], ["PluginStateMachine"]],
+ calibration=True,
+ ),
+ _case(
+ "pp-state-machine-probe",
+ "personal-progress",
+ "one-probe",
+ "E1",
+ "R0",
+ "Run the focused PluginStateMachineTest once to establish the current transition behavior, then report the command and whether the test passed. Do not edit files or run the full build.",
+ [["PluginStateMachineTest"], ["pass", "success", "tests run"]],
+ probe_terms=[["pluginstatemachinetest"], ["mvn", "mvnw"]],
+ calibration=True,
+ ),
+ _case(
+ "pp-operation-failure-diagnosis",
+ "personal-progress",
+ "unexplained-failure",
+ "E2",
+ "R1",
+ "An operation sometimes remains RUNNING after its worker throws. The cause is not established. Inspect the operation executor and its focused tests, identify the earliest incorrect state transition and the cheapest falsifying test. Diagnose only; do not propose unrelated redesigns or edit files.",
+ [["DefaultPluginOperationExecutor"], ["PluginOperationExecutorTest"], ["RUNNING"], ["fail", "exception", "complete"]],
+ capability_path=["diagnosis"],
+ reasoning="DEBUGGING",
+ calibration=True,
+ ),
+ _case(
+ "pp-admin-token-security",
+ "personal-progress",
+ "security-guarantee",
+ "E3",
+ "R2",
+ "Plan a zero-downtime rotation of the bootstrap admin token. Map the authoritative filter/configuration boundary, all protected platform entry points, rejection-before-side-effect behavior, and the focused evidence needed. Report a bounded change plan only; do not edit files.",
+ [["BootstrapAdminTokenFilter"], ["PlatformSecurityConfiguration"], ["401", "unauthorized", "reject"], ["PlatformManagementApiTest", "PluginDispatchApiTest"]],
+ capability_path=["engineering", "security"],
+ reasoning="IMPLEMENTATION",
+ calibration=True,
+ ),
+ _case(
+ "pp-removal-state",
+ "personal-progress",
+ "state-guarantee",
+ "E3",
+ "R2",
+ "Review the uninstall/removal flow for retry, idempotency, ordering, and partial-failure guarantees. Trace lifecycle, operation execution, database provisioning, and the removal integration tests. Report the authoritative state boundary and missing evidence; do not edit files.",
+ [["PluginLifecycleService"], ["PluginOperationExecutor"], ["PluginDatabaseProvisioner"], ["PluginRemovalIT"], ["idempot", "retry", "ordering"]],
+ capability_path=["engineering", "state"],
+ ),
+ _case(
+ "pp-dispatch-compatibility",
+ "personal-progress",
+ "compatibility-guarantee",
+ "E3",
+ "R2",
+ "A public plugin dispatch response must add a required field while old plugins and clients coexist for one release. Map the HTTP contract, dispatcher, response/view types, and contract tests; propose a compatibility window and rollback evidence. Report only; do not edit files.",
+ [["PluginDispatchController"], ["PluginDispatcher"], ["PluginViews", "RegisteredEndpoint"], ["PluginDispatchApiTest", "PluginHttpContractTest"], ["compat", "one release", "rollback"]],
+ capability_path=["engineering", "compatibility"],
+ reasoning="DECISION",
+ ),
+ _case(
+ "ca-export-format-target",
+ "cover-atelier",
+ "known-target",
+ "E0",
+ "R0",
+ "Read src/lib/exportFormat.ts only and report each export format's MIME type, extension, and explicit quality value when present. Do not edit files.",
+ [["image/jpeg", "jpeg"], ["image/webp", "webp"], ["image/avif", "avif"], ["image/png", "png"]],
+ ),
+ _case(
+ "ca-filename-local",
+ "cover-atelier",
+ "local-retrieval",
+ "E0",
+ "R1",
+ "Locate the export filename policy and its focused tests. Report how invalid characters, trailing dots/spaces, empty titles, and maximum length are handled. Do not edit files.",
+ [["getExportFilename"], ["exportFilename.test.ts"], ["cover-atelier"], ["120"]],
+ ),
+ _case(
+ "ca-avif-structural",
+ "cover-atelier",
+ "structural-retrieval",
+ "E0",
+ "R2",
+ "Trace the AVIF export flow from EditorShell through exportCover into the encoder and worker, including progress/cancellation handling and focused tests. Report paths and symbols only; do not edit files.",
+ [["EditorShell"], ["exportCover"], ["encodeAvif", "avifEncoder"], ["avifEncoder.worker.ts"], ["Abort", "signal", "cancel"]],
+ ),
+ _case(
+ "ca-filename-probe",
+ "cover-atelier",
+ "one-probe",
+ "E1",
+ "R0",
+ "Run the focused exportFilename test once to establish the current filename contract, then report the exact command and outcome. Do not edit files or run the full test suite.",
+ [["exportFilename"], ["pass", "passed", "tests"]],
+ probe_terms=[["exportfilename"], ["npm", "vitest"]],
+ ),
+ _case(
+ "ca-export-failure-diagnosis",
+ "cover-atelier",
+ "unexplained-failure",
+ "E2",
+ "R1",
+ "Users report that cancelling an export sometimes still downloads a file. The cause is not established. Inspect the cancellation path and focused tests, identify the earliest observable boundary to probe, and name the single cheapest falsifying test. Diagnose only; do not edit files.",
+ [["AbortController", "AbortSignal", "signal"], ["EditorShell"], ["exportCover"], ["exportProgress.test.ts", "ExportProgressModal.test.tsx"], ["probe", "test"]],
+ capability_path=["diagnosis"],
+ reasoning="DEBUGGING",
+ ),
+ _case(
+ "ca-avif-performance",
+ "cover-atelier",
+ "performance-diagnosis",
+ "E3",
+ "R2",
+ "Large AVIF exports are reported to stall the UI, but no timing evidence exists. Map the main-thread/worker boundary and propose a bounded measurement that separates encode latency, progress delivery, memory pressure, and cancellation. Diagnose and report only; do not edit files.",
+ [["avifEncoder.worker.ts"], ["encodeAvif"], ["performance", "duration", "latency", "measure"], ["memory"], ["cancel", "Abort"]],
+ capability_path=["diagnosis", "performance"],
+ reasoning="DEBUGGING",
+ calibration=True,
+ ),
+ _case(
+ "ca-progress-interface",
+ "cover-atelier",
+ "interface-guarantee",
+ "E3",
+ "R1",
+ "Plan keyboard and screen-reader acceptance for ExportProgressModal without changing its visual direction. Inspect the component and tests; specify focus, labeling, cancel action, progress announcement, and the smallest browser/component evidence. Report only; do not edit files.",
+ [["ExportProgressModal"], ["focus", "keyboard"], ["aria", "screen reader", "label"], ["cancel"], ["progress"]],
+ capability_path=["engineering", "interface"],
+ reasoning="NONE",
+ ),
+ _case(
+ "ca-export-quality",
+ "cover-atelier",
+ "quality-guarantee",
+ "E3",
+ "R2",
+ "Review the export pipeline for duplicated policy or structural coupling that blocks safe extension with another image format. Map format config, filename, export orchestration, encoder boundary, and tests; recommend the smallest coherent refactor with reachability evidence. Report only; do not edit files.",
+ [["exportFormat"], ["exportFilename"], ["exportCover"], ["avifEncoder"], ["test"]],
+ capability_path=["engineering", "quality"],
+ reasoning="DECISION",
+ ),
+ _case(
+ "sa-page-util-target",
+ "super-agent",
+ "known-target",
+ "E0",
+ "R0",
+ "Read super-agent-common/super-agent-common-web/src/main/java/org/javaup/database/page/PageUtil.java only. Report the two supported page abstractions and how records are transformed. Do not edit files.",
+ [["PageInfo"], ["IPage"], ["PageVo"], ["Function", "map"]],
+ ),
+ _case(
+ "sa-memory-structural",
+ "super-agent",
+ "structural-retrieval",
+ "E0",
+ "R2",
+ "Trace the memory comparison HTTP path from MemoryDemoController through MemoryComparisonService to the no-memory, sliding-window, and summary-compression implementations. Report paths and symbols only; do not edit files.",
+ [["MemoryDemoController"], ["MemoryComparisonService"], ["NoMemoryChatService"], ["SlidingWindowMemoryChatService"], ["SummaryCompressionMemoryChatService"]],
+ ),
+ _case(
+ "sa-memory-exhaustive",
+ "super-agent",
+ "bounded-exhaustive",
+ "E0",
+ "R3",
+ "Within ai-example/ai-example-memory/ai-example-spring-ai-memory, exhaustively list every production Java class that directly uses MemoryPromptSupport and state which helper methods each uses. Include the bounded search scope and coverage evidence. Report only; do not edit files.",
+ [["NoMemoryChatService"], ["SlidingWindowMemoryChatService"], ["SummaryCompressionMemoryChatService"], ["estimateTokens"], ["normalizeSessionId", "extractText", "toViews", "toTranscript"]],
+ calibration=True,
+ ),
+ _case(
+ "sa-memory-module-probe",
+ "super-agent",
+ "one-probe",
+ "E1",
+ "R0",
+ "Compile the ai-example-spring-ai-memory module once with its required reactor dependencies to establish current reachability. Report the exact Maven command and outcome; do not edit files or run unrelated modules.",
+ [["ai-example-spring-ai-memory"], ["build success", "success", "compiled"]],
+ probe_terms=[["mvn", "mvnw"], ["ai-example-spring-ai-memory"], ["-pl"]],
+ ),
+ _case(
+ "sa-sensitive-security",
+ "super-agent",
+ "security-guarantee",
+ "E3",
+ "R2",
+ "Review where sensitive-word rejection occurs in the Spring AI Alibaba request path. Map interceptor registration and callers, define rejection-before-model-side-effect behavior, and identify the narrowest security tests needed. Report only; do not edit files.",
+ [["SensitiveWordInterceptor"], ["SpringAiAlibabaAgentService"], ["reject", "before"], ["test"]],
+ capability_path=["engineering", "security"],
+ ),
+ _case(
+ "sa-memory-state",
+ "super-agent",
+ "state-guarantee",
+ "E3",
+ "R2",
+ "Review ResettableMemorySaver and its use by SpringAiAlibabaAgentService for concurrent sessions, reset ordering, and restart semantics. Identify the authoritative state owner and the smallest concurrency evidence. Report only; do not edit files.",
+ [["ResettableMemorySaver"], ["SpringAiAlibabaAgentService"], ["concurrent", "thread"], ["reset", "clear"], ["restart", "durable", "memory"]],
+ capability_path=["engineering", "state"],
+ ),
+]
+
+
+CALIBRATION_IDS = {case["task_id"] for case in CASES if case["calibration"]}
+ABLATION_IDS = {case["task_id"] for case in CASES if len(case["capability_path"]) == 2}
diff --git a/benchmarks/progressive_validation.py b/benchmarks/progressive_validation.py
new file mode 100644
index 0000000..066086a
--- /dev/null
+++ b/benchmarks/progressive_validation.py
@@ -0,0 +1,405 @@
+#!/usr/bin/env python3
+"""Run frozen real-repository held-out tasks against the active event router."""
+
+from __future__ import annotations
+
+import argparse
+import concurrent.futures
+import datetime as dt
+import hashlib
+import json
+import os
+import re
+import shutil
+import statistics
+import sys
+import threading
+import time
+from pathlib import Path
+from typing import Any
+
+HERE = Path(__file__).resolve().parent
+ROOT = HERE.parent
+if str(HERE) not in sys.path:
+ sys.path.insert(0, str(HERE))
+
+import run_benchmarks as bench
+from progressive_cases import CASES, REPOSITORIES
+
+
+VERSION = "2.0"
+MODEL = bench.MODEL
+REASONING = bench.REASONING
+TRACE_RE = re.compile(
+ r"BENCHMARK_TRACE\s+reasoning=(NONE|DEBUGGING|DECISION|IMPLEMENTATION)\s+"
+ r"retrieval=(NONE|TARGETED|BOUNDED|STRUCTURAL)\s+refs=([^\r\n]+)",
+ re.I,
+)
+
+
+def sha256(path: Path) -> str:
+ digest = hashlib.sha256()
+ digest.update(path.read_bytes())
+ return digest.hexdigest()
+
+
+def parse_trace(answer: str) -> dict[str, Any]:
+ matches = list(TRACE_RE.finditer(answer))
+ if not matches:
+ return {"reasoning": None, "retrieval": None, "references_loaded": []}
+ match = matches[-1]
+ refs_raw = match.group(3).strip().strip("<>")
+ refs = [] if refs_raw.lower() == "none" else [part.strip().strip("<>") for part in refs_raw.split(",") if part.strip()]
+ return {
+ "reasoning": match.group(1).upper(),
+ "retrieval": match.group(2).upper(),
+ "references_loaded": refs,
+ }
+
+
+def validate_trace(trace: dict[str, Any]) -> bool:
+ return (
+ trace["reasoning"] in {"NONE", "DEBUGGING", "DECISION", "IMPLEMENTATION"}
+ and trace["retrieval"] in {"NONE", "TARGETED", "BOUNDED", "STRUCTURAL"}
+ )
+
+
+def score_answer(case: dict[str, Any], answer: str, commands: list[str], workspace: Path) -> dict[str, Any]:
+ lower = answer.lower()
+ missing = [group for group in case["required"] if not any(term.lower() in lower for term in group)]
+ command_text = "\n".join(commands).lower()
+ probe_groups = [group if isinstance(group, list) else [group] for group in case["probe_terms"]]
+ probe_missing = [group for group in probe_groups if not any(term.lower() in command_text for term in group)]
+ status = bench.run_command(["git", "status", "--porcelain"], workspace)
+ clean = status.returncode == 0 and not status.stdout.strip()
+ manual_markers = (
+ "references/manual/clarification.md",
+ "requirements interview",
+ "grill me",
+ )
+ spontaneous_manual = any(marker in lower for marker in manual_markers)
+ passed = not missing and not probe_missing and clean
+ return {
+ "passed": passed,
+ "missing_evidence_groups": missing,
+ "missing_probe_terms": probe_missing,
+ "workspace_clean": clean,
+ "spontaneous_manual_mode": spontaneous_manual,
+ }
+
+
+def resolve_repositories(repository_root: Path, overrides: list[str]) -> dict[str, Path]:
+ mapped = {name: (repository_root / data["local_name"]).resolve() for name, data in REPOSITORIES.items()}
+ for raw in overrides:
+ if "=" not in raw:
+ raise ValueError(f"repository override must be NAME=PATH: {raw}")
+ name, value = raw.split("=", 1)
+ if name not in REPOSITORIES:
+ raise ValueError(f"unknown repository override: {name}")
+ mapped[name] = Path(value).resolve()
+ for name, path in mapped.items():
+ commit = REPOSITORIES[name]["commit"]
+ if not path.is_dir():
+ raise FileNotFoundError(f"held-out repository unavailable: {name}: {path}")
+ check = bench.run_command(["git", "cat-file", "-e", f"{commit}^{{commit}}"], path)
+ if check.returncode:
+ raise RuntimeError(f"{name} does not contain frozen commit {commit}: {check.stderr}")
+ return mapped
+
+
+def prepare_workspace(source: Path, commit: str, workspace: Path) -> None:
+ clone = bench.run_command(["git", "clone", "-q", "--shared", "--no-checkout", str(source), str(workspace)], workspace.parent)
+ if clone.returncode:
+ raise RuntimeError(clone.stderr)
+ configure = bench.run_command(["git", "config", "core.longpaths", "true"], workspace)
+ if configure.returncode:
+ raise RuntimeError(configure.stderr)
+ checkout = bench.run_command(["git", "checkout", "-q", "--detach", commit], workspace)
+ if checkout.returncode:
+ raise RuntimeError(checkout.stderr)
+
+
+def task_prompt(case: dict[str, Any], loaded: str, variant: str) -> str:
+ trace = (
+ "After the evidence-backed report, append exactly one final line using this benchmark-only instrumentation: "
+ "BENCHMARK_TRACE reasoning=NONE retrieval=TARGETED refs=none. "
+ "Replace reasoning with NONE, DEBUGGING, DECISION, or IMPLEMENTATION and retrieval with NONE, TARGETED, "
+ "BOUNDED, or STRUCTURAL according to the behavior actually used. NONE means no repository source acquisition; "
+ "TARGETED means only already-known paths or symbols; BOUNDED means lexical/filename/symbol search in a limited "
+ "scope; STRUCTURAL means relationship/flow mapping or bounded exhaustive discovery, even when reconstructed "
+ "from source because no graph index is available. List comma-separated Practical Coding "
+ "reference paths after refs=. The reasoning value must be one of the four listed values; STRUCTURAL is only a "
+ "retrieval value. Requirements interviewing is never a reasoning route. "
+ "Do not mention this instrumentation elsewhere."
+ )
+ return (
+ f"Frozen held-out task {case['task_id']} ({case['family']}).\n\n{case['prompt']}\n\n"
+ "Use PowerShell-compatible commands. Stay within this repository and preserve a clean working tree. "
+ "Cite concrete source paths/symbols and fresh command evidence.\n\n"
+ f"{trace}\n\n{variant}\n{loaded}"
+ )
+
+
+def build_specs(phases: list[str], runs: int, *, current_only: bool = False) -> list[tuple[str, str, str, int]]:
+ specs: list[tuple[str, str, str, int]] = []
+ selected = set(phases)
+ if "all" in selected:
+ selected = {"heldout"}
+ if "heldout" in selected:
+ for case in CASES:
+ variants = ("adaptive",) if current_only else ("no-skill", "previous", "adaptive")
+ for variant in variants:
+ for repetition in range(1, runs + 1):
+ specs.append(("heldout", case["task_id"], variant, repetition))
+ return specs
+
+
+def run_cell(
+ spec: tuple[str, str, str, int],
+ args: argparse.Namespace,
+ repositories: dict[str, Path],
+ previous: Path | None,
+ eval_home: Path,
+ output: Path,
+) -> dict[str, Any]:
+ phase, task_id, variant, repetition = spec
+ case = next(item for item in CASES if item["task_id"] == task_id)
+ cell = output / "cells" / phase / task_id / variant / f"r{repetition:03d}"
+ result_path = cell / "result.json"
+ if result_path.is_file():
+ return json.loads(result_path.read_text(encoding="utf-8"))
+ cell.mkdir(parents=True, exist_ok=True)
+ workspace = cell / "workspace"
+ if workspace.exists():
+ shutil.rmtree(workspace)
+ prepare_workspace(repositories[case["repository"]], REPOSITORIES[case["repository"]]["commit"], workspace)
+
+ if phase == "heldout":
+ if variant == "no-skill":
+ loaded = ""
+ elif variant == "previous":
+ if previous is None:
+ raise RuntimeError("previous Skill is unavailable")
+ loaded = bench.skill_text("practical-previous", {}, previous)
+ else:
+ loaded = bench.skill_text("practical-current", {}, None)
+ else:
+ raise ValueError(phase)
+
+ prompt = task_prompt(case, loaded, variant)
+ (cell / "prompt.txt").write_text(prompt, encoding="utf-8")
+ env = os.environ.copy()
+ env["CODEX_HOME"] = str(eval_home)
+ codex = bench.resolve_codex(args.codex)
+ stdout = cell / "round1.jsonl"
+ stderr = cell / "round1.stderr.txt"
+ code, timed_out, forced, duration = bench.run_codex(
+ bench.codex_command(codex, workspace), prompt, workspace, env, stdout, stderr, args.timeout
+ )
+ parsed = bench.parse_transcript(stdout)
+ trace = parse_trace(parsed["answer"])
+ trace_valid = validate_trace(trace)
+ record: dict[str, Any] = {
+ "phase": phase,
+ "task_id": task_id,
+ "repository": case["repository"],
+ "family": case["family"],
+ "variant": variant,
+ "repetition": repetition,
+ "expected_reasoning": case["expected_reasoning"],
+ "expected_retrieval": case["expected_retrieval_mode"],
+ "exit_status": code,
+ "timed_out": timed_out,
+ "forced_after_completion": forced,
+ "duration_seconds": duration,
+ "tool_calls": parsed["tool_calls"],
+ **parsed["usage"],
+ "answer": parsed["answer"],
+ "tool_commands": parsed["tool_commands"],
+ "selected_reasoning": trace["reasoning"],
+ "selected_retrieval": trace["retrieval"],
+ "references_loaded": trace["references_loaded"],
+ "routing_trace_valid": trace_valid,
+ "routing_exact": (
+ trace_valid
+ and trace["reasoning"] == case["expected_reasoning"]
+ and trace["retrieval"] == case["expected_retrieval_mode"]
+ ),
+ }
+ infrastructure_error = "timeout" if timed_out else (f"codex exit status {code}" if code and not forced else None)
+ if infrastructure_error:
+ record.update({"passed": None, "verdict": "indeterminate", "error": infrastructure_error})
+ else:
+ record.update(score_answer(case, parsed["answer"], parsed["tool_commands"], workspace))
+ record["verdict"] = "pass" if record["passed"] else "fail"
+ (cell / "answer.md").write_text(parsed["answer"] + "\n", encoding="utf-8")
+ result_path.write_text(json.dumps(record, ensure_ascii=False, indent=2) + "\n", encoding="utf-8")
+ return record
+
+
+def _mean(records: list[dict[str, Any]], key: str) -> float | None:
+ values = [float(record[key]) for record in records if record.get(key) is not None]
+ return statistics.mean(values) if values else None
+
+
+def heldout_report(records: list[dict[str, Any]], runs: int) -> dict[str, Any]:
+ rows = [record for record in records if record["phase"] == "heldout"]
+ arms: dict[str, Any] = {}
+ for variant in sorted({record["variant"] for record in rows}):
+ selected = [record for record in rows if record["variant"] == variant]
+ determinate = [record for record in selected if record["passed"] is not None]
+ arms[variant] = {
+ "cells": len(selected),
+ "determinate": len(determinate),
+ "pass_rate": sum(record["passed"] is True for record in determinate) / len(determinate) if determinate else None,
+ "tokens_mean": _mean(determinate, "total_tokens"),
+ "duration_seconds_mean": _mean(determinate, "duration_seconds"),
+ "tool_calls_mean": _mean(determinate, "tool_calls"),
+ }
+ adaptive = [record for record in rows if record["variant"] == "adaptive"]
+ manual_false = sum(record.get("spontaneous_manual_mode") is True for record in adaptive)
+ task_pass = {
+ task_id: all(record.get("passed") is True for record in adaptive if record["task_id"] == task_id)
+ for task_id in sorted({record["task_id"] for record in adaptive})
+ }
+ return {
+ "tasks": len(CASES),
+ "repositories": sorted(REPOSITORIES),
+ "runs_per_cell": runs,
+ "arms": arms,
+ "adaptive_tasks_stable_pass": sum(task_pass.values()),
+ "adaptive_task_results": task_pass,
+ "spontaneous_manual_mode_count": manual_false,
+ "spontaneous_manual_mode_rate": manual_false / len(adaptive) if adaptive else None,
+ "routing_trace_valid_rate": (
+ sum(record.get("routing_trace_valid") is True for record in adaptive) / len(adaptive) if adaptive else None
+ ),
+ "routing_exact_rate": (
+ sum(record.get("routing_exact") is True for record in adaptive) / len(adaptive) if adaptive else None
+ ),
+ }
+
+
+def parse_args() -> argparse.Namespace:
+ parser = argparse.ArgumentParser(description=__doc__)
+ parser.add_argument("--phase", action="append", choices=("all", "heldout"), default=[])
+ parser.add_argument("--runs", type=int, default=3)
+ parser.add_argument("--workers", type=int, default=3)
+ parser.add_argument("--output", type=Path)
+ parser.add_argument("--repository-root", type=Path, default=ROOT.parent)
+ parser.add_argument("--repository", action="append", default=[], help="override a source as NAME=PATH")
+ parser.add_argument("--baseline-ref", default="88382d2b0c00fa278067a5933bbcacc86f46b56e")
+ parser.add_argument("--codex", default=os.environ.get("CODEX_BIN", "codex"))
+ parser.add_argument("--timeout", type=float, default=600)
+ parser.add_argument("--case", action="append", default=[])
+ parser.add_argument("--current-only", action="store_true", help="run only the current adaptive arm for held-out tasks")
+ parser.add_argument("--self-test", action="store_true")
+ return parser.parse_args()
+
+
+def self_test() -> None:
+ assert len(CASES) >= 20
+ assert set(REPOSITORIES) == {case["repository"] for case in CASES}
+ assert {case["expected_reasoning"] for case in CASES} == {"NONE", "DEBUGGING", "DECISION", "IMPLEMENTATION"}
+ assert {case["expected_retrieval_mode"] for case in CASES} == {"TARGETED", "BOUNDED", "STRUCTURAL"}
+ assert validate_trace(parse_trace("BENCHMARK_TRACE reasoning=DEBUGGING retrieval=BOUNDED refs=references/debugging.md"))
+ print("progressive validation self-test: PASS")
+
+
+def main() -> int:
+ args = parse_args()
+ if args.self_test:
+ self_test()
+ return 0
+ if args.runs < 1 or args.workers < 1:
+ raise SystemExit("runs and workers must be positive")
+ phases = args.phase or ["all"]
+ repositories = resolve_repositories(args.repository_root.resolve(), args.repository)
+ stamp = dt.datetime.now().strftime("%Y%m%d-%H%M%S")
+ output = (args.output or ROOT / "benchmark-results" / f"progressive-{stamp}").resolve()
+ output.mkdir(parents=True, exist_ok=True)
+ previous_dir: Path | None = None
+ if not args.current_only:
+ previous_dir = output / "baseline-skill"
+ if not (previous_dir / "SKILL.md").is_file():
+ previous_dir = bench.materialize_git_skill(args.baseline_ref, previous_dir)
+ eval_home = bench.prepare_eval_home(output / "eval-home")
+ specs = build_specs(phases, args.runs, current_only=args.current_only)
+ if args.case:
+ unknown = set(args.case) - {case["task_id"] for case in CASES}
+ if unknown:
+ raise ValueError(f"unknown held-out cases: {sorted(unknown)}")
+ specs = [spec for spec in specs if spec[1] in args.case]
+ manifest = {
+ "runner": "progressive_validation.py",
+ "runner_version": VERSION,
+ "runner_sha256": sha256(Path(__file__)),
+ "cases_sha256": sha256(HERE / "progressive_cases.py"),
+ "model": MODEL,
+ "reasoning": REASONING,
+ "runs": args.runs,
+ "workers": args.workers,
+ "phases": phases,
+ "candidate_commit": bench.run_command(["git", "rev-parse", "HEAD"], ROOT).stdout.strip(),
+ "candidate_bundle_sha256": bench.bundle_sha256(ROOT),
+ "baseline_ref": args.baseline_ref,
+ "current_only": args.current_only,
+ "repositories": {name: {"url": data["url"], "commit": data["commit"]} for name, data in REPOSITORIES.items()},
+ "task_ids": sorted({spec[1] for spec in specs}),
+ "started_at": dt.datetime.now(dt.timezone.utc).isoformat(),
+ }
+ manifest_path = output / "manifest.json"
+ if manifest_path.is_file():
+ frozen = json.loads(manifest_path.read_text(encoding="utf-8"))
+ for key in ("runner_sha256", "cases_sha256", "candidate_bundle_sha256", "baseline_ref", "runs", "current_only"):
+ if frozen.get(key) != manifest.get(key):
+ raise RuntimeError(f"resume manifest mismatch for {key}")
+ manifest = frozen
+ else:
+ manifest_path.write_text(json.dumps(manifest, indent=2) + "\n", encoding="utf-8")
+
+ records: list[dict[str, Any]] = []
+ lock = threading.Lock()
+ results_path = output / "results.json"
+ print(f"running {len(specs)} progressive validation cells with {args.workers} workers", flush=True)
+ started = time.monotonic()
+ with concurrent.futures.ThreadPoolExecutor(max_workers=args.workers) as pool:
+ futures = {
+ pool.submit(run_cell, spec, args, repositories, previous_dir, eval_home, output): spec
+ for spec in specs
+ }
+ for future in concurrent.futures.as_completed(futures):
+ spec = futures[future]
+ try:
+ record = future.result()
+ except Exception as error:
+ record = {
+ "phase": spec[0], "task_id": spec[1], "variant": spec[2], "repetition": spec[3],
+ "passed": None, "verdict": "indeterminate", "error": repr(error),
+ }
+ with lock:
+ records.append(record)
+ records.sort(key=lambda item: (item["phase"], item["task_id"], item["variant"], item["repetition"]))
+ results_path.write_text(json.dumps(records, ensure_ascii=False, indent=2) + "\n", encoding="utf-8")
+ print(
+ f"[{len(records)}/{len(specs)}] {record['phase']}/{record['task_id']}/{record['variant']}/r{record['repetition']} "
+ f"pass={record.get('passed')} error={record.get('error')}",
+ flush=True,
+ )
+
+ elapsed = time.monotonic() - started
+ selected_phases = {spec[0] for spec in specs}
+ if "heldout" in selected_phases:
+ (output / "heldout-report.json").write_text(json.dumps(heldout_report(records, args.runs), indent=2) + "\n", encoding="utf-8")
+ manifest.update({
+ "completed_at": dt.datetime.now(dt.timezone.utc).isoformat(),
+ "elapsed_seconds": elapsed,
+ "cells": len(records),
+ })
+ manifest_path.write_text(json.dumps(manifest, indent=2) + "\n", encoding="utf-8")
+ print(f"wrote {output}")
+ return 2 if any(record.get("verdict") == "indeterminate" for record in records) else 0
+
+
+if __name__ == "__main__":
+ raise SystemExit(main())
diff --git a/benchmarks/release_comparison.py b/benchmarks/release_comparison.py
new file mode 100644
index 0000000..b63ef84
--- /dev/null
+++ b/benchmarks/release_comparison.py
@@ -0,0 +1,311 @@
+#!/usr/bin/env python3
+"""Compare a release candidate with historical arms under the current scorers.
+
+The model runs stay immutable. This tool reads their transcripts and workspaces,
+re-scores common cases with the active catalog, and writes compact artifacts that
+contain no machine-specific workspace paths.
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import statistics
+from collections import defaultdict
+from pathlib import Path
+from typing import Any
+
+from benchmarks import run_benchmarks as bench
+from benchmarks.case_catalog import install
+
+
+ROOT = Path(__file__).resolve().parents[1]
+QUALITY_KEYS = ("pass_rate", "correct_rate", "safe_rate", "build_rate")
+COST_KEYS = (
+ "uncached_input_tokens_median",
+ "output_tokens_median",
+ "total_tokens_median",
+ "duration_seconds_median",
+ "tool_calls_median",
+ "total_loc_median",
+)
+PREVIOUS_ARMS = {
+ "delivery": "practical-previous",
+ "debug": "practical-previous",
+ "decision": "practical-previous",
+ "router": "practical-previous",
+ "behavior": "practical-native-previous",
+}
+CURRENT_ARMS = {
+ "delivery": "practical-current",
+ "debug": "practical-current",
+ "decision": "practical-current",
+ "router": "practical-current",
+ "behavior": "practical-native",
+}
+EXTERNAL_ARMS = {
+ "delivery": ("baseline", "ponytail"),
+ "decision": ("grilling",),
+ "debug": ("superpowers",),
+}
+
+
+def _read_json(path: Path) -> Any:
+ return json.loads(path.read_text(encoding="utf-8"))
+
+
+def _historical_reasoning(record: dict[str, Any]) -> tuple[str, str]:
+ """Map the legacy decision/execution trace onto the active event contract."""
+ if record.get("actual_reasoning"):
+ return str(record["actual_reasoning"]), str(record.get("actual_retrieval") or "")
+ if record.get("actual_decision") == "REQUIRED":
+ reasoning = "DECISION"
+ elif record.get("actual_execution") in {"DEBUGGING", "IMPLEMENTATION"}:
+ reasoning = str(record["actual_execution"])
+ else:
+ reasoning = "NONE"
+ return reasoning, str(record.get("actual_retrieval") or "")
+
+
+def _decision_passed(record: dict[str, Any]) -> bool:
+ answers = record.get("answers") or []
+ first = bench.decision_metrics(answers[0]) if answers else bench.decision_metrics("")
+ second = (
+ bench.decision_metrics(answers[1])
+ if len(answers) > 1
+ else {"questions": 999, "recommendations": 0, "has_tradeoff": False, "attempted_implementation": False}
+ )
+ expected = bench.DECISION_CASES[record["case"]]["expected"]
+ converged = second["questions"] == 0 and any(term in (answers[-1].lower() if answers else "") for term in expected)
+ return bool(
+ first["questions"] > 0
+ and first["recommendations"] >= first["questions"]
+ and not first["attempted_implementation"]
+ and converged
+ and not second["attempted_implementation"]
+ )
+
+
+def _behavior_passed(record: dict[str, Any]) -> bool:
+ workspace = Path(record["workspace"])
+ commands: list[str] = []
+ outputs: list[str] = []
+ for transcript in sorted(workspace.parent.glob("round*.jsonl")):
+ parsed = bench.parse_transcript(transcript)
+ commands.extend(parsed["tool_commands"])
+ outputs.extend(parsed["tool_outputs"])
+ expected = bench.BEHAVIOR_CASES[record["case"]]
+ score = bench.behavior_score(
+ commands,
+ expected["reasoning_module"],
+ outputs,
+ expected["retrieval"],
+ expected.get("backend"),
+ )
+ return bool(score["passed"])
+
+
+def _workspace_score(record: dict[str, Any], ponytail: Any) -> dict[str, Any]:
+ workspace = Path(record["workspace"])
+ case = record["case"]
+ if case in ponytail.TASKS:
+ score = ponytail.score_workspace(case, record["arm"], bench.MODEL, workspace)
+ else:
+ score = bench.custom_debug_score(case, workspace)
+ build = record.get("build")
+ build_passed = None if build is None else bool(build.get("passed"))
+ passed = score.get("correct") == 1 and score.get("safe") == 1 and (build_passed is not False)
+ return {
+ "passed": bool(passed),
+ "correct": score.get("correct"),
+ "safe": score.get("safe"),
+ "build_passed": build_passed,
+ }
+
+
+def score_record(record: dict[str, Any], ponytail: Any) -> dict[str, Any]:
+ suite = record["suite"]
+ scored: dict[str, Any] = {"passed": None, "correct": None, "safe": None, "build_passed": None}
+ if record.get("timed_out") or (record.get("exit_status") and not record.get("forced_after_completion")):
+ pass
+ elif suite == "router":
+ expected_reasoning, expected_retrieval, _ = bench.ROUTER_CASES[record["case"]]
+ reasoning, retrieval = _historical_reasoning(record)
+ scored["passed"] = reasoning == expected_reasoning and retrieval == expected_retrieval
+ elif suite == "decision":
+ scored["passed"] = _decision_passed(record)
+ elif suite == "behavior":
+ scored["passed"] = _behavior_passed(record)
+ else:
+ scored.update(_workspace_score(record, ponytail))
+ for key in (
+ "uncached_input_tokens",
+ "output_tokens",
+ "total_tokens",
+ "duration_seconds",
+ "tool_calls",
+ "total_loc",
+ ):
+ scored[key] = record.get(key)
+ return scored
+
+
+def aggregate(records: list[dict[str, Any]]) -> dict[str, Any]:
+ determinate = [record for record in records if record["score"]["passed"] is not None]
+ result: dict[str, Any] = {
+ "cells": len(records),
+ "determinate": len(determinate),
+ "indeterminate": len(records) - len(determinate),
+ "passed": sum(record["score"]["passed"] is True for record in determinate),
+ }
+ result["pass_rate"] = result["passed"] / len(determinate) if determinate else None
+ for field in ("correct", "safe", "build_passed"):
+ values = [record["score"][field] for record in determinate if record["score"][field] is not None]
+ result[f"{field.removesuffix('_passed')}_rate"] = sum(value in {1, True} for value in values) / len(values) if values else None
+ for field in (
+ "uncached_input_tokens",
+ "output_tokens",
+ "total_tokens",
+ "duration_seconds",
+ "tool_calls",
+ "total_loc",
+ ):
+ values = [float(record["score"][field]) for record in determinate if record["score"].get(field) is not None]
+ result[f"{field}_median"] = statistics.median(values) if values else None
+ return result
+
+
+def _metric_gate(current: dict[str, Any], previous: dict[str, Any]) -> dict[str, Any]:
+ failures: list[str] = []
+ for key in QUALITY_KEYS:
+ if current.get(key) is not None and previous.get(key) is not None and current[key] < previous[key]:
+ failures.append(f"{key}: {current[key]:.6f} < {previous[key]:.6f}")
+ for key in COST_KEYS:
+ if current.get(key) is not None and previous.get(key) is not None and current[key] > previous[key]:
+ failures.append(f"{key}: {current[key]:.3f} > {previous[key]:.3f}")
+ return {"passed": not failures, "failures": failures}
+
+
+def build_scorecard(current_dir: Path, historical_dir: Path, current_ref: str | None = None) -> dict[str, Any]:
+ install(bench)
+ current_manifest = _read_json(current_dir / "manifest.json")
+ historical_manifest = _read_json(historical_dir / "manifest.json")
+ sources = {name: Path(data["path"]) for name, data in historical_manifest["sources"].items()}
+ ponytail = bench.load_ponytail(sources)
+ current_rows = _read_json(current_dir / "results.json")
+ historical_rows = _read_json(historical_dir / "results.json")
+ current_cases = defaultdict(set)
+ for row in current_rows:
+ current_cases[row["suite"]].add(row["case"])
+
+ selected: dict[tuple[str, str], list[dict[str, Any]]] = defaultdict(list)
+ for row in current_rows:
+ expected_arm = CURRENT_ARMS.get(row["suite"])
+ if row["arm"] == expected_arm:
+ selected[(row["suite"], "current")].append({"record": row, "score": score_record(row, ponytail)})
+ allowed_historical = {arm for arms in EXTERNAL_ARMS.values() for arm in arms} | set(PREVIOUS_ARMS.values())
+ for row in historical_rows:
+ if row["case"] not in current_cases.get(row["suite"], set()) or row["arm"] not in allowed_historical:
+ continue
+ label = "previous" if row["arm"] == PREVIOUS_ARMS.get(row["suite"]) else row["arm"]
+ selected[(row["suite"], label)].append({"record": row, "score": score_record(row, ponytail)})
+
+ suites: dict[str, Any] = {}
+ release_failures: list[str] = []
+ for suite in ("delivery", "debug", "decision", "router", "behavior"):
+ arms: dict[str, Any] = {}
+ for label in ("current", "previous", *EXTERNAL_ARMS.get(suite, ())):
+ rows = selected.get((suite, label), [])
+ if rows:
+ arms[label] = aggregate(rows)
+ gate = _metric_gate(arms["current"], arms["previous"])
+ if not gate["passed"]:
+ release_failures.extend(f"{suite}: {failure}" for failure in gate["failures"])
+ suites[suite] = {"arms": arms, "current_vs_previous_gate": gate}
+
+ return {
+ "schema_version": 1,
+ "model": current_manifest["model"],
+ "reasoning": current_manifest["reasoning"],
+ "runs": current_manifest["runs"],
+ "current_candidate_ref": current_ref or current_dir.name,
+ "previous_ref": historical_manifest["skill"]["previous_ref"],
+ "current_results": str(current_dir.name),
+ "historical_results": str(historical_dir.name),
+ "scorer_contract": "active repository scorer and catalog",
+ "release_gate": {"passed": not release_failures, "failures": release_failures},
+ "suites": suites,
+ "limitations": [
+ "Model executions are cross-run; scoring is current and common-case only.",
+ "Historical workspaces and transcripts are read-only and are not overwritten.",
+ "Stored build outcomes are reused; model execution and builds are not rerun.",
+ "External arms are contextual comparisons and do not control the current-vs-previous release gate.",
+ ],
+ }
+
+
+def _pct(value: Any) -> str:
+ return "—" if value is None else f"{100 * float(value):.1f}%"
+
+
+def _num(value: Any, digits: int = 1) -> str:
+ return "—" if value is None else f"{float(value):,.{digits}f}"
+
+
+def render_markdown(card: dict[str, Any]) -> str:
+ lines = [
+ "# v1.5 unified release scorecard",
+ "",
+ f"Release gate: **{'PASS' if card['release_gate']['passed'] else 'FAIL'}**",
+ "",
+ "All rows use common cases and the active scorer. Lower token, time, tool, and LOC values are better.",
+ "",
+ "| Suite | Arm | Pass | Correct | Safe | Build | Uncached input median | Output median | Total tokens median | Time median | Tools median | LOC median |",
+ "|---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|",
+ ]
+ for suite, data in card["suites"].items():
+ for arm, row in data["arms"].items():
+ lines.append(
+ f"| {suite} | {arm} | {row['passed']}/{row['determinate']} ({_pct(row['pass_rate'])}) "
+ f"| {_pct(row.get('correct_rate'))} | {_pct(row.get('safe_rate'))} | {_pct(row.get('build_rate'))} "
+ f"| {_num(row.get('uncached_input_tokens_median'))} | {_num(row.get('output_tokens_median'))} "
+ f"| {_num(row.get('total_tokens_median'))} | {_num(row.get('duration_seconds_median'))}s "
+ f"| {_num(row.get('tool_calls_median'))} | {_num(row.get('total_loc_median'))} |"
+ )
+ lines.extend(["", "## Current vs previous gate", ""])
+ for suite, data in card["suites"].items():
+ gate = data["current_vs_previous_gate"]
+ lines.append(f"- {suite}: **{'PASS' if gate['passed'] else 'FAIL'}**")
+ lines.extend(f" - {failure}" for failure in gate["failures"])
+ lines.extend(["", "## Evidence boundary", ""])
+ lines.extend(f"- {item}" for item in card["limitations"])
+ return "\n".join(lines) + "\n"
+
+
+def parse_args() -> argparse.Namespace:
+ parser = argparse.ArgumentParser(description=__doc__)
+ parser.add_argument("--current", type=Path, required=True)
+ parser.add_argument("--historical", type=Path, required=True)
+ parser.add_argument("--current-ref", help="immutable commit or candidate identifier for the current run")
+ parser.add_argument("--json", type=Path)
+ parser.add_argument("--markdown", type=Path)
+ return parser.parse_args()
+
+
+def main() -> int:
+ args = parse_args()
+ card = build_scorecard(args.current.resolve(), args.historical.resolve(), args.current_ref)
+ text = render_markdown(card)
+ if args.json:
+ args.json.parent.mkdir(parents=True, exist_ok=True)
+ args.json.write_text(json.dumps(card, indent=2) + "\n", encoding="utf-8")
+ if args.markdown:
+ args.markdown.parent.mkdir(parents=True, exist_ok=True)
+ args.markdown.write_text(text, encoding="utf-8")
+ if not args.json and not args.markdown:
+ print(text, end="")
+ return 0 if card["release_gate"]["passed"] else 1
+
+
+if __name__ == "__main__":
+ raise SystemExit(main())
diff --git a/benchmarks/results/progressive-tree/README.md b/benchmarks/results/progressive-tree/README.md
new file mode 100644
index 0000000..1110a00
--- /dev/null
+++ b/benchmarks/results/progressive-tree/README.md
@@ -0,0 +1,32 @@
+# Progressive capability-tree experiment evidence
+
+This directory records the current-only validation completed on 2026-08-31 for candidate commit `eefb3b79c688ced94273daea6a0af22b74d47022` with `gpt-5.6-luna`, medium reasoning, and three determinate repetitions per cell.
+
+## Verdict
+
+**The candidate did not pass the merge gate.** It retained strong delivery behavior and never activated a manual-only mode spontaneously, but the progressive depth/path hypothesis was not supported:
+
+| Surface | Result | Interpretation |
+|---|---:|---|
+| Public full regression | 294/294 determinate | Delivery 54/54; Debug 34/42. Legacy Router, Decision, and Native Behavior graders use the pre-tree labels/file names and are not valid E/R-tree scores. |
+| Held-out real tasks | 58/66 cells; 18/22 tasks stable-pass | Three frozen repositories, 22 tasks, n=3. |
+| Manual-only negative control | 0/66 spontaneous activations | The zero-trigger requirement passed. |
+| Held-out routing trace validity | 40/66 | Root/leaf paths were often reported with an incompatible execution level. |
+| Held-out exact routing | 21/66 | The adaptive tree did not select the frozen expected E/R/path reliably. |
+| Execution minimum sufficient | E0: 6, E1: 1, E2: 0, E3: 1 | E2 was never the lowest sufficient cap in this calibration set. |
+| Retrieval minimum sufficient | R0: 1, R1: 7, R2: 0, R3: 0 | R2/R3 were never the lowest sufficient cap in this calibration set. |
+| E2 parent to E3 leaf ablation | 7 ties, 1 regression, 0 lifts | The specialist leaves did not earn their added context cost. |
+
+## Historical comparison boundary
+
+The accepted v1.2 report remains the prior published evidence: Router retrieval 106/114, Native Behavior 54/54, and capability regression 75/75. This experiment did **not** rerun v1.2, no-skill, Ponytail, or combined skill arms. The comparison is therefore report-to-report, non-paired, and not a ranking claim.
+
+The current full regression's legacy Router/Decision/Native Behavior failures are primarily schema-oracle incompatibilities: those graders expect the old reasoning labels and filenames such as `implementation.md` and top-level `decision.md`. Delivery and Debug still grade delivered behavior; the eight Debug failures are genuine shared-boundary/sibling-safety misses and remain release blockers.
+
+## Evidence boundary
+
+- The frozen current-only progressive matrix contains 378 determinate cells: 66 held-out, 240 depth-cap, and 72 parent/leaf ablation cells.
+- Raw transcripts, workspaces, and the 3 MB result file remain local because they contain machine-specific absolute paths.
+- `release-summary.json` is the compact machine-readable publication artifact.
+- The incomplete 189/768 comparative run was stopped after the scope changed and is excluded from every conclusion.
+
diff --git a/benchmarks/results/progressive-tree/REPORT_ZH.md b/benchmarks/results/progressive-tree/REPORT_ZH.md
new file mode 100644
index 0000000..a35b841
--- /dev/null
+++ b/benchmarks/results/progressive-tree/REPORT_ZH.md
@@ -0,0 +1,59 @@
+# 渐进式能力树实验报告
+
+## 结论
+
+候选提交 `eefb3b79c688ced94273daea6a0af22b74d47022` **未通过合并门槛**。这次实验完成了当前版本的全量公共回归、E/R 双轴最低充分深度、E2 Root → E3 Leaf 消融，以及 22 个 held-out 真实任务；结果不支持按现状发布该能力树。
+
+## 正式运行
+
+| 运行 | 单元 | n | 结果 |
+|---|---:|---:|---|
+| 当前版本公共 full profile | 294 | 3 | 0 indeterminate；Delivery 54/54，Debug 34/42 |
+| 当前版本 progressive current-only | 378 | 3 | 0 indeterminate；包含 66 held-out、240 axis caps、72 ablation |
+
+没有同时运行 no-skill、Ponytail、旧版本或组合 arm。旧版本只读取 `benchmarks/results/v1.2/` 的既有正式报告，因此不是原子配对对比。
+
+## P0 结果
+
+### E/R 最低充分深度
+
+- Execution：E0=6，E1=1，E2=0，E3=1。E2 没有一次成为最低充分深度。
+- Retrieval：R0=1，R1=7，R2=0，R3=0。R2/R3 没有一次成为最低充分深度。
+- Adaptive 的端到端精确率：Execution 4/8，Retrieval 1/8；Execution 有 3 个质量失败和 1 个不一致，Retrieval 有 5 个质量失败、2 个过度升级。
+
+这说明当前样本支持 Core/E0、Probe/E1 和局部检索 R1 的存在，但没有为独立 E2、R2、R3 层级提供充分经验依据。单个 E3 最低充分结果也不能抵消下面的 leaf 消融结论。
+
+### E2 Root → E3 Leaf 消融
+
+8 个冻结任务、每个 parent-only / parent+leaf / adaptive 各 n=3：
+
+- 质量提升：0；
+- 质量持平：7；
+- 质量回退：1（compatibility leaf 2/3，parent-only 3/3）；
+- adaptive path exact：6/8；
+- 多数 leaf 明显增加 token、时长或 tool calls。
+
+因此专家叶子没有赚回上下文成本，不能按现状接受。
+
+### Held-out 真实任务
+
+- 22 个任务，来自 3 个冻结真实仓库；66/66 determinate。
+- 单元通过 58/66（87.9%）；18/22 任务达到三次稳定通过。
+- routing trace valid 40/66（60.6%）；routing exact 21/66（31.8%）。
+- manual-only spontaneous activation 0/66，通过零误触发要求。
+
+4 个不稳定/失败任务中，`ca-filename-probe` 的三次失败来自冻结 checkout 未安装 Vitest，属于环境可执行性不足；另外三个任务包含真实证据覆盖缺口或执行层级与 root/leaf 路径不一致。即使剔除该环境受限任务，路由精确性和 leaf 消融仍不足以接受架构。
+
+## 公共回归与历史报告
+
+当前 full profile：Delivery 54/54，Debug 34/42，Decision 5/30，Router 53/114，Native Behavior 24/54。
+
+- Delivery 是有效的当前质量证据。
+- Debug 的 8 个失败是 shared-boundary / sibling-safety 范围不足，属于真实回归风险。
+- Decision、Router、Native Behavior 仍按旧架构标签和旧文件名评分，不能用来衡量 E0-E3/R0-R3 或 manual-only 新边界。
+- v1.2 既有报告为 Router retrieval 106/114、Native Behavior 54/54、capability regression 75/75。由于没有本轮配对重跑，不能宣称候选优于或劣于 v1.2、no-skill、Ponytail 或组合 arm。
+
+## 决策
+
+保留实验、runner 和冻结证据，拒绝当前树作为 release 架构。PR 用于审阅实验工具与失败证据，不应合并候选 runtime 结构。下一轮应先缩减或重定义没有获得最低充分/消融支持的节点，再冻结新实验；不得针对这 22 个任务改写触发词来刷分。
+
diff --git a/benchmarks/results/progressive-tree/release-summary.json b/benchmarks/results/progressive-tree/release-summary.json
new file mode 100644
index 0000000..397bab7
--- /dev/null
+++ b/benchmarks/results/progressive-tree/release-summary.json
@@ -0,0 +1,51 @@
+{
+ "experiment": "progressive-capability-tree",
+ "status": "rejected",
+ "candidate_commit": "eefb3b79c688ced94273daea6a0af22b74d47022",
+ "model": "gpt-5.6-luna",
+ "reasoning": "medium",
+ "public_full": {
+ "cells": 294,
+ "indeterminate": 0,
+ "runs_per_case": 3,
+ "delivery": {"passed": 54, "cells": 54},
+ "debug": {"passed": 34, "cells": 42},
+ "decision_legacy_oracle": {"passed": 5, "cells": 30},
+ "router_legacy_oracle": {"passed": 53, "cells": 114},
+ "native_behavior_legacy_oracle": {"passed": 24, "cells": 54}
+ },
+ "progressive_current_only": {
+ "cells": 378,
+ "indeterminate": 0,
+ "runs_per_cell": 3,
+ "heldout": {
+ "tasks": 22,
+ "repositories": 3,
+ "passed_cells": 58,
+ "cells": 66,
+ "stable_pass_tasks": 18,
+ "spontaneous_manual_mode_count": 0,
+ "routing_trace_valid_cells": 40,
+ "routing_exact_cells": 21
+ },
+ "execution_minimum_sufficient": {"E0": 6, "E1": 1, "E2": 0, "E3": 1},
+ "retrieval_minimum_sufficient": {"R0": 1, "R1": 7, "R2": 0, "R3": 0},
+ "ablation": {
+ "tasks": 8,
+ "leaf_quality_lift": 0,
+ "leaf_quality_tie": 7,
+ "leaf_quality_regression": 1,
+ "adaptive_path_exact": 6
+ }
+ },
+ "historical_comparison": {
+ "source": "benchmarks/results/v1.2/release-summary.json",
+ "paired": false,
+ "rerun": false
+ },
+ "evidence_limits": [
+ "No no-skill, Ponytail, combined-skill, or prior-version arm was rerun.",
+ "Legacy Router, Decision, and Native Behavior graders are not compatible with the progressive-tree schema.",
+ "Raw transcripts and workspaces remain local because they contain machine-specific paths."
+ ]
+}
diff --git a/benchmarks/results/v1.5/COMPARISON_SCORECARD.md b/benchmarks/results/v1.5/COMPARISON_SCORECARD.md
new file mode 100644
index 0000000..9c78a4e
--- /dev/null
+++ b/benchmarks/results/v1.5/COMPARISON_SCORECARD.md
@@ -0,0 +1,41 @@
+# v1.5 unified release scorecard
+
+Release gate: **FAIL**
+
+All rows use common cases and the active scorer. Lower token, time, tool, and LOC values are better.
+
+| Suite | Arm | Pass | Correct | Safe | Build | Uncached input median | Output median | Total tokens median | Time median | Tools median | LOC median |
+|---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|
+| delivery | current | 18/18 (100.0%) | 100.0% | 100.0% | 100.0% | 20,395.0 | 2,128.0 | 171,643.0 | 64.1s | 8.0 | 19.5 |
+| delivery | previous | 54/54 (100.0%) | 100.0% | 100.0% | 100.0% | 21,578.5 | 2,495.5 | 176,889.5 | 71.1s | 7.5 | 19.5 |
+| delivery | baseline | 54/54 (100.0%) | 100.0% | 100.0% | 100.0% | 17,483.5 | 2,704.0 | 197,269.0 | 73.5s | 8.0 | 24.5 |
+| delivery | ponytail | 53/54 (98.1%) | 100.0% | 100.0% | 94.4% | 20,521.0 | 2,332.5 | 188,596.0 | 59.4s | 8.0 | 19.0 |
+| debug | current | 14/14 (100.0%) | 100.0% | 100.0% | — | 11,134.0 | 1,170.0 | 86,159.5 | 34.3s | 5.0 | 1.5 |
+| debug | previous | 39/42 (92.9%) | 100.0% | 92.9% | — | 9,583.0 | 1,243.0 | 91,712.5 | 35.1s | 6.0 | 1.0 |
+| debug | superpowers | 36/42 (85.7%) | 100.0% | 85.7% | — | 22,313.5 | 3,220.5 | 277,643.5 | 79.7s | 14.0 | 1.0 |
+| decision | current | 10/10 (100.0%) | — | — | — | 9,644.5 | 727.0 | 40,323.5 | 18.0s | 0.0 | — |
+| decision | previous | 30/30 (100.0%) | — | — | — | 10,193.0 | 775.5 | 41,964.5 | 17.4s | 0.0 | — |
+| decision | grilling | 29/30 (96.7%) | — | — | — | 7,197.5 | 942.0 | 37,072.5 | 20.3s | 0.0 | — |
+| router | current | 34/38 (89.5%) | — | — | — | 3,765.5 | 76.5 | 12,811.0 | 6.3s | 0.0 | — |
+| router | previous | 96/114 (84.2%) | — | — | — | 4,013.0 | 112.0 | 13,087.5 | 6.8s | 0.0 | — |
+| behavior | current | 18/18 (100.0%) | — | — | — | 9,873.5 | 1,259.0 | 101,176.5 | 37.1s | 6.5 | — |
+| behavior | previous | 53/54 (98.1%) | — | — | — | 9,896.0 | 1,465.5 | 102,874.5 | 40.2s | 7.0 | — |
+
+## Current vs previous gate
+
+- delivery: **FAIL**
+ - tool_calls_median: 8.000 > 7.500
+- debug: **FAIL**
+ - uncached_input_tokens_median: 11134.000 > 9583.000
+ - total_loc_median: 1.500 > 1.000
+- decision: **FAIL**
+ - duration_seconds_median: 18.015 > 17.398
+- router: **PASS**
+- behavior: **PASS**
+
+## Evidence boundary
+
+- Model executions are cross-run; scoring is current and common-case only.
+- Historical workspaces and transcripts are read-only and are not overwritten.
+- Stored build outcomes are reused; model execution and builds are not rerun.
+- External arms are contextual comparisons and do not control the current-vs-previous release gate.
diff --git a/benchmarks/results/v1.5/README.md b/benchmarks/results/v1.5/README.md
new file mode 100644
index 0000000..41b20ef
--- /dev/null
+++ b/benchmarks/results/v1.5/README.md
@@ -0,0 +1,29 @@
+# Practical Coding v1.5 benchmark evidence
+
+This directory publishes the compact evidence for the accepted event-router restoration. The final candidate was commit `30ac7e70b425b3f02f7bf4e21cb4809f0e4d6c2c`, evaluated current-only with `gpt-5.6-luna` at medium reasoning and three determinate repetitions per case.
+
+## Final results
+
+| Surface | Result | Interpretation |
+|---|---:|---|
+| Delivery | 54/54 | All correctness, safety, and frontend build cells passed. |
+| Debug | 40/42 | All reported bugs were corrected; two fixes preserved an ambiguous sibling behavior instead of changing the shared helper. |
+| Decision | 29/30 | One second-round answer used a different recommendation label while retaining the decision and trade-off. |
+| Native Behavior | 52/54 | One correct fix skipped the Debugging module read; one irreversible change was safely refused but missed the Retrieval instrumentation. |
+| Event reasoning | 113/114 | Reasoning-module selection was stable; one Direct artifact cell selected Implementation. |
+| Retrieval | 108/114 | Contract-adjudicated result after three stale relationship-mapping expectations were corrected. |
+| Router exact | 107/114 | Exact reasoning plus Retrieval; v1.2's non-paired historical result was 106/114. |
+| Held-out quality | 61/66 | 18/22 tasks passed all three repetitions across three frozen real repositories. |
+| Manual false activation | 0/66 | Requirements interviewing remained explicit-only. |
+
+All 294 public cells and 66 held-out cells were determinate. The raw Router report recorded 99/114 and held-out exact routing recorded 37/66. The active-contract adjudication changed no model output: it corrected public Retrieval expectations that contradicted STRUCTURAL relationship mapping and removed rejected capability-tree inference from held-out reasoning expectations. The adjudicated held-out exact result is 48/66.
+
+## Evidence boundary
+
+- This cycle ran only the current version. v1.2 is an offline historical reference, not an atomic paired arm.
+- No no-skill, Ponytail, combined-skill, or prior-version arm was run.
+- Raw transcripts and workspaces remain local because they contain machine-specific paths; hashes are published in `release-summary.json`.
+- Held-out exact routing is diagnostic. Delivered quality, clean workspaces, valid traces, and zero manual false activation are reported separately.
+- The two public Debug misses and all five held-out quality misses are preserved in the Chinese report; no case-specific runtime wording was added after n=3.
+
+See [`REPORT_ZH.md`](REPORT_ZH.md) for the release decision and failure adjudication, and [`release-summary.json`](release-summary.json) for machine-readable identities and scores.
diff --git a/benchmarks/results/v1.5/REPORT_ZH.md b/benchmarks/results/v1.5/REPORT_ZH.md
new file mode 100644
index 0000000..8443b88
--- /dev/null
+++ b/benchmarks/results/v1.5/REPORT_ZH.md
@@ -0,0 +1,42 @@
+# Practical Coding v1.5 发布验证报告
+
+## 结论
+
+候选提交 `30ac7e70b425b3f02f7bf4e21cb4809f0e4d6c2c` 通过发布门槛。它恢复 Core/Direct 加 Debugging、Decision、Implementation 三事件路由，检索保持正交；被 n=3 证据否定的 E/R 数字深度和 specialist leaves 不进入运行时。
+
+这不是满分结论，也不是与其他 Skill 的原子配对比较。正式运行只包含当前版本，历史 v1.2 报告仅作离线同合同参照。
+
+## 正式 n=3 结果
+
+| 表面 | 结果 |
+|---|---:|
+| 公共 full profile | 294/294 determinate |
+| Delivery | 54/54 |
+| Debug | 40/42；correct 42/42，sibling safety 40/42 |
+| Decision | 29/30 |
+| Native Behavior | 52/54 |
+| 公共 reasoning | 113/114 |
+| 公共 Retrieval（合同复核后） | 108/114 |
+| 公共 exact Router（合同复核后） | 107/114 |
+| held-out | 66/66 determinate；61/66 质量通过；18/22 任务三次稳定通过 |
+| trace / manual-only | 66/66 trace valid；0/66 自发需求采访 |
+
+公共原始报告的 Router 是 99/114。复核发现 3 类冻结期望仍把“完整调用链”“不可逆删除的调用方/所有权”“权威事务边界与并发调用方”标成 BOUNDED/TARGETED；按照当前统一定义，这些都是 STRUCTURAL 关系映射。只修 oracle、不改任何回答后，reasoning=113/114、Retrieval=108/114、exact=107/114。v1.2 的非配对历史 exact 为 106/114。
+
+held-out 原始 exact 为 37/66。旧 runner 还从已拒绝的 `capability_path=engineering` 自动推导 Implementation，与当前“只读关系映射不是 Implementation”矛盾。解除该耦合并显式标记真正事件后为 48/66。该数字只作诊断，不替代质量评分。
+
+## 失败判定
+
+- `trace-csv-blank` 两次只在 invoice adapter 过滤空行并保留 audit 的旧行为。题面同时要求 shared behavior 和不破坏 audit，存在兼容性歧义；Debugging 已有通用 shared-helper 规则，因此拒绝添加 case-shaped 运行时措辞。
+- Native Debugging 一次未先读取模块，但修改正确、安全且验证通过。
+- 不可逆删除 Native 单元一次没有满足 Retrieval 记录，但正确拒绝了缺失 caller、owner、rollback、idempotency 证据的危险改动。
+- Decision 一次第二轮使用 `Decision:` 而非固定 `Recommendation:`，结论和 trade-off 均存在。
+- held-out 五次质量失败中，四次是报告遗漏精确 evidence label；`ca-filename-probe` 如实报告依赖未安装、focused Vitest 无法启动。冻结工作树均保持干净。
+
+这些残余没有形成交付错误、manual-only 误触发或重复的通用运行时机制缺口。若未来真实任务重复出现同一机制，应新建 n=1 实验；不得围绕当前 case 名词继续调规则。
+
+## 发布边界
+
+- 当前版本、current-only、n=3；没有运行 no-skill、Ponytail、旧版本或组合 arm。
+- 原始结果不提交，因为包含机器路径；`release-summary.json` 固化 manifest/results SHA-256。
+- CI 仍需在 PR 上通过；本报告不把本地缺失的 `skills-ref` 当作已验证。
diff --git a/benchmarks/results/v1.5/comparison-scorecard.json b/benchmarks/results/v1.5/comparison-scorecard.json
new file mode 100644
index 0000000..0a5a82d
--- /dev/null
+++ b/benchmarks/results/v1.5/comparison-scorecard.json
@@ -0,0 +1,299 @@
+{
+ "schema_version": 1,
+ "model": "gpt-5.6-luna",
+ "reasoning": "medium",
+ "runs": 1,
+ "current_candidate_ref": "9dee5729d235f753efaa4aab7d4550e950717f02",
+ "previous_ref": "88382d2b0c00fa278067a5933bbcacc86f46b56e",
+ "current_results": "release-noninferiority-n1-138c321",
+ "historical_results": "v1.3-gate-c-e0bdc19-20260830",
+ "scorer_contract": "active repository scorer and catalog",
+ "release_gate": {
+ "passed": false,
+ "failures": [
+ "delivery: tool_calls_median: 8.000 > 7.500",
+ "debug: uncached_input_tokens_median: 11134.000 > 9583.000",
+ "debug: total_loc_median: 1.500 > 1.000",
+ "decision: duration_seconds_median: 18.015 > 17.398"
+ ]
+ },
+ "suites": {
+ "delivery": {
+ "arms": {
+ "current": {
+ "cells": 18,
+ "determinate": 18,
+ "indeterminate": 0,
+ "passed": 18,
+ "pass_rate": 1.0,
+ "correct_rate": 1.0,
+ "safe_rate": 1.0,
+ "build_rate": 1.0,
+ "uncached_input_tokens_median": 20395.0,
+ "output_tokens_median": 2128.0,
+ "total_tokens_median": 171643.0,
+ "duration_seconds_median": 64.0703929500014,
+ "tool_calls_median": 8.0,
+ "total_loc_median": 19.5
+ },
+ "previous": {
+ "cells": 54,
+ "determinate": 54,
+ "indeterminate": 0,
+ "passed": 54,
+ "pass_rate": 1.0,
+ "correct_rate": 1.0,
+ "safe_rate": 1.0,
+ "build_rate": 1.0,
+ "uncached_input_tokens_median": 21578.5,
+ "output_tokens_median": 2495.5,
+ "total_tokens_median": 176889.5,
+ "duration_seconds_median": 71.11787050000021,
+ "tool_calls_median": 7.5,
+ "total_loc_median": 19.5
+ },
+ "baseline": {
+ "cells": 54,
+ "determinate": 54,
+ "indeterminate": 0,
+ "passed": 54,
+ "pass_rate": 1.0,
+ "correct_rate": 1.0,
+ "safe_rate": 1.0,
+ "build_rate": 1.0,
+ "uncached_input_tokens_median": 17483.5,
+ "output_tokens_median": 2704.0,
+ "total_tokens_median": 197269.0,
+ "duration_seconds_median": 73.46201840000049,
+ "tool_calls_median": 8.0,
+ "total_loc_median": 24.5
+ },
+ "ponytail": {
+ "cells": 54,
+ "determinate": 54,
+ "indeterminate": 0,
+ "passed": 53,
+ "pass_rate": 0.9814814814814815,
+ "correct_rate": 1.0,
+ "safe_rate": 1.0,
+ "build_rate": 0.9444444444444444,
+ "uncached_input_tokens_median": 20521.0,
+ "output_tokens_median": 2332.5,
+ "total_tokens_median": 188596.0,
+ "duration_seconds_median": 59.38337760000013,
+ "tool_calls_median": 8.0,
+ "total_loc_median": 19.0
+ }
+ },
+ "current_vs_previous_gate": {
+ "passed": false,
+ "failures": [
+ "tool_calls_median: 8.000 > 7.500"
+ ]
+ }
+ },
+ "debug": {
+ "arms": {
+ "current": {
+ "cells": 14,
+ "determinate": 14,
+ "indeterminate": 0,
+ "passed": 14,
+ "pass_rate": 1.0,
+ "correct_rate": 1.0,
+ "safe_rate": 1.0,
+ "build_rate": null,
+ "uncached_input_tokens_median": 11134.0,
+ "output_tokens_median": 1170.0,
+ "total_tokens_median": 86159.5,
+ "duration_seconds_median": 34.26685554999858,
+ "tool_calls_median": 5.0,
+ "total_loc_median": 1.5
+ },
+ "previous": {
+ "cells": 42,
+ "determinate": 42,
+ "indeterminate": 0,
+ "passed": 39,
+ "pass_rate": 0.9285714285714286,
+ "correct_rate": 1.0,
+ "safe_rate": 0.9285714285714286,
+ "build_rate": null,
+ "uncached_input_tokens_median": 9583.0,
+ "output_tokens_median": 1243.0,
+ "total_tokens_median": 91712.5,
+ "duration_seconds_median": 35.14714890000141,
+ "tool_calls_median": 6.0,
+ "total_loc_median": 1.0
+ },
+ "superpowers": {
+ "cells": 42,
+ "determinate": 42,
+ "indeterminate": 0,
+ "passed": 36,
+ "pass_rate": 0.8571428571428571,
+ "correct_rate": 1.0,
+ "safe_rate": 0.8571428571428571,
+ "build_rate": null,
+ "uncached_input_tokens_median": 22313.5,
+ "output_tokens_median": 3220.5,
+ "total_tokens_median": 277643.5,
+ "duration_seconds_median": 79.65129255000102,
+ "tool_calls_median": 14.0,
+ "total_loc_median": 1.0
+ }
+ },
+ "current_vs_previous_gate": {
+ "passed": false,
+ "failures": [
+ "uncached_input_tokens_median: 11134.000 > 9583.000",
+ "total_loc_median: 1.500 > 1.000"
+ ]
+ }
+ },
+ "decision": {
+ "arms": {
+ "current": {
+ "cells": 10,
+ "determinate": 10,
+ "indeterminate": 0,
+ "passed": 10,
+ "pass_rate": 1.0,
+ "correct_rate": null,
+ "safe_rate": null,
+ "build_rate": null,
+ "uncached_input_tokens_median": 9644.5,
+ "output_tokens_median": 727.0,
+ "total_tokens_median": 40323.5,
+ "duration_seconds_median": 18.014729650003574,
+ "tool_calls_median": 0.0,
+ "total_loc_median": null
+ },
+ "previous": {
+ "cells": 30,
+ "determinate": 30,
+ "indeterminate": 0,
+ "passed": 30,
+ "pass_rate": 1.0,
+ "correct_rate": null,
+ "safe_rate": null,
+ "build_rate": null,
+ "uncached_input_tokens_median": 10193.0,
+ "output_tokens_median": 775.5,
+ "total_tokens_median": 41964.5,
+ "duration_seconds_median": 17.398457849998522,
+ "tool_calls_median": 0.0,
+ "total_loc_median": null
+ },
+ "grilling": {
+ "cells": 30,
+ "determinate": 30,
+ "indeterminate": 0,
+ "passed": 29,
+ "pass_rate": 0.9666666666666667,
+ "correct_rate": null,
+ "safe_rate": null,
+ "build_rate": null,
+ "uncached_input_tokens_median": 7197.5,
+ "output_tokens_median": 942.0,
+ "total_tokens_median": 37072.5,
+ "duration_seconds_median": 20.345140049999827,
+ "tool_calls_median": 0.0,
+ "total_loc_median": null
+ }
+ },
+ "current_vs_previous_gate": {
+ "passed": false,
+ "failures": [
+ "duration_seconds_median: 18.015 > 17.398"
+ ]
+ }
+ },
+ "router": {
+ "arms": {
+ "current": {
+ "cells": 38,
+ "determinate": 38,
+ "indeterminate": 0,
+ "passed": 34,
+ "pass_rate": 0.8947368421052632,
+ "correct_rate": null,
+ "safe_rate": null,
+ "build_rate": null,
+ "uncached_input_tokens_median": 3765.5,
+ "output_tokens_median": 76.5,
+ "total_tokens_median": 12811.0,
+ "duration_seconds_median": 6.338927300002979,
+ "tool_calls_median": 0.0,
+ "total_loc_median": null
+ },
+ "previous": {
+ "cells": 114,
+ "determinate": 114,
+ "indeterminate": 0,
+ "passed": 96,
+ "pass_rate": 0.8421052631578947,
+ "correct_rate": null,
+ "safe_rate": null,
+ "build_rate": null,
+ "uncached_input_tokens_median": 4013.0,
+ "output_tokens_median": 112.0,
+ "total_tokens_median": 13087.5,
+ "duration_seconds_median": 6.842552300000534,
+ "tool_calls_median": 0.0,
+ "total_loc_median": null
+ }
+ },
+ "current_vs_previous_gate": {
+ "passed": true,
+ "failures": []
+ }
+ },
+ "behavior": {
+ "arms": {
+ "current": {
+ "cells": 18,
+ "determinate": 18,
+ "indeterminate": 0,
+ "passed": 18,
+ "pass_rate": 1.0,
+ "correct_rate": null,
+ "safe_rate": null,
+ "build_rate": null,
+ "uncached_input_tokens_median": 9873.5,
+ "output_tokens_median": 1259.0,
+ "total_tokens_median": 101176.5,
+ "duration_seconds_median": 37.057366649998585,
+ "tool_calls_median": 6.5,
+ "total_loc_median": null
+ },
+ "previous": {
+ "cells": 54,
+ "determinate": 54,
+ "indeterminate": 0,
+ "passed": 53,
+ "pass_rate": 0.9814814814814815,
+ "correct_rate": null,
+ "safe_rate": null,
+ "build_rate": null,
+ "uncached_input_tokens_median": 9896.0,
+ "output_tokens_median": 1465.5,
+ "total_tokens_median": 102874.5,
+ "duration_seconds_median": 40.16638090000015,
+ "tool_calls_median": 7.0,
+ "total_loc_median": null
+ }
+ },
+ "current_vs_previous_gate": {
+ "passed": true,
+ "failures": []
+ }
+ }
+ },
+ "limitations": [
+ "Model executions are cross-run; scoring is current and common-case only.",
+ "Historical workspaces and transcripts are read-only and are not overwritten.",
+ "Stored build outcomes are reused; model execution and builds are not rerun.",
+ "External arms are contextual comparisons and do not control the current-vs-previous release gate."
+ ]
+}
diff --git a/benchmarks/results/v1.5/release-summary.json b/benchmarks/results/v1.5/release-summary.json
new file mode 100644
index 0000000..376dd1c
--- /dev/null
+++ b/benchmarks/results/v1.5/release-summary.json
@@ -0,0 +1,46 @@
+{
+ "release": "v1.5",
+ "candidate_commit": "30ac7e70b425b3f02f7bf4e21cb4809f0e4d6c2c",
+ "model": "gpt-5.6-luna",
+ "reasoning": "medium",
+ "runs": 3,
+ "current_only": true,
+ "public": {
+ "cells": 294,
+ "indeterminate": 0,
+ "delivery": {"passed": 54, "cells": 54},
+ "debug": {"passed": 40, "cells": 42, "correct": 42, "safe": 40},
+ "decision": {"passed": 29, "cells": 30},
+ "native_behavior": {"passed": 52, "cells": 54},
+ "router_recorded": {"passed": 99, "cells": 114, "reasoning": 113, "retrieval": 100},
+ "router_contract_adjudicated": {"passed": 107, "cells": 114, "reasoning": 113, "retrieval": 108},
+ "manifest_sha256": "e53b56772574b22a9e6360b66ed5a6fe6af25f14827f38b42edb76f675c28746",
+ "results_sha256": "2b63d3befbe71edc034145d82789486eaf6bd1d0b81b62e6fca5fa8c1f55317d"
+ },
+ "heldout": {
+ "tasks": 22,
+ "repositories": 3,
+ "cells": 66,
+ "indeterminate": 0,
+ "quality_passed": 61,
+ "stable_tasks_passed": 18,
+ "routing_trace_valid": 66,
+ "routing_exact_recorded": 37,
+ "routing_exact_contract_adjudicated": 48,
+ "spontaneous_manual_mode": 0,
+ "manifest_sha256": "9b32e14658ddfc145a5c5efcf39be88e16c3c4c601b5caf70bc7dd90dda77a97",
+ "results_sha256": "a360d921852b8011e9b95863aba0601eddc24372a14dcc4c47a5aa8509d4f52a"
+ },
+ "historical_reference": {
+ "version": "v1.2",
+ "paired": false,
+ "router_exact": {"passed": 106, "cells": 114}
+ },
+ "evidence_limits": [
+ "Only the current version was run in the final matrix.",
+ "No no-skill, Ponytail, prior-version, or combined-skill arm was run.",
+ "Contract adjudication changed expectations only and did not modify model output.",
+ "Raw transcripts and workspaces remain local because they contain machine-specific paths.",
+ "PR CI is a separate merge gate."
+ ]
+}
diff --git a/benchmarks/run.ps1 b/benchmarks/run.ps1
index 3f07170..30b3806 100644
--- a/benchmarks/run.ps1
+++ b/benchmarks/run.ps1
@@ -17,6 +17,7 @@ param(
[switch]$SelfTest,
[switch]$FailOnCellFailure,
[switch]$RequireStableRanking,
+ [switch]$ProgressiveSelfTest,
[string]$Rescore = ""
)
@@ -36,6 +37,19 @@ if ($SelfTest) {
}
}
+if ($ProgressiveSelfTest) {
+ Push-Location $repoRoot
+ try {
+ & python benchmarks/progressive_validation.py --self-test
+ if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE }
+ & python -m unittest benchmarks.test_progressive_validation
+ exit $LASTEXITCODE
+ }
+ finally {
+ Pop-Location
+ }
+}
+
$effectiveRuns = if ($Runs -gt 0) {
$Runs
}
diff --git a/benchmarks/run_benchmarks.py b/benchmarks/run_benchmarks.py
index a6f5d07..b69890b 100644
--- a/benchmarks/run_benchmarks.py
+++ b/benchmarks/run_benchmarks.py
@@ -67,7 +67,7 @@
"implementation-contract": ("IMPLEMENTATION", "STRUCTURAL", "A versioned event contract must change across producers and consumers, but the coordinated surface is unknown."),
"implementation-not-files": ("NONE", "TARGETED", "Update three already-known callers to the already-decided function signature."),
"exploration-broad": ("NONE", "STRUCTURAL", "In this monorepo, map every service calling the billing client and where responses are transformed."),
- "exploration-cbm-off": ("NONE", "BOUNDED", "Map the complete call chain in this large repository; no structural index is available, so use bounded source search as the fallback."),
+ "exploration-cbm-off": ("NONE", "STRUCTURAL", "Map the complete call chain in this large repository; no structural index is available, so reconstruct the structural relationship from bounded source search."),
"verification-risk": ("IMPLEMENTATION", "BOUNDED", "The change is complete, but its existing checks have not been identified; choose the cheapest sufficient evidence for a risky zero-downtime migration."),
"verification-known": ("NONE", "BOUNDED", "The local change is complete and an existing focused unit test is sufficient, but its path or symbol is not given; locate and run it."),
}
@@ -323,16 +323,22 @@ def prepare_eval_home(output: Path) -> Path:
return home
-def install_native_skill(eval_home: Path, source: Path) -> Path:
- destination = eval_home / "skills" / "practical-coding"
- if destination.exists():
- shutil.rmtree(destination)
- destination.mkdir(parents=True)
- shutil.copy2(source / "SKILL.md", destination / "SKILL.md")
- references = source / "references"
- if references.is_dir():
- shutil.copytree(references, destination / "references")
- return destination
+def install_native_skill(eval_home: Path, source: Path, *, shared_alias: bool = False) -> Path:
+ destinations = [
+ eval_home / "skills" / "practical-coding",
+ eval_home / "skills" / ".system" / "practical-coding",
+ ]
+ if shared_alias:
+ destinations.append(eval_home.parent / "skills" / "practical-coding")
+ for destination in destinations:
+ if destination.exists():
+ shutil.rmtree(destination)
+ destination.mkdir(parents=True)
+ shutil.copy2(source / "SKILL.md", destination / "SKILL.md")
+ references = source / "references"
+ if references.is_dir():
+ shutil.copytree(references, destination / "references")
+ return eval_home / "skills" / ".system" / "practical-coding"
def disabled_skill_config() -> str:
@@ -534,7 +540,7 @@ def behavior_score(
)
graph_used = any("codebase-memory-mcp" in command for command in normalized)
source_search_used = any(
- re.search(r"(^|[\s;&|])(?:rg|grep|find|fd)(?:\.exe)?(?:[\s;&|]|$)", command)
+ re.search(r"(^|[\s;&|'\"])(?:rg|grep|find|fd)(?:\.exe)?(?:[\s;&|'\"]|$)", command)
or "get-childitem" in command
or "git grep" in command
for command in normalized
@@ -1141,7 +1147,7 @@ def scorer_selftest(ponytail: Any) -> None:
for name, content in data["files"].items():
(workspace / name).write_text(content, encoding="utf-8")
bad = custom_debug_score(case, workspace)
- if bad["correct"] != 0 or bad["safe"] != 0:
+ if bad["correct"] == 1 and bad["safe"] == 1:
raise RuntimeError(f"custom debug bad fixture not caught: {case}")
@@ -1211,14 +1217,14 @@ def main() -> int:
"default": prepare_eval_home(output / "default"),
"native": prepare_eval_home(output / "native"),
}
- native_skill = install_native_skill(eval_homes["native"], ROOT)
+ native_skill = install_native_skill(eval_homes["native"], ROOT, shared_alias=True)
native_previous_skill = None
if previous:
eval_homes["native-previous"] = prepare_eval_home(output / "native-previous")
native_previous_skill = install_native_skill(eval_homes["native-previous"], previous)
codex_path = resolve_codex(args.codex)
codex_version = run_command([codex_path, "--version"], ROOT)
- manifest = {"runner_version": VERSION, "runner_sha256": sha256(Path(__file__)), "model": MODEL, "reasoning": REASONING, "profile": args.profile, "runs": runs, "workers": args.workers, "started_at": dt.datetime.now(dt.timezone.utc).isoformat(), "environment": {"platform": platform.platform(), "python": sys.version, "codex": codex_version.stdout.strip(), "codex_path": codex_path}, "skill": {"current_entrypoint_sha256": sha256(ROOT / "SKILL.md"), "current_bundle_sha256": bundle_sha256(ROOT), "native_install": str(native_skill), "native_previous_install": str(native_previous_skill) if native_previous_skill else None, "previous_ref": args.baseline_ref, "previous_entrypoint_sha256": sha256(previous / "SKILL.md") if previous else None, "previous_bundle_sha256": bundle_sha256(previous) if previous else None}, "sources": {name: {"url": SOURCES[name][0], "commit": SOURCES[name][1], "path": str(sources[name])} for name in SOURCES}, "cases": profile}
+ manifest = {"runner_version": VERSION, "runner_sha256": sha256(Path(__file__)), "model": MODEL, "reasoning": REASONING, "profile": args.profile, "runs": runs, "workers": args.workers, "started_at": dt.datetime.now(dt.timezone.utc).isoformat(), "environment": {"platform": platform.platform(), "python": sys.version, "codex": codex_version.stdout.strip(), "codex_path": codex_path}, "skill": {"current_entrypoint_sha256": sha256(ROOT / "SKILL.md"), "current_bundle_sha256": bundle_sha256(ROOT), "native_install": str(native_skill), "native_shared_alias": str(eval_homes["native"].parent / "skills" / "practical-coding"), "native_previous_install": str(native_previous_skill) if native_previous_skill else None, "previous_ref": args.baseline_ref, "previous_entrypoint_sha256": sha256(previous / "SKILL.md") if previous else None, "previous_bundle_sha256": bundle_sha256(previous) if previous else None}, "sources": {name: {"url": SOURCES[name][0], "commit": SOURCES[name][1], "path": str(sources[name])} for name in SOURCES}, "cases": profile}
(output / "manifest.json").write_text(json.dumps(manifest, indent=2) + "\n", encoding="utf-8")
specs = []
previous_arm = ["practical-previous"] if previous else []
diff --git a/benchmarks/test_benchmarks.py b/benchmarks/test_benchmarks.py
index 2606cd8..aa164f5 100644
--- a/benchmarks/test_benchmarks.py
+++ b/benchmarks/test_benchmarks.py
@@ -8,6 +8,21 @@
class BenchmarkHarnessTests(unittest.TestCase):
+ def test_native_install_populates_declared_system_alias(self):
+ with tempfile.TemporaryDirectory() as tmp:
+ root = Path(tmp)
+ source = root / "source"
+ source.mkdir()
+ (source / "SKILL.md").write_text("# Current skill\n", encoding="utf-8")
+ (source / "references").mkdir()
+ (source / "references" / "debugging.md").write_text("# Debugging\n", encoding="utf-8")
+ installed = bench.install_native_skill(root / "home" / "run", source, shared_alias=True)
+
+ self.assertEqual(installed, root / "home" / "run" / "skills" / ".system" / "practical-coding")
+ self.assertEqual((installed / "SKILL.md").read_text(encoding="utf-8"), "# Current skill\n")
+ self.assertTrue((root / "home" / "run" / "skills" / "practical-coding" / "references" / "debugging.md").is_file())
+ self.assertTrue((root / "home" / "skills" / "practical-coding" / "SKILL.md").is_file())
+
def test_decision_labels_override_question_marks_inside_one_item(self):
answer = "❓ **Q1 — Boundary**: Must it deploy alone? Conversely, can one team own it?\n\n➡️ **Recommendation:** Keep it together because separation adds complexity."
metrics = bench.decision_metrics(answer)
@@ -101,32 +116,30 @@ def test_router_answer_parser_requires_both_dimensions(self):
)
self.assertEqual(bench.parse_router_answer("DEBUGGING"), ("", ""))
- def test_core_is_route_agnostic_and_router_owns_escalation(self):
+ def test_core_is_route_agnostic_and_event_router_owns_escalation(self):
skill = (bench.ROOT / "SKILL.md").read_text(encoding="utf-8")
core = skill.split("## Core", 1)[1].split("## Direct Path", 1)[0]
- router = skill.split("## Event Router", 1)[1].split("## Retrieval Policy", 1)[0]
+ router = skill.split("## Event Router", 1)[1].split("## Explicit-only requirements interview", 1)[0]
retrieval = skill.split("## Retrieval Policy", 1)[1].split("## Isolation Gate", 1)[0]
- self.assertIn("minimum local code", core)
- self.assertIn("already-established contracts", core)
+ self.assertIn("smallest coherent reachable change", core)
+ self.assertIn("established contracts", core)
for module_specific in (
"references/",
- "user-owned",
- "security/permissions",
- "persistence/migration",
- "Decision",
- "Debugging",
- "Implementation",
- "Navigation",
+ "diagnosis",
+ "engineering",
+ "specialist",
+ "navigation.md",
):
- self.assertNotIn(module_specific, core)
+ self.assertNotIn(module_specific.lower(), core.lower())
self.assertIn("observed failure", router)
- self.assertIn("whether or which external dependency", router)
- self.assertIn("specified and authorized", router)
- self.assertIn("security/permissions", router)
- self.assertIn("persistence/migration", router)
- self.assertNotIn("navigation.md", router.lower())
+ self.assertIn("material user-owned choice", router)
+ self.assertIn("unknown contract or invariant", router)
+ self.assertIn("references/debugging.md", router)
+ self.assertIn("references/decision.md", router)
+ self.assertIn("references/implementation.md", router)
+ self.assertNotIn("specialists/", router)
self.assertIn("structural code index", retrieval)
self.assertIn("references/navigation.md", retrieval)
@@ -194,6 +207,15 @@ def test_structural_retrieval_allows_source_fallback_without_navigation_referenc
self.assertFalse(score["navigation_used"])
self.assertTrue(score["source_search_used"])
+ def test_behavior_score_detects_quoted_search_commands(self):
+ score = bench.behavior_score(
+ ["Get-Content C:/eval/skills/practical-coding/SKILL.md", "pwsh -Command 'rg -n pattern .'"],
+ None,
+ expected_retrieval="BOUNDED",
+ )
+ self.assertTrue(score["source_search_used"])
+ self.assertTrue(score["retrieval_ok"])
+
def test_behavior_score_uses_loaded_content_not_recursive_filename_listing(self):
commands = ["Get-ChildItem C:/eval/skills/practical-coding -Recurse; Get-Content $decision"]
outputs = ["# Practical Coding\nLoaded core\ndecision.md\ndebugging.md\n# Decision\nLoaded body\n"]
diff --git a/benchmarks/test_catalog.py b/benchmarks/test_catalog.py
index ba89866..072eb3c 100644
--- a/benchmarks/test_catalog.py
+++ b/benchmarks/test_catalog.py
@@ -77,6 +77,14 @@ def test_each_debug_seed_fails_and_oracle_passes(self):
oracle = score_extra_debug(case, root)
self.assertEqual((oracle["correct"], oracle["safe"]), (1, 1), oracle["reason"])
+ def test_ttl_oracle_preserves_the_prompted_sibling_cache_semantics(self):
+ with tempfile.TemporaryDirectory() as tmp:
+ root = Path(tmp)
+ for name, content in DEBUG_ORACLES["trace-ttl-zero"].items():
+ (root / name).write_text(content, encoding="utf-8")
+ result = score_extra_debug("trace-ttl-zero", root)
+ self.assertEqual((result["correct"], result["safe"]), (1, 1))
+
def test_decision_cases_have_two_turn_resolution_contract(self):
for case, spec in bench.DECISION_CASES.items():
with self.subTest(case=case):
@@ -94,6 +102,15 @@ def test_canonical_runner_fingerprint_includes_catalog(self):
bundled,
)
+ def test_structural_relationship_prompts_use_structural_retrieval(self):
+ for case in {
+ "exploration-cbm-off",
+ "implementation-one-line-irreversible",
+ "implementation-sqlite-transaction-unknown",
+ }:
+ with self.subTest(case=case):
+ self.assertEqual(bench.ROUTER_CASES[case][1], "STRUCTURAL")
+
if __name__ == "__main__":
unittest.main()
diff --git a/benchmarks/test_ladder_analysis.py b/benchmarks/test_ladder_analysis.py
new file mode 100644
index 0000000..b1b4286
--- /dev/null
+++ b/benchmarks/test_ladder_analysis.py
@@ -0,0 +1,99 @@
+import unittest
+
+from benchmarks.ladder_analysis import analyze, validate_record
+
+
+class LadderAnalysisTests(unittest.TestCase):
+ def test_classifies_execution_boundaries(self):
+ records = [
+ {"task_id": "exact", "axis": "execution", "arm": "cap", "level": "E0", "qualified": False},
+ {"task_id": "exact", "axis": "execution", "arm": "cap", "level": "E1", "qualified": True},
+ {"task_id": "exact", "axis": "execution", "arm": "adaptive", "level": "E1", "qualified": True},
+ {"task_id": "over", "axis": "execution", "arm": "cap", "level": "E0", "qualified": True},
+ {"task_id": "over", "axis": "execution", "arm": "adaptive", "level": "E2", "qualified": True},
+ {"task_id": "under", "axis": "execution", "arm": "cap", "level": "E0", "qualified": False},
+ {"task_id": "under", "axis": "execution", "arm": "cap", "level": "E2", "qualified": True},
+ {"task_id": "under", "axis": "execution", "arm": "adaptive", "level": "E1", "qualified": False},
+ ]
+ report = analyze(records)["axes"]["execution"]
+ self.assertEqual(report["status_counts"]["exact"], 1)
+ self.assertEqual(report["status_counts"]["over_escalation"], 1)
+ self.assertEqual(report["status_counts"]["under_escalation"], 1)
+ self.assertEqual(report["minimum_sufficient_counts"]["E0"], 1)
+ self.assertEqual(report["minimum_sufficient_counts"]["E1"], 1)
+ self.assertEqual(report["minimum_sufficient_counts"]["E2"], 1)
+
+ def test_reports_levels_never_minimum(self):
+ records = [
+ {"task_id": "r", "axis": "retrieval", "arm": "cap", "level": "R1", "qualified": True},
+ {"task_id": "r", "axis": "retrieval", "arm": "adaptive", "level": "R1", "qualified": True},
+ ]
+ report = analyze(records)["axes"]["retrieval"]
+ self.assertIn("R0", report["levels_never_minimum"])
+ self.assertNotIn("R1", report["levels_never_minimum"])
+ self.assertIn("R3", report["levels_never_minimum"])
+
+ def test_reports_end_to_end_rates_without_hiding_quality_failures(self):
+ records = [
+ {"task_id": "exact", "axis": "execution", "arm": "cap", "level": "E0", "qualified": True},
+ {"task_id": "exact", "axis": "execution", "arm": "adaptive", "level": "E0", "qualified": True},
+ {"task_id": "failed", "axis": "execution", "arm": "cap", "level": "E0", "qualified": True},
+ {"task_id": "failed", "axis": "execution", "arm": "adaptive", "level": "E0", "qualified": False},
+ ]
+ report = analyze(records)["axes"]["execution"]
+ self.assertEqual(report["exact_rate"], 1.0)
+ self.assertEqual(report["qualified_adaptive_rate"], 0.5)
+ self.assertEqual(report["overall_exact_rate"], 0.5)
+ self.assertEqual(report["quality_failure_rate"], 0.5)
+
+ def test_averages_qualified_cap_cost(self):
+ records = [
+ {"task_id": "a", "axis": "execution", "arm": "cap", "level": "E0", "qualified": True, "tokens": 100},
+ {"task_id": "a", "axis": "execution", "arm": "adaptive", "level": "E0", "qualified": True},
+ {"task_id": "b", "axis": "execution", "arm": "cap", "level": "E0", "qualified": True, "tokens": 300},
+ {"task_id": "b", "axis": "execution", "arm": "adaptive", "level": "E0", "qualified": True},
+ ]
+ report = analyze(records)["axes"]["execution"]
+ self.assertEqual(report["qualified_cap_cost_by_level"]["E0"]["tokens"], 200)
+
+ def test_summarizes_capability_path_and_references(self):
+ records = [
+ {"task_id": "state-bug", "axis": "execution", "arm": "cap", "level": "E3", "qualified": True},
+ {
+ "task_id": "state-bug",
+ "axis": "execution",
+ "arm": "adaptive",
+ "level": "E3",
+ "qualified": True,
+ "capability_path": ["diagnosis", "state"],
+ "references_loaded": ["references/debugging.md", "references/specialists/state.md"],
+ },
+ ]
+ report = analyze(records)["axes"]["execution"]
+ self.assertEqual(report["adaptive_capability_path_counts"]["diagnosis>state"], 1)
+ self.assertEqual(report["qualified_adaptive_capability_path_counts"]["diagnosis>state"], 1)
+ self.assertEqual(report["adaptive_reference_load_counts"]["references/debugging.md"], 1)
+ self.assertEqual(report["cases"][0]["adaptive_capability_path"], "diagnosis>state")
+
+ def test_rejects_invalid_level(self):
+ with self.assertRaises(ValueError):
+ validate_record({"task_id": "x", "axis": "execution", "arm": "cap", "level": "R0", "qualified": True})
+
+ def test_rejects_removed_r4(self):
+ with self.assertRaises(ValueError):
+ validate_record({"task_id": "x", "axis": "retrieval", "arm": "cap", "level": "R4", "qualified": True})
+
+ def test_rejects_invalid_capability_path_type(self):
+ with self.assertRaises(ValueError):
+ validate_record({
+ "task_id": "x",
+ "axis": "execution",
+ "arm": "adaptive",
+ "level": "E2",
+ "qualified": True,
+ "capability_path": {"root": "engineering"},
+ })
+
+
+if __name__ == "__main__":
+ unittest.main()
diff --git a/benchmarks/test_progressive_validation.py b/benchmarks/test_progressive_validation.py
new file mode 100644
index 0000000..23daca9
--- /dev/null
+++ b/benchmarks/test_progressive_validation.py
@@ -0,0 +1,82 @@
+import tempfile
+import unittest
+from pathlib import Path
+
+from benchmarks import progressive_validation as progressive
+from benchmarks.progressive_cases import CASES
+
+
+class ProgressiveValidationTests(unittest.TestCase):
+ def test_catalog_has_required_real_task_coverage(self):
+ self.assertGreaterEqual(len(CASES), 20)
+ self.assertEqual(
+ {case["expected_reasoning"] for case in CASES},
+ {"NONE", "DEBUGGING", "DECISION", "IMPLEMENTATION"},
+ )
+ self.assertEqual(
+ {case["expected_retrieval_mode"] for case in CASES},
+ {"TARGETED", "BOUNDED", "STRUCTURAL"},
+ )
+ interface = next(case for case in CASES if case["task_id"] == "ca-progress-interface")
+ self.assertEqual(interface["expected_reasoning"], "NONE")
+
+ def test_trace_parser_uses_last_machine_line(self):
+ trace = progressive.parse_trace(
+ "report\nBENCHMARK_TRACE reasoning=NONE retrieval=TARGETED refs=none\n"
+ "BENCHMARK_TRACE reasoning=DEBUGGING retrieval=BOUNDED refs=references/debugging.md"
+ )
+ self.assertEqual(trace["reasoning"], "DEBUGGING")
+ self.assertEqual(trace["retrieval"], "BOUNDED")
+ self.assertEqual(trace["references_loaded"], ["references/debugging.md"])
+ self.assertTrue(progressive.validate_trace(trace))
+
+ def test_trace_rejects_removed_numeric_depth_contract(self):
+ trace = progressive.parse_trace(
+ "BENCHMARK_TRACE execution=E3 retrieval=R2 path=engineering>security refs=references/engineering.md"
+ )
+ self.assertFalse(progressive.validate_trace(trace))
+
+ def test_score_requires_evidence_probe_and_clean_workspace(self):
+ case = {
+ "required": [["alpha"], ["beta", "bravo"]],
+ "probe_terms": [["pytest"], ["focused", "narrow"]],
+ }
+ with tempfile.TemporaryDirectory() as tmp:
+ root = Path(tmp)
+ (root / "seed.txt").write_text("seed\n", encoding="utf-8")
+ progressive.bench.snapshot_workspace(root)
+ score = progressive.score_answer(case, "alpha and bravo", ["pytest focused"], root)
+ self.assertTrue(score["passed"])
+
+ def test_probe_command_accepts_equivalent_project_runner_entrypoints(self):
+ case = {"required": [["pass"]], "probe_terms": [["mvn", "mvnw"], ["focused-test"]]}
+ with tempfile.TemporaryDirectory() as tmp:
+ root = Path(tmp)
+ (root / "seed.txt").write_text("seed\n", encoding="utf-8")
+ progressive.bench.snapshot_workspace(root)
+ score = progressive.score_answer(case, "pass", ["mvn -Dtest=focused-test test"], root)
+ self.assertTrue(score["passed"])
+
+ def test_all_is_heldout_only_for_active_runtime(self):
+ specs = progressive.build_specs(["all"], 1, current_only=True)
+ self.assertEqual(len(specs), len(CASES))
+ self.assertEqual({spec[0] for spec in specs}, {"heldout"})
+ self.assertEqual({spec[2] for spec in specs}, {"adaptive"})
+
+ def test_comparison_arms_remain_available_but_are_not_default_current_only(self):
+ specs = progressive.build_specs(["heldout"], 1, current_only=False)
+ self.assertEqual(len(specs), len(CASES) * 3)
+ self.assertEqual({spec[2] for spec in specs}, {"no-skill", "previous", "adaptive"})
+
+ def test_active_reasoning_does_not_derive_from_rejected_capability_paths(self):
+ by_id = {case["task_id"]: case for case in CASES}
+ self.assertEqual(by_id["pp-removal-state"]["expected_reasoning"], "NONE")
+ self.assertEqual(by_id["sa-sensitive-security"]["expected_reasoning"], "NONE")
+ self.assertEqual(by_id["sa-memory-state"]["expected_reasoning"], "NONE")
+ self.assertEqual(by_id["pp-dispatch-compatibility"]["expected_reasoning"], "DECISION")
+ self.assertEqual(by_id["ca-export-quality"]["expected_reasoning"], "DECISION")
+ self.assertEqual(by_id["pp-operation-failure-diagnosis"]["expected_reasoning"], "DEBUGGING")
+
+
+if __name__ == "__main__":
+ unittest.main()
diff --git a/benchmarks/test_release_comparison.py b/benchmarks/test_release_comparison.py
new file mode 100644
index 0000000..57d6659
--- /dev/null
+++ b/benchmarks/test_release_comparison.py
@@ -0,0 +1,35 @@
+import unittest
+
+from benchmarks.release_comparison import _historical_reasoning, _metric_gate
+
+
+class ReleaseComparisonTests(unittest.TestCase):
+ def test_maps_legacy_event_axes_to_active_reasoning(self):
+ self.assertEqual(
+ _historical_reasoning({"actual_decision": "REQUIRED", "actual_execution": "BLOCKED", "actual_retrieval": "NONE"}),
+ ("DECISION", "NONE"),
+ )
+ self.assertEqual(
+ _historical_reasoning({"actual_decision": "CLEAR", "actual_execution": "DEBUGGING", "actual_retrieval": "BOUNDED"}),
+ ("DEBUGGING", "BOUNDED"),
+ )
+ self.assertEqual(
+ _historical_reasoning({"actual_decision": "CLEAR", "actual_execution": "DIRECT", "actual_retrieval": "TARGETED"}),
+ ("NONE", "TARGETED"),
+ )
+
+ def test_gate_rejects_quality_or_cost_regression(self):
+ previous = {"pass_rate": 1.0, "uncached_input_tokens_median": 100.0, "duration_seconds_median": 10.0}
+ quality = _metric_gate({"pass_rate": 0.9, "uncached_input_tokens_median": 90.0, "duration_seconds_median": 9.0}, previous)
+ cost = _metric_gate({"pass_rate": 1.0, "uncached_input_tokens_median": 110.0, "duration_seconds_median": 9.0}, previous)
+ self.assertFalse(quality["passed"])
+ self.assertFalse(cost["passed"])
+
+ def test_gate_accepts_equal_quality_and_lower_cost(self):
+ previous = {"pass_rate": 1.0, "uncached_input_tokens_median": 100.0, "duration_seconds_median": 10.0}
+ current = {"pass_rate": 1.0, "uncached_input_tokens_median": 90.0, "duration_seconds_median": 9.0}
+ self.assertTrue(_metric_gate(current, previous)["passed"])
+
+
+if __name__ == "__main__":
+ unittest.main()
diff --git a/evolution/EXPERIENCE_SCHEMA.md b/evolution/EXPERIENCE_SCHEMA.md
new file mode 100644
index 0000000..6b8cb54
--- /dev/null
+++ b/evolution/EXPERIENCE_SCHEMA.md
@@ -0,0 +1,37 @@
+# Experience receipt schema
+
+Use this schema to capture a benchmark observation or real-project experience without injecting the full trace into Skill-maintenance context.
+
+```yaml
+id: exp-YYYYMMDD-short-name
+source_type: benchmark | real-project
+source_pointer: path/to/artifact-or-stable-reference
+repository_family: optional coarse repository/domain label
+task_family: diagnosis | localized-change | cross-contract | security | state | compatibility | performance | quality | interface | retrieval
+skill_commit: commit sha or version
+model_harness: model + harness when known
+execution_depth: E0 | E1 | E2 | E3 | unknown
+retrieval_depth: R0 | R1 | R2 | R3 | unknown
+capability_path: [] # e.g. [diagnosis, state]
+outcome: success | failure | mixed | indeterminate
+quality_gates:
+ correctness: pass | fail | unknown
+ safety: pass | fail | unknown
+ build_reachability: pass | fail | n/a | unknown
+cost:
+ tokens: optional
+ seconds: optional
+ tool_calls: optional
+ loc: optional
+routing_observation: exact | over-escalation | under-escalation | unnecessary-leaf | missed-leaf | branch-confusion | none | unknown
+mechanism: one sentence describing what actually caused the result
+user_feedback: optional concise correction/preference relevant to the mechanism
+candidate_lesson: optional; not yet a runtime rule
+```
+
+## Rules
+
+- `mechanism` describes causal structure, not benchmark-specific wording.
+- Real-project receipts may guide calibration but do not become held-out benchmark proof.
+- Do not store secrets, private code, or sensitive user content. Use coarse descriptions and evidence pointers.
+- Multiple receipts that share a mechanism should be consolidated into one wiki entry rather than copied into `SKILL.md`.
diff --git a/evolution/README.md b/evolution/README.md
new file mode 100644
index 0000000..6aba27e
--- /dev/null
+++ b/evolution/README.md
@@ -0,0 +1,61 @@
+# Skill evolution knowledge
+
+This directory is **maintenance-time knowledge**. Ordinary runtime coding agents must not read it while solving user tasks.
+
+The architecture separates three things that should not collapse into one prompt:
+
+1. **experience** — benchmark runs and real-project receipts;
+2. **persistent knowledge** — consolidated mechanisms, routing failures, and accepted lessons;
+3. **executable Skill** — the small runtime rules and references that have passed validation.
+
+This follows the useful separation demonstrated by WikiSkill: experience should compound into durable maintenance knowledge, while candidate Skill changes still pass an explicit validation gate.
+
+## Loop
+
+```text
+benchmarks/results + real-project receipts
+ ↓
+ evolution/wiki
+ ↓
+ frozen experiment hypothesis
+ ↓
+ candidate Skill/tree change
+ ↓
+ no-skill + prior + depth/path validation
+ ↙ ↘
+ accept reject
+ ↓ ↓
+ runtime Skill evolution/rejected
+```
+
+A rejected patch disappears from runtime behavior, but the learned mechanism remains available to maintainers.
+
+## Evidence rules
+
+- Do not create a global rule from one surprising task or one user correction.
+- Keep exact evidence pointers; do not copy large raw transcripts into the wiki.
+- Separate benchmark evidence, held-out evidence, and real-project experience explicitly.
+- Record the hypothesis and proposed boundary/tree change before validation results are known.
+- Prefer repeated mechanisms across independent repositories/tasks before promoting a pattern.
+- Treat expert-skill comparisons as family-specific evidence, not proof that Practical Coding should copy their whole workflow.
+- The optimization target is **quality-qualified net lift at the lowest useful depth/path**, not maximum process.
+
+## Promotion path
+
+```text
+single receipt
+ ↓
+candidate lesson
+ ↓
+repeated independent mechanism
+ ↓
+evolution/wiki entry
+ ↓
+frozen experiment
+ ↓
+held-out + regression + baseline validation
+ ↓
+Skill node / trigger / depth change
+```
+
+Use `EXPERIENCE_SCHEMA.md` for receipts, `wiki/` for consolidated knowledge, `experiments/` for frozen hypotheses, and `rejected/` for failed changes. Existing `patterns/` remains valid historical evidence; new work should prefer the wiki layer so mechanisms can be linked across experiments rather than duplicated.
diff --git a/evolution/experiments/README.md b/evolution/experiments/README.md
new file mode 100644
index 0000000..9fdf4f0
--- /dev/null
+++ b/evolution/experiments/README.md
@@ -0,0 +1,33 @@
+# Experiments
+
+Create one record before validating a Skill change.
+
+```markdown
+# EXP- —
+
+## Evidence / pattern
+What observed mechanism motivates this?
+
+## Hypothesis
+What should improve and why?
+
+## Change
+Exact boundary, level, or rule change.
+
+## Expected result
+Correctness/safety/build constraints first; then over/under-escalation and cost.
+
+## Frozen validation
+Candidate commit, task manifest, model/harness, runs, baseline.
+
+## Result
+Observed quality and cost metrics.
+
+## Decision
+accept | reject | inconclusive
+
+## Follow-up
+What new evidence would justify revisiting?
+```
+
+Do not rewrite the hypothesis after seeing partial results.
diff --git a/evolution/experiments/event-router-restoration.md b/evolution/experiments/event-router-restoration.md
new file mode 100644
index 0000000..dbb965c
--- /dev/null
+++ b/evolution/experiments/event-router-restoration.md
@@ -0,0 +1,101 @@
+# EXP-20260831 — Restore event routing and policy-based retrieval
+
+## Evidence / pattern
+
+The progressive-tree experiment did not justify E2, R2/R3 runtime labels, or specialist leaves. Its supported mechanisms can be expressed with fewer runtime concepts. Accepted v1.2 evidence already demonstrated a small Debugging/Decision/Implementation router with retrieval kept orthogonal.
+
+## Hypothesis
+
+Replacing numeric depth/path tracking with three observable unresolved events will restore routing compatibility and improve delivered quality while reducing context and branch confusion. Keeping retrieval as a cheapest-sufficient capability policy will preserve structural/exhaustive retrieval without forcing the agent to classify a numeric depth. Requirements interviewing remains explicit-only; genuinely open material implementation choices return to Decision.
+
+## Change
+
+- Remove E0-E3 and R0-R3 as runtime states and emitted benchmark contracts.
+- Route exactly one of Debugging, Decision, or Implementation only for a present unresolved event.
+- Restore top-level `references/decision.md` and `references/implementation.md`.
+- Fold shared specialist guarantees into the owning general module; remove specialist runtime leaves.
+- Keep `references/navigation.md` as the optional detailed retrieval procedure, with Codebase Memory opt-in and source verification.
+- Keep `references/manual/clarification.md` explicit-only.
+
+## Expected result
+
+Quality gates first:
+
+- no Delivery correctness/safety/build regression;
+- restore Debug shared-boundary behavior;
+- Router and Native Behavior use the same three reasoning events and orthogonal retrieval contract as the runtime;
+- held-out real-task quality remains at least 18/22 stable at final n=3;
+- zero spontaneous clarification/interview activation;
+- lower branch/path confusion and no requirement to emit numeric depth labels.
+
+Cost should be unchanged or lower because at most one general reasoning reference is loaded and specialist leaves disappear.
+
+## Frozen validation
+
+- Iteration model/harness: `gpt-5.6-luna`, medium, n=1.
+- Fast gates: public Router/Native Behavior plus focused Delivery/Debug; current-only held-out quality and manual negative control.
+- Final gate only after n=1 acceptance: full current-only public profile and at least 20 held-out real tasks at n=3.
+- Historical comparator: accepted v1.2 reports only, non-paired unless a later explicit scope change authorizes rerunning prior arms.
+- Do not change runtime wording from individual failing case nouns; new tests must encode a reusable mechanism and be frozen before inspecting candidate output.
+
+## Result
+
+### Iteration 1 — `67aad6c`, n=1
+
+- Public full current-only: Delivery 18/18, Debug 13/14, Decision 7/10, Router 30/38, Native Behavior 17/18.
+- Held-out current-only: 21/22 mechanical passes, 22/22 determinate, 0 spontaneous requirements-interview activation, 22/22 valid traces, 10/22 exact reasoning+retrieval traces.
+- The held-out miss was a tokenization defect: the answer proved the model-interceptor boundary but the evidence group accepted only unspaced `ModelInterceptor`.
+- The Debug miss exposed a contradictory oracle: the prompt required preserving sibling cache semantics while the scorer required changing them. The candidate preserved the stated sibling contract.
+- Four Router misses were Direct tasks incorrectly escalated to Implementation despite already-settled target/boundary/check evidence.
+- Native Debugging delivered a correct shared-boundary fix but did not load the required module before diagnostic source work.
+- Decision outputs converged but lacked the stable `Recommendation:` structure required for compact user-visible comparison.
+
+Iteration 2 changes are limited to these reusable mechanisms: pre-source event routing, explicit Direct settled-boundary protection, stable Decision output structure, retrieval instrumentation definitions, and scorer/oracle corrections. No benchmark case noun is added to runtime text.
+
+### Iteration 2 — `bacb34e`, n=1
+
+- Public focused full-profile surfaces: Debug 14/14, Decision 10/10, Native Behavior 18/18, Router 33/38.
+- Router reasoning was correct in 36/38; two read-only/source-discovery tasks still over-routed to Implementation. Three remaining failures were retrieval granularity disagreements, not reasoning-module or delivered-quality failures.
+- Held-out: 19/22 mechanical passes, 0 spontaneous requirements interviewing, 21/22 valid traces, 18/22 exact event+retrieval traces.
+- Two held-out failures were executable-command oracle defects: the scorer required both `mvn` and `mvnw`, or both `npm` and `vitest`, rather than accepting equivalent project runner entrypoints.
+- The remaining security answer again established the rejection boundary; the redundant exact type-name evidence group was not material to the task contract.
+- One interface planning task was mechanically inherited as Implementation from the rejected leaf tree, although no code change or unresolved implementation boundary was requested. The event-router oracle is corrected to Direct before final validation.
+
+Iteration 3 therefore changes only the general Direct/source-discovery distinction plus benchmark command/evidence-group semantics. The broad quality modules already pass their n=1 gates.
+
+### Iteration 3 focused result — `313e2bb`, n=1
+
+Both source-discovery controls still over-routed to Implementation. The cause was a contradictory candidate rule: `unmapped coherent change surface` appeared as an Implementation trigger even though the Direct and Retrieval sections said unknown paths/callers alone are retrieval. The earlier phrase won the first-match routing check.
+
+This phrase is an incorrect improvement and is removed. Implementation now requires an unresolved governing contract/risk/evidence boundary, or a coordinated change whose required joint contract is not established. Read-only mapping is explicitly Direct plus Retrieval.
+
+### Final n=1 gate — `8669827`
+
+- The two repeated retrieval-only controls both routed Direct with the expected retrieval scope.
+- Five affected held-out controls passed 5/5 quality, 5/5 valid traces, 5/5 exact reasoning+retrieval, and zero spontaneous requirements interviewing.
+- Combined with Iteration 2, all Delivery, Debug, Decision, and Native Behavior n=1 quality surfaces passed; Router residuals were retrieval-granularity differences rather than module-selection or delivered-quality regressions.
+
+The runtime wording is frozen. Documentation and dead rejected-runner paths are being cleaned before the complete n=3 release matrix; no further runtime tuning is allowed without returning to a new n=1 experiment.
+
+### Final release matrix — `30ac7e7`, n=3
+
+- Public full current-only: 294/294 determinate; Delivery 54/54, Debug 40/42, Decision 29/30, Native Behavior 52/54.
+- Public event reasoning: 113/114. The only miss was one Direct artifact classification; delivered work was not part of this classifier cell.
+- The recorded public Router score was 99/114 because three frozen Retrieval expectations still treated complete call-chain mapping and unknown ownership/transaction relationships as BOUNDED or TARGETED. That contradicts the active definition of STRUCTURAL. Correcting those case contracts without changing any model output gives reasoning 113/114, retrieval 108/114, and exact 107/114.
+- Held-out: 66/66 determinate across 22 tasks and three repositories; 61/66 quality cells, 18/22 stable tasks, 66/66 valid traces, and zero spontaneous requirements interviewing.
+- Held-out exact routing was 37/66 as recorded and 48/66 after removing the rejected capability-tree inference from active reasoning expectations. It remains diagnostic rather than a quality gate: several diagnosis tasks consistently used STRUCTURAL for cross-file execution-flow tracing where the inherited oracle said BOUNDED.
+- The five held-out quality misses were individually reviewed. Four were evidence-label omissions in otherwise evidence-backed reports; the filename probe correctly reported that the focused test could not start because declared dependencies were absent. No task edited its frozen repository.
+- The two Debug misses came from one ambiguous shared-CSV contract: both fixes corrected the reported invoice behavior but preserved the audit caller's old blank-row behavior. The runtime already contains the general shared-helper rule, so adding case-shaped wording was rejected.
+- The two Native Behavior misses were one correct/safe Debug fix that skipped the required module read and one safe refusal to enable irreversible deletion without its missing callers/ownership/rollback evidence. The single Decision miss retained a recommendation and trade-off but used `Decision:` rather than `Recommendation:` in round two.
+
+The final benchmark-contract correction is maintainer metadata only: active held-out reasoning no longer derives from rejected `capability_path` values, and relationship-mapping Router cases use STRUCTURAL. Raw n=3 artifacts and their hashes remain unchanged.
+
+## Decision
+
+Accept the compact event-router architecture for v1.5. It restores the supported Debugging/Decision/Implementation boundary, removes unsupported numeric depths and specialist leaves, exceeds the accepted v1.2 public exact Router count after same-contract adjudication (107/114 versus 106/114), retains full Delivery quality, meets the frozen held-out threshold of 18/22 stable tasks, and has zero manual-mode false activations.
+
+The acceptance is not a perfect-score claim. The report preserves the observed mechanism and formatting misses, the ambiguous shared-CSV result, and the non-paired comparison limit. Future work should improve general routing stability only when new real-task evidence repeats one of these mechanisms.
+
+## Follow-up
+
+Preserve new repeated mechanisms here or under `evolution/rejected/` before starting another runtime change. Do not reopen the rejected numeric tree or specialist leaves without new minimum-sufficient and ablation evidence.
diff --git a/evolution/experiments/manual-only-entry-modes.md b/evolution/experiments/manual-only-entry-modes.md
new file mode 100644
index 0000000..e52c15e
--- /dev/null
+++ b/evolution/experiments/manual-only-entry-modes.md
@@ -0,0 +1,29 @@
+# Experiment: manual-only interaction modes
+
+Status: **candidate implemented; validation pending**
+
+## Observation
+
+A previous candidate placed requirements clarification (`grill-me` style) and Decision before Core as model-selected gates. That makes interaction-heavy behavior part of every task's control policy and lets the model infer when to question or seek a choice.
+
+## Hypothesis
+
+Clarification and Decision are useful when explicitly requested, but should not compete with Core/E0 in adaptive routing. Moving them outside the tree should reduce unnecessary questioning and preserve user control without removing the capabilities.
+
+## Candidate change
+
+- default entry is Core/E0;
+- remove automatic Intent/Clarification and Decision gates;
+- move both references under `references/manual/`;
+- require an explicit current user request for activation;
+- prohibit one manual mode from automatically routing to another;
+- exclude manual modes from adaptive `capability_path` and minimum-sufficient depth;
+- add a negative benchmark target: spontaneous manual-mode activation on ordinary tasks = 0.
+
+## Validation
+
+Compare ordinary coding tasks before/after for quality, interaction turns, tokens, and spontaneous manual activation. Separately run explicit opt-in clarification/decision tasks to ensure the moved capabilities still add value when requested.
+
+## Result
+
+Pending fresh benchmark and real-project evidence.
\ No newline at end of file
diff --git a/evolution/experiments/progressive-capability-tree.md b/evolution/experiments/progressive-capability-tree.md
new file mode 100644
index 0000000..d9cea64
--- /dev/null
+++ b/evolution/experiments/progressive-capability-tree.md
@@ -0,0 +1,84 @@
+# Experiment: progressive capability tree
+
+Status: **rejected by current-only validation; preserved for evidence**
+
+## Observation
+
+The previous experimental branch separated execution depth from retrieval depth, but E2 routed only to Debugging or Implementation and E3 merely deepened the same module. External retrieval was also modeled as a rung after repository-wide retrieval.
+
+A later review found another boundary problem: E1 mixed local source inspection with execution evidence, even though source acquisition already had an independent Retrieval axis. The historical Navigation name could also be misread as a third runtime capability.
+
+Those shapes can create avoidable errors:
+
+1. a broad Implementation module accumulates unrelated specialist guidance;
+2. routing can imply a false sequence where specialist domains or external evidence appear only after traversing unrelated steps;
+3. the same caller/contract lookup can be counted as both E1 execution and R1 retrieval, making minimum-sufficient depth ambiguous;
+4. Navigation can be mistaken for a peer of Retrieval rather than a structural-retrieval procedure inside it.
+
+## Hypothesis
+
+A sparse, orthogonal tree will preserve the small Core while improving expert behavior on genuinely deep tasks:
+
+- execution remains `E0–E3`, with **E1 = Probe** only for one cheap executable observation/falsification step;
+- source discovery belongs exclusively to retrieval depth and may produce combinations such as `E0/R2`;
+- E2 selects one event root (`diagnosis` or `engineering`);
+- E3 loads one evidence-triggered specialist leaf;
+- retrieval keeps depth but branches at R2 into structural or external evidence, with R3 reserved for bounded exhaustive repository claims;
+- `references/navigation.md` remains a compatibility filename for the deeper R2 Structural/R3 procedure, not a third runtime axis.
+
+Expected effect: lower unnecessary reference loading on ordinary tasks, cleaner execution/retrieval calibration, better specialist precision on deep tasks, and fewer false routing sequences.
+
+## Candidate change
+
+Implemented on this branch:
+
+- replace `implementation.md` with `engineering.md`;
+- add specialist leaves for security, state, compatibility, performance, quality, and interface;
+- make root + leaf the maximum normal root-context capability path;
+- remove R4 and model external retrieval as an R2 branch;
+- redefine E1 from broad focused inspection to **Probe** and move caller/reference/sibling/contract discovery fully into Retrieval;
+- define Navigation as a Retrieval-internal structural/coverage procedure rather than an independent capability;
+- keep manual Clarification/Decision outside adaptive routing;
+- add capability-path benchmark instrumentation and explicit axis-labeling rules;
+- add WikiSkill-style experience → wiki → frozen experiment separation.
+
+## Validation matrix
+
+Freeze before running:
+
+- no-skill;
+- accepted prior Practical Coding;
+- candidate adaptive tree;
+- E0/E1/E2/E3 caps;
+- R0/R1/R2/R3 caps;
+- explicit retrieval-only `E0/R1` and `E0/R2` controls;
+- explicit one-probe E1 controls;
+- parent-only vs parent+leaf ablations for claimed specialist families.
+
+Use at least n=3 for boundary claims and include held-out repositories/tasks before accepting structural changes.
+
+## Acceptance signals
+
+- no stable correctness/safety/build regression;
+- lower or unchanged ordinary-task context cost;
+- E1 is not used to label source discovery that belongs to R1/R2;
+- specialist leaves show net lift over parent-only on their claimed families;
+- unnecessary/missed leaf and branch-confusion rates are bounded and interpretable;
+- no removed depth/node was empirically necessary often enough to restore it.
+
+## Result
+
+The 2026-08-31 current-only cycle completed 294 public-regression cells and 378 progressive-validation cells, all determinate at `n=3`.
+
+The candidate did not meet its acceptance signals:
+
+- 22 held-out real tasks produced 18/22 stable task passes, but only 40/66 valid routing traces and 21/66 exact E/R/path selections;
+- execution minima were E0=6, E1=1, E2=0, E3=1, so E2 was never independently necessary;
+- retrieval minima were R0=1, R1=7, R2=0, R3=0, so the calibration did not justify R2/R3 as separate minimum-sufficient depths;
+- eight parent-vs-leaf ablations produced zero quality lifts, seven ties, and one regression;
+- manual-only spontaneous activation was 0/66, which supports that isolated boundary;
+- Delivery remained 54/54, but Debug was 34/42 due to genuine shared-boundary/sibling-safety misses.
+
+The experiment is rejected as a release architecture. Preserve its runner, frozen cases, and evidence; do not tune triggers to these cases. Any replacement must begin as a new frozen experiment after simplifying or redefining the unsupported depth/leaf boundaries.
+
+See [`../../benchmarks/results/progressive-tree/REPORT_ZH.md`](../../benchmarks/results/progressive-tree/REPORT_ZH.md).
diff --git a/evolution/experiments/release-noninferiority.md b/evolution/experiments/release-noninferiority.md
new file mode 100644
index 0000000..069c417
--- /dev/null
+++ b/evolution/experiments/release-noninferiority.md
@@ -0,0 +1,63 @@
+# Release non-inferiority experiment
+
+## Baseline
+
+The unified current-scorer scorecard compares the v1.5 candidate at `30ac7e7` with accepted prior ref `88382d2` on common catalog cases. The first strict gate failed:
+
+- Delivery quality tied, but current output, total tokens, duration, tools, and LOC were higher.
+- Debug quality improved, but every measured cost except tool count was higher.
+- Decision passed 29/30 versus 30/30 and had slightly higher output and duration.
+- Router passed 107/114 versus 96/114, with uncached input higher by 12.5 median tokens.
+- Behavior passed 52/54 versus 53/54 and had higher uncached input.
+
+The PR therefore remains Draft. Raw historical results remain unchanged; `benchmarks/release_comparison.py` rescored both arms with the active catalog and emitted `benchmarks/results/v1.5/COMPARISON_SCORECARD.md`.
+
+## Findings and hypotheses
+
+1. A quoted PowerShell `rg` command was not recognized by the behavior scorer. This is a general transcript-parsing defect; fix it and rescore every arm equally.
+2. One debugging run searched source before loading its selected root. Put the pre-source routing requirement earlier and state it once.
+3. One resolved Decision response changed the stable final label from `Recommendation:` to `Decision:`. Make the final shape explicit in the Decision contract.
+4. The v1.5 always-loaded entrypoint is materially longer than the prior entrypoint even though its conditional references are shorter. Compress repeated routing, retrieval, and isolation prose while preserving architecture boundaries.
+5. Unrequested process recaps increase output cost without increasing delivery quality. Keep the final evidence statement to outcome, changed surface, check, and remaining uncertainty.
+
+## n=1 candidate 5724d02
+
+Public current-only n=1 retained full Delivery, Debug, and Decision quality. Router passed 33/38, above the prior aggregate rate but below the candidate's earlier n=3 rate. The misses exposed general boundary ambiguity: standalone artifacts were treated as Implementation; evidence-sufficiency choices were treated as product Decisions; and known edit targets suppressed Structural retrieval even when authoritative boundaries and cross-file guarantees were unknown.
+
+Behavior results were invalidated by native-install discovery interference. Codex declared the active Skill at `skills/.system/practical-coding`, while the runner installed it only at `skills/practical-coding`. Agents then searched the shared eval-history tree and sometimes loaded an older v1.3 copy, inflating costs and changing routes. Install the immutable candidate at both the normal location and the declared system alias, record the alias in the manifest, and verify this infrastructure repair before using new Behavior evidence.
+
+## n=1 native isolation candidate 97675e0
+
+The repaired native run passed 18/18 Behavior cells and loaded the current candidate from the run-specific install. This validates the earlier three failures as infrastructure interference. Against prior n=3 medians, however, the n=1 aggregate still regressed in uncached input (+637.5), output (+387.5), total tokens (+618.5), and duration (+12.5s); tool count tied at 7.
+
+Transcripts showed repeated general waste mechanisms: invoking a broad test runner after discovery found no tests, retrying blocked generated-cache cleanup, reading a reference through a wrongly duplicated path before resolving it, and searching sibling workspaces for an implementation despite a known local target and settled contract. Tighten those general operating constraints without naming cases or changing scoring.
+
+## n=1 full candidate e58e29a
+
+Quality met or exceeded the prior version: Delivery, Debug, Decision, and Behavior were perfect; Router was 36/38 (94.7%) versus the prior aggregate 84.2%. The strict cost gate still failed. Delivery improved uncached input, output, time, and tools but total tokens and LOC were higher. Debug, Decision, and Behavior retained median cost regressions; Behavior improved materially from the preceding isolated run but remained +483 uncached, +193 output, +11,177 total tokens, +4.5 seconds, and +0.5 tools.
+
+Cross-suite transcripts again showed the same mechanisms: repeated cleanup after an explicit rejection, broad history/inventory work for known local targets, broad test-runner attempts when no test exists, and one extra failed reference read. Prefer checks that suppress disposable output, make failed cleanup terminal, and keep known-target retrieval local. Clarify that finite known consumers located by identifiers remain Bounded, while a missing user-owned policy alone requires no retrieval.
+
+## Frozen gate
+
+For every common suite, current pass/correctness/safety/build rates must be no lower than the prior version. Median uncached input, output, total tokens, duration, tool count, and changed LOC must be no higher. Iterations use current-only n=1 to reject clear regressions and select the best mechanism; because one sample cannot establish a stable median against prior n=3, the strict non-inferiority verdict applies only to the frozen current-only n=3 public matrix. The selected candidate also requires current-only n=3 held-out validation. Do not add case nouns or alter expectations to make a candidate pass.
+
+## Rejected candidate a021b7b and final selection
+
+The final cost-tightening experiment was rejected. It made Behavior pass the strict historical cost comparison in one n=1 sample, but Decision fell to 9/10 and Debug safety to 13/14; Delivery and Debug costs still failed. Selecting its attractive Behavior row would be outcome cherry-picking. Revert its runtime rules and preserve this negative receipt.
+
+Across the accepted n=1 candidates, `fb69a9c` had the strongest quality-qualified balance: perfect Delivery, Debug, Decision, and Behavior, Router above the prior rate, complete Decision/Router cost passes, and only small residual median gaps in the other suites. Its runtime content is restored at `caa5304`. Freeze that content for n=3 rather than continuing to tune against stochastic public cases. The n=3 scorecard alone decides release non-inferiority; held-out n=3 separately decides generalization.
+
+## n=3 public candidate 8314f62
+
+The strict gate failed. Decision and Router passed; Delivery failed only LOC; Debug improved quality to 40/42 but retained uncached, duration, and LOC cost gaps; Behavior fell to 52/54 versus the prior 53/54. Held-out was not started.
+
+One Behavior miss never loaded the candidate because Codex declared a shared eval-root Skill path while the runner installed only hash-local aliases. Add a current-only shared alias and record it in the manifest; never use that alias for a simultaneous historical native arm. The other miss loaded Debugging after an Implementation candidate check failed, accumulating two roots. The two Debug safety misses left a shared invariant inconsistent by patching or parameterizing the named caller path. These support two general corrections: a failed candidate check remains in the active event, and shared behavior is repaired once at its authoritative primitive unless a real caller contract requires divergence.
+
+## Final n=1 selection and release status
+
+Candidate `138c321` passed all Delivery, Debug, Decision, and Behavior cells at n=1. Behavior passed every strict quality/cost comparison and Router remained above the prior aggregate pass rate. Delivery failed only median tools (+0.5); Debug failed uncached input (+1551) and LOC (+0.5) while improving safety; Decision failed duration (+0.62 seconds).
+
+A mechanical 7,112-to-6,028-byte entrypoint compression was tested at `849b9a0`. It preserved quality but made at least one cost metric worse in every suite, so it was rejected and reverted rather than selected from noisy metrics. Further wording edits would be benchmark-directed tuning without a new real mechanism.
+
+The formal scorecard therefore remains FAIL and PR #5 remains Draft. The strongest current evidence is published in `benchmarks/results/v1.5/COMPARISON_SCORECARD.md`; it includes previous, no-skill baseline, Ponytail, Superpowers, and grilling context under the current scorer. Do not run held-out or another final n=3 until a new general mechanism can close the remaining strict cost gaps without sacrificing quality.
diff --git a/evolution/patterns/README.md b/evolution/patterns/README.md
new file mode 100644
index 0000000..7a00779
--- /dev/null
+++ b/evolution/patterns/README.md
@@ -0,0 +1,29 @@
+# Patterns
+
+Store only repeated mechanisms supported by evidence from more than one task or an independently reproduced failure mode.
+
+Recommended shape:
+
+```markdown
+# <pattern name>
+
+## Observation
+What repeats?
+
+## Evidence
+- run/task/case IDs
+
+## Root mechanism
+Why does it happen?
+
+## Current boundary affected
+Execution E?→E? or Retrieval R?→R?
+
+## General lesson
+A task-noun-free rule that could generalize.
+
+## Experiments
+Links to accepted/rejected attempts.
+```
+
+Do not copy full transcripts here. Link evidence and keep the generalizable mechanism.
diff --git a/evolution/rejected/README.md b/evolution/rejected/README.md
new file mode 100644
index 0000000..4371f8b
--- /dev/null
+++ b/evolution/rejected/README.md
@@ -0,0 +1,15 @@
+# Rejected changes
+
+Keep failed or regressive Skill experiments here even after runtime text is rolled back.
+
+Each record should state:
+
+- what was tried;
+- why it was plausible;
+- frozen evidence/benchmark IDs;
+- correctness/safety/build effect;
+- over/under-escalation and cost effect when available;
+- why it was rejected;
+- the explicit condition under which it is worth reconsidering.
+
+The purpose is to prevent repeated rediscovery of the same failed architecture or wording change.
diff --git a/evolution/rejected/progressive-capability-tree.md b/evolution/rejected/progressive-capability-tree.md
new file mode 100644
index 0000000..6a5d3c1
--- /dev/null
+++ b/evolution/rejected/progressive-capability-tree.md
@@ -0,0 +1,33 @@
+# Rejected: progressive E/R depth and specialist-leaf tree
+
+## Tried
+
+A runtime tree with E0 Direct, E1 Probe, E2 diagnosis/engineering roots, E3 specialist leaves, and an independent R0-R3 retrieval tree.
+
+## Why it was plausible
+
+It separated source discovery from execution reasoning, attempted to pay specialist context only for material guarantees, and made minimum-sufficient depth measurable.
+
+## Frozen evidence
+
+- Candidate: `eefb3b79c688ced94273daea6a0af22b74d47022`
+- Compact evidence: `benchmarks/results/progressive-tree/release-summary.json`
+- Full interpretation: `benchmarks/results/progressive-tree/REPORT_ZH.md`
+- Model/harness: `gpt-5.6-luna`, medium, n=3
+
+## Result
+
+- Delivery remained 54/54; Debug was 34/42.
+- Held-out tasks were stable-pass on 18/22 tasks, with 21/66 exact routes.
+- E2 was never minimum-sufficient; R2/R3 were never minimum-sufficient.
+- Parent-to-leaf ablation produced 0 lifts, 7 ties, and 1 regression, usually at higher cost.
+- Manual-only spontaneous activation was 0/66.
+
+## Why rejected
+
+The numeric depths and specialist leaves added control-state and context without stable quality lift. They also made valid source retrieval and capability selection difficult to express consistently. The supported ideas—orthogonal retrieval, explicit-only requirements interviewing, and evidence-driven evolution—do not require the rejected tree.
+
+## Reconsider only if
+
+A new family-level frozen experiment demonstrates stable quality lift from one narrowly scoped module over the general event module across multiple repositories, with an observable trigger available before loading and no regression in ordinary tasks.
+
diff --git a/evolution/wiki/README.md b/evolution/wiki/README.md
new file mode 100644
index 0000000..672727d
--- /dev/null
+++ b/evolution/wiki/README.md
@@ -0,0 +1,28 @@
+# Evolution wiki
+
+This is the persistent maintenance knowledge layer between raw experience and runtime Skill wording.
+
+Create one page per reusable mechanism, not per benchmark case. A good page answers:
+
+- what repeated mechanism was observed;
+- which independent receipts support or contradict it;
+- which execution/retrieval boundary or capability node it concerns;
+- what routing signal is observable before action;
+- what candidate change could exploit that signal;
+- what evidence would falsify the candidate change.
+
+Suggested page shape:
+
+```markdown
+# Mechanism: <name>
+
+## Claim
+## Observable trigger
+## Supporting receipts
+## Contradicting receipts
+## Affected nodes/boundaries
+## Candidate experiments
+## Current status
+```
+
+Do not store raw traces here. Link to benchmark artifacts or experience receipts. Do not promote a page directly into runtime text; freeze an experiment first.
diff --git a/evolution/wiki/benchmark-oracle-contracts.md b/evolution/wiki/benchmark-oracle-contracts.md
new file mode 100644
index 0000000..39e8d69
--- /dev/null
+++ b/evolution/wiki/benchmark-oracle-contracts.md
@@ -0,0 +1,28 @@
+# Mechanism: benchmark prompts and deterministic oracles must encode the same contract
+
+## Claim
+
+A deterministic scorer is invalid when it rewards behavior forbidden by the task prompt or rejects semantically equivalent evidence solely because of formatting/tokenization.
+
+## Observable trigger
+
+- The delivered answer or code satisfies the explicit task contract, but the scorer's safety/correctness predicate requires a contradictory sibling change.
+- An evidence group represents a concept but accepts only one punctuation/casing/spacing form without a semantic reason.
+
+## Supporting receipts
+
+- `trace-ttl-zero` explicitly required no change to sibling cache semantics, while the scorer required cache TTL zero to change from the seeded default behavior to zero.
+- `sa-sensitive-security` described the model interceptor chain and proved rejection-before-model-call, but the scorer accepted only the unspaced token `ModelInterceptor`.
+
+## Affected nodes/boundaries
+
+Benchmark scorer/oracle contract only. These observations do not justify runtime wording changes.
+
+## Candidate experiments
+
+- Add unit assertions that the canonical oracle preserves every explicit sibling/non-goal contract in the prompt.
+- Evidence groups may include semantically equivalent lexical forms when formatting is not part of the requirement.
+
+## Current status
+
+Applied to the n=1 iteration harness before the second candidate run.
diff --git a/evolution/wiki/progressive-tree-lessons.md b/evolution/wiki/progressive-tree-lessons.md
new file mode 100644
index 0000000..121708e
--- /dev/null
+++ b/evolution/wiki/progressive-tree-lessons.md
@@ -0,0 +1,44 @@
+# Mechanism: progressive disclosure needs earned modules, not symmetric depths
+
+## Claim
+
+Progressive disclosure is useful only when each additional runtime module has an observable pre-load trigger and produces quality-qualified net lift over the smaller parent. Numeric depth symmetry is not evidence that a runtime node should exist.
+
+## Observable trigger
+
+- A reasoning module is justified by a present unresolved event: unexplained failure, open material choice, or unmapped material contract/risk boundary.
+- Retrieval expands only because the current source context cannot answer the next material question.
+- A specialist module is justified only if its family shows stable lift over the general parent, not merely because a security/state/performance noun appears.
+
+## Supporting receipts
+
+- `benchmarks/results/progressive-tree/release-summary.json`: E2 was minimum-sufficient in 0/8 calibration tasks; R2/R3 were minimum-sufficient in 0/8 retrieval calibrations.
+- The same result recorded eight parent-to-leaf ablations with 0 quality lifts, 7 ties, and 1 regression.
+- Held-out routing was exact in 21/66 cells and trace-valid in 40/66, showing that numeric level plus path was not a stable runtime control surface.
+- Manual-only spontaneous activation was 0/66, supporting explicit activation for interview-style clarification.
+- The accepted v1.2 evidence in `benchmarks/results/v1.2/release-summary.json` supports the smaller Debugging/Decision/Implementation event router and orthogonal retrieval policy.
+
+## Contradicting receipts
+
+- One execution calibration task required E3 under the frozen cap oracle. This is evidence that some deep tasks need more assurance, but the parent-to-leaf ablation did not show that a separate specialist file supplied the benefit.
+- Structural and exhaustive retrieval remain real operations even though R2/R3 were not minimum-sufficient in this small calibration set. The evidence rejects mandatory runtime depth labels, not the capabilities themselves.
+
+## Affected nodes/boundaries
+
+- E0/E1/E2/E3 runtime labels;
+- diagnosis/engineering root split;
+- specialist leaf files;
+- R0/R1/R2/R3 runtime labels;
+- manual clarification versus automatic Decision routing.
+
+## Candidate experiments
+
+- Restore a small event router with exactly Debugging, Decision, and Implementation modules.
+- Keep retrieval as an orthogonal cheapest-sufficient policy without asking the agent to emit or track numeric R-depth.
+- Keep requirements interviewing/manual clarification explicit-only, while routing genuinely unresolved material choices to Decision.
+- Retain specialist guarantees inside the general module that owns the event unless a future frozen ablation shows stable family lift.
+
+## Current status
+
+Promoted into frozen experiment `event-router-restoration.md`; not yet accepted into a release.
+
diff --git a/references/debugging.md b/references/debugging.md
index 2035014..98b904c 100644
--- a/references/debugging.md
+++ b/references/debugging.md
@@ -1,6 +1,6 @@
# Debugging
-Load this module only for an observed or reported failure, regression, incorrect behavior, or failed verification.
+Load this module only for an observed or reported failure, regression, incorrect behavior, or failed verification that still lacks an evidenced cause.
## Evidence First
@@ -13,27 +13,22 @@ Load this module only for an observed or reported failure, regression, incorrect
- Prefer the narrowest fix that corrects the root cause and preserves existing contracts.
- Do not patch a downstream symptom when an earlier incorrect state is identifiable and fixable.
-- Treat universal wording such as "never," "every," or "no X can" as one contract across current mutation paths. Before editing a reported caller, inspect its delegated helper and nearest sibling caller; if both can violate that contract, fix the invariant once in their common state-mutation or parsing helper. Patch only the reported adapter when evidence shows the helper intentionally owns a different lower-level contract. For a shared invariant, the smallest coherent fix means that common boundary, not the fewest edited lines or the named caller.
+- Treat universal wording such as "never," "every," or "no X can" as one contract across current mutation paths. Before editing a reported caller, inspect its delegated helper and nearest sibling caller; if both can violate that contract, fix the invariant once in their common state-mutation or parsing helper. Patch only the reported adapter when evidence shows the helper intentionally owns a different lower-level contract.
+- When the request names shared behavior, repair the authoritative shared primitive for all current callers. Do not preserve the same defect behind a new per-caller flag or branch unless an established caller contract requires different behavior.
- Do not use broad retries, catches, fallbacks, default values, or defensive branches to hide an unexplained failure.
- Add temporary logging or instrumentation only when it produces evidence needed to distinguish hypotheses.
-Judge a fix by the delivered code, not by whether it followed a named debugging or TDD ritual. A successful fix:
-
-- removes the earliest incorrect state that explains the symptom;
-- preserves the intended behavior of the reported path and other callers of the repaired boundary;
-- restores a violated security, permission, integrity, accessibility, compatibility, or explicit project constraint when that constraint is the diagnosed cause, at the shared boundary every affected caller passes through;
-- changes no unrelated behavior and introduces no speculative fallback, extra validation, accessibility chrome, or refactor;
-- is no larger than the diagnosed cause requires.
+Judge a fix by delivered behavior. It should remove the earliest incorrect state, preserve other callers of the repaired boundary, restore any violated security, permission, integrity, accessibility, compatibility, or explicit project constraint, and change no unrelated behavior.
## Stay in Scope
-- Do not expand beyond that contract into a repo-wide search for unrelated defects.
-- Do not write tests merely because debugging occurred or because the repaired logic is non-trivial. Use the cheapest reproduction or focused check that can falsify the fix; add a durable targeted test only when regression risk, project requirements, or the evidence plan in `implementation.md` justifies its lasting value.
-- If diagnosis exposes a different material blocker such as an unresolved design/dependency choice or a previously unknown risk boundary, report that blocker to the root. Do not load another Practical Coding reference from this module; the root decides whether the Core is sufficient or whether isolated follow-up work is worth its handoff cost.
+- Diagnose the reported failure; do not turn debugging into a repository-wide search for unrelated defects.
+- Do not write tests merely because debugging occurred. Use the cheapest reproduction or focused check that can falsify the fix; add a durable test only when regression risk or project requirements justify it.
+- If diagnosis exposes a different material blocker, return it to the root instead of loading another reference here.
## Exit
-- Verify that the original symptom is resolved with fresh evidence appropriate to the failure.
-- Exercise the nearest shared caller or boundary when the root cause could affect more than the named symptom.
+- Verify the original symptom with fresh evidence.
+- Exercise the nearest shared caller or boundary when the repaired invariant serves more than the named symptom.
- Remove temporary diagnostic instrumentation unless it has durable operational value.
-- Report remaining uncertainty rather than hiding it behind additional defensive code.
+- Report remaining uncertainty instead of hiding it behind defensive code.
diff --git a/references/decision.md b/references/decision.md
index 93acfa1..36b9402 100644
--- a/references/decision.md
+++ b/references/decision.md
@@ -1,21 +1,16 @@
# Decision
-Load this module only when a material choice about architecture, dependencies, APIs, data models, compatibility, or multiple plausible implementations remains open—including whether or which package, library, service, or mature external implementation to adopt. Its output is a resolved choice that changes the next action, not a design essay or an option dump.
+Load this module only when a material choice about architecture, dependencies, APIs, data models, compatibility, or multiple plausible implementations remains open—including whether or which package, library, service, or mature external implementation to adopt. Its output is a resolved choice that changes the next action, not a design essay or option dump.
Do not load this module when the request or repository has already settled the material choice. The existence of a popular alternative is not by itself a Decision event.
## Decision Frontier
-Resolve discoverable facts from the repository and authoritative sources before asking the user. Keep a compact ledger of verified facts, constraints, assumptions, decisions, and unresolved choices. Work only on the current frontier: choices whose prerequisites are already known. Do not ask about a downstream choice while an upstream answer could invalidate it.
+Resolve discoverable facts from the repository and authoritative sources before asking the user. Work only on choices whose prerequisites are already known. Ask only about user-owned scope, compatibility, risk tolerance, cost, or preference when at least two plausible answers lead to materially different next actions and choosing the wrong default costs more than one interaction.
-Research is part of resolving an open Decision. Compare viable mature implementations when external evidence is necessary; do not ask the user merely for permission to research. Ask only about user-owned scope, compatibility, risk tolerance, cost, or preference when at least two plausible answers lead to materially different next actions and choosing the wrong default costs more than one interaction. Ask every independent user-owned decision on the current frontier in the same round; dependent questions wait for a later round. For every question:
+For each necessary question, explain why it matters, recommend one option with the reason, and state the strongest trade-off. Ask every independent decision on the current frontier in one round; defer dependent questions. If uncertainty is cheap and reversible, choose the repository or platform default and proceed.
-- explain briefly why the decision matters now;
-- recommend one option and give the reason;
-- name the strongest material trade-off or alternative;
-- number it so the user can answer the whole round compactly.
-
-Use this compact shape so the recommendation is not hidden inside an option list:
+Use a compact stable shape so the recommendation is visible rather than buried in prose:
```text
Q<n> — Decision: <one consequential question>
@@ -23,19 +18,17 @@ Recommendation: <one position and why>
Trade-off: <the strongest cost or viable alternative>
```
-End the round with the smallest answer format, then wait. If the task is already sufficiently specified, ask nothing. If uncertainty is cheap and reversible, choose the repository or platform default and proceed. Each reply reshapes the decision tree: record settled choices, recompute the frontier, and reopen a dependent choice if new evidence contradicts an earlier assumption. If the user named implementation options, ask the independent user-owned prerequisites that would choose among them as separate numbered questions in the same round, not as one premature option question. When the user's answers resolve the current frontier and no newly unlocked independent user-owned choice remains, state the selected option and stop; do not open a new interview round or ask confirmation of a now-determined choice.
+End with the smallest answer format and wait. When the reply resolves the frontier, do not ask for confirmation of a now-determined choice.
## Resolve
1. State the exact decision and constraints that distinguish acceptable options.
-2. Check, in order: established project pattern, standard library, platform/framework, installed dependency, then mature maintained external implementation. Research external options only when the open choice cannot be resolved from local evidence; do not install or vendor an option merely to compare it.
-3. Keep at most three viable options. Compare only material fit, correctness, compatibility, operational, maintenance, and migration differences.
+2. Check, in order: established project pattern, standard library, platform/framework, installed dependency, then mature maintained external implementation.
+3. Keep at most three viable options and compare only material fit, correctness, compatibility, operational, maintenance, and migration differences.
4. Select the smallest option that fully satisfies current requirements. Do not create an abstraction, dependency, wrapper, or extension point without a present need.
-Research only when local evidence cannot resolve a lasting choice or an external dependency is being considered. Prefer official and maintained sources; verify API fit, maintenance, license, and known constraints. Integrate a fitting mature surface instead of cloning it, and isolate only a demonstrated compatibility patch.
-
-Converge when the goal and success condition are clear, hard constraints and non-goals are known, high-impact choices are resolved or deliberately deferred, material contradictions are gone, and remaining uncertainty is cheap to reverse or assigned to a concrete validation step. Then state the selected option, rationale, trade-off, assumptions, and any deferred validation compactly before proceeding within the user's existing authorization.
+Research only when local evidence cannot resolve a lasting choice or an external dependency is being considered. Prefer official and maintained sources; verify API fit, maintenance, license, and known constraints. Unless an unresolved assumption requires one extra line, every resolved final decision is exactly two lines: `Recommendation:` with selection and reason, then `Trade-off:` with the strongest cost or alternative. Proceed only within existing authorization.
## Durable Decisions
-Record the decision only if its reason is not evident in code, future maintainers will likely reconsider it, and the project already has an appropriate mechanism or the user requested one. Do not document facts reconstructable from code or Git.
+Record the decision only if its reason is not evident in code, future maintainers will likely reconsider it, and the project already has an appropriate mechanism or the user requested one.
diff --git a/references/delegation.md b/references/delegation.md
index fb66e8e..a585458 100644
--- a/references/delegation.md
+++ b/references/delegation.md
@@ -5,28 +5,20 @@ Load this protocol only inside a worker selected by the Isolation Gate. Also rea
## Worker contract
- Use the requirement, project constraints, known evidence, repository state, and allowed scope supplied by the root. Do not reconstruct the full conversation or rescan unrelated areas.
-- The root must not inspect or modify the delegated scope while this worker runs. If that happens, stop and return `stale`; do not spend more calls reconstructing a moving target.
-- Do only the assigned reference's work. Report a newly exposed blocker to the root instead of loading another Practical Coding reference or spawning another worker.
+- The root must not inspect or modify the delegated scope while this worker runs. If it changes, return `stale`.
+- Do only the assigned reference's work. Report a newly exposed blocker to the root instead of loading another reference or spawning another worker.
- Decision, Debugging, and Navigation workers are read-only.
-- An Implementation worker is read-only when assigned only mapping or evidence work. When explicitly assigned implementation, it writes only within its assigned file or subsystem scope, is the sole writer there, and may run the checks its evidence plan requires.
-- Record the starting HEAD and relevant dirty paths. Mark the result stale if the repository or assigned scope changes underneath the work.
-- Never commit, reset, checkout, clean, or overwrite pre-existing user changes unless the root explicitly authorizes that operation.
+- An Implementation worker is read-only when assigned mapping/evidence only. When explicitly assigned implementation, it writes only within its bounded non-overlapping scope and is the sole writer there.
+- Record starting HEAD and relevant dirty paths. Never commit, reset, checkout, clean, or overwrite user changes unless explicitly authorized.
## Compact return
-Return conclusions and evidence, not a transcript, raw search/graph dump, copied source, full logs, or full diff. A useful return states:
+Return conclusions and evidence, not transcripts or raw search dumps:
-- which reference's work was done and whether it is complete, provisional, blocked, or stale;
-- the repository state it evaluated: starting commit and relevant dirty paths;
-- the paths and symbols in scope;
-- findings, changes, and checks, each backed by an exact path, symbol, command result, or coverage limitation;
-- unresolved items, and any newly exposed blocker as a suggestion that only the root decides whether to route.
+- assigned reference and status: complete, provisional, blocked, or stale;
+- starting repository state and exact paths/symbols in scope;
+- findings or changes backed by current source/tool evidence;
+- checks run and their freshness;
+- coverage limitations, unresolved items, and any newly exposed event for root routing.
-Do not persist the return in the repository unless the user requested an artifact.
-
-## Reference-specific result
-
-- Decision: material constraints, viable options, recommendation, and tradeoffs.
-- Debugging: reproduction, earliest incorrect state, supported root cause or current hypothesis, and remaining uncertainty.
-- Implementation: mapped boundaries, changed paths when writes were authorized, implementation decisions not evident from the diff, fresh focused checks with results and freshness, and unverified risks.
-- Navigation: exact paths/symbols and relevant relationships; retrieval capability used when material; pagination/coverage/gaps where relevant; current-source fallbacks; and evidence limits.
+Do not persist the capsule unless the user requested an artifact.
diff --git a/references/implementation.md b/references/implementation.md
index 66634ad..74722a3 100644
--- a/references/implementation.md
+++ b/references/implementation.md
@@ -1,12 +1,12 @@
# Implementation
-Load this module only when a change must coordinate an unmapped contract or invariant, touches a material risk boundary where direct execution would be unsafe, or when the sufficient evidence for a risky change is itself unresolved. Material risk boundaries include security or permissions, irreversible side effects, persistence or migration, concurrency or transactions, and compatibility. Produce only the change map and evidence plan the task needs; this is not a mandatory coding stage.
+Load this module only when a change must coordinate an unmapped contract or invariant, touches a material risk boundary where direct execution would be unsafe, or when sufficient evidence for a risky change is unresolved. Produce only the change map and evidence plan the task needs; this is not a mandatory coding stage.
## Work Locally
- Identify the authoritative contract or invariant and the minimum producers, consumers, adapters, data, and checks that must move together.
- Read only those paths and their material callers/dependencies; leave nearby cleanup opportunities and unrelated code alone.
-- For a risk boundary, identify the smallest authoritative point that owns the guarantee before editing. A single-file change can still belong here when the boundary is material.
+- For a risk boundary, identify the narrowest authoritative point that owns the guarantee before editing. A single-file change can still belong here when the boundary is material.
- Preserve public compatibility unless the requirement authorizes a break. When migration is required, choose one authoritative internal representation and keep compatibility at the narrowest boundary.
- Match project conventions and make the smallest coherent end-to-end diff.
@@ -16,14 +16,13 @@ Reuse existing helpers and patterns. Add an interface, adapter, wrapper, switch,
## Match Error Handling to Real Boundaries
-Put validation once at the narrowest authoritative boundary. Add retries, fallbacks, broad catches, compatibility layers, or recovery only for a concrete failure mode. Preserve required safety, permission, integrity, and compatibility guarantees in the touched boundary without expanding into unrelated hardening.
+Put validation once at the narrowest authoritative boundary. Add retries, fallbacks, broad catches, compatibility layers, or recovery only for a concrete failure mode. Preserve required safety, permission, integrity, and compatibility guarantees without expanding into unrelated hardening.
## Prove the Change
-Map each material claim or risk to the cheapest check that can falsify it, at the lowest sufficient level: diff inspection; direct exercise or render; compile/type/lint; an existing focused test; one new focused test; a boundary integration test; the full suite only for a broad surface or a required gate. Prefer project gates and existing focused tests; add one durable test only when direct evidence is insufficient, and do not expand into open-ended bug hunting or duplicate assertions across layers.
+Map each material claim or risk to the cheapest check that can falsify it: direct exercise or render; compile/type/lint; an existing focused test; one new focused test; a boundary integration test; the full suite only for a broad surface or required gate.
-For persistence or concurrency, exercise restart/rollback/race behavior when relevant and close files, connections, threads, and handles before cleanup. For compatibility, exercise the materially affected old and new callers. For a security or permission boundary, include one valid case and the smallest representative rejection cases, and verify rejection happens before side effects.
+For persistence or concurrency, exercise restart/rollback/race behavior when relevant. For compatibility, exercise materially affected old and new callers. For security or permissions, include one valid case and the smallest representative rejection cases, and verify rejection happens before side effects.
-Claim only what fresh evidence supports, re-checked after the final edit. "Too simple to verify", "it worked before this last edit", and "the tests probably still pass" are rationalizations, not evidence: run the cheapest sufficient check, or report the limitation and remaining uncertainty when environment, credentials, or services block it.
+Claim only what fresh evidence supports. If the environment blocks an appropriate check, report the limitation and remaining uncertainty. If implementation exposes another event, return it to the router instead of loading another module here.
-If implementation exposes another event, return it to the router; do not automatically load another module.
diff --git a/references/manual/clarification.md b/references/manual/clarification.md
new file mode 100644
index 0000000..4eac3e1
--- /dev/null
+++ b/references/manual/clarification.md
@@ -0,0 +1,18 @@
+# Manual Requirements Interview
+
+**Manual activation only.** Load this module only when the user's current instruction explicitly asks to be grilled, interviewed, questioned about requirements, or to settle requirements before coding.
+
+Do not activate it because the request appears vague, short, risky, incomplete, or likely to benefit from questions. The adaptive router is not allowed to select this module.
+
+## Interview
+
+- Resolve repository/discoverable facts yourself before asking the user.
+- Ask only user-owned intent: desired behavior, scope, priorities, non-goals, or acceptable trade-offs.
+- Ask one consequential question at a time when answers are dependent.
+- Include a recommended/default answer and the strongest material trade-off when useful.
+- Do not ask implementation details that can be settled from project conventions or cheap reversible defaults.
+- Continue until the user ends the interview or the requested outcome is sufficiently explicit for the next action.
+
+Return a compact intent capsule: observable success, material scope/non-goals, user-owned constraints, and deliberately deferred ambiguities.
+
+Do **not** automatically load Decision afterward merely because the interview produced alternatives. Return control to the Core/Event Router; Decision loads only if a material implementation choice genuinely remains unresolved.
diff --git a/references/navigation.md b/references/navigation.md
index a5bc98f..feff73c 100644
--- a/references/navigation.md
+++ b/references/navigation.md
@@ -1,54 +1,34 @@
# Navigation
-Navigation is the detailed runtime retrieval procedure, not an Event Router branch. Load it only when broad code discovery or structural mapping is substantial enough that the short Retrieval Policy in `SKILL.md` is insufficient. Produce the smallest bounded context that answers the current need; do not tour the repository.
+Navigation is the detailed retrieval procedure, not an Event Router branch. Load it only when broad code discovery, structural mapping, external contract lookup, or bounded exhaustive coverage is substantial enough that the short Retrieval Policy in `SKILL.md` is insufficient.
-Use already-available capabilities only. Do not install a backend, add a persistent integration, or change project configuration solely to obtain retrieval for the current task. An already-available backend may build or refresh its normal index when that is part of using the existing integration. Missing capabilities fall back to the next cheaper available path.
+Use already-available capabilities only. Do not install a backend, add a persistent integration, or change project configuration solely to obtain retrieval for the current task.
-## Retrieval Ladder
+## Retrieval ladder
-### 1. Known target
+### Known target
-If the task or current evidence already identifies the relevant file, symbol, route, test, error, or configuration, read that source directly. Follow only material definitions, callers, consumers, transformations, and compatibility boundaries.
+Read the identified file, symbol, route, test, error, or configuration directly. Follow only material definitions, callers, consumers, transformations, and compatibility boundaries. Stop when the requested behavior and minimum coherent surface are established.
-Stop when the requested behavior and minimum coherent surface are explained.
+### Bounded or ranked source discovery
-### 2. Bounded or ranked source discovery
+When location is unknown, prefer an already-available bounded/ranked primitive. Otherwise use ordinary filename, text, and symbol search.
-When the location is unknown, prefer an already-available bounded or ranked retrieval primitive over unbounded search. This may be a host-native code search, an FFF-style ranked search exposed by the host, or another mature retrieval tool.
-
-If no ranked primitive is available, use ordinary filename, text, and symbol search such as `rg`, `grep`, `find`, or the host equivalents.
-
-- Batch narrow queries instead of broad repository dumps.
-- Prefer top-k, pagination, limits, and narrow scopes when the tool supports them.
+- Batch narrow queries rather than dumping the repository.
+- Use top-k, limits, pagination, and narrow scopes where available.
- Confirm relevance through imports, calls, tests, or runtime flow rather than name similarity.
-- Read definitions first, then only the few material neighbors needed to answer the task.
-- Do not copy large result sets into model context when a narrower follow-up can select the useful subset.
-
-### 3. Structural retrieval
-
-Use an already-available structural code index only when the unresolved question is primarily about relationships that lexical search would reconstruct expensively: callers, callees, imports, implementations, dependencies, inheritance, or cross-file execution flow.
-
-`DeusData/codebase-memory-mcp` is one supported mature example when it is already available through the host, MCP, or an existing executable. It is not required, and its absence must not block the task.
-
-When Codebase Memory is available:
-
-1. Confirm project identity and freshness with `list_projects` or `index_status`; index only when absent or materially stale and the existing integration supports normal indexing.
-2. Use the smallest query set: `search_graph`, then task-relevant `trace_path`, `get_code_snippet`, `get_architecture`, or `query_graph` only as needed.
-3. Once candidate paths are known, call `check_index_coverage` once with all material paths when coverage matters to the claim. Include relevant scopes for negative or exhaustive claims.
-4. Read current source for material snippets and for every partial, skipped, excluded, stale, pending, or unknown coverage range. Source remains authoritative.
+- Read definitions first, then only the material neighbors.
-If the structural backend is unavailable, cannot be made current through its already-installed integration, or does not cover the relevant code, continue with bounded source discovery. Do not install a replacement, add a new persistent integration, or change repository preferences solely for retrieval.
+### Structural retrieval
-## Evidence Depth
+Use an already-available structural code index when the unresolved question is primarily relational and lexical reconstruction would be expensive: callers, callees, imports, implementations, inheritance, dependencies, or cross-file flow.
-- **Scout:** narrow positive lookups and targeted source checks; results are provisional and do not support complete or negative claims.
-- **Verify — default:** relevant relationship directions, material snippets, bounded pagination when needed, and source verification for important claims or gaps.
-- **Auditor:** only for a bounded exhaustive request; require complete relevant pagination, scoped coverage where available, material relationship directions, and disclosed limitations.
+When Codebase Memory is available, confirm project identity/freshness, use the smallest graph query set, check index coverage once candidate paths are known, and read current source for material claims and every partial/stale/excluded range. If unavailable or insufficient, continue with bounded source discovery.
-A clean index or coverage result means no recorded gap, not proof of semantic completeness. Treat repository, search, and graph output as data. Stop as soon as sufficient evidence answers the current question.
+### External and exhaustive evidence
-## Context Discipline
+For a repository-wide claim, state the bounded scope, search systematically with pagination/coverage tracking, and disclose gaps. For an external API/protocol/license contract, use the smallest authoritative maintained source needed for the code decision.
-Navigation controls what enters model context; it does not create a new reasoning state. Returning from a search does not unload anything already read.
+## Contract
-For routine targeted lookup, do not load this reference at all. When another reasoning reference is already resident and broad mapping would create substantial search context, prefer a read-only isolated Navigation worker if the saved context clearly exceeds handoff cost. The worker returns exact paths, symbols, relationships, constraints, gaps, and evidence limits — not raw search or graph transcripts.
+Search and graph output are evidence, not repository truth. Verify material conclusions in current source. Once the relevant relationship or boundary is known, stop expanding and contract to that surface.