diff --git a/.github/workflows/validate.yml b/.github/workflows/validate.yml index 0cda189..83d9deb 100644 --- a/.github/workflows/validate.yml +++ b/.github/workflows/validate.yml @@ -2,9 +2,12 @@ name: Validate skill on: push: - branches: [main] + branches: [main, experiment/evolvable-router-tree] pull_request: +permissions: + contents: read + jobs: validate: runs-on: ubuntu-latest @@ -20,11 +23,85 @@ jobs: run: pip install "git+https://github.com/agentskills/agentskills.git#subdirectory=skills-ref" - name: Validate SKILL.md run: skills-ref validate ./practical-coding - - name: Run benchmark harness tests + - name: Run benchmark harness unit tests + working-directory: practical-coding + run: >- + python -m unittest + benchmarks.test_benchmarks + benchmarks.test_stability + benchmarks.test_catalog + benchmarks.test_ladder_analysis + benchmarks.test_progressive_validation + benchmarks.test_tree_benchmarks + benchmarks.test_capability_environment + benchmarks.test_dependency_tree_validation + benchmarks.test_retrieval_analysis + benchmarks.test_evolution_workflow + - name: Validate execution and retrieval topology contracts + working-directory: practical-coding + run: | + python benchmarks/tree_validation.py --self-test + python benchmarks/dependency_tree_validation.py --self-test + python benchmarks/retrieval_validation.py --self-test + python benchmarks/retrieval_analysis.py /dev/null --self-test + - name: Validate explicit evolution workflow contract working-directory: practical-coding - run: python -m unittest benchmarks.test_benchmarks benchmarks.test_stability benchmarks.test_catalog benchmarks.test_ladder_analysis benchmarks.test_progressive_validation + run: python benchmarks/evolution_workflow_validation.py --self-test --output benchmark-results/evolution-workflow-contract.json + - name: Check manual-only Decision layout + run: | + test ! -e practical-coding/references/decision.md + test -e practical-coding/references/manual/decision.md + - name: Check progressive Retrieval layout + run: | + test -e practical-coding/references/retrieval/SKILL.md + test -e practical-coding/references/retrieval/direct.md + test -e practical-coding/references/retrieval/discovery.md + test -e practical-coding/references/retrieval/evidence.md + test -e practical-coding/references/retrieval/structural.md + test -e practical-coding/benchmarks/capability_manifest.json + - name: Ensure removed ranked-search integration is absent from active surfaces + run: | + ! grep -R -i -E 'fff-style|pi-fff|\bFFF\b' \ + practical-coding/SKILL.md \ + practical-coding/AGENTS.md \ + practical-coding/README.md \ + practical-coding/README_zh.md \ + practical-coding/CONTRIBUTING.md \ + practical-coding/examples \ + practical-coding/references \ + practical-coding/agents \ + practical-coding/docs/CAPABILITY_LAYER.md \ + practical-coding/benchmarks/README.md \ + practical-coding/benchmarks/capability_manifest.json - name: Check Codex default_prompt references the skill as $skill-name run: grep -qF '$practical-coding' practical-coding/agents/openai.yaml + - name: Ensure retired execution-state experiment is absent + run: | + retired_files="$({ + find practical-coding/runtime practical-coding/tests practical-coding/benchmarks \ + -maxdepth 1 -type f \ + \( -name '*skill_state*' -o -name 'test_skill_state*' \) \ + -print 2>/dev/null || true + find practical-coding/evolution/experiments \ + -maxdepth 1 -type f -name 'skill-state-*' -print 2>/dev/null || true + })" + if test -n "$retired_files"; then + printf 'retired execution-state files remain on an active surface:\n%s\n' "$retired_files" + exit 1 + fi + test ! -e practical-coding/docs/SKILL_STATE.md + test ! -e practical-coding/docs/SKILL_STATE_HOST.md + test ! -e practical-coding/docs/SKILL_STATE_INVARIANTS.md + test -e practical-coding/evolution/rejected/execution-state/README.md + ! grep -R -E \ + 'runtime/skill_state|SKILL_STATE_MODEL_GATE|state-history-free|Execution State Projection|"execution_state"' \ + practical-coding/SKILL.md \ + practical-coding/AGENTS.md \ + practical-coding/README.md \ + practical-coding/README_zh.md \ + practical-coding/agents/openai.yaml \ + practical-coding/benchmarks/README.md \ + practical-coding/benchmarks/tree_topology.json - name: Ensure legacy local graph runtime is not reintroduced run: | test ! -e practical-coding/runtime/codebase_memory.py diff --git a/AGENTS.md b/AGENTS.md index c2a52b3..18fdac1 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -4,30 +4,50 @@ This repository is an Agent Skill. Apply [`SKILL.md`](SKILL.md) when working fro ## Runtime model -1. Apply the Core and stay Direct unless one present unresolved event matches the Router. -2. Route only Debugging, Decision, or Implementation; load at most one reasoning reference for the current event. -3. Complete routing before diagnostic, decision-research, or change-mapping source work. The selected reference is the next read. -4. Keep retrieval orthogonal. Unknown paths, callers, consumers, and data flow are retrieval questions, not Implementation events. -5. Contract to the smallest affected surface as soon as the cause, choice, contract, invariant, or evidence boundary is established. +1. Apply Core at execution-tree depth 0. +2. Core knows only its immediate automatic execution children: Debugging and Implementation. +3. A loaded execution node owns only its own next-level router. Do not preload siblings or descendants and do not send descendant selection back to Core. +4. Current Debugging and Implementation nodes are leaves until benchmark evidence earns a child. +5. Retrieval is a separate progressive tree. Its depth describes the unresolved information problem, not execution complexity or tool strength. +6. Host capabilities such as ranked search, graph retrieval, and output compaction are replaceable infrastructure outside both trees. +7. Automatic routing must converge toward resolving the current blocker; it must not reopen deliberation. -## Event Router +## Root Router -| Present unresolved event | Reference | +| Present unresolved blocker | Immediate child | |---|---| | Observed failure still lacks an evidenced cause | [`references/debugging.md`](references/debugging.md) | -| Material user-owned implementation choice changes the next action | [`references/decision.md`](references/decision.md) | -| Unknown contract/invariant, coordinated guarantee, material risk boundary, or evidence plan blocks safe execution | [`references/implementation.md`](references/implementation.md) | +| Unknown contract/invariant, coordinated guarantee, material risk boundary, or evidence requirement blocks safe execution | [`references/implementation.md`](references/implementation.md) | -A known target and settled behavior/boundary/check stay Direct even when risk nouns are present. A read-only mapping request is Direct plus Retrieval. +A known target and settled behavior/boundary/check stay at Core even when risk nouns are present. A read-only mapping request is Core plus Retrieval. -Requirements interviewing is explicit-only through [`references/manual/clarification.md`](references/manual/clarification.md). +## Manual modes -## Retrieval +Manual modes are outside the automatic trees: -Use known source, then bounded/ranked search, then an already-available structural capability when it materially reduces relationship discovery. Use exhaustive coverage or external authoritative sources only when the claim requires them. Source remains authoritative. +- [`references/manual/decision.md`](references/manual/decision.md) only for an explicit current request to compare options, choose a technology/architecture/dependency/API/data model, or perform decision analysis; +- [`references/manual/clarification.md`](references/manual/clarification.md) only for an explicit current request to be interviewed, grilled, questioned, or to clarify requirements before implementation. -Read [`references/navigation.md`](references/navigation.md) only for substantial retrieval. Missing graph/ranked capabilities fall back without installing or persisting tooling solely for retrieval. +No automatic node may route to a manual mode. Ordinary technical choices discovered during execution use the established project convention or the smallest sufficient reversible option. If a user-owned choice has no safe default, ask the minimum blocking question in the current context without opening Decision. + +## Navigation and Retrieval + +Navigation answers **which bounded repository area** should be searched. Load [`references/navigation.md`](references/navigation.md) only when that map is genuinely unresolved; it must return a compact topology and stop. + +Retrieval answers **which concrete evidence** resolves the current claim. Load [`references/retrieval/SKILL.md`](references/retrieval/SKILL.md), then follow only the immediate child declared by the currently loaded node. The runtime root must not reproduce the complete topology from the benchmark manifest or select a distant descendant directly. + +Do not route by provider name. Runtime fallback remains lossless when a ranked or graph provider is unavailable. The dependency-enabled benchmark is different: it fails closed unless every provider in [`benchmarks/capability_manifest.json`](benchmarks/capability_manifest.json) is installed and successfully pre-initialized. + +## Execution output + +Output compaction is a cross-cutting execution layer. A host adapter should make it transparent where command hooks exist; otherwise use the thinnest wrapper instruction available. It may reduce noisy shell, test, build, and Git output, but it must preserve semantics, exit status, failures, and material verification evidence. It is never a Retrieval or execution-tree node. ## Evolution -`evolution/` is maintainer knowledge and must not enter ordinary runtime context. During Skill maintenance, record mechanisms and failed changes there before modifying another runtime rule. Iterations use n=1; only a frozen release candidate receives the complete n=3 matrix. +`evolution/` is maintainer knowledge and must not enter ordinary runtime context. The trees are experiment results, not fixed taxonomies. + +Use [`benchmarks/tree_topology.json`](benchmarks/tree_topology.json), [`benchmarks/dependency_tree_validation.py`](benchmarks/dependency_tree_validation.py), [`benchmarks/retrieval_validation.py`](benchmarks/retrieval_validation.py), [`benchmarks/tree_analysis.py`](benchmarks/tree_analysis.py), and [`benchmarks/retrieval_analysis.py`](benchmarks/retrieval_analysis.py) for active dependency-enabled topology work. Cases must not encode a gold automatic node or fixed numeric execution level. Derive minimum-sufficient nodes by capability ablation, then use repeated routing ambiguity or quality failures to propose add/split/merge/promote/collapse/remove changes. + +Iterations use n=1. Only a frozen candidate receives the complete n=3 baseline/no-skill comparison. Provider installation, model download, first index, dependency resolution, and first build warm-up are setup work and are never included in benchmark token, duration, or tool-call comparisons. + +Preserve v1.5 and rejected experiments as historical evidence rather than rewriting them for the current topology. The execution-state/history-free proposal is retired under [`evolution/rejected/execution-state/`](evolution/rejected/execution-state/); do not restore it without a new frozen hypothesis and independent evidence. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 4e12bd6..3bcbf7e 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -4,14 +4,37 @@ Practical Coding optimizes for the smallest quality-qualified runtime contract, ## Runtime boundaries -- Keep `SKILL.md` compact and route-agnostic outside the Event Router. -- Route only a present unresolved Debugging, Decision, or Implementation event. -- Load at most one reasoning reference for the current event. -- Keep source/context retrieval independent from reasoning selection. -- Requirements interviewing remains explicit-only. +- Keep `SKILL.md` compact. Core knows only the immediate automatic execution children. +- Route only a present unresolved Debugging or Implementation blocker. +- Decision and requirements interviewing remain explicit-only manual modes. +- Keep the execution tree, Retrieval tree, capability providers, output transport, and maintenance workflows separate. +- A loaded node may name only its immediate child. Do not place every depth decision in Core. - Do not introduce mandatory plans, reviews, tests, documents, Git workflows, workers, or lifecycle ceremony. -Unknown locations, callers, consumers, or relationships are retrieval questions. They become Implementation only when a requested coordinated change has an unresolved governing contract or material risk/evidence boundary. +Unknown locations, callers, consumers, or relationships are Retrieval questions. They become Implementation only when a requested coordinated change has an unresolved governing contract or material risk/evidence boundary. + +## Retrieval-tree discipline + +The active Retrieval path is: + +`Retrieval Root -> R0 Direct Locate -> R1 Ranked Discovery -> R2 Evidence Expansion -> R3 Structural Trace`. + +- R0 handles known or narrowly identifiable targets. +- R1 finds ranked candidates when intent is known but location is not. +- R2 builds the smallest distributed evidence set required by unresolved claims. +- R3 resolves relationship, flow, dependency, ownership, and impact questions. + +Depth represents the unresolved information problem. Tool names, brands, and installation methods do not belong in node identities or routing conditions. Every material conclusion must be verified in current source. + +Navigation has one narrower job: answer which bounded repository area should be searched. It must not absorb semantic search, evidence expansion, or graph tracing. + +## Capability providers + +Normal runtime use remains portable: use the strongest already-available provider appropriate to the current node, then fall back losslessly to bounded source search. + +Provider-enabled benchmark claims use `benchmarks/capability_manifest.json` and `benchmarks/dependency_tree_validation.py`. That profile requires `zg`, `codebase-memory-mcp`, and `rtk`; missing or failed providers abort the run. Provider probes, indexes, dependency resolution, and first-build warm-up occur before Codex starts and are never merged into compared token, duration, or tool-call fields. + +Do not weaken the benchmark by adding an allow-missing flag, silently falling back, estimating setup tokens, or resuming a measured cell without its matching `capability-setup.json` receipt. ## Evolution before wording @@ -26,18 +49,16 @@ Do not add benchmark case nouns to runtime wording. A module must have an observ ## Benchmark discipline -- Quality, safety, and build/reachability precede routing and cost. +- Quality, safety, build/reachability, and environment parity precede routing and cost. - Iteration runs use n=1. Run n=3 only for a frozen candidate believed ready to release. - Current-only runs may compare against prior published reports offline, but are not paired ranking evidence. - Keep deterministic prompts and oracles contract-consistent; do not reward behavior the prompt forbids. +- Every arm in a paired task sees the same initialized providers and repository warm-up. +- Setup is separately auditable but excluded from comparison; measured execution begins only after setup succeeds. - Add a test when a newly discovered mechanism or scorer invariant would otherwise regress. -Public regression covers Delivery, Debug, Decision, Router, and Native Behavior. Real-repository held-out coverage validates delivered evidence, zero spontaneous requirements interviewing, event selection, and retrieval scope. - -## Retrieval capabilities - -Known source → bounded/ranked search → structural capability when useful → bounded exhaustive or authoritative external evidence only when required. FFF-style search, ordinary search, LSP/AST, and Codebase Memory are optional capabilities. Verify material claims against current source and disclose coverage gaps. +Public regression covers Delivery, Debug, Decision, Router, and Native Behavior. Real-repository held-out coverage validates delivered evidence, zero spontaneous manual activation, execution-node selection, progressive Retrieval disclosure, and provider use. ## Mature implementation first -For a non-trivial new capability, inspect maintained prior art, extract the smallest fitting mechanism, verify maintenance/license/API fit, and keep the result removable. Do not copy an entire expert workflow into the Core. +For a non-trivial new capability, inspect maintained prior art, extract the smallest fitting mechanism, verify maintenance/license/API fit, and keep the result removable. Do not copy an entire expert workflow into Core. diff --git a/README.md b/README.md index 18f5813..9c06402 100644 --- a/README.md +++ b/README.md @@ -2,73 +2,180 @@ Practical Coding is an Agent Skill for producing the smallest reliable coding change without turning every task into a heavyweight workflow. -It uses one compact Core, three evidence-triggered reasoning modules, and an orthogonal retrieval policy: - -```text -Core / Direct -├─ unresolved observed failure → Debugging -├─ unresolved material implementation choice → Decision -└─ unresolved contract, invariant, or risk boundary → Implementation - -Retrieval (independent): -known target → bounded/ranked search → structural or authoritative evidence → bounded exhaustive coverage +The active experiment now separates three concerns: + +1. an evolvable **execution tree** for engineering depth; +2. an independent progressive **Retrieval tree** for unresolved information problems; +3. a replaceable **capability layer** for ranked search, graph retrieval, and command-output compaction. + +```mermaid +flowchart TD + Core[Core · execution depth 0] + Core -->|unexplained observed failure| Debugging[Debugging · current leaf] + Core -->|unknown contract / coordinated risk boundary| Implementation[Implementation · current leaf] + + Retrieval[Retrieval Root] --> Direct[R0 Direct Locate] + Direct -->|target unresolved| Discovery[R1 Ranked Discovery] + Discovery -->|distributed evidence unresolved| Evidence[R2 Evidence Expansion] + Evidence -->|relationship is the unresolved answer| Structural[R3 Structural Trace · leaf] + + ZG[ranked retrieval provider: zg] -. implements .-> Discovery + ZG -. supports .-> Evidence + CBM[graph provider: codebase-memory-mcp] -. implements .-> Structural + RTK[execution output layer: rtk] -. compacts .-> Commands[shell / test / build / Git output] ``` +Decision and Clarification remain explicit-only manual modes outside both automatic trees. + ## Runtime contract -The Core applies to every task: +Core applies to every coding task: - define the smallest observable success; -- reuse established project primitives; +- reuse established project primitives and contracts; - add no speculative abstractions, dependencies, configuration, validation, tests, or documentation; - preserve unrelated behavior and user changes; - verify with the cheapest check that can falsify the material claim. -If no unresolved Event Router condition matches, stay Direct. A risk-related noun, multiple files, unknown paths, or caller discovery does not itself justify a reasoning module. - -When an event is present, load exactly one reference: +Core knows only its immediate automatic execution children: - [`references/debugging.md`](references/debugging.md) — an observed failure still lacks an evidenced cause; -- [`references/decision.md`](references/decision.md) — a material user-owned implementation choice remains open; -- [`references/implementation.md`](references/implementation.md) — safe execution is blocked by an unresolved contract, coordinated invariant, material risk boundary, or evidence plan. +- [`references/implementation.md`](references/implementation.md) — safe execution is blocked by an unresolved contract, coordinated invariant, material risk boundary, or evidence requirement. + +Each loaded node owns only its own next-level router. A node with no benchmark-earned children declares itself a leaf. Automatic routing may deepen to resolve a blocker but must not reopen deliberation. + +## Manual modes + +- [`references/manual/decision.md`](references/manual/decision.md) loads only when the current user explicitly asks to compare options, select a technology/architecture/dependency/API/data model, or perform decision analysis. +- [`references/manual/clarification.md`](references/manual/clarification.md) loads only when the current user explicitly asks to be interviewed, grilled, questioned, or to clarify requirements before implementation. + +No automatic node routes to a manual mode. When a manual request finishes, its settled result returns to Core as input. + +## Retrieval tree + +Retrieval depth represents **what information remains unresolved**, not which tool is available. + +[`references/retrieval/SKILL.md`](references/retrieval/SKILL.md) is the Retrieval root and knows only R0: + +| Stage | Question answered | Next escalation | +|---|---|---| +| [`R0 Direct Locate`](references/retrieval/direct.md) | Can a known file, symbol, identifier, or narrow literal establish the target? | Target still unknown → R1 | +| [`R1 Ranked Discovery`](references/retrieval/discovery.md) | Where are the strongest candidates when intent is known but location is not? | Answer needs distributed evidence → R2 | +| [`R2 Evidence Expansion`](references/retrieval/evidence.md) | What is the smallest cross-file evidence set needed for the unresolved claims? | The answer is fundamentally relational → R3 | +| [`R3 Structural Trace`](references/retrieval/structural.md) | What call, dependency, ownership, control/data-flow, or impact relationship establishes the answer? | Leaf; stop when the relationship is proved | + +The root does not choose R0–R3 globally. Every node knows only its immediate child and returns as soon as the current claim has enough current-source evidence. + +At runtime, providers are optional accelerators and every stage has a bounded source-search fallback. In the dependency-enabled benchmark, concrete providers are mandatory so the experiment measures the intended capability surface rather than a mixture of installed and missing tools. + +## Navigation boundary + +[`references/navigation.md`](references/navigation.md) answers only: **which bounded repository area should be searched?** It creates a small topology map from module declarations, package metadata, and maintained architecture evidence. + +Retrieval answers: **which concrete source evidence resolves the claim?** Navigation does not perform semantic discovery, expand related evidence, or trace graph relationships. Known targets skip Navigation and start at R0. + +## Capability and output layers + +[`docs/CAPABILITY_LAYER.md`](docs/CAPABILITY_LAYER.md) defines the provider boundary. + +The active dependency profile pins and requires: + +- `zg` from zvec-grep `0.2.0` for ranked hybrid retrieval at R1/R2; +- `codebase-memory-mcp` `0.10.8` for graph-aware R3 retrieval; +- `rtk` `0.47.0` for compact shell, test, build, and Git output. -Requirements interviewing and `grill-me` behavior are explicit-only through [`references/manual/clarification.md`](references/manual/clarification.md). One unavoidable blocking question in an ordinary task is normal interaction, not an interview mode. +These names never become tree nodes. A future provider can replace one without changing Retrieval policy. -## Retrieval policy +Output compaction is cross-cutting infrastructure. It must preserve command semantics, exit status, failures, and material verification evidence. The agent does not route to RTK. A host with a command hook may make this transparent; the Codex benchmark exposes the wrapper through one equal capability note because RTK's Codex integration is instruction-based rather than a hard pre-execution hook, and records whether `rtk` was actually used. -Retrieval is separate from reasoning. Use the cheapest available capability that supplies enough current context: +## Dependency-enabled benchmark -1. read a known path or symbol; -2. use bounded/ranked filename, text, or symbol search; -3. use an already-available structural index for relationship questions when it saves work; -4. use bounded exhaustive coverage only for explicit exhaustive claims, and authoritative external sources only for contracts the repository cannot establish; -5. verify material conclusions against current source. +The machine-readable profile is [`benchmarks/capability_manifest.json`](benchmarks/capability_manifest.json). Both [`benchmarks/dependency_tree_validation.py`](benchmarks/dependency_tree_validation.py) and [`benchmarks/retrieval_validation.py`](benchmarks/retrieval_validation.py) fail before comparison when any required binary or probe is unavailable. The former preserves the execution-tree ceiling experiment; the latter runs independent `NONE/R0/R1/R2/R3` Retrieval ceilings. -[`references/navigation.md`](references/navigation.md) is the optional detailed procedure for substantial retrieval. Codebase Memory, LSP/AST, ranked search, and ordinary search are capabilities, not required dependencies. +Verify the frozen profile first. The preflight runner enforces the provider-version regular expressions recorded in the manifest and records the observed output: -## Evolution discipline +```powershell +zg --version +codebase-memory-mcp --version +rtk --version +git --version +node --version +npm --version +java -version +mvn --version +``` + +Install the providers from their maintained upstream distributions before running the model benchmark. The repository does not silently install or substitute them during a measured cell. + +### Measurement boundary + +Every cell has two phases: + +1. **setup, excluded** — versioned provider probes, local model/assets, `zg` indexing plus a first query, Codebase Memory indexing plus daemon warm-up, dependency resolution, first test/build warm-up, and workspace cleanliness checks; +2. **measured execution** — Codex starts only after setup succeeds; transcript tokens, model-visible tool calls, duration, answer quality, and routing trace are collected here. -Runtime agents do not read `evolution/`. Maintainers record experiences, consolidate repeated mechanisms, freeze experiments before changing runtime rules, and preserve rejected changes. +Setup details are written to each cell's `capability-setup.json` with `included_in_comparison: false`. The setup report contains output byte counts and elapsed time for auditability but no token estimate. Because Codex is not running during setup, those operations cannot enter measured input/output tokens, tool calls, or wall time. Every paired arm receives the same initialized environment. -The rejected E/R depth and specialist-leaf experiment is retained under [`evolution/rejected/`](evolution/rejected/) with its n=3 evidence in [`benchmarks/results/progressive-tree/`](benchmarks/results/progressive-tree/). The replacement event-router experiment is documented in [`evolution/experiments/event-router-restoration.md`](evolution/experiments/event-router-restoration.md). +A measured attempt to run `zg index`, Codebase Memory indexing, `rtk init`, or package installation is a contract violation rather than an accepted cold-start cost. -The accepted v1.5 release evidence is published under [`benchmarks/results/v1.5/`](benchmarks/results/v1.5/). Its frozen current-only n=3 matrix had zero indeterminate cells: Delivery 54/54, Debug 40/42, Decision 29/30, Native Behavior 52/54, and 61/66 held-out quality cells across 22 real tasks. Event reasoning was 113/114; after correcting three retrieval expectations that contradicted the current structural-mapping contract, the public Router result was 107/114 (reasoning 113/114, retrieval 108/114). These are non-paired release results; they do not claim superiority over other skills. +## Benchmark-driven evolution + +The current topology lives in [`benchmarks/tree_topology.json`](benchmarks/tree_topology.json). Cases in [`benchmarks/tree_cases.py`](benchmarks/tree_cases.py) contain no expected automatic execution route or fixed depth. + +The benchmark may add, split, merge, promote, collapse, move, or remove nodes when evidence supports the mutation: + +- **add/deepen** when a repeatable pre-load signal exists and a child adds stable quality-qualified lift over its parent; +- **merge/move boundary** when nodes are repeatedly ambiguous without net value; +- **promote/collapse** when a child is required for most of its parent's useful scope; +- **remove** when a node has no independent minimum-sufficient or marginal-lift cases; +- **split** when a leaf has a repeated failure cluster with an observable pre-load boundary. + +Execution depth and Retrieval depth describe disclosure only. They are not universal task-complexity scores. ## Validation -Public regression and real-repository held-out validation use `gpt-5.6-luna` at medium reasoning. Iteration runs use `n=1`; release claims require the complete current-only matrix at `n=3`. +Deterministic contract checks require no external providers: ```powershell -pwsh -NoProfile -File benchmarks/run.ps1 -SelfTest -pwsh -NoProfile -File benchmarks/run.ps1 -ProgressiveSelfTest +python benchmarks/dependency_tree_validation.py --self-test +python benchmarks/retrieval_validation.py --self-test +python benchmarks/retrieval_analysis.py /dev/null --self-test +python -m unittest ` + benchmarks.test_tree_benchmarks ` + benchmarks.test_capability_environment ` + benchmarks.test_dependency_tree_validation ` + benchmarks.test_retrieval_analysis +``` -python benchmarks/run_catalog.py --profile full --runs 3 --workers 3 ` - --arm practical-current --arm practical-native --output benchmark-results/public-final +A model-backed Retrieval iteration requires every dependency and uses `n=1`: -python benchmarks/progressive_validation.py --phase all --current-only --runs 3 --workers 3 ` - --output benchmark-results/heldout-final +```powershell +python benchmarks/retrieval_validation.py --current-only --runs 1 --workers 3 ` + --output benchmark-results/retrieval-tree-n1 +python benchmarks/retrieval_analysis.py benchmark-results/retrieval-tree-n1/results.jsonl ` + --output benchmark-results/retrieval-tree-n1/analysis.json +``` + +Run `dependency_tree_validation.py` separately when the execution-tree wording or boundaries also changed. It preserves the Core/Debugging/Implementation ceiling experiment under the same provider and warm-up contract. + +After freezing the candidate, run the paired `n=3` Retrieval comparison against no-skill and the v1.5 baseline: + +```powershell +python benchmarks/retrieval_validation.py --runs 3 --workers 3 ` + --output benchmark-results/retrieval-tree-final +python benchmarks/retrieval_analysis.py benchmark-results/retrieval-tree-final/results.jsonl ` + --output benchmark-results/retrieval-tree-final/analysis.json +``` + +For a release candidate that changes both trees, also run: + +```powershell +python benchmarks/dependency_tree_validation.py --runs 3 --workers 3 ` + --output benchmark-results/execution-tree-final +python benchmarks/tree_analysis.py benchmark-results/execution-tree-final/results.jsonl ` + --output benchmark-results/execution-tree-final/analysis.json ``` -Historical published evidence remains version-specific and non-paired unless its arms are rerun in one frozen matrix. +The accepted v1.5 flat Event Router and rejected fixed-depth, specialist-leaf, and execution-state experiments remain historical evidence under `benchmarks/results/` and `evolution/rejected/`. Historical reports are not rewritten to fit the new topology. -MIT License. See `THIRD_PARTY_NOTICES.md` for attribution. +MIT License. See [`THIRD_PARTY_NOTICES.md`](THIRD_PARTY_NOTICES.md) for provider attribution. diff --git a/README_zh.md b/README_zh.md index e53b50a..c359d92 100644 --- a/README_zh.md +++ b/README_zh.md @@ -2,73 +2,180 @@ Practical Coding 是一个 Agent Skill：目标是交付最小、可靠的代码修改，同时避免把所有任务都变成重量级流程。 -运行时只有一个 Core、三个由证据触发的推理模块，以及一条独立的检索策略： - -```text -Core / Direct -├─ 已观察失败但原因未证实 → Debugging -├─ 会改变实现方向的重大选择尚未解决 → Decision -└─ 契约、不变量、风险边界或证据计划未解决 → Implementation - -检索独立： -已知目标 → 有界/排序搜索 → 结构或权威证据 → 有界穷举覆盖 +当前实验把三类问题彻底分开： + +1. 决定工程化深度的**执行树**； +2. 按“尚未解决的信息问题”逐步展开的独立 **Retrieval 树**； +3. 提供排序检索、图检索和命令输出压缩的可替换**能力层**。 + +```mermaid +flowchart TD + Core[Core · 执行 depth 0] + Core -->|已观察失败但原因未证实| Debugging[Debugging · 当前叶子] + Core -->|未知契约 / 协同风险边界| Implementation[Implementation · 当前叶子] + + Retrieval[Retrieval Root] --> Direct[R0 Direct Locate] + Direct -->|目标仍无法定位| Discovery[R1 Ranked Discovery] + Discovery -->|分布式证据仍不足| Evidence[R2 Evidence Expansion] + Evidence -->|答案本质是关系| Structural[R3 Structural Trace · 叶子] + + ZG[排序检索 provider: zg] -. 实现 .-> Discovery + ZG -. 支持 .-> Evidence + CBM[图检索 provider: codebase-memory-mcp] -. 实现 .-> Structural + RTK[执行输出层: rtk] -. 压缩 .-> Commands[shell / test / build / Git 输出] ``` +Decision 与 Clarification 继续是显式手动模式，不属于任何自动树。 + ## 运行时契约 Core 始终适用： - 先定义最小可观察成功； -- 复用项目已经存在的 primitive； +- 复用项目已有 primitive 与 contract； - 不添加推测性的抽象、依赖、配置、验证、测试或文档； - 保留无关行为和用户已有修改； -- 用能证伪关键结论的最便宜检查验证。 +- 用能够证伪关键结论的最便宜检查验证。 -没有 Event Router 条件时保持 Direct。风险名词、文件数量、路径未知或需要找 caller，本身都不是推理升级理由。 - -存在未解决事件时只加载一个 reference： +Core 只知道两个直接自动执行子节点： - [`references/debugging.md`](references/debugging.md)：已观察失败仍没有证据化原因； -- [`references/decision.md`](references/decision.md)：会改变实现方向的重大用户选择尚未解决； -- [`references/implementation.md`](references/implementation.md)：安全执行被未知契约、协同不变量、重大风险边界或证据计划阻塞。 +- [`references/implementation.md`](references/implementation.md)：安全执行被未知契约、协同不变量、重大风险边界或证据要求阻塞。 + +每个被加载的节点只拥有自己的下一层 Router。没有通过 benchmark 证明有价值的 child 时，节点必须明确声明为叶子。自动路由只能为了消除当前 blocker 而加深，不能重新打开 deliberation。 + +## 手动模式 + +- [`references/manual/decision.md`](references/manual/decision.md) 只在用户当前明确要求比较方案、技术选型、推荐架构/依赖/API/数据模型或决策分析时加载； +- [`references/manual/clarification.md`](references/manual/clarification.md) 只在用户明确要求先采访、grill、提问或澄清需求时加载。 + +任何自动节点都不能路由到手动模式。手动任务完成后，把已确定的结果作为输入返回 Core。 + +## Retrieval 树 + +Retrieval depth 表示**当前还缺哪一种信息**，不表示工具品牌或能力强弱。 + +[`references/retrieval/SKILL.md`](references/retrieval/SKILL.md) 是 Retrieval 根节点，只知道 R0： + +| 阶段 | 回答的问题 | 何时进入下一层 | +|---|---|---| +| [`R0 Direct Locate`](references/retrieval/direct.md) | 已知文件、symbol、identifier 或窄 literal 能否直接定位目标？ | 目标仍未知 → R1 | +| [`R1 Ranked Discovery`](references/retrieval/discovery.md) | 已知语义意图但不知道位置时，最可能的候选在哪里？ | 回答依赖跨文件证据 → R2 | +| [`R2 Evidence Expansion`](references/retrieval/evidence.md) | 当前 claim 所需的最小跨文件证据集是什么？ | 答案本质是关系 → R3 | +| [`R3 Structural Trace`](references/retrieval/structural.md) | 哪条调用、依赖、所有权、控制/数据流或影响关系能够证明答案？ | 叶子；关系成立后停止 | + +Core 不一次性选择 R0–R3。每个 Retrieval 节点只知道自己的直接 child，并在当前 claim 获得最小充分源码证据后立即返回。 + +普通运行时中，provider 只是可选加速器；缺失时无损回退到有界源码检索。依赖启用 benchmark 则强制要求具体 provider，避免把“装了工具”和“没装工具”的结果混成一组成本比较。 + +## Navigation 边界 + +[`references/navigation.md`](references/navigation.md) 只回答：**应该去哪个有界仓库区域找？** 它根据 module 声明、包元数据和维护中的架构证据建立小型拓扑图。 + +Retrieval 回答：**哪一段具体源码证据能够解决问题？** Navigation 不执行语义发现、不扩展相关证据，也不追踪调用图。目标已经明确时跳过 Navigation，直接从 R0 开始。 + +## 能力层与输出层 + +[`docs/CAPABILITY_LAYER.md`](docs/CAPABILITY_LAYER.md) 定义 provider 边界。 + +当前依赖 profile 固定版本并强制要求： + +- zvec-grep `0.2.0` 的 `zg`：实现 R1/R2 的混合排序检索； +- `codebase-memory-mcp` `0.10.8`：实现 R3 的图关系检索； +- `rtk` `0.47.0`：压缩 shell、test、build 与 Git 输出。 -需求采访和 `grill-me` 只能由用户显式激活 [`references/manual/clarification.md`](references/manual/clarification.md)。普通任务里一个不可避免的阻塞问题不算进入采访模式。 +这些名称都不是树节点。未来替换 provider 时，不需要重构 Retrieval policy。 -## 检索策略 +输出压缩是横切基础设施，必须保留命令语义、退出码、失败信息和关键验证证据。模型不会“路由到 RTK”。支持 command hook 的宿主可以做到透明改写；RTK 对 Codex 的上游集成属于规则/提示词而不是强制 pre-execution hook，因此 Benchmark 会通过所有 arm 相同的一条 capability note 暴露 wrapper，并记录 `rtk` 是否真的被调用。 -检索与推理正交，始终使用能提供充分当前上下文的最便宜能力： +## 依赖启用 Benchmark -1. 读取已知路径或 symbol； -2. 使用有界/排序的文件名、文本或 symbol 搜索； -3. 关系问题在确实节省探索成本时使用已经可用的结构索引； -4. 只有明确穷举结论才做有界覆盖，仓库无法建立的外部契约才查询权威来源； -5. 重要结论必须回到当前源码验证。 +机器可读 profile 位于 [`benchmarks/capability_manifest.json`](benchmarks/capability_manifest.json)。[`benchmarks/dependency_tree_validation.py`](benchmarks/dependency_tree_validation.py) 与 [`benchmarks/retrieval_validation.py`](benchmarks/retrieval_validation.py) 都会在任何依赖缺失或 probe 失败时，在创建比较 cell 前直接失败。前者保留执行树 ceiling；后者独立运行 `NONE/R0/R1/R2/R3` Retrieval ceiling。 -[`references/navigation.md`](references/navigation.md) 只用于较重的检索过程。Codebase Memory、LSP/AST、排序搜索和普通搜索都是可选能力，不是依赖。 +先验证冻结的 profile。preflight 会按 manifest 中的版本正则强制校验 provider，并保存实际版本输出： -## 演化纪律 +```powershell +zg --version +codebase-memory-mcp --version +rtk --version +git --version +node --version +npm --version +java -version +mvn --version +``` + +请先按照各上游项目的维护方式安装依赖。仓库不会在 measured cell 内静默安装或替换 provider。 + +### 计量边界 + +每个 cell 分成两个阶段： + +1. **setup，不参与比较**：带版本校验的 provider probe、本地模型/资源初始化、`zg` 索引与首次 query、Codebase Memory 建图与 daemon warm-up、项目依赖解析、首次测试/构建 warm-up、工作区洁净检查； +2. **measured execution**：只有 setup 成功后才启动 Codex，此时才采集 transcript token、模型可见 tool call、时长、答案质量和路由 trace。 -普通运行时不读取 `evolution/`。维护阶段才记录体验、合并重复机制、先冻结实验再修改运行时规则，并保留失败改进。 +每个 cell 的 setup 详情写入 `capability-setup.json`，并明确标记 `included_in_comparison: false`。setup 报告只保留输出字节数和耗时用于审计，不估算 token。由于 setup 时 Codex 尚未启动，因此这些操作不可能进入 measured input/output token、tool call 或 wall time。配对比较的所有 arm 使用完全相同的预初始化环境。 -被拒绝的 E/R 深度与专家叶子实验保存在 [`evolution/rejected/`](evolution/rejected/)，其 n=3 证据位于 [`benchmarks/results/progressive-tree/`](benchmarks/results/progressive-tree/)。替代实验记录在 [`evolution/experiments/event-router-restoration.md`](evolution/experiments/event-router-restoration.md)。 +如果模型在 measured 阶段再次执行 `zg index`、Codebase Memory 建图、`rtk init` 或包安装，这会被判为契约违规，而不是把冷启动成本混入结果。 -已接受的 v1.5 发布证据位于 [`benchmarks/results/v1.5/`](benchmarks/results/v1.5/)。冻结的 current-only n=3 矩阵没有 indeterminate：Delivery 54/54、Debug 40/42、Decision 29/30、Native Behavior 52/54，22 个真实任务的 held-out 质量为 61/66。事件推理为 113/114；修正 3 个与当前“结构关系映射”合同矛盾的 Retrieval 期望后，公共 Router 为 107/114（reasoning 113/114、retrieval 108/114）。这些是非配对发布结果，不用于宣称优于其他 Skill。 +## Benchmark 驱动演化 + +当前拓扑位于 [`benchmarks/tree_topology.json`](benchmarks/tree_topology.json)。[`benchmarks/tree_cases.py`](benchmarks/tree_cases.py) 不保存 expected automatic route 或固定 execution depth。 + +Benchmark 可以在证据支持时新增、拆分、合并、提升、折叠、移动或删除节点： + +- **新增/加深**：存在可观察 pre-load signal，且 child 相比 parent 有稳定、通过质量门槛的净收益； +- **合并/移动边界**：节点长期难区分且分离没有净价值； +- **提升/折叠**：child 对 parent 的大多数有效任务都不可缺少； +- **删除**：节点没有独立 minimum-sufficient 或 marginal-lift 案例； +- **拆分**：叶子出现重复失败簇，并且能在加载前观察到稳定边界。 + +执行 depth 与 Retrieval depth 都只表示渐进披露，不是通用任务复杂度分数。 ## 验证 -公共回归与真实仓库 held-out 使用 `gpt-5.6-luna`、medium reasoning。迭代阶段使用 `n=1`；发布结论必须完成 current-only 全矩阵 `n=3`。 +不需要真实外部 provider 的确定性契约检查： ```powershell -pwsh -NoProfile -File benchmarks/run.ps1 -SelfTest -pwsh -NoProfile -File benchmarks/run.ps1 -ProgressiveSelfTest +python benchmarks/dependency_tree_validation.py --self-test +python benchmarks/retrieval_validation.py --self-test +python benchmarks/retrieval_analysis.py /dev/null --self-test +python -m unittest ` + benchmarks.test_tree_benchmarks ` + benchmarks.test_capability_environment ` + benchmarks.test_dependency_tree_validation ` + benchmarks.test_retrieval_analysis +``` -python benchmarks/run_catalog.py --profile full --runs 3 --workers 3 ` - --arm practical-current --arm practical-native --output benchmark-results/public-final +Retrieval 树模型迭代必须存在全部依赖，先跑 `n=1`： -python benchmarks/progressive_validation.py --phase all --current-only --runs 3 --workers 3 ` - --output benchmark-results/heldout-final +```powershell +python benchmarks/retrieval_validation.py --current-only --runs 1 --workers 3 ` + --output benchmark-results/retrieval-tree-n1 +python benchmarks/retrieval_analysis.py benchmark-results/retrieval-tree-n1/results.jsonl ` + --output benchmark-results/retrieval-tree-n1/analysis.json +``` + +如果执行树文案或边界也发生变化，另跑 `dependency_tree_validation.py`，在同一 provider 与 warm-up 契约下保留 Core/Debugging/Implementation ceiling。 + +候选冻结后，再执行 no-skill、v1.5 baseline 与当前版本的 `n=3` Retrieval 配对比较： + +```powershell +python benchmarks/retrieval_validation.py --runs 3 --workers 3 ` + --output benchmark-results/retrieval-tree-final +python benchmarks/retrieval_analysis.py benchmark-results/retrieval-tree-final/results.jsonl ` + --output benchmark-results/retrieval-tree-final/analysis.json +``` + +如果 release candidate 同时修改两棵树，还要运行： + +```powershell +python benchmarks/dependency_tree_validation.py --runs 3 --workers 3 ` + --output benchmark-results/execution-tree-final +python benchmarks/tree_analysis.py benchmark-results/execution-tree-final/results.jsonl ` + --output benchmark-results/execution-tree-final/analysis.json ``` -历史报告只证明生成它的版本；除非在同一冻结矩阵中重跑，否则只能做非配对参照。 +已接受的 v1.5 扁平 Event Router，以及被拒绝的固定深度、专家叶子和 execution-state 实验，都继续作为历史证据保存在 `benchmarks/results/` 与 `evolution/rejected/`。历史报告不会为了适配新拓扑而重写。 -MIT License。第三方归属见 `THIRD_PARTY_NOTICES.md`。 +MIT License。Provider 归属见 [`THIRD_PARTY_NOTICES.md`](THIRD_PARTY_NOTICES.md)。 diff --git a/SKILL.md b/SKILL.md index 66d07fc..c751e31 100644 --- a/SKILL.md +++ b/SKILL.md @@ -1,70 +1,100 @@ --- name: practical-coding -description: "Use for implementing, fixing, refactoring, or reviewing code with the smallest correct change; routes only unresolved debugging, decision, or execution-boundary blockers, while retrieval expands independently." +description: "Use for implementing, fixing, refactoring, reviewing, or explaining code with the smallest correct change; execution and retrieval disclose through separate local trees while host capabilities remain replaceable infrastructure." license: MIT metadata: author: Hubujiu - version: "1.5" + version: "2.0" --- # Practical Coding -Use the Core for every coding task. Load one reasoning reference only for a present unresolved event; expand retrieval independently. +Use Core for every coding task. Core is the root of the automatic execution tree. Retrieval is a separate progressive tree that expands only when current source evidence is insufficient. Host tools are capabilities beneath those policies, not routing nodes. -Before the first diagnostic, decision-research, or change-mapping source command, apply the Event Router. If a condition matches, its reference is the next read. Otherwise stay Direct. +A loaded node may disclose only its own immediate children. It must not know or select descendants owned by another node. ## Core +**Execution tree depth: 0** + - Read the request and touched code; define the smallest observable success. - Stop at the first rung that works: do nothing; reuse the nearest project primitive; standard library; platform feature; available dependency; one line; otherwise minimum local code. -- Reuse established APIs and established contracts. Build only behavior required by a current caller or requirement; nearby richness is not a requirement. -- When one established primitive owns a shared behavior, repair it once instead of adding caller-specific branches or modes. +- Reuse established APIs and contracts. Build only behavior required by a current caller or requirement; nearby richness is not a requirement. +- When one established primitive owns shared behavior, repair it once instead of adding caller-specific branches or modes. - Add no speculative options, wrappers, aliases, configuration, scaffolding, helper layers, or one-implementation interfaces. - Make the smallest coherent reachable change. A standalone artifact needs no demo; a user-facing feature must be reachable. Preserve unrelated code and user changes. - Prefer deletion. Remove each new dependency, file, option, wrapper, comment, fallback, retry, test, or document not required by behavior, project contract, or verification. -- Run the cheapest focused check once after the final edit. If no repository test exists, run one focused executable check, not a broad test runner. Prefer no-write check modes (for example Python `-B`). Never repeat an unchanged check or replace a required build gate with diff inspection. If disposable output remains, clean it once when safe; after a blocked or failed cleanup, stop and report it without another inspection or command. Install declared dependencies only as a bounded prerequisite in isolation; otherwise report the missing prerequisite. +- Run the cheapest focused check once after the final edit. If no repository test exists, run one focused executable check, not a broad test runner. Prefer no-write check modes. Never repeat an unchanged check or replace a required build gate with diff inspection. Install declared project dependencies only as a bounded prerequisite in isolation; otherwise report the missing prerequisite. - State only fresh evidence. Unless requested, finish with the outcome, changed surface, check, and remaining uncertainty—no process recap. -## Direct Path +## Root Router + +Route only when Core cannot safely resolve the present execution blocker. These are the only automatic execution children known at depth 0: -When no Event Router condition matches, proceed with the Core alone. Targeted reads and searches are ordinary Direct work. +1. An observed failure, regression, incorrect behavior, or failed check still lacks an evidenced cause: load `references/debugging.md`. +2. Safe execution is blocked by an unknown contract or invariant; required producers and consumers must change together but their joint contract is unknown; a material security, irreversible-effect, persistence/migration, concurrency/transaction, or compatibility boundary remains unresolved; or sufficient evidence for a risky material claim is unknown: load `references/implementation.md`. -## Event Router +Otherwise stay at Core. Unknown locations, callers, consumers, file count, or data flow are retrieval questions, not automatic execution children. -Route only a present unresolved blocker. Settled facts and choices are inputs, not events; risk or technology nouns do not route by themselves. +A routed node owns its next decision. Do not return to Core merely to discover a descendant. Do not preload siblings or descendants. If a node declares itself a leaf, resolve there unless the task becomes a genuinely different top-level blocker. -Use this first-match ladder: +## Convergence Rule -1. An observed failure, regression, incorrect behavior, or failed check still lacks an evidenced cause: read `references/debugging.md`. -2. A material user-owned choice about architecture, dependency, implementation, API, data model, or compatibility remains unresolved and would change the next action: read `references/decision.md`. -3. Safe execution is blocked by an unknown contract or invariant; required producers and consumers must change together but their joint contract is unknown; a material security, irreversible-effect, persistence/migration, concurrency/transaction, or compatibility boundary remains unresolved; or evidence sufficient for a risky material claim is unknown: read `references/implementation.md`. +Automatic execution routing may deepen only to resolve a current blocker. It must not reopen deliberation. -Read exactly that reference plus the Core. Resolve the blocker, then contract. Do not preload candidates. A failed check of your proposed change stays inside the active event; correct the candidate without loading Debugging. Reassess only for a different later blocker; handle a trivial one with the Core or isolate a substantial one when the saved context exceeds handoff cost. +- Do not automatically load Decision from Core or from any execution node. +- When implementation exposes an ordinary technical choice, reuse the established project convention or choose the smallest sufficient reversible option and continue. +- When a genuinely user-owned choice blocks progress and no safe default exists, ask only the minimum blocking question in the current context. Do not activate the Decision workflow unless the user explicitly requested decision analysis. +- A failed check of the current candidate stays inside the active node when its cause is the candidate itself; correct it there instead of opening a fresh routing cycle. -Stay Direct when the cause, choice, governing boundary, affected surface, and sufficient check are already established. A named target with settled behavior remains Direct even when it concerns risk; a requested standalone artifact with no integration remains Direct. Unknown locations, file count, callers, consumers, and data flow are Retrieval questions, not Implementation events. Read-only source mapping is never an Implementation event. Choosing evidence sufficient to support a material risk or performance claim is an Implementation boundary, not a user-owned product Decision. +## Manual Modes -## Explicit-only requirements interview +Manual modes are outside both automatic trees and never appear in an automatic capability path. -Load `references/manual/clarification.md` only when the current instruction explicitly asks to be interviewed, grilled, or questioned before implementation. Ambiguity, importance, risk, or one unavoidable blocking question does not activate it. Decision resolves a genuinely open material choice; alternatives alone do not activate it. +- Load `references/manual/decision.md` only when the current user explicitly asks to compare options, make a technical choice, recommend an architecture/dependency/API/data-model approach, or otherwise perform decision analysis. +- Load `references/manual/clarification.md` only when the current user explicitly asks to be interviewed, grilled, questioned, or to clarify requirements before implementation. +- A manual mode must not automatically route to another manual mode or into an automatic descendant. After the requested manual work is resolved, return to Core with the settled result as input. ## Retrieval Policy -Retrieval is orthogonal to execution. Stop at the first sufficient rung: +Retrieval is orthogonal to execution. Its depth represents the unresolved information problem, not the strength or brand of an available tool. + +When source evidence is needed, load `references/retrieval/SKILL.md`. Core knows only the Retrieval root; it does not know or select that root's descendants. Every retrieval node owns only its immediate child decision and returns as soon as the minimum evidence needed for the current claim has been established. + +Do not choose a retrieval depth from Core in one global decision. Do not route by tool name. Do not preload deeper retrieval modules or copy the full benchmark topology into a runtime node. + +Runtime retrieval uses the strongest already-available capability appropriate to the current node and falls back losslessly to bounded repository-native search. Material conclusions must be verified in current source. A benchmark profile may deliberately require concrete providers; that requirement belongs to the benchmark environment, not to the runtime tree. + +Once candidate paths or symbols are known, stop inventory and switch to bounded line ranges or symbol reads; do not dump whole files or repeat broad discovery. Batch independent bounded reads only when each source is required by a current claim. + +Use a structural code index only at R3, when the unresolved answer is a call, dependency, ownership, control/data-flow, or impact relationship. Provider output proposes evidence; current source establishes it. + +## Navigation Boundary + +Load `references/navigation.md` only when the unresolved question is which bounded repository area should be searched. Navigation returns a small topology map; it does not perform semantic evidence discovery, choose a search provider, or tour the repository. + +After the area is bounded, use the Retrieval tree to identify the concrete evidence. If the target is already known, skip Navigation, load the Retrieval root, and let that root start at R0. + +## Execution Output Layer -1. Read a known path or symbol directly. Do not inventory history, branches, or unrelated files, or search outside the project for an implementation unless the blocker or request requires it. -2. Otherwise use an available bounded/ranked source search, falling back to filename, text, or symbol search. -3. For unknown callers, dependencies, authoritative boundaries, or cross-file guarantees, prefer an available structural code index when it materially reduces exploration. Stay Bounded when known identifiers or a finite known consumer set can be located by text search; a known edit target alone is not Targeted when relationships are unknown. -4. For bounded exhaustive repository claims, use coverage-aware discovery and disclose gaps. For external contracts, use the smallest authoritative current source. -5. Fall back without installing retrieval tooling; verify material conclusions in current source. Use NONE when only user-owned policy is missing, and retrieve only facts needed to resolve it. +Shell, test, build, and Git output may pass through an already-configured output-compaction layer. This is cross-cutting infrastructure, not Navigation, Retrieval, Verification, or execution depth. A host adapter should make it transparent when the host supports command hooks; otherwise expose only the thinnest wrapper instruction needed to use it. -Routine lookup stays here; do not load `references/navigation.md`. Load it only for substantial broad structural mapping or bounded exhaustive discovery. Do not add Navigation beside a reasoning reference merely to search; use this policy or isolate the mapping when worthwhile. +Compaction must preserve command semantics, exit status, failures, and enough evidence to verify the material claim. Never change the requested check merely to obtain shorter output. If compact output omits evidence needed for diagnosis, retrieve that bounded evidence without disabling compaction globally. ## Isolation Gate -Direct work and one routed event in small context use no worker. Keep the root at Core plus one active reasoning reference. +Core and one small routed node use no worker. Use `references/delegation.md` only when isolation saves more context than the handoff costs. Navigation and Debugging workers are read-only. Implementation writes only an assigned non-overlapping scope as sole writer. Manual Decision is read-only unless the user separately authorizes implementation. Never overlap writers or build worker pipelines. -When isolation saves more context than its handoff cost, dispatch one worker with `references/delegation.md` and one assigned reference. Navigation and Debugging workers are read-only. Decision is read-only unless the root authorizes settled implementation. Implementation writes only an assigned non-overlapping scope as sole writer. Never overlap writers or build worker pipelines. +## Evolution Contract -## Evolution contract +Runtime agents do not read `evolution/`. Neither tree is a fixed taxonomy. -Runtime agents do not read `evolution/`. Maintenance records benchmark and real-project receipts there, freezes experiments before changing runtime rules, and preserves rejected changes. Never add benchmark-specific nouns or keep a module for symmetry; each runtime module must earn quality-qualified net lift over its smaller parent. +- Every runtime node owns its behavior, current depth, and only its immediate-child router; a leaf says so explicitly. +- Retrieval policy, capability providers, output transport, and maintenance workflows remain separate concerns. A provider must not become a tree node merely to expose a tool. +- On an `experiment/*` branch, a proposed child may be staged only to collect controlled parent-versus-child and adaptive-routing evidence. Staging is not promotion. +- Promote a staged child into a release topology only when a repeatable pre-load signal exists and parent-versus-child ablation shows quality-qualified net lift across multiple tasks or repositories. +- Merge siblings when their boundary is persistently ambiguous and separation adds no net value. +- Promote a child into its parent when the child is needed for most parent tasks. +- Remove a child that does not independently improve qualified outcomes enough to justify context and routing cost. +- Split or deepen a node only when failures form a stable, observable task cluster that a narrower capability fixes. +- Benchmark evidence may change node names, boundaries, branching factor, or depth. Do not preserve symmetry, numeric levels, or historical route labels for compatibility. diff --git a/THIRD_PARTY_NOTICES.md b/THIRD_PARTY_NOTICES.md index 49371d5..4155a27 100644 --- a/THIRD_PARTY_NOTICES.md +++ b/THIRD_PARTY_NOTICES.md @@ -1,42 +1,42 @@ # Third-Party Notices -## DeusData/codebase-memory-mcp +Practical Coding does not vendor the provider source trees or release binaries listed below. Normal Skill use can fall back when a provider is absent. The dependency-enabled benchmark deliberately requires the declared executables so paired runs share one concrete capability surface. + +## zvec-ai/zvec-grep -Practical Coding recognizes `DeusData/codebase-memory-mcp` as one optional mature structural-retrieval backend when it is already available in the host environment. +- Project: `zvec-ai/zvec-grep` +- Executable used by the benchmark: `zg` +- Frozen benchmark profile version: `0.2.0` +- Source: https://github.com/zvec-ai/zvec-grep +- License: Apache License 2.0 +- Role: local ranked lexical + semantic retrieval for R1 Ranked Discovery and bounded R2 Evidence Expansion. + +Provider results are candidate evidence, not repository truth. Material conclusions are checked in current source. + +## DeusData/codebase-memory-mcp - Project: `DeusData/codebase-memory-mcp` +- Executable used by the benchmark: `codebase-memory-mcp` +- Frozen benchmark profile version: `0.10.8` - Source: https://github.com/DeusData/codebase-memory-mcp - License: MIT -- Upstream revision reviewed when the direct-backend policy was established: `010569fa6ce1bc5d6430f858129243ea1a2e3fd5` - -Practical Coding does not vendor the upstream source tree or release binaries, and it does not require Codebase Memory for normal operation. The Skill does not automatically install or persist the backend solely for retrieval; if no structural index is already available, retrieval falls back to bounded source search. - -This choice keeps parser accuracy, Tree-sitter grammars, Hybrid LSP resolution, semantic search, indexing, coverage reporting, concurrency, and graph queries owned and maintained upstream instead of being copied into a divergent Practical Coding implementation. +- Role: graph-aware callers, callees, dependencies, flows, and impact evidence for the R3 Structural Trace leaf. -If Practical Coding later vendors upstream code or carries a source patch, retain the upstream copyright and MIT license terms with the copied/substantial portions. +Benchmark cells create distinct per-workspace graph identities while sharing one explicit account-daemon cache cohort for the run. The selected cohort is recorded; graph identity, freshness, and coverage must be checked, and material paths are verified in current source. -The upstream MIT license is reproduced below for attribution. +## rtk-ai/rtk -```text -MIT License +- Project: `rtk-ai/rtk` +- Executable used by the benchmark: `rtk` +- Frozen benchmark profile version: `0.47.0` +- Source: https://github.com/rtk-ai/rtk +- License: Apache License 2.0 +- Role: cross-cutting compaction of noisy shell, test, build, and Git output. -Copyright (c) 2025 DeusData +Output compaction is infrastructure rather than a Retrieval node. It must preserve command semantics, exit status, failures, and sufficient evidence for the current claim. -Permission is hereby granted, free of charge, to any person obtaining a copy -of this software and associated documentation files (the "Software"), to deal -in the Software without restriction, including without limitation the rights -to use, copy, modify, merge, publish, distribute, sublicense, and/or sell -copies of the Software, and to permit persons to whom the Software is -furnished to do so, subject to the following conditions: +## Distribution boundary -The above copyright notice and this permission notice shall be included in all -copies or substantial portions of the Software. +The repository records executable probes and setup commands in `benchmarks/capability_manifest.json`, but it does not redistribute provider binaries, embedding models, or cached indexes. Install each provider from its maintained upstream distribution and review its own license, security, data-handling, and configuration documentation. -THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR -IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, -FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE -AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER -LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, -OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE -SOFTWARE. -``` +If Practical Coding later vendors upstream code or carries a source patch, retain all required copyright and license notices with the copied or substantial portions. diff --git a/agents/openai.yaml b/agents/openai.yaml index 43911f5..344b9b1 100644 --- a/agents/openai.yaml +++ b/agents/openai.yaml @@ -1,6 +1,6 @@ interface: display_name: "Practical Coding" - short_description: "Small correct changes with event routing and bounded retrieval" - default_prompt: "Use $practical-coding. Apply the Core, route only a present unresolved Debugging, Decision, or Implementation event, and use the cheapest sufficient retrieval capability. Load at most one reasoning reference for the current event." + short_description: "Small correct changes with progressive execution and retrieval" + default_prompt: "Use $practical-coding. Apply Core as the automatic execution root and route only a present Debugging or Implementation blocker. Keep Retrieval as its own progressive path: start at Direct Locate, then disclose only the current node's immediate child when the information problem remains unresolved. Use Navigation only to bound a repository area. Treat ranked search, graph retrieval, and command-output compaction as replaceable capabilities outside both trees. Decision and Clarification are explicit-only manual modes. Stop at the minimum current-source evidence and run the cheapest focused check that can falsify the material claim." policy: allow_implicit_invocation: true diff --git a/benchmarks/HIGH_STAR_FUSION.md b/benchmarks/HIGH_STAR_FUSION.md new file mode 100644 index 0000000..7b857d6 --- /dev/null +++ b/benchmarks/HIGH_STAR_FUSION.md @@ -0,0 +1,153 @@ +# High-star skill fusion experiment + +Date: 2026-09-01 +Branch: `experiment/evolvable-router-tree` + +## Goal + +Use strong external coding-skill patterns as **candidate capabilities**, not as a larger always-on workflow. The experiment asks whether a narrower descendant actually earns its context cost over the parent node. + +The active root remains intentionally small: + +```text +Core +├── Debugging +│ └── Dynamic Evidence [staged] +└── Implementation + ├── Security Boundary [staged] + ├── Migration & Compatibility [staged] + └── State & Concurrency [staged] +``` + +Decision and Clarification remain manual-only. Retrieval remains orthogonal. + +## External mechanisms reviewed + +The research pool includes SkillsBench / Skill-Use, Superpowers, Ponytail, Addy Osmani's `agent-skills`, Matt Pocock's `skills`, Anthropic's public skill examples, wshobson/agents, Thermos, swell-agents/coding-skills, and other large engineering-skill collections. + +The experiment deliberately **does not** import their full lifecycle taxonomies. + +| Mechanism | Placement | Decision | +|---|---|---| +| smallest-correct-change / delete-first | Core | Already present; keep always-on and tiny | +| systematic root-cause tracing | Debugging | Already present | +| runtime instrumentation / condition-based async evidence | Debugging → Dynamic Evidence | Stage as a depth-2 candidate | +| security hardening | Implementation → Security Boundary | Stage only for explicit trust-boundary invariants | +| deprecation / migration compatibility | Implementation → Migration & Compatibility | Stage for coexistence and rollback surfaces | +| transactions / idempotency / concurrency | Implementation → State & Concurrency | Stage for ordering/atomicity/state-owner problems | +| TDD / verification-before-completion | Cross-cutting evidence | Do not create a node merely because testing is useful | +| context engineering / source-driven lookup | Retrieval | Keep orthogonal to execution depth | +| spec / grill / architectural choice | Manual Decision or Clarification | Never restore automatic deliberation | +| code review / simplification | Core for ordinary explicit review | Do not auto-run a review phase after every change | +| worktrees / multi-agent review | Delegation / harness capability | Do not make repository mechanics an execution node | +| performance optimization | Parent or Dynamic Evidence when an observed regression lacks measurements | Do not create a noun-only Performance node yet | +| shipping / CI / release orchestration | Outside the automatic tree until a repeatable coding blocker earns it | Avoid importing an SDLC pipeline | + +## Why this topology + +Three findings drive the experiment: + +1. **Focused skills beat broad bundles.** SkillsBench reports that compact curated skills outperform exhaustive bundles, so depth should buy specificity without turning the root into a catalog. +2. **Routing is a separate capability.** Skill-Use decomposes skill use into Trigger, Compliance, and Boundary. A child that helps when forced but is triggered badly is not a good runtime node. +3. **Minimalism and rigor are not competing roots.** Ponytail-style minimalism belongs in Core; deeper engineering discipline should appear only at evidence-backed boundaries. + +## Complexity contract for staged children + +Every staged child must satisfy all of these: + +- one observable pre-load signal stated in its parent; +- one narrow reference file; +- no mandatory new dependency or tool; +- a lightweight fallback: if the signal disappears, stay in the parent; +- no sibling preloading; +- no new automatic Decision path; +- no broad test suite unless the material surface requires it. + +The benchmark records token, duration, and tool-call cost. A child with no independent quality lift is removed even if its advice is individually reasonable. + +## Benchmark protocol + +### A. Main paired quality benchmark + +Use the existing frozen real repositories and every root-to-node capability ceiling: + +```powershell +python benchmarks/tree_validation.py --runs 3 --workers 3 +python benchmarks/tree_analysis.py /results.jsonl --topology benchmarks/tree_topology.json --output /tree-analysis.json +python benchmarks/tree_skilluse_analysis.py /results.jsonl --topology benchmarks/tree_topology.json --output /skill-use.json +``` + +Run `--runs 1 --current-only` only while debugging the harness. Freeze prompts, topology, repositories, and scorer before the n=3 run. + +Each ordinary task is evaluated under: + +- no skill; +- frozen previous Practical baseline; +- adaptive current tree; +- Core-only ceiling; +- every root-to-node ceiling. + +Manual tasks remain no-skill / baseline / adaptive and must never contaminate the automatic path. + +### B. Capability-derived Trigger / Compliance / Boundary + +`tree_skilluse_analysis.py` avoids a human-authored gold automatic route. + +For a node `C` with parent `P`: + +- **positive Trigger opportunity**: `cap:C` is stable-passing and `cap:P` is not; +- **negative Boundary opportunity**: `cap:P` is already stable-passing; +- **Trigger recall**: adaptive runs select `C` or a descendant on positive opportunities; +- **Compliance**: adaptive runs that selected `C`/descendants still deliver a passing result; +- **Boundary specificity**: adaptive runs do **not** enter `C`/descendants when the parent was already sufficient. + +This preserves the branch's principle that topology is inferred from capability evidence rather than scored against a predefined taxonomy. + +### C. Candidate promotion gate + +A staged child is eligible for promotion only when all are true: + +1. at least 2 stable marginal-lift tasks; +2. those tasks span at least 2 repositories; +3. Trigger recall ≥ 0.80; +4. Boundary specificity ≥ 0.90; +5. Compliance when triggered ≥ 0.90; +6. the main adaptive quality/non-inferiority gate passes; +7. zero spontaneous manual-mode activation; +8. all adaptive paths are valid parent-child paths; +9. cost is reviewed against the quality gained; +10. no benchmark case leaks child wording, file-specific answers, or expected constants into the skill. + +These thresholds are experiment defaults, not permanent product constants. + +### D. Removal / merge rules + +Remove a staged child when it has no independent minimum-sufficient or marginal-lift cases. Merge or move sibling boundaries when capability ceilings repeatedly make siblings co-minimum and adaptive routing confuses them without net quality benefit. Promote behavior into the parent if the child becomes necessary for most parent-scope tasks. + +## Required benchmark expansion before release promotion + +The current frozen tree suite is useful for routing and repository-evidence behavior, but descendant promotion should not rely only on keyword evidence. Before release promotion, add executable tasks with deterministic verifiers for each surviving child, following the SkillsBench pattern: + +```text +task/ +├── task.md +├── environment/ +├── oracle/ +│ └── solve.* +└── verifier/ + └── test.* +``` + +Minimum target inventory: + +- Dynamic Evidence: 4 tasks / 2 repositories or fixtures +- Security Boundary: 4 tasks / 2 repositories or fixtures +- Migration & Compatibility: 4 tasks / 2 repositories or fixtures +- State & Concurrency: 4 tasks / 2 repositories or fixtures +- 1–2 hard negatives per child that look topically similar but should stop at the parent + +Every oracle must pass before agent runs. Prefer behavior checks over LLM judges. Skills must encode reusable procedure, never benchmark-specific filenames, constants, or solution commands. + +## Expected outcomes + +This experiment is allowed to conclude that **none** of the four descendants should survive. A useful external practice is not automatically a useful runtime node. The target is the smallest topology on the quality/cost Pareto frontier, not the deepest tree. diff --git a/benchmarks/NEXT_VALIDATION.md b/benchmarks/NEXT_VALIDATION.md index a8f89cc..cf5392c 100644 --- a/benchmarks/NEXT_VALIDATION.md +++ b/benchmarks/NEXT_VALIDATION.md @@ -1,45 +1,45 @@ -# Release validation protocol — event-router restoration +# Release validation protocol — evolvable local-router tree -This protocol freezes the final validation for `experiment/progressive-ladders` after n=1 iteration. +This protocol freezes final validation for `experiment/evolvable-router-tree` after n=1 mechanism iteration. -## Candidate contract +## Frozen candidate -- Core plus Direct default; -- exactly three adaptive reasoning modules: Debugging, Decision, Implementation; -- retrieval orthogonal and cheapest-sufficient; -- requirements interviewing explicit-only; -- no numeric execution/retrieval runtime depths or specialist leaves. +- Core owns only Debugging and Implementation as automatic children. +- Debugging and Implementation are leaves; repeated ablation did not earn a depth-2 child. +- Decision and Clarification remain explicit-only manual modes. +- Retrieval remains orthogonal to execution-tree depth. +- The deterministic scorer normalizes equivalent semantic acts, outcome wording, ordinary inflections, and Windows/POSIX reference paths without changing the frozen task prompts. + +Qualified n=1 artifact: `benchmark-results/tree-delivery-n1-retired-isolated-20260902` (58/58 determinate cells; adaptive 15/15; every capability ceiling 13/13; all traces/manual contracts valid). ## Iteration gate -Use n=1 while changing a mechanism. Save the full result, classify failures as infrastructure, scorer/oracle, stochastic, routing, or genuine capability failures, and record reusable lessons under `evolution/`. Never add case-specific nouns to runtime text. +Use n=1 while changing runtime wording, topology, cases, scorer contracts, or evidence normalization. Save each complete artifact, classify every failure, and write an immutable receipt before consolidating the reusable mechanism into `evolution/wiki/`. A scorer correction invalidates all affected model-backed results; rerun the complete matrix in a fresh directory. -## Final gate +## Final paired gate -The candidate must be committed and unchanged before both commands run: +Commit and freeze runtime, topology, cases, scorer, repository refs, model, and harness before running: ```powershell -python benchmarks/run_catalog.py --profile full --runs 3 --workers 3 ` - --arm practical-current --arm practical-native ` - --output benchmark-results/event-router-final-public +python benchmarks/tree_validation.py --runs 3 --workers 3 ` + --output benchmark-results/tree-final -python benchmarks/progressive_validation.py --phase all --current-only --runs 3 --workers 3 ` - --output benchmark-results/event-router-final-heldout +python benchmarks/tree_analysis.py benchmark-results/tree-final/results.jsonl ` + --output benchmark-results/tree-final/analysis.json ``` Required evidence: -- zero indeterminate cells and at least three determinate repetitions per cell; -- no Delivery correctness/safety/build regression; -- Debug, Decision, and Native Behavior stable enough for a release claim; -- Router reasoning and retrieval reported separately; -- at least 20 held-out real tasks across multiple repositories; -- zero spontaneous requirements-interview activation; -- held-out quality and routing failures individually classified; -- raw machine paths excluded from published compact artifacts. +- every expected cell exists, is determinate, and has three unique repetitions; +- adaptive delivered quality is strictly better than the frozen v1.5 baseline on the same cases and no worse than no-skill on required correctness/safety/reachability; +- zero adaptive trace failures and zero spontaneous manual-mode activation; +- both explicit manual tasks load the requested manual mode in every repetition; +- parent-versus-child capability evidence uses all three repetitions before retaining, removing, promoting, or merging a node; +- raw machine paths remain in ignored artifacts and are excluded from published compact reports; +- report version, commit, model, harness, frozen repository refs, cell counts, pass counts, noninferiority results, and limitations separately. -Historical v1.2 reports may be compared offline, but this current-only cycle cannot make a paired superiority claim against v1.2, no-skill, Ponytail, or combined skill arms. +Router exactness and topology diagnostics do not override delivered quality. A favorable n=1 cell or incomplete split rerun is not release evidence. ## Merge gate -Update the formal README and compact result artifacts from the final reports, run all unit/self/Skill validation, push the branch, and require PR CI success. If a genuine quality or stable reasoning regression remains, return to n=1 iteration and freeze the next mechanism change before editing runtime rules. +Update the formal README and compact result artifacts from the complete n=3 report, run all unit/self/Skill validation, and require PR CI success before merge. If a genuine stable quality regression or unearned staged node remains, return to a new frozen n=1 hypothesis and repeat the gate; do not edit the n=3 artifact in place. diff --git a/benchmarks/README.md b/benchmarks/README.md index 6ad3ada..187a6a6 100644 --- a/benchmarks/README.md +++ b/benchmarks/README.md @@ -1,46 +1,174 @@ # Practical Coding benchmark chain -The active release candidate uses one Core, a Debugging/Decision/Implementation Event Router, and orthogonal retrieval. The rejected E/R depth and specialist-leaf experiment remains historical evidence under [`results/progressive-tree/`](results/progressive-tree/) and [`../evolution/rejected/`](../evolution/rejected/). +The active experiment uses two independent evolvable trees: + +- the automatic execution tree starts at Core and currently exposes Debugging and Implementation leaves; +- the Retrieval tree progresses from R0 Direct Locate through R1 Ranked Discovery and R2 Evidence Expansion to the R3 Structural Trace leaf. + +Decision and requirements interviewing are explicit-only manual modes. Ranked search, graph retrieval, and execution-output compaction are providers outside both trees. + +The accepted v1.5 flat Event Router and rejected fixed E/R ladder remain historical baselines. Do not use their fixed labels, numeric depths, or gold routes as the acceptance oracle for this experiment. ## Active questions -1. Does the Skill deliver a correct, safe, reachable result? -2. Does it load the one reasoning module required by the present unresolved event—and no module for Direct work? -3. Does retrieval stop at the cheapest sufficient capability? -4. Does requirements interviewing remain at zero spontaneous activation? +1. Does the candidate deliver a correct, safe, evidence-backed result at least as reliably as the v1.5 baseline and no-skill arm? +2. Which automatic execution nodes are minimum-sufficient under parent-versus-child ceilings? +3. Does adaptive execution disclosure stop without spontaneous manual Decision or Clarification activation? +4. Does Retrieval start at R0 and escalate only through the current node's immediate child? +5. Does each Retrieval stage stop at the minimum current-source evidence required by the task? +6. With the required provider surface held constant, does the candidate improve quality or measured context cost? +7. Do repeated failures or boundary ambiguity justify growing, splitting, merging, promoting, collapsing, moving, or removing a node? + +## Runtime topology + +- `tree_topology.json` — execution nodes, manual modes, Retrieval nodes and edges, trace modes, dependency profile, and frozen baseline ref; +- `tree_cases.py` — topology-neutral real-repository tasks with no expected automatic execution route; +- `tree_validation.py` — underlying quality/scoring runner and historical-compatible execution trace parser; +- `dependency_tree_validation.py` — required active execution-tree runner that injects fail-closed provider setup before every measured turn; +- `retrieval_trace.py` — canonical R0–R3 parser plus observed Retrieval-reference extraction; +- `retrieval_validation.py` — independent Retrieval-stage ceiling runner; +- `retrieval_analysis.py` — minimum-sufficient Retrieval-depth and provider-use analysis; +- `tree_analysis.py` — minimum-sufficient execution-node and topology-change analysis; +- `TREE_EVOLUTION.md` — interpretation and mutation rules. + +`tree_validation.py` remains directly runnable for historical reproduction and deterministic topology self-tests. Provider-enabled cost claims must use `dependency_tree_validation.py`. + +## Required capability profile + +`capability_manifest.json` declares the exact environment: + +| Role | Required executable | Purpose | +|---|---|---| +| ranked retrieval | `zg` | R1 candidate discovery and bounded R2 support | +| graph retrieval | `codebase-memory-mcp` | R3 relationship tracing | +| execution output | `rtk` | compact noisy shell/test/build/Git evidence | +| repository warm-up | `node`/`npm`, `java`/`mvn` where declared | dependency and first-build parity | + +The active runner has no allow-missing mode. It resolves and probes every required executable before model cells are created. A provider setup or repository warm-up failure aborts the run rather than silently switching capability surfaces. + +## Measurement contract -## Iteration versus release +Every cell has an auditable setup receipt at: -Use `n=1` while changing mechanisms or scorer contracts. Run the complete `n=3` matrices only after focused n=1 evidence supports release. +```text +cells///rNNN/capability-setup.json +``` + +The setup phase includes: + +- provider probes and local runtime/model initialization; +- workspace `zg` indexing; +- a per-workspace Codebase Memory graph in one explicit shared daemon/cache cohort; +- `rtk` command-path verification; +- repository-specific dependency resolution and first test/build warm-up; +- post-setup clean-tree validation. + +Setup is marked `included_in_comparison: false`. It occurs before `run_codex`, produces no token estimate, and is absent from the `results.jsonl` measured usage fields. Only the later Codex transcript contributes input/output tokens, tool calls, and measured duration. + +All paired arms receive the same initialized providers and repository warm-up. A baseline may choose not to use a provider, but it may not receive a colder environment. + +The runner rejects reuse of a measured result without a matching setup receipt. It also marks measured provider installation/indexing commands as contract violations, including `zg index`, Codebase Memory indexing, `rtk init`, and package installation. + +## Retrieval trace contract + +The dependency runner emits only canonical modes: + +```text +NONE +R0_DIRECT +R1_DISCOVERY +R2_EVIDENCE +R3_STRUCTURAL +``` + +A trace that reports a stage must list the actually loaded Retrieval references as a complete root-to-stage prefix. For example, `R2_EVIDENCE` requires: + +```text +references/retrieval/SKILL.md +references/retrieval/direct.md +references/retrieval/discovery.md +references/retrieval/evidence.md +``` + +Legacy `TARGETED`, `BOUNDED`, and `STRUCTURAL` values remain parser-compatible only so historical result files can still be read. The active dependency runner does not emit them. + +`NONE` means no Retrieval policy reference was loaded; repository-native exact reads remain available as the no-tree control. `R0_DIRECT` begins by loading the Retrieval root followed by `direct.md`. + +For active arms, the declared Retrieval prefix must exactly match Retrieval reference paths observed in command execution. A self-reported stage cannot stand in for an unread node, and a hidden deeper read is a trace failure. + +## Deterministic validation + +These checks do not claim that external providers are installed; they validate topology, fail-closed preflight, setup separation and shared-cohort handling, receipt structure, and measurement boundaries with controlled shims: ```powershell -pwsh -NoProfile -File benchmarks/run.ps1 -SelfTest -pwsh -NoProfile -File benchmarks/run.ps1 -ProgressiveSelfTest +python benchmarks/dependency_tree_validation.py --self-test +python benchmarks/retrieval_validation.py --self-test +python benchmarks/retrieval_analysis.py /dev/null --self-test +python -m unittest ` + benchmarks.test_tree_benchmarks ` + benchmarks.test_capability_environment ` + benchmarks.test_dependency_tree_validation ` + benchmarks.test_retrieval_analysis ``` -Current-only public matrix: +CI runs these deterministic checks. The full model benchmark is intentionally not disguised as a unit test. + +## Model-backed iteration + +Install and verify the frozen dependency profile first. Exact accepted provider versions live in `capability_manifest.json`; preflight rejects a different provider version instead of mixing it into an older result set: ```powershell -python benchmarks/run_catalog.py --profile full --runs 1 --workers 3 ` - --arm practical-current --arm practical-native ` - --output benchmark-results/public-n1 +zg --version +codebase-memory-mcp --version +rtk --version +git --version +node --version +npm --version +java -version +mvn --version ``` -Current-only real-repository held-out: +Use `n=1` while changing topology, node content, provider contracts, cases, or scoring: ```powershell -python benchmarks/progressive_validation.py --phase all --current-only --runs 1 --workers 3 ` - --output benchmark-results/heldout-n1 +python benchmarks/retrieval_validation.py --current-only --runs 1 --workers 3 ` + --output benchmark-results/retrieval-tree-n1 +python benchmarks/retrieval_analysis.py benchmark-results/retrieval-tree-n1/results.jsonl ` + --output benchmark-results/retrieval-tree-n1/analysis.json ``` -Change `--runs 1` to `--runs 3` only for the frozen final candidate. +Only after freezing the candidate should it run `n=3` with baseline and no-skill arms: + +```powershell +python benchmarks/retrieval_validation.py --runs 3 --workers 3 ` + --output benchmark-results/retrieval-tree-final +python benchmarks/retrieval_analysis.py benchmark-results/retrieval-tree-final/results.jsonl ` + --output benchmark-results/retrieval-tree-final/analysis.json +``` ## Interpretation -- Delivery and Debug grade delivered behavior, safety, and build evidence. -- Decision grades compact two-turn convergence. -- Router grades reasoning selection and retrieval separately. -- Native Behavior verifies actual Skill discovery and module isolation. -- Held-out tasks use frozen commits from three real repositories and mechanically grade evidence coverage, executable probes, clean workspaces, event/retrieval traces, and spontaneous requirements interviewing. +Delivered quality gates the candidate. Exact historical route labels do not. + +For each non-manual task, the execution-tree runner exposes Core and every root-to-node capability ceiling. The analyzer marks stable passing ceilings, removes qualified descendants whose ancestor already passes, and reports the remaining set as the task's minimum-sufficient set. More than one minimum node is allowed. + +Adaptive execution traces are reported as: + +- `exact_minimum` — stopped on a derived minimum node; +- `over_disclosure` — went deeper than a sufficient ancestor; +- `under_disclosure` — stopped above a node needed by ceiling evidence; +- `alternate_branch` — selected a different branch; +- `quality_gap` — no current node ceiling solves the task reliably. + +Retrieval disclosure is analyzed separately through canonical stage traces, loaded-reference prefixes, provider-use counts, quality, and measured cost. A provider can be present without being used; presence is held constant, while stage and provider selection remain behavior under test. + +Manual modes retain a separate contract: ordinary tasks must have zero spontaneous manual activation; explicit requests must load the corresponding `references/manual/` mode. + +## Historical baselines + +- `progressive_validation.py`, `progressive_cases.py`, and `ladder_analysis.py` reproduce previous fixed E/R and flat Event Router experiments. +- `results/progressive-tree/` and `../evolution/rejected/` preserve rejected fixed-depth evidence. +- `results/v1.5/` preserves the accepted flat-router evidence frozen by `tree_topology.json`. +- `../evolution/rejected/execution-state/` preserves the retired execution-state/history-free experiment. -Historical reports are version-specific. Offline comparison with v1.2 is non-paired unless old and new arms are rerun together in one frozen matrix. +Do not rewrite historical contracts to make the current tree appear better. New topology or capability-policy claims require a frozen candidate, appropriate ablation, identical provider setup across arms, and real-repository evidence. diff --git a/benchmarks/TREE_DISCRIMINATOR.md b/benchmarks/TREE_DISCRIMINATOR.md new file mode 100644 index 0000000..eebf243 --- /dev/null +++ b/benchmarks/TREE_DISCRIMINATOR.md @@ -0,0 +1,21 @@ +# Tree discriminator benchmark + +This is the cheap routing-language diagnostic for staged automatic children. It complements, but never replaces, `tree_validation.py` capability ceilings and executable outcome verifiers. + +Run a harness check first: + +```powershell +python benchmarks/tree_discriminator_validation.py --self-test +``` + +Then run the frozen discriminator matrix: + +```powershell +python benchmarks/tree_discriminator_validation.py --runs 3 --workers 3 +``` + +The suite exposes only one parent node and a task summary. It does **not** expose child bodies. The model must return the immediate child name or `parent`. + +Cases include positive signals, ordinary parent-stay negatives, and sibling-confusion hard negatives. The report emits per-parent accuracy plus per-child Trigger recall, Boundary specificity, false-trigger counts, token use, tool calls, and duration. + +These labels are allowed here because the suite is testing whether the written Local Router distinguishes deliberately constructed boundary examples. They are diagnostic only. A child still needs empirically minimum-sufficient parent-vs-child lift in the real tree benchmark before promotion. diff --git a/benchmarks/TREE_EVOLUTION.md b/benchmarks/TREE_EVOLUTION.md new file mode 100644 index 0000000..62f431b --- /dev/null +++ b/benchmarks/TREE_EVOLUTION.md @@ -0,0 +1,228 @@ +# Evolvable local-router trees + +This experiment treats progressive disclosure topology as a learned maintenance artifact rather than a permanent taxonomy. Execution reasoning and source Retrieval are independent trees with different signals, ceilings, and analysis. + +## Invariants + +1. Core is the automatic execution root at depth 0. +2. Every automatic execution node owns its behavior, current depth, and only its immediate-child router. +3. A parent does not know grandchildren. A leaf says it has no earned children. +4. Manual Decision and Clarification are outside both automatic trees and require an explicit current user request. +5. Automatic execution routing may deepen to resolve a blocker but may not reopen deliberation. +6. Retrieval starts at its own root and progresses only through the current node's immediate child. +7. Retrieval depth describes the unresolved information problem, not repository size, execution risk, or provider strength. +8. Navigation only bounds the repository area. It is not semantic discovery, evidence expansion, or graph tracing. +9. Ranked retrieval, graph retrieval, and execution-output compaction are capability providers outside both trees. +10. Provider setup, indexing, dependency resolution, and first-build warm-up occur before model measurement and never enter compared token, duration, or tool-call fields. +11. Depth means disclosure depth only; branches do not need equal depth or symmetric children. + +## Current seed topologies + +### Execution + +```text +Core +├── Debugging (leaf) +└── Implementation (leaf) +``` + +### Retrieval + +```text +Retrieval Root +└── R0 Direct Locate + └── R1 Ranked Discovery + └── R2 Evidence Expansion + └── R3 Structural Trace (leaf) +``` + +The execution tree branches by blocker type. The Retrieval seed is a monotonic path because each stage answers a strictly deeper unresolved information question. Benchmark evidence may still merge, split, reorder, promote, or remove these nodes. + +## Why the old E0-E3 result does not reject trees + +The rejected experiment froze numeric levels and specialist families before evidence existed, then scored the model against those labels. That tested one predefined taxonomy. It did not test whether local progressive disclosure itself was useful. + +The active experiment reverses the dependency: + +```text +small candidate topology + ↓ +parent/stage capability ceilings + ↓ +minimum-sufficient node or Retrieval stage + ↓ +adaptive traces + delivered quality + measured cost + ↓ +topology mutation candidate + ↓ +new frozen experiment +``` + +The benchmark is therefore allowed to conclude that a node should disappear, move, merge, split, or gain a child. + +## Execution capability ceilings + +`dependency_tree_validation.py` wraps the existing execution-tree runner with the mandatory provider and warm-up contract. For every ordinary task it exposes: + +- Core only; +- every root-to-execution-node path in `tree_topology.json`; +- adaptive execution disclosure with the full current execution tree. + +A capability ceiling is not an expected route. It asks: *if no execution capability below this node were available, could the task still be delivered correctly?* + +At n=3 a ceiling is stable passing only when every determinate repetition passes. `tree_analysis.py` removes stable-passing descendants whose ancestor already passes. The remaining nodes form the task's minimum-sufficient execution-node set. Multiple minimum nodes are valid evidence of alternate sufficient branches or a weak boundary. + +## Retrieval capability ceilings + +`retrieval_validation.py` keeps the automatic execution tree adaptive while running the same task under these Retrieval ceilings: + +- `NONE` — current Skill without loading the Retrieval tree; +- `R0_DIRECT` — Retrieval root plus Direct Locate; +- `R1_DISCOVERY` — adds ranked candidate discovery; +- `R2_EVIDENCE` — adds bounded cross-file evidence construction; +- `R3_STRUCTURAL` — adds graph relationship tracing. + +All required provider binaries remain installed for every ceiling. The ceiling restricts policy references and deeper-stage provider use: + +- `rtk` is available at every stage because output compaction is not Retrieval; +- `zg` becomes available to Retrieval at R1; +- `codebase-memory-mcp` becomes available at R3; +- repository-native exact reads/search remain available at every stage. + +`NONE` is the no-Retrieval-policy control: no Retrieval reference is loaded, while repository-native exact reads remain available. `R0_DIRECT` is the first loaded policy prefix (`Retrieval root -> Direct Locate`). + +The active runner compares the declared prefix with reference paths actually observed in commands. Missing parent reads, hidden deeper reads, or claimed-but-unread nodes invalidate the trace. + +`retrieval_analysis.py` selects the shallowest stage whose repetitions all pass. That is the task's minimum-sufficient Retrieval stage. Adaptive disclosure is then classified as exact, over-disclosed, under-disclosed, invalid, or a quality gap. + +R2 and R3 are not justified merely because an agent loaded them. They survive only if their ceilings solve stable task clusters that shallower stages cannot solve with equal delivered quality. + +## Required provider environment + +`capability_manifest.json` requires three roles: + +- ranked retrieval through `zg`; +- graph retrieval through `codebase-memory-mcp`; +- execution-output compaction through `rtk`. + +The benchmark fails closed when a binary is missing, a probe fails, indexing fails, repository warm-up fails, or setup dirties the frozen workspace. There is no allow-missing path for provider-enabled claims. + +Normal Skill runtime remains portable and retains bounded fallbacks. Provider absence is a separate runtime condition, not noise mixed into this experiment. + +## Measurement boundary + +Each comparison cell has two phases. + +### Unmeasured setup + +Before Codex starts, the runner: + +1. probes required providers; +2. initializes local provider assets; +3. creates the workspace ranked index; +4. creates a per-workspace graph in one explicit shared Codebase Memory daemon/cache cohort; +5. warms each provider query path, repository dependencies, and the first focused test/build path; +6. verifies a clean worktree; +7. writes `capability-setup.json` with `included_in_comparison: false`. + +Setup elapsed time and output bytes are auditable but never merged into model records. Setup reports contain no token estimate. + +### Measured execution + +Only `run_codex` and the transcript after setup contribute: + +- input/cached-input/output/reasoning/total tokens; +- model-visible tool calls; +- measured duration; +- answer quality and routing trace. + +Every paired arm receives the same initialized environment. A measured attempt to reinstall packages, rebuild provider indexes, or initialize the output adapter is a contract failure rather than accepted cold-start cost. + +## What gates a candidate + +Release quality is primary: + +- adaptive delivered quality must remain non-inferior to the frozen v1.5 baseline and no-skill within the configured margin; +- ordinary tasks must have zero spontaneous manual-mode activation; +- explicit manual tasks must activate the requested manual mode; +- execution traces must describe a valid parent-child path; +- Retrieval traces must use canonical stages and list a complete loaded root-to-stage prefix; +- a Retrieval ceiling must not use a provider owned by a deeper stage; +- measured setup-violation count must be zero; +- all setup receipts must match the frozen capability manifest. + +Exact agreement with one human-authored execution node is deliberately not a release gate. Retrieval minimum stage is derived from quality-qualified ceilings, not prompt nouns. + +## Diagnostics + +### Execution + +- `exact_minimum` — adaptive execution stopped on a minimum-sufficient node; +- `over_disclosure` — a sufficient execution ancestor existed; +- `under_disclosure` — execution stopped above a required capability; +- `alternate_branch` — adaptive execution chose another branch; +- `quality_gap` — no current execution ceiling solves the task reliably. + +### Retrieval + +- `exact_minimum` — adaptive Retrieval stopped on the shallowest stable-passing stage; +- `over_disclosure` — a shallower stage already passed; +- `under_disclosure` — adaptive stopped before the minimum stage; +- `invalid_trace` — the canonical stage/reference-prefix contract failed; +- `quality_gap` — no current Retrieval ceiling solves the task reliably; +- `adaptive_quality_failure` — the adaptive result itself failed quality. + +These labels diagnose topology. They do not justify adding benchmark case nouns to runtime prompts. + +## Cross-cutting infrastructure + +A mechanism belongs outside both trees when it changes how every node executes rather than which task or information capability is selected. + +RTK-style output compaction is such infrastructure: + +- it may wrap shell, test, build, and Git output at any execution or Retrieval depth; +- a `Core -> Output Compression` or `R2 -> RTK` tree edge is meaningless; +- compaction must preserve semantics, exit status, failures, and material evidence; +- provider initialization is setup, while model-visible compact commands remain measured execution; +- replacing the provider should not require Retrieval-policy changes. + +Do not add a cross-cutting provider as a child merely to make it visible in a diagram. + +## Mutation rules + +### ADD / DEEPEN + +Add a child only when all are true: + +- a repeated failure cluster exists under one parent; +- the cluster has an observable pre-load signal that does not require loading the proposed child first; +- the child adds stable quality-qualified lift over the parent across multiple tasks or repositories; +- ordinary parent tasks do not pay the child context cost. + +### SPLIT + +Split a node when distinct failure clusters require materially different behavior and can be distinguished before loading either child. Do not split merely because domain nouns are recognizable. + +### MERGE / MOVE BOUNDARY + +Merge siblings or move their boundary when they are repeatedly co-minimum-sufficient, frequently confused by adaptive routing, and their separation does not produce net quality or context value. + +For the linear Retrieval seed, merge adjacent stages when the deeper stage has no independent minimum-sufficient cluster or the boundary cannot be observed before loading it. + +### PROMOTE / COLLAPSE + +Promote child behavior into its parent when the child is required for most of the parent's useful scope. A nearly mandatory child is not progressive disclosure. + +### REMOVE + +Remove a node when it has no independent minimum-sufficient cases and no stable marginal lift over its parent. Historical symmetry or a provider's existence is not a retention reason. + +## Experiment discipline + +- Use n=1 only for mechanism iteration and scorer correctness. +- Freeze runtime wording, both topologies, capability manifest, warm-up commands, cases, repositories, and scorer contracts before n=3. +- Compare a topology mutation against its immediate parent topology, not only against old public releases. +- Preserve raw outputs, setup receipts, provider preflight, and topology manifests with results. +- Do not edit a frozen case after seeing candidate output unless the oracle itself is demonstrably contradictory; record such corrections separately. +- Do not compare a warm candidate with a cold baseline or count setup output as model tokens. +- Do not reopen the rejected numeric E/R taxonomy merely to make the new trees look familiar. diff --git a/benchmarks/capability_environment.py b/benchmarks/capability_environment.py new file mode 100644 index 0000000..beeb75a --- /dev/null +++ b/benchmarks/capability_environment.py @@ -0,0 +1,60 @@ +#!/usr/bin/env python3 +"""Fail-closed capability setup for dependency-enabled model benchmarks. + +Provider probes, indexes, dependency resolution, and first-build warm-up run +before Codex starts. Their output and elapsed time are recorded separately and +never merged into measured token, duration, or tool-call data. +""" + +from __future__ import annotations + +try: + from .capability_manifest import ( + SCHEMA_VERSION, + CapabilityError, + CapabilityManifestError, + CapabilitySetupError, + MissingCapabilityError, + load_manifest, + manifest_fingerprint, + ) + from .capability_process import Run, preflight + from .capability_workspace import ( + contains_token_key, + prepare_workspace, + workspace_environment, + write_report, + ) +except ImportError: # direct script imports from the benchmarks directory + from capability_manifest import ( + SCHEMA_VERSION, + CapabilityError, + CapabilityManifestError, + CapabilitySetupError, + MissingCapabilityError, + load_manifest, + manifest_fingerprint, + ) + from capability_process import Run, preflight + from capability_workspace import ( + contains_token_key, + prepare_workspace, + workspace_environment, + write_report, + ) + +__all__ = [ + "SCHEMA_VERSION", + "Run", + "CapabilityError", + "CapabilityManifestError", + "CapabilitySetupError", + "MissingCapabilityError", + "load_manifest", + "manifest_fingerprint", + "preflight", + "prepare_workspace", + "workspace_environment", + "write_report", + "contains_token_key", +] diff --git a/benchmarks/capability_manifest.json b/benchmarks/capability_manifest.json new file mode 100644 index 0000000..61bc376 --- /dev/null +++ b/benchmarks/capability_manifest.json @@ -0,0 +1,117 @@ +{ + "schema_version": 1, + "profile": "retrieval-dependencies-v1", + "required_roles": [ + "ranked_retrieval", + "graph_retrieval", + "execution_output" + ], + "providers": [ + { + "id": "zvec-grep", + "role": "ranked_retrieval", + "binary": "zg", + "probe": ["zg", "--version"], + "prepare": ["zg", "index", "--embedding", "local/potion-code-16m-v2"], + "timeout_seconds": 1200, + "retrieval_stages": ["R1_DISCOVERY", "R2_EVIDENCE"], + "workspace_owned_paths": [".zvec-grep/"], + "version_regex": "(? list[str]: + if not isinstance(value, list) or not all(isinstance(item, str) and item for item in value): + raise CapabilityManifestError(f"{label} must be a list of non-empty strings") + if not allow_empty and not value: + raise CapabilityManifestError(f"{label} must not be empty") + return list(value) + + +def load_manifest(path: Path) -> dict[str, Any]: + data = json.loads(path.read_text(encoding="utf-8")) + if data.get("schema_version") != SCHEMA_VERSION: + raise CapabilityManifestError(f"unsupported capability manifest schema: {data.get('schema_version')!r}") + if not isinstance(data.get("profile"), str) or not data["profile"]: + raise CapabilityManifestError("profile must be a non-empty string") + + required_roles = set(_string_list(data.get("required_roles"), "required_roles", allow_empty=False)) + _string_list(data.get("runner_required_binaries", []), "runner_required_binaries") + providers = data.get("providers") + if not isinstance(providers, list) or not providers: + raise CapabilityManifestError("providers must be a non-empty list") + + ids: set[str] = set() + roles: set[str] = set() + for index, provider in enumerate(providers): + label = f"providers[{index}]" + if not isinstance(provider, dict): + raise CapabilityManifestError(f"{label} must be an object") + provider_id = provider.get("id") + role = provider.get("role") + binary = provider.get("binary") + if not all(isinstance(value, str) and value for value in (provider_id, role, binary)): + raise CapabilityManifestError(f"{label} requires non-empty id, role, and binary") + if provider_id in ids: + raise CapabilityManifestError(f"duplicate provider id: {provider_id}") + if role in roles: + raise CapabilityManifestError(f"duplicate provider role: {role}") + ids.add(provider_id) + roles.add(role) + _string_list(provider.get("probe"), f"{label}.probe", allow_empty=False) + _string_list(provider.get("prepare"), f"{label}.prepare", allow_empty=False) + version_regex = provider.get("version_regex") + if not isinstance(version_regex, str) or not version_regex: + raise CapabilityManifestError(f"{label}.version_regex must be a non-empty string") + try: + re.compile(version_regex) + except re.error as exc: + raise CapabilityManifestError(f"{label}.version_regex is invalid: {exc}") from exc + warmup_commands = provider.get("warmup_commands", []) + if not isinstance(warmup_commands, list): + raise CapabilityManifestError(f"{label}.warmup_commands must be a list") + for warmup_index, item in enumerate(warmup_commands): + warmup_label = f"{label}.warmup_commands[{warmup_index}]" + if not isinstance(item, dict): + raise CapabilityManifestError(f"{warmup_label} must be an object") + _string_list(item.get("command"), f"{warmup_label}.command", allow_empty=False) + warmup_timeout = item.get("timeout_seconds") + if not isinstance(warmup_timeout, (int, float)) or warmup_timeout <= 0: + raise CapabilityManifestError(f"{warmup_label}.timeout_seconds must be positive") + _string_list(provider.get("retrieval_stages", []), f"{label}.retrieval_stages") + _string_list(provider.get("workspace_owned_paths", []), f"{label}.workspace_owned_paths") + timeout = provider.get("timeout_seconds") + if not isinstance(timeout, (int, float)) or timeout <= 0: + raise CapabilityManifestError(f"{label}.timeout_seconds must be positive") + + if roles != required_roles: + missing = sorted(required_roles - roles) + unexpected = sorted(roles - required_roles) + raise CapabilityManifestError(f"provider roles mismatch; missing={missing}, unexpected={unexpected}") + + warmups = data.get("repository_warmups") + if not isinstance(warmups, dict): + raise CapabilityManifestError("repository_warmups must be an object") + for repository, spec in warmups.items(): + if not isinstance(repository, str) or not repository or not isinstance(spec, dict): + raise CapabilityManifestError("repository_warmups entries must be named objects") + _string_list(spec.get("required_binaries", []), f"repository_warmups.{repository}.required_binaries") + commands = spec.get("commands", []) + if not isinstance(commands, list): + raise CapabilityManifestError(f"repository_warmups.{repository}.commands must be a list") + for index, item in enumerate(commands): + if not isinstance(item, dict): + raise CapabilityManifestError(f"repository_warmups.{repository}.commands[{index}] must be an object") + _string_list(item.get("command"), f"repository_warmups.{repository}.commands[{index}].command", allow_empty=False) + timeout = item.get("timeout_seconds") + if not isinstance(timeout, (int, float)) or timeout <= 0: + raise CapabilityManifestError( + f"repository_warmups.{repository}.commands[{index}].timeout_seconds must be positive" + ) + + contract = data.get("measurement_contract") + if not isinstance(contract, dict): + raise CapabilityManifestError("measurement_contract must be an object") + if contract.get("setup_phase") != "unmeasured": + raise CapabilityManifestError("setup_phase must be unmeasured") + if contract.get("setup_included_in_comparison") is not False: + raise CapabilityManifestError("setup_included_in_comparison must be false") + if contract.get("setup_token_estimate") is not False: + raise CapabilityManifestError("setup_token_estimate must be false") + if contract.get("measured_phase_starts") != "after_workspace_prepare": + raise CapabilityManifestError("measured phase must start after workspace preparation") + _string_list(contract.get("measured_fields"), "measurement_contract.measured_fields", allow_empty=False) + _string_list( + contract.get("forbidden_measured_setup_commands", []), + "measurement_contract.forbidden_measured_setup_commands", + ) + return data + + +def manifest_fingerprint(manifest: Mapping[str, Any]) -> str: + encoded = json.dumps(manifest, sort_keys=True, separators=(",", ":"), ensure_ascii=False).encode("utf-8") + return hashlib.sha256(encoded).hexdigest() diff --git a/benchmarks/capability_process.py b/benchmarks/capability_process.py new file mode 100644 index 0000000..0562dfc --- /dev/null +++ b/benchmarks/capability_process.py @@ -0,0 +1,152 @@ +"""Executable resolution, probes, and auditable command records.""" + +from __future__ import annotations + +import os +import re +import shutil +import subprocess +import time +from pathlib import Path +from typing import Any, Callable, Mapping, Sequence + +try: + from .capability_manifest import ( + SCHEMA_VERSION, CapabilityError, MissingCapabilityError, manifest_fingerprint + ) +except ImportError: # direct script imports from the benchmarks directory + from capability_manifest import ( + SCHEMA_VERSION, CapabilityError, MissingCapabilityError, manifest_fingerprint + ) + +OUTPUT_TAIL_LIMIT = 12_000 +Run = Callable[[Sequence[str], Path, Mapping[str, str], float], subprocess.CompletedProcess[str]] + +def _default_run(command: Sequence[str], cwd: Path, env: Mapping[str, str], timeout: float) -> subprocess.CompletedProcess[str]: + return subprocess.run( + list(command), + cwd=str(cwd), + env=dict(env), + text=True, + encoding="utf-8", + errors="replace", + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + timeout=timeout, + check=False, + ) + + +def _tail(value: str) -> str: + return value if len(value) <= OUTPUT_TAIL_LIMIT else value[-OUTPUT_TAIL_LIMIT:] + + +def _command_record(command: Sequence[str], result: subprocess.CompletedProcess[str], elapsed: float) -> dict[str, Any]: + stdout = result.stdout or "" + stderr = result.stderr or "" + return { + "command": list(command), + "returncode": result.returncode, + "elapsed_seconds": elapsed, + "stdout_bytes": len(stdout.encode("utf-8", errors="replace")), + "stderr_bytes": len(stderr.encode("utf-8", errors="replace")), + "stdout_tail": _tail(stdout), + "stderr_tail": _tail(stderr), + } + + +def _run_checked( + command: Sequence[str], + *, + cwd: Path, + env: Mapping[str, str], + timeout: float, + runner: Run, + error_type: type[CapabilityError], + label: str, +) -> dict[str, Any]: + started = time.monotonic() + try: + result = runner(command, cwd, env, timeout) + except (OSError, subprocess.SubprocessError) as exc: + raise error_type(f"{label} could not run: {exc}") from exc + record = _command_record(command, result, time.monotonic() - started) + if result.returncode != 0: + detail = (result.stderr or result.stdout or "no output").strip() + raise error_type(f"{label} failed with exit {result.returncode}: {detail[-2000:]}") + return record + + +def _resolve(binary: str, which: Callable[[str], str | None]) -> str: + resolved = which(binary) + if not resolved: + raise MissingCapabilityError(f"required benchmark executable is missing: {binary}") + return str(Path(resolved).resolve()) + + +def preflight( + manifest: Mapping[str, Any], + *, + cwd: Path | None = None, + env: Mapping[str, str] | None = None, + runner: Run = _default_run, + which: Callable[[str], str | None] = shutil.which, +) -> dict[str, Any]: + """Resolve and probe every required provider and warm-up executable.""" + + root = (cwd or Path.cwd()).resolve() + base_env = dict(os.environ if env is None else env) + resolved: dict[str, str] = {} + probes: list[dict[str, Any]] = [] + + for binary in manifest.get("runner_required_binaries", []): + resolved[binary] = _resolve(binary, which) + + for provider in manifest["providers"]: + binary = provider["binary"] + resolved[binary] = _resolve(binary, which) + command = list(provider["probe"]) + command[0] = resolved[binary] + record = _run_checked( + command, + cwd=root, + env=base_env, + timeout=float(provider["timeout_seconds"]), + runner=runner, + error_type=MissingCapabilityError, + label=f"provider probe {provider['id']}", + ) + observed_version = "\n".join( + part for part in (record.get("stdout_tail", ""), record.get("stderr_tail", "")) if part + ) + version_regex = provider["version_regex"] + if re.search(version_regex, observed_version) is None: + raise MissingCapabilityError( + f"provider probe {provider['id']} returned an unapproved version; " + f"expected /{version_regex}/, observed: {observed_version.strip() or 'no output'}" + ) + record.update( + { + "provider": provider["id"], + "role": provider["role"], + "version_regex": version_regex, + "observed_version_output": observed_version.strip(), + } + ) + probes.append(record) + + for repository, spec in manifest["repository_warmups"].items(): + for binary in spec.get("required_binaries", []): + if binary not in resolved: + resolved[binary] = _resolve(binary, which) + + return { + "schema_version": SCHEMA_VERSION, + "phase": "setup-preflight", + "included_in_comparison": False, + "profile": manifest["profile"], + "manifest_sha256": manifest_fingerprint(manifest), + "resolved_executables": resolved, + "provider_probes": probes, + } + diff --git a/benchmarks/capability_workspace.py b/benchmarks/capability_workspace.py new file mode 100644 index 0000000..8a09c02 --- /dev/null +++ b/benchmarks/capability_workspace.py @@ -0,0 +1,196 @@ +"""Pre-measurement provider, index, dependency, and first-build setup.""" + +from __future__ import annotations + +import json +import os +import threading +from pathlib import Path +from typing import Any, Mapping, Sequence + +try: + from .capability_manifest import SCHEMA_VERSION, CapabilitySetupError, manifest_fingerprint + from .capability_process import Run, _default_run, _run_checked +except ImportError: # direct script imports from the benchmarks directory + from capability_manifest import SCHEMA_VERSION, CapabilitySetupError, manifest_fingerprint + from capability_process import Run, _default_run, _run_checked + +_SETUP_LOCK = threading.Lock() + +def workspace_environment( + workspace: Path, + base_env: Mapping[str, str] | None = None, +) -> dict[str, str]: + """Return benchmark-only environment additions for one workspace. + + Codebase Memory uses an account daemon. Giving concurrent cells different + ``CBM_CACHE_DIR`` values can split the daemon cohort and make a paired run + fail for an infrastructure reason. Therefore the benchmark inherits the + host's existing cache cohort by default. Operators that need a dedicated + cohort can set ``PRACTICAL_BENCHMARK_CBM_CACHE_DIR`` once for the whole run; + every cell receives that same value. + """ + + source = dict(os.environ if base_env is None else base_env) + state = workspace.parent / "capability-state" + state.mkdir(parents=True, exist_ok=True) + additions = {"PRACTICAL_CAPABILITY_STATE": str(state.resolve())} + cbm_cache = source.get("PRACTICAL_BENCHMARK_CBM_CACHE_DIR") or source.get("CBM_CACHE_DIR") + if cbm_cache: + cache = Path(cbm_cache).expanduser().resolve() + cache.mkdir(parents=True, exist_ok=True) + additions["CBM_CACHE_DIR"] = str(cache) + return additions + + +def _substitute(command: Sequence[str], workspace: Path) -> list[str]: + value = str(workspace.resolve()) + return [part.replace("{workspace}", value) for part in command] + + +def _exclude_owned_paths(workspace: Path, paths: Sequence[str]) -> None: + if not paths: + return + info = workspace / ".git" / "info" + if not info.is_dir(): + raise CapabilitySetupError(f"workspace is not a normal Git checkout: {workspace}") + exclude = info / "exclude" + existing = exclude.read_text(encoding="utf-8", errors="replace").splitlines() if exclude.exists() else [] + additions = [path for path in paths if path not in existing] + if additions: + with exclude.open("a", encoding="utf-8") as handle: + if existing and existing[-1] != "": + handle.write("\n") + handle.write("\n".join(additions) + "\n") + + +def prepare_workspace( + workspace: Path, + repository: str, + manifest: Mapping[str, Any], + preflight_report: Mapping[str, Any], + *, + runner: Run = _default_run, + base_env: Mapping[str, str] | None = None, +) -> dict[str, Any]: + """Prepare one frozen cell before model timing begins. + + Setup is serialized to avoid concurrent first-download/cache races. The + resulting report deliberately contains no token field. + """ + + workspace = workspace.resolve() + if repository not in manifest["repository_warmups"]: + raise CapabilitySetupError(f"repository has no warm-up contract: {repository}") + if preflight_report.get("manifest_sha256") != manifest_fingerprint(manifest): + raise CapabilitySetupError("preflight report does not match capability manifest") + + env = dict(os.environ if base_env is None else base_env) + env.update(workspace_environment(workspace, env)) + resolved = dict(preflight_report["resolved_executables"]) + owned_paths = [ + path + for provider in manifest["providers"] + for path in provider.get("workspace_owned_paths", []) + ] + _exclude_owned_paths(workspace, owned_paths) + + provider_records: list[dict[str, Any]] = [] + provider_warmup_records: list[dict[str, Any]] = [] + warmup_records: list[dict[str, Any]] = [] + with _SETUP_LOCK: + for provider in manifest["providers"]: + command = _substitute(provider["prepare"], workspace) + command[0] = resolved[provider["binary"]] + record = _run_checked( + command, + cwd=workspace, + env=env, + timeout=float(provider["timeout_seconds"]), + runner=runner, + error_type=CapabilitySetupError, + label=f"provider setup {provider['id']}", + ) + record.update({"provider": provider["id"], "role": provider["role"]}) + provider_records.append(record) + for warmup_index, item in enumerate(provider.get("warmup_commands", [])): + warmup_command = _substitute(item["command"], workspace) + if warmup_command[0] in resolved: + warmup_command[0] = resolved[warmup_command[0]] + warmup_record = _run_checked( + warmup_command, + cwd=workspace, + env=env, + timeout=float(item["timeout_seconds"]), + runner=runner, + error_type=CapabilitySetupError, + label=f"provider warm-up {provider['id']}[{warmup_index}]", + ) + warmup_record.update( + {"provider": provider["id"], "role": provider["role"], "index": warmup_index} + ) + provider_warmup_records.append(warmup_record) + + warmup = manifest["repository_warmups"][repository] + for index, item in enumerate(warmup.get("commands", [])): + command = _substitute(item["command"], workspace) + if command[0] in resolved: + command[0] = resolved[command[0]] + record = _run_checked( + command, + cwd=workspace, + env=env, + timeout=float(item["timeout_seconds"]), + runner=runner, + error_type=CapabilitySetupError, + label=f"repository warm-up {repository}[{index}]", + ) + record.update({"repository": repository, "index": index}) + warmup_records.append(record) + + _exclude_owned_paths(workspace, owned_paths) + git = resolved.get("git") + if not git: + raise CapabilitySetupError("preflight receipt did not resolve required benchmark executable: git") + clean_record = _run_checked( + [git, "status", "--porcelain", "--untracked-files=all"], + cwd=workspace, + env=env, + timeout=60, + runner=runner, + error_type=CapabilitySetupError, + label="post-setup git status", + ) + if clean_record["stdout_tail"].strip(): + raise CapabilitySetupError(f"capability setup dirtied frozen workspace: {clean_record['stdout_tail'].strip()}") + + return { + "schema_version": SCHEMA_VERSION, + "phase": "setup", + "included_in_comparison": False, + "profile": manifest["profile"], + "manifest_sha256": manifest_fingerprint(manifest), + "repository": repository, + "workspace": str(workspace), + "cbm_cache_cohort": env.get("CBM_CACHE_DIR", "provider-default"), + "provider_setup": provider_records, + "provider_warmup": provider_warmup_records, + "repository_warmup": warmup_records, + "post_setup_clean_check": clean_record, + "measurement_begins_after_report": True, + } + + +def write_report(path: Path, report: Mapping[str, Any]) -> None: + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text(json.dumps(report, ensure_ascii=False, indent=2) + "\n", encoding="utf-8") + + +def contains_token_key(value: Any) -> bool: + """Test helper enforcing that setup reports never estimate model tokens.""" + + if isinstance(value, dict): + return any("token" in str(key).lower() or contains_token_key(item) for key, item in value.items()) + if isinstance(value, list): + return any(contains_token_key(item) for item in value) + return False diff --git a/benchmarks/dependency_tree_contract.py b/benchmarks/dependency_tree_contract.py new file mode 100644 index 0000000..067a0c2 --- /dev/null +++ b/benchmarks/dependency_tree_contract.py @@ -0,0 +1,179 @@ +"""Shared topology, trace, and measured-command contracts for dependency runs.""" + +from __future__ import annotations + +import re +from typing import Any, Callable, Mapping + +try: + from . import retrieval_trace +except ImportError: # direct script imports from the benchmarks directory + import retrieval_trace + + +CANONICAL_RETRIEVAL_MODES = ( + "NONE", + "R0_DIRECT", + "R1_DISCOVERY", + "R2_EVIDENCE", + "R3_STRUCTURAL", +) +RETRIEVAL_REFERENCE_TO_MODE = { + "references/retrieval/skill.md": "NONE", + "references/retrieval/direct.md": "R0_DIRECT", + "references/retrieval/discovery.md": "R1_DISCOVERY", + "references/retrieval/evidence.md": "R2_EVIDENCE", + "references/retrieval/structural.md": "R3_STRUCTURAL", +} +SETUP_COMMAND_RE = re.compile( + r"(?:^|[\s;&|])(?:[^\s;&|]*[/\\])?(?:zg(?:\.exe)?\s+index|" + r"codebase-memory-mcp(?:\.exe)?\s+cli\s+index_repository|" + r"rtk(?:\.exe)?\s+init|npm(?:\.cmd|\.exe)?\s+(?:ci|install))\b", + re.I, +) + + +def retrieval_nodes(topology: Mapping[str, Any]) -> dict[str, Any]: + tree = topology.get("retrieval_tree") + if not isinstance(tree, dict): + raise ValueError("topology requires retrieval_tree") + nodes = tree.get("nodes") + root = tree.get("root") + if not isinstance(nodes, dict) or root not in nodes: + raise ValueError("retrieval_tree root must name a node") + for name, spec in nodes.items(): + if not isinstance(spec, dict): + raise ValueError(f"invalid retrieval node: {name}") + parent = spec.get("parent") + children = spec.get("children") + depth = spec.get("depth") + reference = spec.get("reference") + if not isinstance(depth, int) or depth < 0: + raise ValueError(f"invalid retrieval depth: {name}") + if not isinstance(reference, str) or not reference: + raise ValueError(f"invalid retrieval reference: {name}") + if not isinstance(children, list) or not all(child in nodes for child in children): + raise ValueError(f"invalid retrieval children: {name}") + if name == root: + if parent is not None or depth != 0: + raise ValueError("retrieval root must have parent=null and depth=0") + else: + if parent not in nodes or name not in nodes[parent].get("children", []): + raise ValueError(f"invalid retrieval parent edge: {name}") + if depth != nodes[parent]["depth"] + 1: + raise ValueError(f"retrieval depth must be parent depth + 1: {name}") + return nodes + + +def retrieval_reference_prefix( + topology: Mapping[str, Any], + mode: str, + canonicalize: Callable[[str], str], +) -> list[str]: + if mode == "NONE": + return [] + nodes = retrieval_nodes(topology) + target = next((name for name, spec in nodes.items() if spec.get("trace_mode") == mode), None) + if target is None: + return [] + path: list[str] = [] + current: str | None = target + while current is not None: + path.append(current) + current = nodes[current]["parent"] + path.reverse() + return [canonicalize(nodes[name]["reference"]) for name in path] + + +def capability_note() -> str: + return ( + "\n" + "The paired environment already contains and has preinitialized all required providers before this measured turn: " + "ranked retrieval via `zg query --human --limit `, structural retrieval via " + "`codebase-memory-mcp cli` (use `list_projects` before a project query), and noisy command output compaction via `rtk`. " + "Do not install, initialize, download models, or build indexes during measured execution. " + "Select Retrieval depth by the unresolved information problem, never by provider name. " + "All benchmark arms receive this same capability note.\n" + "" + ) + + +def instrumentation(topology: Mapping[str, Any]) -> str: + nodes = ", ".join(sorted(topology["automatic_nodes"])) + manuals = ", ".join(sorted(topology.get("manual_modes", {}))) + retrieval = ", ".join(topology.get("retrieval_trace_modes", CANONICAL_RETRIEVAL_MODES)) + return ( + "After the evidence-backed report, append exactly one final benchmark-only line: " + "TREE_TRACE path= retrieval= manual= refs=. " + f"Automatic node names are: {nodes}. A path starts at {topology['root']} and uses '>' between nodes; " + f"use path={topology['root']} when no automatic child was loaded. " + f"Retrieval mode must be one of: {retrieval}. Manual mode must be none or one of: {manuals}. " + "Manual modes are not path nodes. Retrieval references must be the actually loaded progressive prefix; " + "refs=none when no Practical Coding reference beyond SKILL.md was loaded. " + "Report behavior actually used; do not infer a preferred route from task wording. " + "Do not mention this instrumentation elsewhere." + ) + + +def extend_allowed_references( + topology: Mapping[str, Any], + original: Callable[[dict[str, Any]], set[str]], + canonicalize: Callable[[str], str], +) -> set[str]: + refs = set(original(dict(topology))) + refs.update(canonicalize(spec["reference"]) for spec in retrieval_nodes(topology).values()) + return refs + + +def validate_trace( + topology: Mapping[str, Any], + trace: Mapping[str, Any], + original: Callable[[dict[str, Any], dict[str, Any]], bool], + canonicalize: Callable[[str], str], +) -> bool: + if not original(dict(topology), dict(trace)): + return False + mode = trace.get("retrieval") + if mode not in CANONICAL_RETRIEVAL_MODES: + return False + retrieval_refs = [ + canonicalize(ref) + for ref in trace.get("references_loaded", []) + if canonicalize(ref).startswith("references/retrieval/") + ] + expected = retrieval_reference_prefix(topology, str(mode), canonicalize) + if mode == "NONE": + return not retrieval_refs or retrieval_refs == expected[: len(retrieval_refs)] + return retrieval_refs == expected + + +def infer_trace( + topology: Mapping[str, Any], + commands: list[str], + original: Callable[[dict[str, Any], list[str]], dict[str, Any]], + canonicalize: Callable[[str], str], +) -> dict[str, Any]: + trace = original(dict(topology), commands) + observed = retrieval_trace.observed_references(commands) + modes = [RETRIEVAL_REFERENCE_TO_MODE[ref] for ref in observed if ref in RETRIEVAL_REFERENCE_TO_MODE] + if modes: + order = {mode: index for index, mode in enumerate(CANONICAL_RETRIEVAL_MODES)} + trace["retrieval"] = max(modes, key=lambda mode: order[mode]) + else: + trace["retrieval"] = "NONE" + non_retrieval = [ + canonicalize(reference) + for reference in trace.get("references_loaded", []) + if not canonicalize(reference).startswith("references/retrieval/") + ] + trace["references_loaded"] = [*non_retrieval, *observed] + return trace + + +def provider_usage(commands: list[str]) -> dict[str, bool]: + text = "\n".join(str(command) for command in commands).lower() + return { + "zvec-grep": bool(re.search(r"(?:^|[\s;&|])(?:[^\s;&|]*[/\\])?zg(?:\.exe)?\s+(?:query|search)\b", text)), + "codebase-memory-mcp": "codebase-memory-mcp" in text, + "rtk": bool(re.search(r"(?:^|[\s;&|])(?:[^\s;&|]*[/\\])?rtk(?:\.exe)?\s+", text)), + } diff --git a/benchmarks/dependency_tree_runtime.py b/benchmarks/dependency_tree_runtime.py new file mode 100644 index 0000000..b2ea431 --- /dev/null +++ b/benchmarks/dependency_tree_runtime.py @@ -0,0 +1,154 @@ +"""Monkey-patch the historical execution-tree runner with dependency setup.""" + +from __future__ import annotations + +import json +from pathlib import Path +from typing import Any, Mapping + +try: + from . import capability_environment as capabilities + from . import dependency_tree_contract as contract + from . import retrieval_trace + from . import tree_validation as base +except ImportError: # direct script imports from the benchmarks directory + import capability_environment as capabilities + import dependency_tree_contract as contract + import retrieval_trace + import tree_validation as base + + +def patch_runner( + manifest: Mapping[str, Any], + preflight_report: Mapping[str, Any], + manifest_path: Path, +) -> None: + original_prepare_workspace = base.prepare_workspace + original_allowed_references = base.allowed_references + original_task_prompt = base.task_prompt + original_validate_trace = base.validate_trace + original_infer_trace = base.infer_trace_from_commands + original_score_answer = base.score_answer + original_summary = base.summary + original_run_cell = base.run_cell + original_run_codex = base.bench.run_codex + + repository_by_local_name = {str(spec["local_name"]): name for name, spec in base.REPOSITORIES.items()} + repository_by_commit = {str(spec["commit"]): name for name, spec in base.REPOSITORIES.items()} + + def prepare_workspace(source: Path, commit: str, workspace: Path) -> None: + original_prepare_workspace(source, commit, workspace) + repository = repository_by_commit.get(commit) or repository_by_local_name.get(source.name) + if repository is None: + raise capabilities.CapabilitySetupError(f"no capability warm-up mapping for source: {source}") + report = capabilities.prepare_workspace(workspace, repository, manifest, preflight_report) + capabilities.write_report(workspace.parent / "capability-setup.json", report) + + def allowed_references(topology: dict[str, Any]) -> set[str]: + return contract.extend_allowed_references(topology, original_allowed_references, base.canonical_reference) + + note = contract.capability_note() + + def task_prompt(case: dict[str, Any], loaded: str, variant: str, topology: dict[str, Any]) -> str: + enriched = note if not loaded else loaded + "\n\n" + note + return original_task_prompt(case, enriched, variant, topology) + + def validate_trace(topology: dict[str, Any], trace: dict[str, Any]) -> bool: + return contract.validate_trace(topology, trace, original_validate_trace, base.canonical_reference) + + def infer_trace_from_commands(topology: dict[str, Any], commands: list[str]) -> dict[str, Any]: + return contract.infer_trace(topology, commands, original_infer_trace, base.canonical_reference) + + def score_answer(*args: Any, **kwargs: Any) -> dict[str, Any]: + result = original_score_answer(*args, **kwargs) + commands = args[2] if len(args) >= 3 else kwargs.get("commands", []) + violation = bool(contract.SETUP_COMMAND_RE.search("\n".join(str(command) for command in commands))) + result["measured_setup_violation"] = violation + if violation: + result["passed"] = False + return result + + def summary(records: list[dict[str, Any]], runs: int) -> dict[str, Any]: + report = original_summary(records, runs) + measured = [record for record in records if record.get("measurement_phase") == "measured"] + report["capability_profile"] = { + "profile": manifest["profile"], + "manifest": str(manifest_path), + "manifest_sha256": capabilities.manifest_fingerprint(manifest), + "required_roles": list(manifest["required_roles"]), + "preflight": preflight_report, + } + report["measurement_contract"] = dict(manifest["measurement_contract"]) + report["measured_cells"] = len(measured) + report["measured_setup_violation_count"] = sum( + record.get("measured_setup_violation") is True for record in measured + ) + report["retrieval_reference_observation_violation_count"] = sum( + record.get("retrieval_reference_observation_ok") is False for record in measured + ) + report["provider_usage_counts"] = { + provider["id"]: sum( + record.get("capability_usage", {}).get(provider["id"]) is True for record in measured + ) + for provider in manifest["providers"] + } + return report + + def run_codex(command: list[str], prompt: str, workspace: Path, env: dict[str, str], *args: Any, **kwargs: Any): + measured_env = dict(env) + measured_env.update(capabilities.workspace_environment(workspace, measured_env)) + return original_run_codex(command, prompt, workspace, measured_env, *args, **kwargs) + + def run_cell(*args: Any, **kwargs: Any) -> dict[str, Any]: + spec = args[0] if args else kwargs["spec"] + output = args[6] if len(args) >= 7 else kwargs["output"] + task_id, variant, repetition = spec + cell = output / "cells" / task_id / variant.replace(":", "-") / f"r{repetition:03d}" + result_path = cell / "result.json" + setup_path = cell / "capability-setup.json" + if result_path.is_file() and not setup_path.is_file(): + raise capabilities.CapabilitySetupError( + f"refusing to reuse measured result without capability setup receipt: {result_path}" + ) + + record = original_run_cell(*args, **kwargs) + record["measurement_phase"] = "measured" + record["setup_included_in_comparison"] = False + record["capability_setup_file"] = str(setup_path) + record["capability_usage"] = contract.provider_usage(record.get("tool_commands", [])) + current_runtime = variant == "adaptive" or variant.startswith("cap:") + if current_runtime: + declared = [ + base.canonical_reference(reference) + for reference in record.get("references_loaded", []) + if base.canonical_reference(reference).startswith("references/retrieval/") + ] + observed = retrieval_trace.observed_references(record.get("tool_commands", [])) + observation_ok = declared == observed + record["retrieval_reference_observation_ok"] = observation_ok + record["observed_retrieval_references"] = observed + if not observation_ok: + record["passed"] = False + record["verdict"] = "fail" + else: + record["retrieval_reference_observation_ok"] = None + record["observed_retrieval_references"] = [] + if not setup_path.is_file(): + raise capabilities.CapabilitySetupError(f"missing setup receipt after cell execution: {setup_path}") + setup = json.loads(setup_path.read_text(encoding="utf-8")) + if setup.get("manifest_sha256") != capabilities.manifest_fingerprint(manifest): + raise capabilities.CapabilitySetupError(f"stale setup receipt: {setup_path}") + result_path.write_text(json.dumps(record, ensure_ascii=False, indent=2) + "\n", encoding="utf-8") + return record + + base.prepare_workspace = prepare_workspace + base.parse_trace = retrieval_trace.parse_trace + base.allowed_references = allowed_references + base.task_prompt = task_prompt + base.instrumentation = contract.instrumentation + base.validate_trace = validate_trace + base.infer_trace_from_commands = infer_trace_from_commands + base.score_answer = score_answer + base.summary = summary + base.run_cell = run_cell + base.bench.run_codex = run_codex diff --git a/benchmarks/dependency_tree_validation.py b/benchmarks/dependency_tree_validation.py new file mode 100644 index 0000000..93712f1 --- /dev/null +++ b/benchmarks/dependency_tree_validation.py @@ -0,0 +1,80 @@ +#!/usr/bin/env python3 +"""Run the execution-tree benchmark with mandatory preinitialized providers.""" + +from __future__ import annotations + +import argparse +import sys +from pathlib import Path +from typing import Any, Mapping + +HERE = Path(__file__).resolve().parent +if str(HERE) not in sys.path: + sys.path.insert(0, str(HERE)) + +import capability_environment as capabilities +import dependency_tree_contract as contract +import dependency_tree_runtime as runtime +import tree_validation as base + +CANONICAL_RETRIEVAL_MODES = contract.CANONICAL_RETRIEVAL_MODES +RETRIEVAL_REFERENCE_TO_MODE = contract.RETRIEVAL_REFERENCE_TO_MODE +SETUP_COMMAND_RE = contract.SETUP_COMMAND_RE + + +def _extract_wrapper_args(argv: list[str]) -> tuple[argparse.Namespace, list[str]]: + parser = argparse.ArgumentParser(add_help=False) + parser.add_argument("--capability-manifest", type=Path, default=HERE / "capability_manifest.json") + return parser.parse_known_args(argv) + + +def _retrieval_nodes(topology: Mapping[str, Any]) -> dict[str, Any]: + return contract.retrieval_nodes(topology) + + +def retrieval_reference_prefix(topology: Mapping[str, Any], mode: str) -> list[str]: + return contract.retrieval_reference_prefix(topology, mode, base.canonical_reference) + + +def _patch_runner( + manifest: Mapping[str, Any], + preflight_report: Mapping[str, Any], + manifest_path: Path, +) -> None: + runtime.patch_runner(manifest, preflight_report, manifest_path) + + +def main(argv: list[str] | None = None) -> int: + raw = list(sys.argv[1:] if argv is None else argv) + wrapper_args, remaining = _extract_wrapper_args(raw) + manifest_path = wrapper_args.capability_manifest.resolve() + manifest = capabilities.load_manifest(manifest_path) + topology = base.load_topology(HERE / "tree_topology.json") + _retrieval_nodes(topology) + + # Structural self-tests validate the contract without pretending external + # binaries are available in ordinary CI. Every actual model run preflights. + if "--self-test" in remaining: + original_argv = sys.argv + try: + sys.argv = [original_argv[0], *remaining] + return base.main() + finally: + sys.argv = original_argv + + preflight_report = capabilities.preflight(manifest, cwd=HERE.parent) + _patch_runner(manifest, preflight_report, manifest_path) + original_argv = sys.argv + try: + sys.argv = [original_argv[0], *remaining] + return base.main() + finally: + sys.argv = original_argv + + +if __name__ == "__main__": + try: + raise SystemExit(main()) + except capabilities.CapabilityError as exc: + print(f"dependency benchmark setup failed: {exc}", file=sys.stderr) + raise SystemExit(2) diff --git a/benchmarks/evolution_workflow_validation.py b/benchmarks/evolution_workflow_validation.py new file mode 100644 index 0000000..3d7e775 --- /dev/null +++ b/benchmarks/evolution_workflow_validation.py @@ -0,0 +1,128 @@ +#!/usr/bin/env python3 +"""Score the explicit WikiSkill-inspired maintenance workflow contract. + +This benchmark is deterministic and intentionally does not measure runtime coding +quality. Runtime quality remains covered by the tree benchmark. This suite verifies +that maintenance skills are isolated from automatic routing and that the proposer +contract cannot accept a candidate without a frozen, same-evidence non-regression gate. +""" + +from __future__ import annotations + +import argparse +import json +import re +from pathlib import Path +from typing import Callable + +HERE = Path(__file__).resolve().parent +ROOT = HERE.parent + +SESSION_SKILL = ROOT / "evolution/skills/session-to-wiki/SKILL.md" +EVOLVE_SKILL = ROOT / "evolution/skills/evolve-skill/SKILL.md" +TOPOLOGY = HERE / "tree_topology.json" + + +def read(path: Path) -> str: + return path.read_text(encoding="utf-8") + + +def frontmatter_name(text: str) -> str | None: + match = re.search(r"(?ms)^---\s*$.*?^name:\s*([^\n]+?)\s*$.*?^---\s*$", text) + return match.group(1).strip().strip('"\'') if match else None + + +def contains_all(text: str, terms: list[str]) -> bool: + lower = text.lower() + return all(term.lower() in lower for term in terms) + + +def checks() -> list[tuple[str, Callable[[], bool]]]: + session = read(SESSION_SKILL) if SESSION_SKILL.is_file() else "" + evolve = read(EVOLVE_SKILL) if EVOLVE_SKILL.is_file() else "" + topology = json.loads(read(TOPOLOGY)) if TOPOLOGY.is_file() else {} + automatic_refs = { + spec.get("reference") + for spec in topology.get("automatic_nodes", {}).values() + if isinstance(spec, dict) + } + automatic_children = { + child + for spec in topology.get("automatic_nodes", {}).values() + if isinstance(spec, dict) + for child in spec.get("children", []) + } + + return [ + ("session skill exists", lambda: SESSION_SKILL.is_file()), + ("evolve skill exists", lambda: EVOLVE_SKILL.is_file()), + ("session skill has standalone name", lambda: frontmatter_name(session) == "session-to-wiki"), + ("evolve skill has standalone name", lambda: frontmatter_name(evolve) == "evolve-skill"), + ("session activation is explicit-only", lambda: contains_all(session, ["explicit", "outside the automatic runtime router tree"])), + ("evolve activation is explicit-only", lambda: contains_all(evolve, ["explicit", "outside the automatic coding router tree"])), + ("session writes immutable raw receipt first", lambda: contains_all(session, ["immutable receipt", "evolution/raw/sessions/", "before consolidating"])), + ("session sanitizes and avoids transcript storage", lambda: contains_all(session, ["sanitize", "never copy the full transcript", "secrets"])), + ("session reads wiki before consolidation", lambda: contains_all(session, ["read the current wiki before consolidating", "evolution/wiki/index.md"])), + ("session deduplicates mechanisms", lambda: contains_all(session, ["update an existing mechanism", "create a new page only"])), + ("session cannot mutate runtime skill", lambda: contains_all(session, ["must not edit `skill.md`", "stop before runtime mutation"])), + ("evolver reads wiki index and impact history first", lambda: contains_all(evolve, ["evolution/wiki/index.md", "evolution/wiki/skill-impact.md", "first"])), + ("evolver proposes one atomic target", lambda: contains_all(evolve, ["one atomic proposal", "one runtime skill/node/boundary"])), + ("hypothesis frozen before candidate validation", lambda: contains_all(evolve, ["freeze the hypothesis", "before seeing candidate validation"])), + ("benchmark frozen before runtime patch", lambda: contains_all(evolve, ["benchmark before applying the runtime patch", "positive case", "boundary/negative case"])), + ("baseline runs before candidate", lambda: contains_all(evolve, ["run the baseline on the frozen benchmark", "exact commit/ref"])), + ("baseline and candidate use same evidence", lambda: contains_all(evolve, ["same model, harness, repetitions, cases, and scorer"])), + ("scorer fixes invalidate both arms", lambda: contains_all(evolve, ["invalidate both affected results", "rerun baseline and candidate from scratch"])), + ("quality cannot regress", lambda: contains_all(evolve, ["not lower than baseline", "quality regression"])), + ("indeterminate gate cannot accept", lambda: contains_all(evolve, ["gate is indeterminate", "revert the runtime candidate"])), + ("rejection preserves wiki knowledge", lambda: contains_all(evolve, ["keep valid raw receipts/wiki knowledge", "rejected"])), + ("impact tracker records accepted outcome", lambda: contains_all(evolve, ["evolution/wiki/skill-impact.md", "accepted"])), + ("maintenance skills absent from automatic refs", lambda: "evolution/skills/session-to-wiki/SKILL.md" not in automatic_refs and "evolution/skills/evolve-skill/SKILL.md" not in automatic_refs), + ("maintenance skills absent from automatic child names", lambda: "session-to-wiki" not in automatic_children and "evolve-skill" not in automatic_children), + ("wiki index exists", lambda: (ROOT / "evolution/wiki/index.md").is_file()), + ("wiki log exists", lambda: (ROOT / "evolution/wiki/log.md").is_file()), + ("skill impact tracker exists", lambda: (ROOT / "evolution/wiki/skill-impact.md").is_file()), + ("raw session receipt exists", lambda: (ROOT / "evolution/raw/sessions/2026-09-01-wikiskill-maintenance.md").is_file()), + ] + + +def evaluate() -> dict[str, object]: + rows = [] + for name, predicate in checks(): + try: + passed = bool(predicate()) + error = None + except Exception as exc: # benchmark should report malformed inputs, not hide them + passed = False + error = f"{type(exc).__name__}: {exc}" + rows.append({"check": name, "passed": passed, "error": error}) + passed = sum(row["passed"] is True for row in rows) + total = len(rows) + return { + "schema_version": 1, + "benchmark": "evolution-workflow-contract", + "passed": passed, + "total": total, + "score": passed / total if total else 0.0, + "checks": rows, + "note": "Deterministic maintenance-contract benchmark; runtime coding quality is gated separately by tree validation.", + } + + +def main() -> int: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--self-test", action="store_true", help="require a perfect contract score") + parser.add_argument("--output", type=Path) + args = parser.parse_args() + report = evaluate() + payload = json.dumps(report, ensure_ascii=False, indent=2) + "\n" + if args.output: + args.output.parent.mkdir(parents=True, exist_ok=True) + args.output.write_text(payload, encoding="utf-8") + print(payload, end="") + if args.self_test and report["score"] != 1.0: + return 1 + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/benchmarks/results/evolution-workflow/2026-09-01.json b/benchmarks/results/evolution-workflow/2026-09-01.json new file mode 100644 index 0000000..75e4a34 --- /dev/null +++ b/benchmarks/results/evolution-workflow/2026-09-01.json @@ -0,0 +1,38 @@ +{ + "schema_version": 1, + "benchmark": "evolution-workflow-contract", + "passed": 28, + "total": 28, + "score": 1.0, + "checks": [ + {"check": "session skill exists", "passed": true, "error": null}, + {"check": "evolve skill exists", "passed": true, "error": null}, + {"check": "session skill has standalone name", "passed": true, "error": null}, + {"check": "evolve skill has standalone name", "passed": true, "error": null}, + {"check": "session activation is explicit-only", "passed": true, "error": null}, + {"check": "evolve activation is explicit-only", "passed": true, "error": null}, + {"check": "session writes immutable raw receipt first", "passed": true, "error": null}, + {"check": "session sanitizes and avoids transcript storage", "passed": true, "error": null}, + {"check": "session reads wiki before consolidation", "passed": true, "error": null}, + {"check": "session deduplicates mechanisms", "passed": true, "error": null}, + {"check": "session cannot mutate runtime skill", "passed": true, "error": null}, + {"check": "evolver reads wiki index and impact history first", "passed": true, "error": null}, + {"check": "evolver proposes one atomic target", "passed": true, "error": null}, + {"check": "hypothesis frozen before candidate validation", "passed": true, "error": null}, + {"check": "benchmark frozen before runtime patch", "passed": true, "error": null}, + {"check": "baseline runs before candidate", "passed": true, "error": null}, + {"check": "baseline and candidate use same evidence", "passed": true, "error": null}, + {"check": "scorer fixes invalidate both arms", "passed": true, "error": null}, + {"check": "quality cannot regress", "passed": true, "error": null}, + {"check": "indeterminate gate cannot accept", "passed": true, "error": null}, + {"check": "rejection preserves wiki knowledge", "passed": true, "error": null}, + {"check": "impact tracker records accepted outcome", "passed": true, "error": null}, + {"check": "maintenance skills absent from automatic refs", "passed": true, "error": null}, + {"check": "maintenance skills absent from automatic child names", "passed": true, "error": null}, + {"check": "wiki index exists", "passed": true, "error": null}, + {"check": "wiki log exists", "passed": true, "error": null}, + {"check": "skill impact tracker exists", "passed": true, "error": null}, + {"check": "raw session receipt exists", "passed": true, "error": null} + ], + "note": "Deterministic maintenance-contract benchmark; runtime coding quality is gated separately by tree validation." +} diff --git a/benchmarks/results/evolvable-tree/README.md b/benchmarks/results/evolvable-tree/README.md new file mode 100644 index 0000000..c7a4f06 --- /dev/null +++ b/benchmarks/results/evolvable-tree/README.md @@ -0,0 +1,6 @@ +# Evolvable local router tree benchmark evidence + +- [`REPORT_ZH.md`](REPORT_ZH.md): paired n=3 delivery report. +- [`release-summary.json`](release-summary.json): sanitized machine-readable summary. + +Raw cell outputs and machine-specific paths remain in the ignored local `benchmark-results/tree-final-b202f7a-20260902/` artifact. diff --git a/benchmarks/results/evolvable-tree/REPORT_ZH.md b/benchmarks/results/evolvable-tree/REPORT_ZH.md new file mode 100644 index 0000000..8a28ecf --- /dev/null +++ b/benchmarks/results/evolvable-tree/REPORT_ZH.md @@ -0,0 +1,35 @@ +# Evolvable local router tree 发布验证报告 + +## 结论 + +候选提交 `d85c72c` 的树质量 n=1 迭代和冻结 n=3 非回归均通过。最终 n=3 使用两个独立 work（8 case + 7 case），每个 work 使用 8 个并行 worker，单个 work 不超过 10 个 case。 + +合并后的结果为 15 个 case、252/252 个 determinate cell；adaptive、frozen baseline 和 no-skill 均为 45/45 通过，`release_quality_gate` 为 `PASS`。 + +## 配对结果 + +| arm | 质量 | 稳定任务 | 平均 tokens | 平均时长 | 平均工具调用 | +|---|---:|---:|---:|---:|---:| +| adaptive | 45/45 | 15/15 | 253,543.13 | 84.15s | 8.82 | +| frozen baseline | 45/45 | 15/15 | 222,598.13 | 80.31s | 8.02 | +| no-skill | 45/45 | 15/15 | 245,342.04 | 81.66s | 6.33 | + +adaptive 相对两个比较 arm 均通过质量非劣性门禁（margin 0.03）。本轮不宣称成本优势；adaptive 的平均 tokens、时长和工具调用高于 frozen baseline。 + +## 纪律与边界 + +- adaptive trace failures：0；spontaneous manual：0；explicit manual contract failures：0。 +- Core、Debugging、Implementation capability ceiling 均为 39/39。 +- 一个初始不确定 cell `pp-running-after-throw / cap:debugging / repetition 1` 已独立重跑；该 case 的 18 个 cell 全部通过，随后完成 252 cell 合并分析。 +- tree topology 未改变；execution-state 仍是 cross-cutting substrate，不是 Router 节点。 + +## 未完成项 + +本报告只证明树质量 n=3 非回归。execution-state 四臂模型 benchmark（full history、state shadow、state history-free、no-skill full history）以及最终 outbound transport/header/cookie/proxy 审计仍为 `pending`，因为当前仓库没有实现该四臂 transport runner；不能用普通 tree runner 代替。 + +## 可复现边界 + +- 候选：`d85c72cc5aa239da32352309e723ed1e6fc80429`。 +- 冻结 baseline：`ba4058b4ef47a42bf79c9963b25678a2389897c1`。 +- 模型：`gpt-5.6-luna`，reasoning `medium`。 +- 原始 transcript、cell JSON 和机器路径保留在 ignored local artifacts；本目录只包含脱敏汇总。 diff --git a/benchmarks/results/evolvable-tree/release-summary.json b/benchmarks/results/evolvable-tree/release-summary.json new file mode 100644 index 0000000..f9ab464 --- /dev/null +++ b/benchmarks/results/evolvable-tree/release-summary.json @@ -0,0 +1,28 @@ +{ + "experiment": "evolvable-local-router-tree", + "status": "tree-n3-accepted-state-gate-pending", + "candidate_commit": "d85c72cc5aa239da32352309e723ed1e6fc80429", + "baseline_ref": "ba4058b4ef47a42bf79c9963b25678a2389897c1", + "model": "gpt-5.6-luna", + "reasoning": "medium", + "runs_per_cell": 3, + "tasks": 15, + "repositories": 3, + "cells": 252, + "determinate": 252, + "quality": { + "adaptive": {"passed": 45, "cells": 45, "stable_tasks": 15}, + "baseline": {"passed": 45, "cells": 45, "stable_tasks": 15}, + "no_skill": {"passed": 45, "cells": 45, "stable_tasks": 15}, + "release_quality_gate": "PASS" + }, + "discipline": { + "adaptive_trace_failures": 0, + "explicit_manual_contract_failures": 0, + "spontaneous_manual_count": 0 + }, + "parallelism": {"work_count": 2, "work_case_counts": [8, 7], "workers_per_work": 8}, + "execution_state_model_gate": "PENDING", + "pending_reason": "No four-arm model-backed transport runner or final outbound header/cookie/proxy audit exists in the repository.", + "raw_artifacts": "ignored benchmark-results/tree-d85c72c-final-merged/" +} diff --git a/benchmarks/retrieval_analysis.py b/benchmarks/retrieval_analysis.py new file mode 100644 index 0000000..654c6b6 --- /dev/null +++ b/benchmarks/retrieval_analysis.py @@ -0,0 +1,227 @@ +#!/usr/bin/env python3 +"""Derive minimum-sufficient Retrieval stages from dependency-enabled ceilings.""" + +from __future__ import annotations + +import argparse +import json +import statistics +from collections import Counter, defaultdict +from pathlib import Path +from typing import Any, Iterable + + +STAGES = ("NONE", "R0_DIRECT", "R1_DISCOVERY", "R2_EVIDENCE", "R3_STRUCTURAL") +STAGE_INDEX = {stage: index for index, stage in enumerate(STAGES)} + + +def load_rows(path: Path) -> list[dict[str, Any]]: + rows: list[dict[str, Any]] = [] + for line in path.read_text(encoding="utf-8").splitlines(): + if line.strip(): + rows.append(json.loads(line)) + return rows + + +def stable_pass( + records: Iterable[dict[str, Any]], + *, + expected_repetitions: set[int] | None = None, +) -> bool: + selected = list(records) + repetitions = [int(record.get("repetition", 0)) for record in selected] + if expected_repetitions is not None: + if set(repetitions) != expected_repetitions or len(repetitions) != len(expected_repetitions): + return False + determinate = [record for record in selected if record.get("passed") is not None] + return bool(selected) and len(determinate) == len(selected) and all(record.get("passed") is True for record in determinate) + + +def ceiling_repetitions(task_records: list[dict[str, Any]]) -> set[int]: + return { + int(record.get("repetition", 0)) + for record in task_records + if str(record.get("variant", "")).startswith("retrieval-cap:") + } + + +def minimum_stage(task_records: list[dict[str, Any]]) -> str | None: + expected_repetitions = ceiling_repetitions(task_records) + for stage in STAGES: + variant = f"retrieval-cap:{stage}" + if stable_pass( + (record for record in task_records if record.get("variant") == variant), + expected_repetitions=expected_repetitions, + ): + return stage + return None + + +def adaptive_relation(record: dict[str, Any], minimum: str | None) -> str: + if record.get("passed") is not True: + return "adaptive_quality_failure" + selected = record.get("selected_retrieval") + if selected not in STAGE_INDEX: + return "invalid_trace" + if minimum is None: + return "quality_gap" + if selected == minimum: + return "exact_minimum" + if STAGE_INDEX[selected] > STAGE_INDEX[minimum]: + return "over_disclosure" + return "under_disclosure" + + +def _mean(records: list[dict[str, Any]], key: str) -> float | None: + values = [float(record[key]) for record in records if record.get(key) is not None] + return statistics.mean(values) if values else None + + +def analyze(rows: list[dict[str, Any]]) -> dict[str, Any]: + measured = [row for row in rows if row.get("measurement_phase") == "measured"] + by_task: dict[str, list[dict[str, Any]]] = defaultdict(list) + for row in measured: + by_task[row["task_id"]].append(row) + + tasks: dict[str, Any] = {} + minimum_counts: Counter[str] = Counter() + relation_counts: Counter[str] = Counter() + provider_by_stage: dict[str, Counter[str]] = {stage: Counter() for stage in STAGES} + + for task_id, records in sorted(by_task.items()): + ordinary = not any(record.get("manual_request") for record in records) + expected_repetitions = ceiling_repetitions(records) if ordinary else set() + minimum = minimum_stage(records) if ordinary else None + if minimum is not None: + minimum_counts[minimum] += 1 + adaptive = [record for record in records if record.get("variant") == "adaptive"] + relations = [adaptive_relation(record, minimum) for record in adaptive] if ordinary else [] + relation_counts.update(relations) + for record in adaptive: + selected = record.get("selected_retrieval") + if selected in provider_by_stage: + for provider, used in record.get("capability_usage", {}).items(): + if used: + provider_by_stage[selected][provider] += 1 + tasks[task_id] = { + "manual": not ordinary, + "minimum_sufficient_retrieval_stage": minimum, + "stable_ceiling_pass": { + stage: stable_pass( + (record for record in records if record.get("variant") == f"retrieval-cap:{stage}"), + expected_repetitions=expected_repetitions, + ) + for stage in STAGES + } + if ordinary + else {}, + "adaptive": [ + { + "repetition": record.get("repetition"), + "passed": record.get("passed"), + "selected_retrieval": record.get("selected_retrieval"), + "relation": relation, + "capability_usage": record.get("capability_usage", {}), + "total_tokens": record.get("total_tokens"), + "duration_seconds": record.get("duration_seconds"), + "tool_calls": record.get("tool_calls"), + } + for record, relation in zip(adaptive, relations or [None] * len(adaptive)) + ], + } + + arms: dict[str, Any] = {} + for variant in sorted({record["variant"] for record in measured}): + selected = [record for record in measured if record["variant"] == variant] + determinate = [record for record in selected if record.get("passed") is not None] + arms[variant] = { + "cells": len(selected), + "determinate": len(determinate), + "pass_rate": sum(record.get("passed") is True for record in determinate) / len(determinate) + if determinate + else None, + "total_tokens_mean": _mean(determinate, "total_tokens"), + "duration_seconds_mean": _mean(determinate, "duration_seconds"), + "tool_calls_mean": _mean(determinate, "tool_calls"), + } + + setup_leak_count = sum( + record.get("setup_included_in_comparison") is not False + or record.get("measurement_phase") != "measured" + or record.get("measured_setup_violation") is True + for record in rows + ) + return { + "schema_version": "1.0", + "stages": list(STAGES), + "measured_rows": len(measured), + "tasks": tasks, + "minimum_stage_counts": dict(minimum_counts), + "adaptive_relation_counts": dict(relation_counts), + "provider_usage_by_selected_stage": { + stage: dict(counter) for stage, counter in provider_by_stage.items() + }, + "arms": arms, + "setup_measurement_contract_violation_count": setup_leak_count, + } + + +def parse_args() -> argparse.Namespace: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("results", type=Path) + parser.add_argument("--output", type=Path) + parser.add_argument("--self-test", action="store_true") + return parser.parse_args() + + +def self_test() -> None: + rows: list[dict[str, Any]] = [] + for stage in STAGES: + rows.append( + { + "task_id": "task", + "variant": f"retrieval-cap:{stage}", + "repetition": 1, + "passed": STAGE_INDEX[stage] >= STAGE_INDEX["R1_DISCOVERY"], + "measurement_phase": "measured", + "setup_included_in_comparison": False, + "measured_setup_violation": False, + } + ) + rows.append( + { + "task_id": "task", + "variant": "adaptive", + "repetition": 1, + "passed": True, + "selected_retrieval": "R2_EVIDENCE", + "capability_usage": {"zvec-grep": True}, + "measurement_phase": "measured", + "setup_included_in_comparison": False, + "measured_setup_violation": False, + } + ) + report = analyze(rows) + assert report["tasks"]["task"]["minimum_sufficient_retrieval_stage"] == "R1_DISCOVERY" + assert report["adaptive_relation_counts"]["over_disclosure"] == 1 + assert report["setup_measurement_contract_violation_count"] == 0 + print("retrieval analysis self-test: PASS") + + +def main() -> int: + args = parse_args() + if args.self_test: + self_test() + return 0 + rows = load_rows(args.results) + report = analyze(rows) + value = json.dumps(report, ensure_ascii=False, indent=2) + "\n" + if args.output: + args.output.parent.mkdir(parents=True, exist_ok=True) + args.output.write_text(value, encoding="utf-8") + print(value, end="") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/benchmarks/retrieval_cell.py b/benchmarks/retrieval_cell.py new file mode 100644 index 0000000..5b7f2a3 --- /dev/null +++ b/benchmarks/retrieval_cell.py @@ -0,0 +1,174 @@ +"""One dependency-enabled, prewarmed Retrieval benchmark cell.""" + +from __future__ import annotations + +import argparse +import json +import os +import shutil +from pathlib import Path +from typing import Any, Mapping + +try: + from . import capability_environment as capabilities + from . import retrieval_trace + from . import run_benchmarks as bench + from . import tree_validation as base + from .retrieval_prompt import ( + SETUP_COMMAND_RE, _cell_path, _provider_usage, provider_ceiling_violation, task_prompt + ) + from .retrieval_topology import STAGE_INDEX, infer_trace, retrieval_declared_prefix, validate_trace + from .tree_cases import CASES, REPOSITORIES +except ImportError: # direct script imports from the benchmarks directory + import capability_environment as capabilities + import retrieval_trace + import run_benchmarks as bench + import tree_validation as base + from retrieval_prompt import SETUP_COMMAND_RE, _cell_path, _provider_usage, provider_ceiling_violation, task_prompt + from retrieval_topology import STAGE_INDEX, infer_trace, retrieval_declared_prefix, validate_trace + from tree_cases import CASES, REPOSITORIES + +VERSION = "1.0" + +def run_cell( + spec: tuple[str, str, int], + args: argparse.Namespace, + topology: Mapping[str, Any], + manifest: Mapping[str, Any], + preflight_report: Mapping[str, Any], + repositories: Mapping[str, Path], + baseline: Path | None, + eval_home: Path, + output: Path, +) -> dict[str, Any]: + task_id, variant, repetition = spec + case = next(item for item in CASES if item["task_id"] == task_id) + cell = _cell_path(output, spec) + result_path = cell / "result.json" + setup_path = cell / "capability-setup.json" + manifest_sha = capabilities.manifest_fingerprint(manifest) + if result_path.is_file(): + if not setup_path.is_file(): + raise capabilities.CapabilitySetupError(f"measured result has no setup receipt: {result_path}") + setup = json.loads(setup_path.read_text(encoding="utf-8")) + if setup.get("manifest_sha256") != manifest_sha: + raise capabilities.CapabilitySetupError(f"stale setup receipt: {setup_path}") + return json.loads(result_path.read_text(encoding="utf-8")) + + cell.mkdir(parents=True, exist_ok=True) + workspace = cell / "workspace" + if workspace.exists(): + shutil.rmtree(workspace) + base.prepare_workspace(repositories[case["repository"]], REPOSITORIES[case["repository"]]["commit"], workspace) + setup = capabilities.prepare_workspace(workspace, case["repository"], manifest, preflight_report) + capabilities.write_report(setup_path, setup) + + if variant == "no-skill": + loaded = "" + elif variant == "baseline": + if baseline is None: + raise RuntimeError("baseline Skill is unavailable") + loaded = bench.skill_text("practical-previous", {}, baseline) + else: + loaded = bench.skill_text("practical-current", {}, None) + + prompt = task_prompt(case, loaded, variant, topology) + (cell / "prompt.txt").write_text(prompt, encoding="utf-8") + env = os.environ.copy() + env["CODEX_HOME"] = str(eval_home) + env.update(capabilities.workspace_environment(workspace, env)) + codex = bench.resolve_codex(args.codex) + stdout = cell / "round1.jsonl" + stderr = cell / "round1.stderr.txt" + + # Measured time begins here, after every provider/index/build warm-up has + # succeeded and its separate receipt has been written. + code, timed_out, forced, duration = bench.run_codex( + bench.codex_command(codex, workspace), prompt, workspace, env, stdout, stderr, args.timeout + ) + parsed = bench.parse_transcript(stdout) + current_runtime = variant == "adaptive" or variant.startswith("retrieval-cap:") + trace = retrieval_trace.parse_trace(parsed["answer"]) if current_runtime else None + trace_source = "reported" if trace and trace.get("path") else None + if current_runtime and trace and not trace.get("path"): + trace = infer_trace(topology, parsed["tool_commands"]) + trace_source = "observed-commands" + ceiling = variant.split(":", 1)[1] if variant.startswith("retrieval-cap:") else None + trace_valid = validate_trace(topology, trace, ceiling) if current_runtime and trace is not None else None + terminal_node = trace["path"][-1] if trace and trace.get("path") else None + setup_violation = bool(SETUP_COMMAND_RE.search("\n".join(parsed["tool_commands"]))) + provider_usage = _provider_usage(parsed["tool_commands"]) + ceiling_violation = provider_ceiling_violation(provider_usage, ceiling) + observed_retrieval = retrieval_trace.observed_references(parsed["tool_commands"]) + declared_retrieval = [ + base.canonical_reference(reference) + for reference in (trace or {}).get("references_loaded", []) + if base.canonical_reference(reference).startswith("references/retrieval/") + ] + observation_ok = declared_retrieval == observed_retrieval if current_runtime else None + + record: dict[str, Any] = { + "schema_version": VERSION, + "task_id": task_id, + "repository": case["repository"], + "family": case["family"], + "manual_request": case.get("manual_request"), + "variant": variant, + "retrieval_ceiling": ceiling, + "repetition": repetition, + "exit_status": code, + "timed_out": timed_out, + "forced_after_completion": forced, + "duration_seconds": duration, + "tool_calls": parsed["tool_calls"], + **parsed["usage"], + "answer": parsed["answer"], + "tool_commands": parsed["tool_commands"], + "selected_path": trace["path"] if trace else None, + "selected_terminal_node": terminal_node, + "selected_depth": topology["automatic_nodes"].get(terminal_node, {}).get("depth") if terminal_node else None, + "selected_retrieval": trace["retrieval"] if trace else None, + "selected_retrieval_index": STAGE_INDEX.get(trace["retrieval"]) if trace else None, + "selected_retrieval_references": retrieval_declared_prefix(topology, trace["retrieval"]) + if trace and trace.get("retrieval") in STAGE_INDEX + else [], + "selected_manual": trace["manual"] if trace else None, + "references_loaded": trace["references_loaded"] if trace else [], + "routing_trace_valid": trace_valid, + "routing_trace_source": trace_source, + "retrieval_reference_observation_ok": observation_ok, + "observed_retrieval_references": observed_retrieval, + "capability_usage": provider_usage, + "capability_ceiling_violation": ceiling_violation, + "measurement_phase": "measured", + "setup_included_in_comparison": False, + "capability_setup_file": str(setup_path), + "measured_setup_violation": setup_violation, + } + infrastructure_error = "timeout" if timed_out else (f"codex exit status {code}" if code and not forced else None) + if infrastructure_error: + record.update({"passed": None, "verdict": "indeterminate", "error": infrastructure_error}) + else: + record.update( + base.score_answer( + case, + parsed["answer"], + parsed["tool_commands"], + workspace, + trace=trace, + enforce_runtime_contract=current_runtime, + ) + ) + if current_runtime and not trace_valid: + record["passed"] = False + record["routing_trace_error"] = True + if current_runtime and observation_ok is not True: + record["passed"] = False + record["retrieval_reference_observation_error"] = True + if setup_violation or ceiling_violation: + record["passed"] = False + record["verdict"] = "pass" if record["passed"] else "fail" + + (cell / "answer.md").write_text(parsed["answer"] + "\n", encoding="utf-8") + result_path.write_text(json.dumps(record, ensure_ascii=False, indent=2) + "\n", encoding="utf-8") + return record diff --git a/benchmarks/retrieval_prompt.py b/benchmarks/retrieval_prompt.py new file mode 100644 index 0000000..1d5b35f --- /dev/null +++ b/benchmarks/retrieval_prompt.py @@ -0,0 +1,113 @@ +"""Prompt, provider-ceiling, and cell-spec contracts for Retrieval ablation.""" + +from __future__ import annotations + +import re +from pathlib import Path +from typing import Any, Mapping + +try: + from .retrieval_topology import ( + STAGES, STAGE_INDEX, instrumentation, retrieval_declared_prefix + ) + from .tree_cases import CASES +except ImportError: # direct script imports from the benchmarks directory + from retrieval_topology import STAGES, STAGE_INDEX, instrumentation, retrieval_declared_prefix + from tree_cases import CASES + +SETUP_COMMAND_RE = re.compile( + r"(?:^|[\s;&|])(?:[^\s;&|]*[/\\])?(?:zg(?:\.exe)?\s+index|" + r"codebase-memory-mcp(?:\.exe)?\s+cli\s+index_repository|" + r"rtk(?:\.exe)?\s+init|npm(?:\.cmd|\.exe)?\s+(?:ci|install))\b", + re.I, +) + +def allowed_provider_ids(stage: str) -> set[str]: + allowed = {"rtk"} + if STAGE_INDEX[stage] >= STAGE_INDEX["R1_DISCOVERY"]: + allowed.add("zvec-grep") + if STAGE_INDEX[stage] >= STAGE_INDEX["R3_STRUCTURAL"]: + allowed.add("codebase-memory-mcp") + return allowed + + +def retrieval_ceiling_instruction(topology: Mapping[str, Any], stage: str) -> str: + refs = retrieval_declared_prefix(topology, stage) + refs_text = ", ".join(refs) if refs else "none" + providers = ", ".join(sorted(allowed_provider_ids(stage))) + return ( + "\n" + f"This ablation permits Retrieval policy only through {stage}. " + f"Permitted Retrieval references, in progressive order: {refs_text}. " + f"Permitted capability providers at this ceiling: {providers}. " + "Repository-native exact reads/search remain available at every ceiling. " + "Do not load a deeper Retrieval reference or invoke a provider owned by a deeper stage. " + "The automatic execution tree remains adaptive. " + "This is an availability ceiling, not a claim that the ceiling is the correct stage. " + "Stop earlier when the task has enough evidence.\n" + "" + ) + + +def capability_note() -> str: + return ( + "\n" + "The paired environment already contains and has preinitialized all required providers before this measured turn: " + "ranked retrieval via `zg query --human --limit `, structural retrieval via " + "`codebase-memory-mcp cli` (use `list_projects` before a project query), and noisy command output compaction via `rtk`. " + "Do not install, initialize, download models, build indexes, or install project packages during measured execution. " + "Choose Retrieval depth by the unresolved information problem, never by provider name. " + "All benchmark arms receive this same note.\n" + "" + ) + + +def task_prompt(case: Mapping[str, Any], loaded: str, variant: str, topology: Mapping[str, Any]) -> str: + suffix = [capability_note()] + if variant.startswith("retrieval-cap:"): + suffix.append(retrieval_ceiling_instruction(topology, variant.split(":", 1)[1])) + if variant == "adaptive" or variant.startswith("retrieval-cap:"): + suffix.append(instrumentation(topology)) + return ( + f"Frozen retrieval-tree task {case['task_id']} ({case['family']}).\n\n{case['prompt']}\n\n" + "Use PowerShell-compatible commands. Stay within this repository and preserve a clean working tree. " + "Cite concrete current-source paths/symbols and fresh command evidence when the task needs repository evidence.\n\n" + f"{variant}\n{loaded}\n\n" + "\n\n".join(suffix) + ) + + +def build_specs(runs: int, *, current_only: bool, selected_cases: set[str]) -> list[tuple[str, str, int]]: + specs: list[tuple[str, str, int]] = [] + for case in CASES: + if selected_cases and case["task_id"] not in selected_cases: + continue + if case.get("manual_request"): + variants = ["adaptive"] if current_only else ["no-skill", "baseline", "adaptive"] + else: + caps = [f"retrieval-cap:{stage}" for stage in STAGES] + variants = ["adaptive", *caps] if current_only else ["no-skill", "baseline", "adaptive", *caps] + for variant in variants: + for repetition in range(1, runs + 1): + specs.append((case["task_id"], variant, repetition)) + return specs + + +def _cell_path(output: Path, spec: tuple[str, str, int]) -> Path: + task_id, variant, repetition = spec + return output / "cells" / task_id / variant.replace(":", "-") / f"r{repetition:03d}" + + +def _provider_usage(commands: list[str]) -> dict[str, bool]: + text = "\n".join(commands).lower() + return { + "zvec-grep": bool(re.search(r"(?:^|[\s;&|])(?:[^\s;&|]*[/\\])?zg(?:\.exe)?\s+(?:query|search)\b", text)), + "codebase-memory-mcp": "codebase-memory-mcp" in text, + "rtk": bool(re.search(r"(?:^|[\s;&|])(?:[^\s;&|]*[/\\])?rtk(?:\.exe)?\s+", text)), + } + + +def provider_ceiling_violation(usage: Mapping[str, bool], ceiling: str | None) -> bool: + if ceiling is None: + return False + allowed = allowed_provider_ids(ceiling) + return any(used and provider not in allowed for provider, used in usage.items()) diff --git a/benchmarks/retrieval_topology.py b/benchmarks/retrieval_topology.py new file mode 100644 index 0000000..7c8c07f --- /dev/null +++ b/benchmarks/retrieval_topology.py @@ -0,0 +1,133 @@ +"""Progressive R0-R3 topology and trace validation.""" + +from __future__ import annotations + +from typing import Any, Mapping + +try: + from . import retrieval_trace + from . import tree_validation as base +except ImportError: # direct script imports from the benchmarks directory + import retrieval_trace + import tree_validation as base + +STAGES = ("NONE", "R0_DIRECT", "R1_DISCOVERY", "R2_EVIDENCE", "R3_STRUCTURAL") +STAGE_INDEX = {stage: index for index, stage in enumerate(STAGES)} + +def retrieval_nodes(topology: Mapping[str, Any]) -> dict[str, Any]: + tree = topology.get("retrieval_tree") + if not isinstance(tree, dict): + raise ValueError("topology requires retrieval_tree") + nodes = tree.get("nodes") + root = tree.get("root") + if not isinstance(nodes, dict) or root not in nodes: + raise ValueError("retrieval_tree root must name a node") + seen_modes: set[str] = set() + for name, spec in nodes.items(): + if not isinstance(spec, dict): + raise ValueError(f"invalid retrieval node: {name}") + parent = spec.get("parent") + children = spec.get("children") + depth = spec.get("depth") + mode = spec.get("trace_mode") + reference = spec.get("reference") + if not isinstance(depth, int) or depth < 0: + raise ValueError(f"invalid retrieval depth: {name}") + if mode not in STAGE_INDEX or mode in seen_modes: + raise ValueError(f"invalid or duplicate retrieval trace mode: {name}") + seen_modes.add(mode) + if not isinstance(reference, str) or not reference: + raise ValueError(f"invalid retrieval reference: {name}") + if not isinstance(children, list) or not all(child in nodes for child in children): + raise ValueError(f"invalid retrieval children: {name}") + if name == root: + if parent is not None or depth != 0: + raise ValueError("retrieval root must have parent=null and depth=0") + else: + if parent not in nodes or name not in nodes[parent].get("children", []): + raise ValueError(f"invalid retrieval parent edge: {name}") + if depth != nodes[parent]["depth"] + 1: + raise ValueError(f"retrieval depth must equal parent depth + 1: {name}") + if seen_modes != set(STAGES): + raise ValueError(f"retrieval modes mismatch: {sorted(seen_modes)}") + return nodes + + +def retrieval_declared_prefix(topology: Mapping[str, Any], stage: str) -> list[str]: + if stage == "NONE": + return [] + nodes = retrieval_nodes(topology) + target = next(name for name, spec in nodes.items() if spec["trace_mode"] == stage) + path: list[str] = [] + current: str | None = target + while current is not None: + path.append(current) + current = nodes[current]["parent"] + path.reverse() + return [str(nodes[name]["reference"]) for name in path] + + +def retrieval_prefix(topology: Mapping[str, Any], stage: str) -> list[str]: + return [base.canonical_reference(reference) for reference in retrieval_declared_prefix(topology, stage)] + + +def allowed_references(topology: Mapping[str, Any]) -> set[str]: + refs = base.allowed_references(dict(topology)) + refs.update(base.canonical_reference(spec["reference"]) for spec in retrieval_nodes(topology).values()) + return refs + + +def validate_trace(topology: Mapping[str, Any], trace: Mapping[str, Any], ceiling: str | None = None) -> bool: + mode = trace.get("retrieval") + if mode not in STAGE_INDEX: + return False + if ceiling is not None and STAGE_INDEX[mode] > STAGE_INDEX[ceiling]: + return False + if not base.validate_automatic_path(dict(topology), list(trace.get("path") or [])): + return False + manual = trace.get("manual") + if manual != "none" and manual not in topology.get("manual_modes", {}): + return False + refs = [base.canonical_reference(ref) for ref in trace.get("references_loaded", [])] + if any(ref not in allowed_references(topology) for ref in refs): + return False + retrieval_refs = [ref for ref in refs if ref.startswith("references/retrieval/")] + expected = retrieval_prefix(topology, mode) + if mode == "NONE": + return retrieval_refs in ([], expected) + return retrieval_refs == expected + + +def infer_trace(topology: Mapping[str, Any], commands: list[str]) -> dict[str, Any]: + trace = base.infer_trace_from_commands(dict(topology), commands) + observed = retrieval_trace.observed_references(commands) + mode_by_ref = { + base.canonical_reference(spec["reference"]): spec["trace_mode"] + for spec in retrieval_nodes(topology).values() + } + modes = [mode_by_ref[reference] for reference in observed if reference in mode_by_ref] + trace["retrieval"] = max(modes, key=lambda mode: STAGE_INDEX[mode]) if modes else "NONE" + non_retrieval = [ + base.canonical_reference(reference) + for reference in trace.get("references_loaded", []) + if not base.canonical_reference(reference).startswith("references/retrieval/") + ] + trace["references_loaded"] = [*non_retrieval, *observed] + return trace + + +def instrumentation(topology: Mapping[str, Any]) -> str: + nodes = ", ".join(sorted(topology["automatic_nodes"])) + manuals = ", ".join(sorted(topology.get("manual_modes", {}))) + return ( + "After the evidence-backed report, append exactly one final benchmark-only line: " + "TREE_TRACE path= retrieval= manual= refs=. " + f"Automatic node names are: {nodes}. A path starts at {topology['root']} and uses '>' between nodes; " + f"use path={topology['root']} when no automatic child was loaded. " + f"Retrieval mode must be one of: {', '.join(STAGES)}. " + f"Manual mode must be none or one of: {manuals}. " + "Manual modes are not path nodes. Retrieval references must be the complete actually loaded root-to-stage prefix. " + "refs=none only when no Practical Coding reference beyond SKILL.md was loaded. " + "Report behavior actually used; do not infer a preferred route from task wording. " + "Do not mention this instrumentation elsewhere." + ) diff --git a/benchmarks/retrieval_trace.py b/benchmarks/retrieval_trace.py new file mode 100644 index 0000000..3903740 --- /dev/null +++ b/benchmarks/retrieval_trace.py @@ -0,0 +1,62 @@ +"""Canonical dependency-benchmark trace parser. + +The historical tree runner accepted only alphabetic retrieval labels. Active +Retrieval stages contain digits (R0-R3), so dependency-enabled runners share +this parser instead of weakening their trace contract or rewriting old result +files. +""" + +from __future__ import annotations + +import re +from typing import Any + + +TRACE_RE = re.compile( + r"TREE_TRACE\s+path=([^\s]+)\s+retrieval=([A-Z0-9_]+)\s+manual=([a-z_-]+)\s+refs=([^\r\n]+)", + re.I, +) +RETRIEVAL_REF_RE = re.compile( + r"references[/\\]retrieval[/\\](?:skill|direct|discovery|evidence|structural)\.md", + re.I, +) + + +def parse_trace(answer: str) -> dict[str, Any]: + matches = list(TRACE_RE.finditer(answer)) + if not matches: + return {"path": [], "retrieval": None, "manual": None, "references_loaded": []} + match = matches[-1] + raw_path = match.group(1).strip().strip("<>") + path = ( + [] + if raw_path.lower() in {"none", "-"} + else [part.strip().lower() for part in raw_path.split(">") if part.strip()] + ) + refs_raw = match.group(4).strip().strip("<>") + refs = ( + [] + if refs_raw.lower() in {"none", "-"} + else [part.strip().strip("<>") for part in refs_raw.split(",") if part.strip()] + ) + return { + "path": path, + "retrieval": match.group(2).upper(), + "manual": match.group(3).lower(), + "references_loaded": refs, + } + + +def observed_references(commands: list[str]) -> list[str]: + """Return unique Retrieval references in actual command-observation order.""" + + observed: list[str] = [] + seen: set[str] = set() + for command in commands: + normalized = str(command).replace("\\", "/") + for match in RETRIEVAL_REF_RE.finditer(normalized): + reference = match.group(0).lower().replace("\\", "/") + if reference not in seen: + seen.add(reference) + observed.append(reference) + return observed diff --git a/benchmarks/retrieval_validation.py b/benchmarks/retrieval_validation.py new file mode 100644 index 0000000..71cd234 --- /dev/null +++ b/benchmarks/retrieval_validation.py @@ -0,0 +1,206 @@ +#!/usr/bin/env python3 +"""Dependency-enabled R0-R3 Retrieval-tree capability-ceiling benchmark.""" + +from __future__ import annotations + +import argparse +import concurrent.futures +import datetime as dt +import json +import os +import statistics +import sys +from pathlib import Path +from typing import Any, Mapping + +HERE = Path(__file__).resolve().parent +ROOT = HERE.parent +if str(HERE) not in sys.path: + sys.path.insert(0, str(HERE)) + +import capability_environment as capabilities +import retrieval_cell as cell_runner +import retrieval_prompt as prompt_contract +import retrieval_topology as topology_contract +import run_benchmarks as bench +import tree_validation as base +from tree_cases import CASES + +VERSION = "1.0" +MODEL = bench.MODEL +REASONING = bench.REASONING +STAGES = topology_contract.STAGES +STAGE_INDEX = topology_contract.STAGE_INDEX +SETUP_COMMAND_RE = prompt_contract.SETUP_COMMAND_RE +retrieval_nodes = topology_contract.retrieval_nodes +retrieval_declared_prefix = topology_contract.retrieval_declared_prefix +retrieval_prefix = topology_contract.retrieval_prefix +allowed_references = topology_contract.allowed_references +validate_trace = topology_contract.validate_trace +infer_trace = topology_contract.infer_trace +instrumentation = topology_contract.instrumentation +allowed_provider_ids = prompt_contract.allowed_provider_ids +retrieval_ceiling_instruction = prompt_contract.retrieval_ceiling_instruction +capability_note = prompt_contract.capability_note +task_prompt = prompt_contract.task_prompt +build_specs = prompt_contract.build_specs +_cell_path = prompt_contract._cell_path +_provider_usage = prompt_contract._provider_usage +provider_ceiling_violation = prompt_contract.provider_ceiling_violation +run_cell = cell_runner.run_cell + +def _mean(records: list[dict[str, Any]], key: str) -> float | None: + values = [float(record[key]) for record in records if record.get(key) is not None] + return statistics.mean(values) if values else None + + +def summary(records: list[dict[str, Any]], runs: int, manifest: Mapping[str, Any], preflight_report: Mapping[str, Any]) -> dict[str, Any]: + arms: dict[str, Any] = {} + for variant in sorted({record["variant"] for record in records}): + selected = [record for record in records if record["variant"] == variant] + determinate = [record for record in selected if record.get("passed") is not None] + arms[variant] = { + "cells": len(selected), + "determinate": len(determinate), + "pass_rate": sum(record["passed"] is True for record in determinate) / len(determinate) if determinate else None, + "tokens_mean": _mean(determinate, "total_tokens"), + "duration_seconds_mean": _mean(determinate, "duration_seconds"), + "tool_calls_mean": _mean(determinate, "tool_calls"), + } + measured = [record for record in records if record.get("measurement_phase") == "measured"] + return { + "runs_per_cell": runs, + "tasks": len({record["task_id"] for record in records}), + "repositories": sorted({record["repository"] for record in records}), + "arms": arms, + "canonical_retrieval_stages": list(STAGES), + "trace_valid_rate": sum(record.get("routing_trace_valid") is True for record in measured if record.get("routing_trace_valid") is not None) + / max(1, sum(record.get("routing_trace_valid") is not None for record in measured)), + "measured_setup_violation_count": sum(record.get("measured_setup_violation") is True for record in measured), + "capability_ceiling_violation_count": sum(record.get("capability_ceiling_violation") is True for record in measured), + "retrieval_reference_observation_violation_count": sum( + record.get("retrieval_reference_observation_ok") is False for record in measured + ), + "provider_usage_counts": { + provider["id"]: sum(record.get("capability_usage", {}).get(provider["id"]) is True for record in measured) + for provider in manifest["providers"] + }, + "capability_profile": { + "profile": manifest["profile"], + "manifest_sha256": capabilities.manifest_fingerprint(manifest), + "required_roles": list(manifest["required_roles"]), + "preflight": preflight_report, + }, + "measurement_contract": dict(manifest["measurement_contract"]), + } + + +def parse_args() -> argparse.Namespace: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--runs", type=int, default=3) + parser.add_argument("--workers", type=int, default=3) + parser.add_argument("--output", type=Path) + parser.add_argument("--repository-root", type=Path, default=ROOT.parent) + parser.add_argument("--repository", action="append", default=[], help="override a source as NAME=PATH") + parser.add_argument("--topology", type=Path, default=HERE / "tree_topology.json") + parser.add_argument("--capability-manifest", type=Path, default=HERE / "capability_manifest.json") + parser.add_argument("--baseline-ref") + parser.add_argument("--codex", default=os.environ.get("CODEX_BIN", "codex")) + parser.add_argument("--timeout", type=float, default=600) + parser.add_argument("--case", action="append", default=[]) + parser.add_argument("--current-only", action="store_true") + parser.add_argument("--self-test", action="store_true") + return parser.parse_args() + + +def self_test(topology: Mapping[str, Any], manifest: Mapping[str, Any]) -> None: + nodes = retrieval_nodes(topology) + assert [nodes[name]["trace_mode"] for name in ("retrieval", "direct", "discovery", "evidence", "structural")] == list(STAGES) + assert retrieval_prefix(topology, "R3_STRUCTURAL")[-1] == "references/retrieval/structural.md" + assert len(build_specs(1, current_only=True, selected_cases={"pp-known-contract"})) == 6 + assert allowed_provider_ids("R0_DIRECT") == {"rtk"} + assert allowed_provider_ids("R1_DISCOVERY") == {"rtk", "zvec-grep"} + assert allowed_provider_ids("R3_STRUCTURAL") == {"rtk", "zvec-grep", "codebase-memory-mcp"} + assert manifest["measurement_contract"]["setup_included_in_comparison"] is False + assert manifest["measurement_contract"]["setup_token_estimate"] is False + print("retrieval validation self-test: PASS") + + +def main() -> int: + args = parse_args() + topology = base.load_topology(args.topology.resolve()) + manifest = capabilities.load_manifest(args.capability_manifest.resolve()) + retrieval_nodes(topology) + if args.self_test: + self_test(topology, manifest) + return 0 + if args.runs < 1 or args.workers < 1: + raise SystemExit("runs and workers must be positive") + selected_cases = set(args.case) + unknown = selected_cases - {case["task_id"] for case in CASES} + if unknown: + raise SystemExit(f"unknown cases: {', '.join(sorted(unknown))}") + + preflight_report = capabilities.preflight(manifest, cwd=ROOT) + repositories = base.resolve_repositories(args.repository_root.resolve(), args.repository) + stamp = dt.datetime.now().strftime("%Y%m%d-%H%M%S") + output = (args.output or ROOT / "benchmark-results" / f"retrieval-{stamp}").resolve() + output.mkdir(parents=True, exist_ok=True) + capabilities.write_report(output / "capability-preflight.json", preflight_report) + + baseline_ref = args.baseline_ref or topology.get("baseline_ref") + baseline_dir: Path | None = None + if not args.current_only: + if not baseline_ref: + raise RuntimeError("baseline_ref is required unless --current-only is used") + baseline_dir = output / "baseline-skill" + if not (baseline_dir / "SKILL.md").is_file(): + baseline_dir = bench.materialize_git_skill(str(baseline_ref), baseline_dir) + + eval_home = bench.prepare_eval_home(output / "eval-home") + specs = build_specs(args.runs, current_only=args.current_only, selected_cases=selected_cases) + records: list[dict[str, Any]] = [] + with concurrent.futures.ThreadPoolExecutor(max_workers=args.workers) as pool: + futures = [ + pool.submit( + run_cell, + spec, + args, + topology, + manifest, + preflight_report, + repositories, + baseline_dir, + eval_home, + output, + ) + for spec in specs + ] + for future in concurrent.futures.as_completed(futures): + records.append(future.result()) + + records.sort(key=lambda row: (row["task_id"], row["variant"], row["repetition"])) + rows_path = output / "results.jsonl" + rows_path.write_text("".join(json.dumps(row, ensure_ascii=False) + "\n" for row in records), encoding="utf-8") + report = summary(records, args.runs, manifest, preflight_report) + report.update( + { + "schema_version": VERSION, + "model": MODEL, + "reasoning": REASONING, + "topology": topology, + "baseline_ref": baseline_ref, + "results_jsonl": str(rows_path), + } + ) + (output / "report.json").write_text(json.dumps(report, ensure_ascii=False, indent=2) + "\n", encoding="utf-8") + print(json.dumps(report, ensure_ascii=False, indent=2)) + return 0 + + +if __name__ == "__main__": + try: + raise SystemExit(main()) + except capabilities.CapabilityError as exc: + print(f"retrieval benchmark setup failed: {exc}", file=sys.stderr) + raise SystemExit(2) diff --git a/benchmarks/run.ps1 b/benchmarks/run.ps1 index 30b3806..b3b01ac 100644 --- a/benchmarks/run.ps1 +++ b/benchmarks/run.ps1 @@ -18,6 +18,7 @@ param( [switch]$FailOnCellFailure, [switch]$RequireStableRanking, [switch]$ProgressiveSelfTest, + [switch]$TreeSelfTest, [string]$Rescore = "" ) @@ -50,6 +51,29 @@ if ($ProgressiveSelfTest) { } } +if ($TreeSelfTest) { + Push-Location $repoRoot + try { + & python benchmarks/tree_validation.py --self-test + if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + & python benchmarks/dependency_tree_validation.py --self-test + if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + & python benchmarks/retrieval_validation.py --self-test + if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + & python benchmarks/retrieval_analysis.py /dev/null --self-test + if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + & python -m unittest ` + benchmarks.test_tree_benchmarks ` + benchmarks.test_capability_environment ` + benchmarks.test_dependency_tree_validation ` + benchmarks.test_retrieval_analysis + exit $LASTEXITCODE + } + finally { + Pop-Location + } +} + $effectiveRuns = if ($Runs -gt 0) { $Runs } diff --git a/benchmarks/test_benchmarks.py b/benchmarks/test_benchmarks.py index aa164f5..603c523 100644 --- a/benchmarks/test_benchmarks.py +++ b/benchmarks/test_benchmarks.py @@ -116,32 +116,40 @@ def test_router_answer_parser_requires_both_dimensions(self): ) self.assertEqual(bench.parse_router_answer("DEBUGGING"), ("", "")) - def test_core_is_route_agnostic_and_event_router_owns_escalation(self): + def test_core_is_local_tree_root_and_manual_modes_are_separate(self): skill = (bench.ROOT / "SKILL.md").read_text(encoding="utf-8") - core = skill.split("## Core", 1)[1].split("## Direct Path", 1)[0] - router = skill.split("## Event Router", 1)[1].split("## Explicit-only requirements interview", 1)[0] + core = skill.split("## Core", 1)[1].split("## Root Router", 1)[0] + router = skill.split("## Root Router", 1)[1].split("## Convergence Rule", 1)[0] + convergence = skill.split("## Convergence Rule", 1)[1].split("## Manual Modes", 1)[0] + manual = skill.split("## Manual Modes", 1)[1].split("## Retrieval Policy", 1)[0] retrieval = skill.split("## Retrieval Policy", 1)[1].split("## Isolation Gate", 1)[0] self.assertIn("smallest coherent reachable change", core) - self.assertIn("established contracts", core) + self.assertIn("established APIs and contracts", core) for module_specific in ( "references/", "diagnosis", - "engineering", "specialist", "navigation.md", ): self.assertNotIn(module_specific.lower(), core.lower()) self.assertIn("observed failure", router) - self.assertIn("material user-owned choice", router) self.assertIn("unknown contract or invariant", router) self.assertIn("references/debugging.md", router) - self.assertIn("references/decision.md", router) self.assertIn("references/implementation.md", router) - self.assertNotIn("specialists/", router) + self.assertNotIn("references/manual/decision.md", router) + self.assertNotIn("references/decision.md", router) + self.assertIn("must not reopen deliberation", convergence) + self.assertIn("Do not automatically load Decision", convergence) + self.assertIn("references/manual/decision.md", manual) + self.assertIn("references/manual/clarification.md", manual) self.assertIn("structural code index", retrieval) self.assertIn("references/navigation.md", retrieval) + self.assertIn("Once candidate paths or symbols are known, stop inventory", retrieval) + self.assertIn("bounded line ranges", retrieval) + self.assertIn("do not dump whole files or repeat broad discovery", retrieval) + self.assertIn("Batch independent bounded reads", retrieval) def test_decision_suite_inlines_decision_module(self): with tempfile.TemporaryDirectory() as tmp: diff --git a/benchmarks/test_capability_environment.py b/benchmarks/test_capability_environment.py new file mode 100644 index 0000000..ca4c846 --- /dev/null +++ b/benchmarks/test_capability_environment.py @@ -0,0 +1,226 @@ +from __future__ import annotations + +import copy +import json +import subprocess +import tempfile +import unittest +from pathlib import Path + +from benchmarks import capability_environment as cap + + +HERE = Path(__file__).resolve().parent + + +class CapabilityManifestTests(unittest.TestCase): + def test_checked_in_manifest_is_fail_closed_and_unmeasured(self) -> None: + manifest = cap.load_manifest(HERE / "capability_manifest.json") + self.assertEqual( + set(manifest["required_roles"]), + {"ranked_retrieval", "graph_retrieval", "execution_output"}, + ) + self.assertFalse(manifest["measurement_contract"]["setup_included_in_comparison"]) + self.assertFalse(manifest["measurement_contract"]["setup_token_estimate"]) + self.assertEqual( + manifest["measurement_contract"]["measured_phase_starts"], + "after_workspace_prepare", + ) + + def test_manifest_rejects_missing_required_provider_role(self) -> None: + manifest = json.loads((HERE / "capability_manifest.json").read_text(encoding="utf-8")) + manifest["providers"] = manifest["providers"][:-1] + with tempfile.TemporaryDirectory() as directory: + path = Path(directory) / "manifest.json" + path.write_text(json.dumps(manifest), encoding="utf-8") + with self.assertRaises(cap.CapabilityManifestError): + cap.load_manifest(path) + + def test_manifest_rejects_setup_entering_comparison(self) -> None: + manifest = json.loads((HERE / "capability_manifest.json").read_text(encoding="utf-8")) + manifest["measurement_contract"]["setup_included_in_comparison"] = True + with tempfile.TemporaryDirectory() as directory: + path = Path(directory) / "manifest.json" + path.write_text(json.dumps(manifest), encoding="utf-8") + with self.assertRaises(cap.CapabilityManifestError): + cap.load_manifest(path) + + def test_manifest_rejects_invalid_provider_version_regex(self) -> None: + manifest = json.loads((HERE / "capability_manifest.json").read_text(encoding="utf-8")) + manifest["providers"][0]["version_regex"] = "[" + with tempfile.TemporaryDirectory() as directory: + path = Path(directory) / "manifest.json" + path.write_text(json.dumps(manifest), encoding="utf-8") + with self.assertRaises(cap.CapabilityManifestError): + cap.load_manifest(path) + + +class CapabilityPreflightTests(unittest.TestCase): + def setUp(self) -> None: + self.manifest = cap.load_manifest(HERE / "capability_manifest.json") + + def test_missing_binary_fails_before_benchmark(self) -> None: + with self.assertRaisesRegex(cap.MissingCapabilityError, "zg"): + cap.preflight( + self.manifest, + which=lambda binary: None if binary == "zg" else f"/fake/bin/{binary}", + ) + + @staticmethod + def _probe_output(command) -> str: + binary = Path(command[0]).name + return { + "zg": "zg 0.2.0", + "codebase-memory-mcp": "codebase-memory-mcp 0.10.8", + "rtk": "rtk 0.47.0", + }[binary] + + def test_all_providers_are_probed(self) -> None: + commands: list[list[str]] = [] + + def runner(command, cwd, env, timeout): + commands.append(list(command)) + return subprocess.CompletedProcess(command, 0, stdout=self._probe_output(command), stderr="") + + report = cap.preflight( + self.manifest, + runner=runner, + which=lambda binary: f"/fake/bin/{binary}", + ) + self.assertFalse(report["included_in_comparison"]) + self.assertEqual(len(report["provider_probes"]), 3) + self.assertEqual({item["role"] for item in report["provider_probes"]}, set(self.manifest["required_roles"])) + self.assertTrue(all(command[0].startswith("/fake/bin/") for command in commands)) + self.assertTrue(all(item["observed_version_output"] for item in report["provider_probes"])) + self.assertFalse(cap.contains_token_key(report)) + + def test_unapproved_provider_version_fails_before_benchmark(self) -> None: + def runner(command, cwd, env, timeout): + output = self._probe_output(command) + if Path(command[0]).name == "zg": + output = "zg 0.1.0" + return subprocess.CompletedProcess(command, 0, stdout=output, stderr="") + + with self.assertRaisesRegex(cap.MissingCapabilityError, "unapproved version"): + cap.preflight( + self.manifest, + runner=runner, + which=lambda binary: f"/fake/bin/{binary}", + ) + + +class WorkspaceSetupTests(unittest.TestCase): + def setUp(self) -> None: + self.manifest = cap.load_manifest(HERE / "capability_manifest.json") + + @staticmethod + def _preflight(manifest): + return { + "manifest_sha256": cap.manifest_fingerprint(manifest), + "resolved_executables": { + "zg": "/fake/zg", + "codebase-memory-mcp": "/fake/codebase-memory-mcp", + "rtk": "/fake/rtk", + "git": "/fake/git", + "node": "/fake/node", + "npm": "/fake/npm", + "java": "/fake/java", + "mvn": "/fake/mvn", + }, + } + + def test_setup_is_separate_clean_and_has_no_token_field(self) -> None: + commands: list[list[str]] = [] + observed_cache_dirs: list[str] = [] + + def runner(command, cwd, env, timeout): + commands.append(list(command)) + observed_cache_dirs.append(env.get("CBM_CACHE_DIR", "")) + return subprocess.CompletedProcess(command, 0, stdout="", stderr="") + + with tempfile.TemporaryDirectory() as directory: + workspace = Path(directory) / "cell" / "workspace" + shared_cbm_cache = Path(directory) / "shared-cbm-cache" + (workspace / ".git" / "info").mkdir(parents=True) + report = cap.prepare_workspace( + workspace, + "personal-progress", + self.manifest, + self._preflight(self.manifest), + runner=runner, + base_env={"PRACTICAL_BENCHMARK_CBM_CACHE_DIR": str(shared_cbm_cache)}, + ) + self.assertFalse(report["included_in_comparison"]) + self.assertTrue(report["measurement_begins_after_report"]) + self.assertEqual(len(report["provider_setup"]), 3) + self.assertEqual(len(report["provider_warmup"]), 2) + self.assertEqual(report["repository_warmup"], []) + self.assertFalse(cap.contains_token_key(report)) + self.assertTrue(all(observed_cache_dirs)) + self.assertEqual({str(shared_cbm_cache.resolve())}, set(observed_cache_dirs)) + self.assertEqual(report["cbm_cache_cohort"], str(shared_cbm_cache.resolve())) + exclude = (workspace / ".git" / "info" / "exclude").read_text(encoding="utf-8") + self.assertIn(".zvec-grep/", exclude) + self.assertTrue(any(command[:2] == ["/fake/zg", "index"] for command in commands)) + self.assertTrue(any(command[:2] == ["/fake/zg", "query"] for command in commands)) + + def test_repository_warmup_is_executed_before_measurement(self) -> None: + commands: list[list[str]] = [] + + def runner(command, cwd, env, timeout): + commands.append(list(command)) + return subprocess.CompletedProcess(command, 0, stdout="", stderr="") + + with tempfile.TemporaryDirectory() as directory: + workspace = Path(directory) / "cell" / "workspace" + (workspace / ".git" / "info").mkdir(parents=True) + report = cap.prepare_workspace( + workspace, + "cover-atelier", + self.manifest, + self._preflight(self.manifest), + runner=runner, + base_env={}, + ) + self.assertEqual(len(report["repository_warmup"]), 2) + self.assertIn(["/fake/npm", "ci", "--no-audit", "--no-fund"], commands) + self.assertFalse(cap.contains_token_key(report)) + + def test_default_environment_does_not_create_per_cell_cbm_cohorts(self) -> None: + with tempfile.TemporaryDirectory() as directory: + workspace = Path(directory) / "cell" / "workspace" + additions = cap.workspace_environment(workspace, {}) + self.assertNotIn("CBM_CACHE_DIR", additions) + self.assertIn("PRACTICAL_CAPABILITY_STATE", additions) + + def test_failed_provider_setup_aborts(self) -> None: + def runner(command, cwd, env, timeout): + if command[0] == "/fake/codebase-memory-mcp": + return subprocess.CompletedProcess(command, 1, stdout="", stderr="index failed") + return subprocess.CompletedProcess(command, 0, stdout="", stderr="") + + with tempfile.TemporaryDirectory() as directory: + workspace = Path(directory) / "cell" / "workspace" + (workspace / ".git" / "info").mkdir(parents=True) + with self.assertRaisesRegex(cap.CapabilitySetupError, "index failed"): + cap.prepare_workspace( + workspace, + "personal-progress", + self.manifest, + self._preflight(self.manifest), + runner=runner, + base_env={}, + ) + + def test_stale_preflight_receipt_is_rejected(self) -> None: + stale = copy.deepcopy(self._preflight(self.manifest)) + stale["manifest_sha256"] = "0" * 64 + with tempfile.TemporaryDirectory() as directory: + workspace = Path(directory) / "cell" / "workspace" + (workspace / ".git" / "info").mkdir(parents=True) + with self.assertRaisesRegex(cap.CapabilitySetupError, "does not match"): + cap.prepare_workspace(workspace, "personal-progress", self.manifest, stale) + + +if __name__ == "__main__": + unittest.main() diff --git a/benchmarks/test_dependency_tree_validation.py b/benchmarks/test_dependency_tree_validation.py new file mode 100644 index 0000000..4f82f43 --- /dev/null +++ b/benchmarks/test_dependency_tree_validation.py @@ -0,0 +1,182 @@ +from __future__ import annotations + +import unittest +from pathlib import Path + +from benchmarks import dependency_tree_validation as dependency +from benchmarks import retrieval_trace +from benchmarks import retrieval_validation as retrieval +from benchmarks import tree_validation as base + + +HERE = Path(__file__).resolve().parent + + +class RetrievalTopologyContractTests(unittest.TestCase): + @classmethod + def setUpClass(cls) -> None: + cls.topology = base.load_topology(HERE / "tree_topology.json") + cls.nodes = dependency._retrieval_nodes(cls.topology) + + def test_retrieval_tree_is_a_single_progressive_local_path(self) -> None: + self.assertEqual(self.topology["retrieval_tree"]["root"], "retrieval") + self.assertEqual(self.nodes["retrieval"]["children"], ["direct"]) + self.assertEqual(self.nodes["direct"]["children"], ["discovery"]) + self.assertEqual(self.nodes["discovery"]["children"], ["evidence"]) + self.assertEqual(self.nodes["evidence"]["children"], ["structural"]) + self.assertEqual(self.nodes["structural"]["children"], []) + + def test_canonical_trace_modes_match_declared_nodes(self) -> None: + declared = {spec["trace_mode"] for spec in self.nodes.values()} + self.assertEqual(declared, set(dependency.CANONICAL_RETRIEVAL_MODES)) + self.assertEqual( + tuple(self.topology["retrieval_trace_modes"]), + dependency.CANONICAL_RETRIEVAL_MODES, + ) + + def test_none_has_no_loaded_retrieval_policy_prefix(self) -> None: + self.assertEqual(dependency.retrieval_reference_prefix(self.topology, "NONE"), []) + self.assertEqual(retrieval.retrieval_declared_prefix(self.topology, "NONE"), []) + + def test_r2_requires_a_complete_loaded_reference_prefix(self) -> None: + self.assertEqual( + dependency.retrieval_reference_prefix(self.topology, "R2_EVIDENCE"), + [ + "references/retrieval/skill.md", + "references/retrieval/direct.md", + "references/retrieval/discovery.md", + "references/retrieval/evidence.md", + ], + ) + + def test_declared_prefix_preserves_case_sensitive_skill_path(self) -> None: + self.assertEqual( + retrieval.retrieval_declared_prefix(self.topology, "R1_DISCOVERY"), + [ + "references/retrieval/SKILL.md", + "references/retrieval/direct.md", + "references/retrieval/discovery.md", + ], + ) + + def test_structural_is_the_only_leaf(self) -> None: + leaves = {name for name, spec in self.nodes.items() if not spec["children"]} + self.assertEqual(leaves, {"structural"}) + + def test_provider_names_are_not_retrieval_nodes(self) -> None: + self.assertTrue({"zg", "zvec-grep", "codebase-memory-mcp", "rtk"}.isdisjoint(self.nodes)) + + +class CanonicalTraceParserTests(unittest.TestCase): + def test_digit_bearing_retrieval_stage_is_parsed(self) -> None: + trace = retrieval_trace.parse_trace( + "TREE_TRACE path=core>debugging retrieval=R2_EVIDENCE manual=none " + "refs=references/retrieval/SKILL.md,references/retrieval/direct.md" + ) + self.assertEqual(trace["path"], ["core", "debugging"]) + self.assertEqual(trace["retrieval"], "R2_EVIDENCE") + + +class RetrievalReferenceObservationTests(unittest.TestCase): + def test_observed_references_preserve_progressive_command_order(self) -> None: + observed = retrieval_trace.observed_references( + [ + "Get-Content references/retrieval/SKILL.md", + "Get-Content references/retrieval/direct.md", + "Get-Content references/retrieval/discovery.md", + ] + ) + self.assertEqual( + observed, + [ + "references/retrieval/skill.md", + "references/retrieval/direct.md", + "references/retrieval/discovery.md", + ], + ) + + def test_repeated_reference_is_deduplicated_without_reordering(self) -> None: + observed = retrieval_trace.observed_references( + [ + "cat references/retrieval/SKILL.md references/retrieval/direct.md", + "cat references/retrieval/SKILL.md", + ] + ) + self.assertEqual( + observed, + ["references/retrieval/skill.md", "references/retrieval/direct.md"], + ) + + +class MeasuredSetupGuardTests(unittest.TestCase): + def test_forbidden_setup_commands_are_detected(self) -> None: + for command in ( + "zg index --embedding local/potion-code-16m-v2", + "codebase-memory-mcp cli index_repository --repo-path .", + "rtk init -g --codex", + "npm ci", + "npm install", + ): + with self.subTest(command=command): + self.assertRegex(command, dependency.SETUP_COMMAND_RE) + + def test_normal_provider_queries_and_requested_builds_are_not_setup(self) -> None: + for command in ( + 'zg query --human "where is login restored" --limit 5', + "codebase-memory-mcp cli trace_path --project workspace --function-name run", + "rtk git diff", + "mvn -pl ai-example/ai-example-memory/ai-example-spring-ai-memory -am compile", + "npm test -- src/lib/exportFilename.test.ts", + ): + with self.subTest(command=command): + self.assertIsNone(dependency.SETUP_COMMAND_RE.search(command)) + + +class ProviderUsageDetectionTests(unittest.TestCase): + def test_absolute_provider_paths_are_detected(self) -> None: + usage = retrieval._provider_usage( + [ + "/usr/local/bin/zg query --human auth --limit 5", + r"C:\tools\rtk.exe git diff", + "/opt/cbm/codebase-memory-mcp cli list_projects", + ] + ) + self.assertEqual( + usage, + {"zvec-grep": True, "codebase-memory-mcp": True, "rtk": True}, + ) + + +class RetrievalCeilingTests(unittest.TestCase): + @classmethod + def setUpClass(cls) -> None: + cls.topology = base.load_topology(HERE / "tree_topology.json") + + def test_current_only_runs_adaptive_plus_all_five_ceilings(self) -> None: + specs = retrieval.build_specs(1, current_only=True, selected_cases={"pp-known-contract"}) + self.assertEqual( + {variant for _, variant, _ in specs}, + {"adaptive", *(f"retrieval-cap:{stage}" for stage in retrieval.STAGES)}, + ) + + def test_provider_availability_is_owned_by_retrieval_stage(self) -> None: + self.assertEqual(retrieval.allowed_provider_ids("R0_DIRECT"), {"rtk"}) + self.assertEqual(retrieval.allowed_provider_ids("R1_DISCOVERY"), {"rtk", "zvec-grep"}) + self.assertEqual( + retrieval.allowed_provider_ids("R3_STRUCTURAL"), + {"rtk", "zvec-grep", "codebase-memory-mcp"}, + ) + + def test_deeper_provider_use_is_a_ceiling_violation(self) -> None: + usage = {"rtk": False, "zvec-grep": True, "codebase-memory-mcp": False} + self.assertTrue(retrieval.provider_ceiling_violation(usage, "R0_DIRECT")) + self.assertFalse(retrieval.provider_ceiling_violation(usage, "R1_DISCOVERY")) + + def test_ceiling_instruction_uses_real_paths(self) -> None: + instruction = retrieval.retrieval_ceiling_instruction(self.topology, "R0_DIRECT") + self.assertIn("references/retrieval/SKILL.md", instruction) + self.assertNotIn("references/retrieval/skill.md", instruction) + + +if __name__ == "__main__": + unittest.main() diff --git a/benchmarks/test_evolution_workflow.py b/benchmarks/test_evolution_workflow.py new file mode 100644 index 0000000..476b2e2 --- /dev/null +++ b/benchmarks/test_evolution_workflow.py @@ -0,0 +1,15 @@ +from __future__ import annotations + +import unittest + +from benchmarks import evolution_workflow_validation as evolution + + +class EvolutionWorkflowTests(unittest.TestCase): + def test_contract_score_is_perfect(self) -> None: + report = evolution.evaluate() + self.assertEqual(report["score"], 1.0, report["checks"]) + + +if __name__ == "__main__": + unittest.main() diff --git a/benchmarks/test_retrieval_analysis.py b/benchmarks/test_retrieval_analysis.py new file mode 100644 index 0000000..cf901d2 --- /dev/null +++ b/benchmarks/test_retrieval_analysis.py @@ -0,0 +1,74 @@ +from __future__ import annotations + +import unittest + +from benchmarks import retrieval_analysis as analysis + + +def row(task, variant, passed, *, selected=None, repetition=1, violation=False): + return { + "task_id": task, + "variant": variant, + "repetition": repetition, + "passed": passed, + "selected_retrieval": selected, + "capability_usage": {}, + "measurement_phase": "measured", + "setup_included_in_comparison": False, + "measured_setup_violation": violation, + } + + +class RetrievalAnalysisTests(unittest.TestCase): + def test_shallowest_stable_passing_ceiling_is_minimum(self) -> None: + rows = [ + row("t", "retrieval-cap:NONE", False), + row("t", "retrieval-cap:R0_DIRECT", False), + row("t", "retrieval-cap:R1_DISCOVERY", True), + row("t", "retrieval-cap:R2_EVIDENCE", True), + row("t", "retrieval-cap:R3_STRUCTURAL", True), + row("t", "adaptive", True, selected="R1_DISCOVERY"), + ] + report = analysis.analyze(rows) + self.assertEqual(report["tasks"]["t"]["minimum_sufficient_retrieval_stage"], "R1_DISCOVERY") + self.assertEqual(report["adaptive_relation_counts"], {"exact_minimum": 1}) + + def test_any_failed_repetition_prevents_stable_pass(self) -> None: + rows = [ + row("t", "retrieval-cap:R1_DISCOVERY", True, repetition=1), + row("t", "retrieval-cap:R1_DISCOVERY", False, repetition=2), + row("t", "retrieval-cap:R2_EVIDENCE", True, repetition=1), + row("t", "retrieval-cap:R2_EVIDENCE", True, repetition=2), + row("t", "adaptive", True, selected="R1_DISCOVERY", repetition=1), + ] + report = analysis.analyze(rows) + self.assertEqual(report["tasks"]["t"]["minimum_sufficient_retrieval_stage"], "R2_EVIDENCE") + self.assertEqual(report["adaptive_relation_counts"], {"under_disclosure": 1}) + + def test_missing_repetition_prevents_a_false_shallow_minimum(self) -> None: + rows = [ + row("t", "retrieval-cap:R1_DISCOVERY", True, repetition=1), + row("t", "retrieval-cap:R2_EVIDENCE", True, repetition=1), + row("t", "retrieval-cap:R2_EVIDENCE", True, repetition=2), + row("t", "retrieval-cap:R3_STRUCTURAL", True, repetition=1), + row("t", "retrieval-cap:R3_STRUCTURAL", True, repetition=2), + row("t", "adaptive", True, selected="R2_EVIDENCE", repetition=1), + ] + report = analysis.analyze(rows) + self.assertEqual(report["tasks"]["t"]["minimum_sufficient_retrieval_stage"], "R2_EVIDENCE") + self.assertFalse(report["tasks"]["t"]["stable_ceiling_pass"]["R1_DISCOVERY"]) + + def test_no_passing_ceiling_is_quality_gap(self) -> None: + rows = [row("t", f"retrieval-cap:{stage}", False) for stage in analysis.STAGES] + rows.append(row("t", "adaptive", True, selected="R3_STRUCTURAL")) + report = analysis.analyze(rows) + self.assertIsNone(report["tasks"]["t"]["minimum_sufficient_retrieval_stage"]) + self.assertEqual(report["adaptive_relation_counts"], {"quality_gap": 1}) + + def test_setup_violation_is_visible(self) -> None: + report = analysis.analyze([row("t", "adaptive", False, selected="R0_DIRECT", violation=True)]) + self.assertEqual(report["setup_measurement_contract_violation_count"], 1) + + +if __name__ == "__main__": + unittest.main() diff --git a/benchmarks/test_tree_benchmarks.py b/benchmarks/test_tree_benchmarks.py new file mode 100644 index 0000000..a0d0fae --- /dev/null +++ b/benchmarks/test_tree_benchmarks.py @@ -0,0 +1,458 @@ +from __future__ import annotations + +import unittest +from pathlib import Path + +from benchmarks import tree_analysis as analysis +from benchmarks import tree_cases +from benchmarks import tree_skilluse_analysis as skilluse +from benchmarks import tree_validation as validation + + +HERE = Path(__file__).resolve().parent + + +class TreeTopologyTests(unittest.TestCase): + @classmethod + def setUpClass(cls) -> None: + cls.topology = validation.load_topology(HERE / "tree_topology.json") + + def test_seed_paths_are_parent_local(self) -> None: + self.assertEqual(validation.node_path(self.topology, "core"), ["core"]) + self.assertEqual(validation.node_path(self.topology, "debugging"), ["core", "debugging"]) + self.assertEqual(validation.node_path(self.topology, "implementation"), ["core", "implementation"]) + + def test_evidence_rejected_descendants_leave_seed_nodes_as_leaves(self) -> None: + self.assertEqual(self.topology["automatic_nodes"]["debugging"]["children"], []) + self.assertEqual(self.topology["automatic_nodes"]["implementation"]["children"], []) + + def test_cross_sibling_path_is_invalid(self) -> None: + self.assertFalse(validation.validate_automatic_path(self.topology, ["core", "debugging", "implementation"])) + self.assertFalse( + validation.validate_automatic_path( + self.topology, + ["core", "implementation", "debugging"], + ) + ) + + def test_manual_mode_is_not_an_automatic_node(self) -> None: + self.assertNotIn("decision", self.topology["automatic_nodes"]) + self.assertIn("decision", self.topology["manual_modes"]) + trace = validation.parse_trace( + "TREE_TRACE path=core retrieval=NONE manual=decision refs=references/manual/decision.md" + ) + self.assertTrue(validation.validate_trace(self.topology, trace)) + self.assertEqual(trace["path"], ["core"]) + + def test_missing_trace_can_be_recovered_from_observed_reference_reads(self) -> None: + trace = validation.infer_trace_from_commands( + self.topology, + [ + r"Get-Content D:\Workspace\AiProjects\practical-coding\references\implementation.md", + r"Get-Content D:\Workspace\AiProjects\practical-coding\references\manual\decision.md", + ], + ) + self.assertEqual(trace["path"], ["core", "implementation"]) + self.assertEqual(trace["manual"], "decision") + self.assertTrue(validation.validate_trace(self.topology, trace)) + + def test_observed_retired_reference_remains_an_invalid_trace(self) -> None: + trace = validation.infer_trace_from_commands( + self.topology, + [ + r"Get-Content D:\Workspace\AiProjects\practical-coding\references\implementation.md", + r"Get-Content D:\Workspace\AiProjects\practical-coding\references\implementation-state-concurrency.md", + ], + ) + self.assertEqual(trace["path"], ["core", "implementation"]) + self.assertFalse(validation.validate_trace(self.topology, trace)) + + +class MinimumSufficientTests(unittest.TestCase): + @classmethod + def setUpClass(cls) -> None: + cls.topology = validation.load_topology(HERE / "tree_topology.json") + + def test_root_dominates_passing_descendants(self) -> None: + result = analysis.minimum_sufficient_set( + self.topology, + { + "core": True, + "debugging": True, + "implementation": True, + }, + ) + self.assertEqual(result, {"core"}) + + def test_multiple_sibling_minima_are_allowed(self) -> None: + result = analysis.minimum_sufficient_set( + self.topology, + { + "core": False, + "debugging": True, + "implementation": True, + }, + ) + self.assertEqual(result, {"debugging", "implementation"}) + + def test_leaf_minimum_is_derived_when_root_fails(self) -> None: + result = analysis.minimum_sufficient_set( + self.topology, + { + "core": False, + "debugging": True, + "implementation": False, + }, + ) + self.assertEqual(result, {"debugging"}) + + def test_no_passing_capability_is_quality_gap(self) -> None: + result = analysis.minimum_sufficient_set( + self.topology, + { + "core": False, + "debugging": False, + "implementation": False, + }, + ) + self.assertEqual(result, set()) + self.assertEqual( + analysis.relation_to_minimum(self.topology, "core", result, False), + "quality_gap", + ) + + def test_over_and_under_disclosure_are_topology_diagnostics(self) -> None: + self.assertEqual( + analysis.relation_to_minimum(self.topology, "debugging", {"core"}, True), + "over_disclosure", + ) + self.assertEqual( + analysis.relation_to_minimum(self.topology, "core", {"debugging"}, False), + "under_disclosure", + ) + + def test_retired_selected_node_is_an_invalid_trace_not_an_analysis_crash(self) -> None: + self.assertEqual( + analysis.relation_to_minimum(self.topology, "state-concurrency", {"core"}, False), + "invalid_trace", + ) + + +class SkillUseMetricTests(unittest.TestCase): + def test_skilluse_self_test(self) -> None: + skilluse.self_test() + + +class ManualContractTests(unittest.TestCase): + @classmethod + def setUpClass(cls) -> None: + cls.topology = validation.load_topology(HERE / "tree_topology.json") + + def test_automatic_trace_with_manual_reference_is_detectable(self) -> None: + trace = validation.parse_trace( + "TREE_TRACE path=core>implementation retrieval=STRUCTURAL manual=decision refs=references/manual/decision.md,references/implementation.md" + ) + self.assertTrue(validation.validate_trace(self.topology, trace)) + self.assertEqual(trace["manual"], "decision") + self.assertIn("references/manual/decision.md", trace["references_loaded"]) + + def test_explicit_manual_trace_remains_outside_path(self) -> None: + trace = validation.parse_trace( + "TREE_TRACE path=core retrieval=BOUNDED manual=decision refs=references/manual/decision.md" + ) + self.assertEqual(trace["path"], ["core"]) + self.assertEqual(trace["manual"], "decision") + self.assertTrue(validation.validate_trace(self.topology, trace)) + + def test_explicit_manual_contract_normalizes_windows_reference_path(self) -> None: + case = next(item for item in tree_cases.CASES if item["task_id"] == "sa-memory-strategy-manual-decision") + trace = validation.parse_trace( + r"TREE_TRACE path=core retrieval=TARGETED manual=decision refs=D:\repo\references\manual\decision.md" + ) + result = validation.score_answer( + case, + ( + "Recommendation: use SummaryCompressionMemoryChatService instead of " + "SlidingWindowMemoryChatService. Strongest trade-off: lossy recall." + ), + [], + HERE.parent, + trace=trace, + enforce_runtime_contract=True, + ) + self.assertTrue(result["manual_contract_ok"]) + + def test_non_manual_windows_reference_does_not_satisfy_manual_contract(self) -> None: + case = next(item for item in tree_cases.CASES if item["task_id"] == "sa-memory-strategy-manual-decision") + trace = validation.parse_trace( + r"TREE_TRACE path=core retrieval=TARGETED manual=decision refs=D:\repo\references\implementation.md" + ) + result = validation.score_answer( + case, + ( + "Recommendation: use SummaryCompressionMemoryChatService instead of " + "SlidingWindowMemoryChatService. Strongest trade-off: lossy recall." + ), + [], + HERE.parent, + trace=trace, + enforce_runtime_contract=True, + ) + self.assertFalse(result["manual_contract_ok"]) + + def test_manual_contract_accepts_root_elided_identity(self) -> None: + case = next(item for item in tree_cases.CASES if item["task_id"] == "sa-memory-strategy-manual-decision") + trace = validation.parse_trace( + "TREE_TRACE path=core retrieval=TARGETED manual=decision refs=manual/decision.md" + ) + result = validation.score_answer( + case, + ( + "Recommendation: use SummaryCompressionMemoryChatService instead of " + "SlidingWindowMemoryChatService. Strongest trade-off: lossy recall." + ), + [], + HERE.parent, + trace=trace, + enforce_runtime_contract=True, + ) + self.assertTrue(result["manual_contract_ok"]) + + def test_manual_contract_accepts_observed_reference_read(self) -> None: + case = next(item for item in tree_cases.CASES if item["task_id"] == "sa-memory-strategy-manual-decision") + trace = validation.parse_trace( + "TREE_TRACE path=core retrieval=TARGETED manual=decision refs=none" + ) + result = validation.score_answer( + case, + ( + "Recommendation: use SummaryCompressionMemoryChatService instead of " + "SlidingWindowMemoryChatService. Strongest trade-off: lossy recall." + ), + [r"Get-Content D:\repo\references\manual\decision.md"], + HERE.parent, + trace=trace, + enforce_runtime_contract=True, + ) + self.assertTrue(result["manual_contract_ok"]) + + def test_automatic_task_detects_observed_manual_reference_read(self) -> None: + case = next(item for item in tree_cases.CASES if item["task_id"] == "pp-known-contract") + trace = validation.parse_trace( + "TREE_TRACE path=core retrieval=TARGETED manual=none refs=none" + ) + result = validation.score_answer( + case, + "PluginDescriptor status version lifecycle", + [r"Get-Content D:\repo\references\manual\decision.md"], + HERE.parent, + trace=trace, + enforce_runtime_contract=True, + ) + self.assertTrue(result["spontaneous_manual_mode"]) + self.assertFalse(result["manual_contract_ok"]) + + def test_manual_tradeoff_evidence_does_not_require_colon_formatting(self) -> None: + case = next(item for item in tree_cases.CASES if item["task_id"] == "sa-memory-strategy-manual-decision") + trace = validation.parse_trace( + "TREE_TRACE path=core retrieval=TARGETED manual=decision refs=references/manual/decision.md" + ) + result = validation.score_answer( + case, + ( + "Recommendation: use SummaryCompressionMemoryChatService instead of " + "SlidingWindowMemoryChatService for summary-compression. " + "Its strongest trade-off is lossy recall." + ), + [], + HERE.parent, + trace=trace, + enforce_runtime_contract=True, + ) + self.assertEqual(result["missing_evidence_groups"], []) + self.assertTrue(result["manual_contract_ok"]) + + def test_manual_evidence_accepts_equivalent_chinese_labels(self) -> None: + case = next(item for item in tree_cases.CASES if item["task_id"] == "sa-memory-strategy-manual-decision") + trace = validation.parse_trace( + "TREE_TRACE path=core retrieval=TARGETED manual=decision refs=references/manual/decision.md" + ) + result = validation.score_answer( + case, + ( + "推荐：使用 SummaryCompressionMemoryChatService，而不是 SlidingWindowMemoryChatService。" + "summary-compression 的最强权衡是额外模型调用和细节损失。" + ), + [], + HERE.parent, + trace=trace, + enforce_runtime_contract=True, + ) + self.assertEqual(result["missing_evidence_groups"], []) + self.assertTrue(result["manual_contract_ok"]) + + def test_manual_evidence_accepts_decision_and_cost_wording(self) -> None: + case = next(item for item in tree_cases.CASES if item["task_id"] == "sa-memory-strategy-manual-decision") + trace = validation.parse_trace( + "TREE_TRACE path=core retrieval=TARGETED manual=decision refs=references/manual/decision.md" + ) + result = validation.score_answer( + case, + ( + "Decision: choose SummaryCompressionMemoryChatService instead of " + "SlidingWindowMemoryChatService for summary-compression. " + "Its strongest cost is lossy recall." + ), + [], + HERE.parent, + trace=trace, + enforce_runtime_contract=True, + ) + self.assertEqual(result["missing_evidence_groups"], []) + self.assertTrue(result["manual_contract_ok"]) + + def test_manual_evidence_accepts_recommend_as_a_verb(self) -> None: + case = next(item for item in tree_cases.CASES if item["task_id"] == "sa-memory-strategy-manual-decision") + trace = validation.parse_trace( + "TREE_TRACE path=core retrieval=TARGETED manual=decision refs=references/manual/decision.md" + ) + result = validation.score_answer( + case, + ( + "Recommend SummaryCompressionMemoryChatService over SlidingWindowMemoryChatService. " + "The summary-compression trade-off is lossy recall." + ), + [], + HERE.parent, + trace=trace, + enforce_runtime_contract=True, + ) + self.assertEqual(result["missing_evidence_groups"], []) + self.assertTrue(result["manual_contract_ok"]) + + def test_manual_evidence_still_requires_a_tradeoff(self) -> None: + case = next(item for item in tree_cases.CASES if item["task_id"] == "sa-memory-strategy-manual-decision") + trace = validation.parse_trace( + "TREE_TRACE path=core retrieval=TARGETED manual=decision refs=references/manual/decision.md" + ) + result = validation.score_answer( + case, + "推荐：使用 SummaryCompressionMemoryChatService，而不是 SlidingWindowMemoryChatService。", + [], + HERE.parent, + trace=trace, + enforce_runtime_contract=True, + ) + self.assertIn(["trade-off", "tradeoff", "cost", "权衡", "代价"], result["missing_evidence_groups"]) + + +class EvidenceOracleTests(unittest.TestCase): + def test_executor_diagnosis_accepts_a_concrete_focused_test_method(self) -> None: + case = next(item for item in tree_cases.CASES if item["task_id"] == "pp-running-after-throw") + result = validation.score_answer( + case, + ( + "DefaultPluginOperationExecutor runCommand leaves RUNNING after an Error. " + "Strengthen errorIsNotSwallowedAsOperationFailure to assert the record reaches a failed terminal state." + ), + [], + HERE.parent, + trace=None, + enforce_runtime_contract=False, + ) + self.assertEqual(result["missing_evidence_groups"], []) + + def test_cancel_diagnosis_accepts_existing_cancellation_test(self) -> None: + case = next(item for item in tree_cases.CASES if item["task_id"] == "ca-cancel-download") + result = validation.score_answer( + case, + ( + "EditorShell passes an AbortController signal into exportCover. " + "The existing avifEncoder.test.ts covers cancellation, while the cheapest falsifying test " + "should probe before link.click and prove the download side effect is suppressed." + ), + [], + HERE.parent, + trace=None, + enforce_runtime_contract=False, + ) + self.assertEqual(result["missing_evidence_groups"], []) + + def test_cancel_diagnosis_accepts_semantic_operation_and_test_evidence(self) -> None: + case = next(item for item in tree_cases.CASES if item["task_id"] == "ca-cancel-download") + result = validation.score_answer( + case, + ( + "EditorShell calls abort() while exportCover can still reach link.click. " + "Existing tests cover stalled-worker cancellation; add one falsifying test at the download probe." + ), + [], + HERE.parent, + trace=None, + enforce_runtime_contract=False, + ) + self.assertEqual(result["missing_evidence_groups"], []) + + def test_cancel_diagnosis_accepts_authoritative_boundary_without_ui_caller(self) -> None: + case = next(item for item in tree_cases.CASES if item["task_id"] == "ca-cancel-download") + result = validation.score_answer( + case, + ( + "exportCover has no signal.aborted check before link.click. " + "The focused avifEncoder.test.ts covers abort; add one falsifying test at that probe." + ), + [], + HERE.parent, + trace=None, + enforce_runtime_contract=False, + ) + self.assertEqual(result["missing_evidence_groups"], []) + + def test_focused_probe_accepts_an_explicit_blocked_outcome(self) -> None: + case = next(item for item in tree_cases.CASES if item["task_id"] == "ca-export-filename-probe") + result = validation.score_answer( + case, + ( + "Ran npx vitest run src/lib/exportFilename.test.ts once. " + "Outcome: blocked before collection because a required plugin could not be resolved." + ), + ["npx vitest run src/lib/exportFilename.test.ts"], + HERE.parent, + trace=None, + enforce_runtime_contract=False, + ) + self.assertEqual(result["missing_evidence_groups"], []) + + def test_focused_probe_accepts_an_explicit_outcome_field(self) -> None: + case = next(item for item in tree_cases.CASES if item["task_id"] == "ca-export-filename-probe") + result = validation.score_answer( + case, + ( + "Ran npm test -- src/lib/exportFilename.test.ts once. " + "Outcome: Vitest did not start because dependencies are absent." + ), + ["npm test -- src/lib/exportFilename.test.ts"], + HERE.parent, + trace=None, + enforce_runtime_contract=False, + ) + self.assertEqual(result["missing_evidence_groups"], []) + + def test_cancel_diagnosis_still_requires_concrete_test_evidence(self) -> None: + case = next(item for item in tree_cases.CASES if item["task_id"] == "ca-cancel-download") + result = validation.score_answer( + case, + "An AbortController signal reaches exportCover; inspect the download boundary.", + [], + HERE.parent, + trace=None, + enforce_runtime_contract=False, + ) + self.assertIn( + ["focused", "suite", "existing test", ".test."], + result["missing_evidence_groups"], + ) + + +if __name__ == "__main__": + unittest.main() diff --git a/benchmarks/test_tree_discriminator.py b/benchmarks/test_tree_discriminator.py new file mode 100644 index 0000000..76fdf6a --- /dev/null +++ b/benchmarks/test_tree_discriminator.py @@ -0,0 +1,20 @@ +from __future__ import annotations + +import unittest +from pathlib import Path + +from benchmarks import tree_discriminator_validation as discriminator +from benchmarks import tree_validation as validation + + +HERE = Path(__file__).resolve().parent + + +class TreeDiscriminatorTests(unittest.TestCase): + def test_cases_match_local_topology(self) -> None: + topology = validation.load_topology(HERE / "tree_topology.json") + discriminator.self_test(topology) + + +if __name__ == "__main__": + unittest.main() diff --git a/benchmarks/tree_analysis.py b/benchmarks/tree_analysis.py new file mode 100644 index 0000000..e864840 --- /dev/null +++ b/benchmarks/tree_analysis.py @@ -0,0 +1,374 @@ +#!/usr/bin/env python3 +"""Analyze evolvable router-tree benchmark results. + +The analysis derives minimum-sufficient node sets from capability ceilings instead +of comparing adaptive behavior to a predefined gold route. Routing disagreement is +therefore evidence about topology, not automatically a model failure. +""" + +from __future__ import annotations + +import argparse +import json +import statistics +from collections import Counter, defaultdict +from pathlib import Path +from typing import Any, Iterable + + +QUALITY_NONINFERIORITY_MARGIN = 0.03 +PROMOTE_THRESHOLD = 0.80 +MERGE_AMBIGUITY_THRESHOLD = 0.20 +MIN_TOPOLOGY_SAMPLE = 3 + + +def load_jsonl(path: Path) -> list[dict[str, Any]]: + rows: list[dict[str, Any]] = [] + with path.open("r", encoding="utf-8") as handle: + for line_number, raw in enumerate(handle, 1): + line = raw.strip() + if not line: + continue + try: + rows.append(json.loads(line)) + except json.JSONDecodeError as exc: + raise ValueError(f"{path}:{line_number}: invalid JSON: {exc}") from exc + return rows + + +def load_topology(path: Path) -> dict[str, Any]: + return json.loads(path.read_text(encoding="utf-8")) + + +def ancestors(topology: dict[str, Any], node: str, *, include_self: bool = False) -> list[str]: + nodes = topology["automatic_nodes"] + result = [node] if include_self else [] + current = nodes[node].get("parent") + while current is not None: + result.append(current) + current = nodes[current].get("parent") + return result + + +def descendants(topology: dict[str, Any], node: str, *, include_self: bool = False) -> set[str]: + nodes = topology["automatic_nodes"] + result = {node} if include_self else set() + stack = list(nodes[node].get("children", [])) + while stack: + current = stack.pop() + if current in result: + continue + result.add(current) + stack.extend(nodes[current].get("children", [])) + return result + + +def is_ancestor(topology: dict[str, Any], ancestor: str, node: str) -> bool: + return ancestor in ancestors(topology, node) + + +def determinate(rows: Iterable[dict[str, Any]]) -> list[dict[str, Any]]: + return [row for row in rows if row.get("passed") is not None] + + +def stable_pass(rows: Iterable[dict[str, Any]]) -> bool: + selected = determinate(rows) + return bool(selected) and all(row.get("passed") is True for row in selected) + + +def pass_rate(rows: Iterable[dict[str, Any]]) -> float | None: + selected = determinate(rows) + if not selected: + return None + return sum(row.get("passed") is True for row in selected) / len(selected) + + +def mean_or_none(rows: Iterable[dict[str, Any]], key: str) -> float | None: + values = [float(row[key]) for row in rows if row.get(key) is not None] + return statistics.mean(values) if values else None + + +def median_or_none(rows: Iterable[dict[str, Any]], key: str) -> float | None: + values = [float(row[key]) for row in rows if row.get(key) is not None] + return statistics.median(values) if values else None + + +def minimum_sufficient_set(topology: dict[str, Any], cap_status: dict[str, bool]) -> set[str]: + qualified = {node for node, passed in cap_status.items() if passed} + return { + node + for node in qualified + if not any(parent in qualified for parent in ancestors(topology, node)) + } + + +def relation_to_minimum(topology: dict[str, Any], selected: str | None, minimum: set[str], passed: bool | None) -> str: + if not minimum: + return "quality_gap" + if selected is None or selected not in topology["automatic_nodes"]: + return "invalid_trace" + if selected in minimum: + return "exact_minimum" if passed else "quality_failure_at_minimum" + if any(is_ancestor(topology, candidate, selected) for candidate in minimum): + return "over_disclosure" if passed else "quality_failure_after_over_disclosure" + if any(is_ancestor(topology, selected, candidate) for candidate in minimum): + return "under_disclosure" + return "alternate_branch" + + +def task_reports(rows: list[dict[str, Any]], topology: dict[str, Any]) -> list[dict[str, Any]]: + grouped: dict[str, list[dict[str, Any]]] = defaultdict(list) + for row in rows: + grouped[str(row["task_id"])].append(row) + + reports: list[dict[str, Any]] = [] + for task_id, task_rows in sorted(grouped.items()): + sample = task_rows[0] + if sample.get("manual_request"): + adaptive = [row for row in task_rows if row["variant"] == "adaptive"] + reports.append({ + "task_id": task_id, + "repository": sample["repository"], + "family": sample["family"], + "manual_request": sample["manual_request"], + "adaptive_stable_pass": stable_pass(adaptive), + "manual_contract_stable": bool(adaptive) and all(row.get("manual_contract_ok") is True for row in determinate(adaptive)), + }) + continue + + cap_status: dict[str, bool] = {} + for node in topology["automatic_nodes"]: + cap_rows = [row for row in task_rows if row["variant"] == f"cap:{node}"] + cap_status[node] = stable_pass(cap_rows) + minimum = minimum_sufficient_set(topology, cap_status) + adaptive = [row for row in task_rows if row["variant"] == "adaptive"] + terminals = Counter(row.get("selected_terminal_node") for row in determinate(adaptive) if row.get("selected_terminal_node")) + relations = Counter( + relation_to_minimum(topology, row.get("selected_terminal_node"), minimum, row.get("passed")) + for row in determinate(adaptive) + ) + reports.append({ + "task_id": task_id, + "repository": sample["repository"], + "family": sample["family"], + "manual_request": None, + "cap_stable_pass": cap_status, + "minimum_sufficient_set": sorted(minimum, key=lambda name: topology["automatic_nodes"][name]["depth"]), + "minimum_sufficient_depths": sorted({topology["automatic_nodes"][name]["depth"] for name in minimum}), + "adaptive_stable_pass": stable_pass(adaptive), + "adaptive_terminal_counts": dict(sorted(terminals.items())), + "adaptive_relation_counts": dict(sorted(relations.items())), + "adaptive_trace_stable": bool(adaptive) and len(terminals) == 1 and all(row.get("routing_trace_valid") is True for row in determinate(adaptive)), + }) + return reports + + +def node_reports(rows: list[dict[str, Any]], tasks: list[dict[str, Any]], topology: dict[str, Any]) -> dict[str, Any]: + result: dict[str, Any] = {} + automatic_tasks = [task for task in tasks if not task.get("manual_request")] + for node, spec in topology["automatic_nodes"].items(): + parent = spec.get("parent") + cap_rows = [row for row in rows if row["variant"] == f"cap:{node}" and not row.get("manual_request")] + stable_cap_tasks = [task for task in automatic_tasks if task["cap_stable_pass"].get(node)] + minimum_tasks = [task for task in automatic_tasks if node in task["minimum_sufficient_set"]] + marginal_lift_tasks: list[dict[str, Any]] = [] + parent_only_tasks: list[dict[str, Any]] = [] + if parent is not None: + marginal_lift_tasks = [ + task for task in automatic_tasks + if task["cap_stable_pass"].get(node) and not task["cap_stable_pass"].get(parent) + ] + parent_only_tasks = [ + task for task in automatic_tasks + if task["cap_stable_pass"].get(parent) and not task["cap_stable_pass"].get(node) + ] + adaptive_selected = [ + row for row in rows + if row["variant"] == "adaptive" and row.get("selected_terminal_node") == node and not row.get("manual_request") + ] + result[node] = { + "depth": spec["depth"], + "parent": parent, + "children": list(spec.get("children", [])), + "cap_stable_pass_tasks": len(stable_cap_tasks), + "minimum_sufficient_tasks": len(minimum_tasks), + "marginal_lift_over_parent_tasks": len(marginal_lift_tasks), + "parent_only_regression_tasks": len(parent_only_tasks), + "adaptive_selected_cells": len(adaptive_selected), + "adaptive_selected_tasks": len({row["task_id"] for row in adaptive_selected}), + "cap_cost": { + "tokens_mean": mean_or_none(determinate(cap_rows), "total_tokens"), + "duration_seconds_mean": mean_or_none(determinate(cap_rows), "duration_seconds"), + "tool_calls_mean": mean_or_none(determinate(cap_rows), "tool_calls"), + "tokens_median": median_or_none(determinate(cap_rows), "total_tokens"), + }, + "marginal_lift_task_ids": [task["task_id"] for task in marginal_lift_tasks], + "minimum_sufficient_task_ids": [task["task_id"] for task in minimum_tasks], + } + return result + + +def topology_suggestions(tasks: list[dict[str, Any]], nodes: dict[str, Any], topology: dict[str, Any]) -> list[dict[str, Any]]: + suggestions: list[dict[str, Any]] = [] + automatic_tasks = [task for task in tasks if not task.get("manual_request")] + + for node, report in nodes.items(): + if node == topology["root"]: + continue + if report["marginal_lift_over_parent_tasks"] == 0 and report["minimum_sufficient_tasks"] == 0: + suggestions.append({ + "action": "REMOVE_CANDIDATE", + "node": node, + "reason": "Node never becomes minimum-sufficient and shows no stable capability lift over its parent in this sample.", + }) + + parent = report["parent"] + if parent is not None: + parent_subtree = descendants(topology, parent, include_self=True) + scoped = [ + task for task in automatic_tasks + if any(minimum in parent_subtree for minimum in task["minimum_sufficient_set"]) + ] + required = [task for task in scoped if node in task["minimum_sufficient_set"]] + if len(scoped) >= MIN_TOPOLOGY_SAMPLE and len(required) / len(scoped) >= PROMOTE_THRESHOLD: + suggestions.append({ + "action": "PROMOTE_OR_COLLAPSE_CANDIDATE", + "node": node, + "parent": parent, + "support": len(required), + "scope": len(scoped), + "reason": "The child is minimum-sufficient for most tasks in the parent scope; the disclosure boundary may be too shallow to justify a separate node.", + }) + + for parent, parent_spec in topology["automatic_nodes"].items(): + children = list(parent_spec.get("children", [])) + if len(children) < 2: + continue + relevant = [task for task in automatic_tasks if any(child in task["minimum_sufficient_set"] for child in children)] + ambiguous = [task for task in relevant if sum(child in task["minimum_sufficient_set"] for child in children) >= 2] + if len(relevant) >= MIN_TOPOLOGY_SAMPLE and len(ambiguous) / len(relevant) >= MERGE_AMBIGUITY_THRESHOLD: + suggestions.append({ + "action": "MERGE_OR_MOVE_BOUNDARY_CANDIDATE", + "parent": parent, + "children": children, + "ambiguous_tasks": [task["task_id"] for task in ambiguous], + "reason": "Sibling capabilities are repeatedly co-minimum-sufficient; their current boundary may not buy enough specialization.", + }) + + failed_by_leaf_family: dict[tuple[str, str], list[str]] = defaultdict(list) + for task in automatic_tasks: + if task["adaptive_stable_pass"]: + continue + terminals = task.get("adaptive_terminal_counts", {}) + if not terminals: + continue + terminal = max(terminals, key=terminals.get) + if topology["automatic_nodes"][terminal].get("children"): + continue + failed_by_leaf_family[(terminal, task["family"])].append(task["task_id"]) + for (leaf, family), task_ids in sorted(failed_by_leaf_family.items()): + if len(task_ids) >= 2: + suggestions.append({ + "action": "DEEPEN_OR_SPLIT_CANDIDATE", + "node": leaf, + "family": family, + "tasks": task_ids, + "reason": "A stable failure cluster ends at the same leaf; inspect whether an observable pre-load sub-capability earns a child.", + }) + + return suggestions + + +def quality_report(rows: list[dict[str, Any]], tasks: list[dict[str, Any]]) -> dict[str, Any]: + arms: dict[str, Any] = {} + for variant in ("no-skill", "baseline", "adaptive"): + selected = [row for row in rows if row["variant"] == variant] + if not selected: + continue + arms[variant] = { + "cells": len(selected), + "pass_rate": pass_rate(selected), + "stable_tasks": sum( + stable_pass([row for row in selected if row["task_id"] == task_id]) + for task_id in {row["task_id"] for row in selected} + ), + "tokens_mean": mean_or_none(determinate(selected), "total_tokens"), + "duration_seconds_mean": mean_or_none(determinate(selected), "duration_seconds"), + "tool_calls_mean": mean_or_none(determinate(selected), "tool_calls"), + } + + adaptive = [row for row in rows if row["variant"] == "adaptive" and row.get("passed") is not None] + automatic = [row for row in adaptive if not row.get("manual_request")] + manual = [row for row in adaptive if row.get("manual_request")] + adaptive_rate = arms.get("adaptive", {}).get("pass_rate") + baseline_rate = arms.get("baseline", {}).get("pass_rate") + no_skill_rate = arms.get("no-skill", {}).get("pass_rate") + comparisons = [] + for name, comparator in (("baseline", baseline_rate), ("no-skill", no_skill_rate)): + if adaptive_rate is not None and comparator is not None: + comparisons.append({ + "comparator": name, + "pass": adaptive_rate + QUALITY_NONINFERIORITY_MARGIN >= comparator, + "candidate_pass_rate": adaptive_rate, + "comparator_pass_rate": comparator, + "margin": QUALITY_NONINFERIORITY_MARGIN, + }) + manual_false = sum(row.get("spontaneous_manual_mode") is True for row in automatic) + manual_explicit_fail = sum(row.get("manual_contract_ok") is not True for row in manual) + trace_fail = sum(row.get("routing_trace_valid") is not True for row in adaptive) + gate = all(item["pass"] for item in comparisons) and manual_false == 0 and manual_explicit_fail == 0 and trace_fail == 0 + return { + "release_quality_gate": "PASS" if gate else "FAIL", + "arms": arms, + "noninferiority": comparisons, + "automatic_spontaneous_manual_count": manual_false, + "explicit_manual_contract_failures": manual_explicit_fail, + "adaptive_trace_failures": trace_fail, + "note": "Adaptive route exactness is diagnostic topology evidence, not a release gate. Delivered quality and manual-mode discipline gate the candidate.", + } + + +def analyze(rows: list[dict[str, Any]], topology: dict[str, Any]) -> dict[str, Any]: + tasks = task_reports(rows, topology) + nodes = node_reports(rows, tasks, topology) + relation_counts: Counter[str] = Counter() + for task in tasks: + if task.get("manual_request"): + continue + relation_counts.update(task.get("adaptive_relation_counts", {})) + return { + "schema_version": 1, + "quality": quality_report(rows, tasks), + "routing_diagnostics": { + "relation_counts": dict(sorted(relation_counts.items())), + "tasks_with_multiple_minimum_nodes": [ + task["task_id"] for task in tasks if not task.get("manual_request") and len(task["minimum_sufficient_set"]) > 1 + ], + "tasks_without_qualified_cap": [ + task["task_id"] for task in tasks if not task.get("manual_request") and not task["minimum_sufficient_set"] + ], + }, + "nodes": nodes, + "topology_suggestions": topology_suggestions(tasks, nodes, topology), + "tasks": tasks, + } + + +def main() -> int: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("results", type=Path, help="tree_validation.py results.jsonl") + parser.add_argument("--topology", type=Path, default=Path(__file__).resolve().parent / "tree_topology.json") + parser.add_argument("--output", type=Path) + args = parser.parse_args() + + report = analyze(load_jsonl(args.results), load_topology(args.topology)) + text = json.dumps(report, ensure_ascii=False, indent=2, sort_keys=True) + if args.output: + args.output.write_text(text + "\n", encoding="utf-8") + else: + print(text) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/benchmarks/tree_cases.py b/benchmarks/tree_cases.py new file mode 100644 index 0000000..f1851dd --- /dev/null +++ b/benchmarks/tree_cases.py @@ -0,0 +1,166 @@ +"""Frozen real-repository cases for evolvable router-tree experiments. + +These cases score delivered evidence and manual-mode discipline. They intentionally +contain no expected automatic node, depth, or capability path. Automatic topology +is inferred by parent-versus-child capability ceilings and adaptive traces. +""" + +from __future__ import annotations + + +REPOSITORIES = { + "personal-progress": { + "url": "https://github.com/Hubujiu/personal-progress.git", + "commit": "515c2e2193c3d547e04e65687da6666dc877ab61", + "local_name": "personal-progress", + }, + "cover-atelier": { + "url": "https://github.com/Hubujiu/cover-atelier.git", + "commit": "fc3b12b3a944f45b5a1d19963e29307d95b120fb", + "local_name": "cover-atelier", + }, + "super-agent": { + "url": "https://github.com/java-up-up/super-agent.git", + "commit": "d44edf063032a2d8797549411f11923aa4a83ec3", + "local_name": "super-agent", + }, +} + + +def _case( + task_id: str, + repository: str, + family: str, + prompt: str, + required: list[list[str]], + *, + probe_terms: list[list[str]] | None = None, + manual_request: str | None = None, +) -> dict[str, object]: + return { + "task_id": task_id, + "repository": repository, + "family": family, + "prompt": prompt, + "required": required, + "probe_terms": probe_terms or [], + "manual_request": manual_request, + } + + +CASES = [ + _case( + "pp-known-contract", + "personal-progress", + "known-target", + "Read progress-core/src/main/java/com/hubujiu/progress/core/database/PluginDatabaseNames.java only. Report the schema and role naming invariants and why long plugin IDs cannot collide. Do not edit files.", + [["63", "max_identifier_bytes"], ["sha-256", "digest"], ["plugin_", "plugin_role_"]], + ), + _case( + "pp-lifecycle-map", + "personal-progress", + "structural-read", + "Trace install/start/stop from the platform API into lifecycle execution. Identify the controller, lifecycle service, operation executor, and state machine with source paths. Report only; do not edit files.", + [["PluginManagementController"], ["PluginLifecycleService"], ["PluginOperationExecutor", "DefaultPluginOperationExecutor"], ["PluginStateMachine"]], + ), + _case( + "pp-running-after-throw", + "personal-progress", + "unexplained-failure", + "An operation sometimes remains RUNNING after its worker throws. The cause is not established. Inspect the operation executor and focused tests, identify the earliest incorrect state transition, and name the cheapest falsifying test. Diagnose only; do not edit files.", + [["DefaultPluginOperationExecutor", "runCommand"], ["PluginOperationExecutorTest", "errorIsNotSwallowedAsOperationFailure", "commandErrorDoesNotLeaveOperationRunning"], ["RUNNING"], ["fail", "exception", "complete"]], + ), + _case( + "pp-token-rotation-boundary", + "personal-progress", + "risk-boundary", + "Plan a zero-downtime rotation of the bootstrap admin token. Map the authoritative filter/configuration boundary, protected platform entry points, rejection-before-side-effect behavior, and focused evidence. Use existing project boundaries rather than opening a technology-selection discussion. Report only; do not edit files.", + [["BootstrapAdminTokenFilter"], ["PlatformSecurityConfiguration"], ["401", "unauthorized", "reject"], ["PlatformManagementApiTest", "PluginDispatchApiTest"]], + ), + _case( + "pp-compatibility-manual-decision", + "personal-progress", + "manual-decision", + "Explicit decision request: compare a breaking rename versus a one-release compatibility alias for a required public plugin-dispatch response field while old plugins and clients coexist. Inspect the current HTTP/view contract as needed and recommend one option with its strongest trade-off. Do not implement.", + [["Recommendation:", "recommendation", "recommend", "Decision:", "choose", "推荐", "建议", "决定", "选择"], ["trade-off", "tradeoff", "cost", "权衡", "代价"], ["alias", "compatib"], ["one release", "one-release", "release window"]], + manual_request="decision", + ), + _case( + "ca-export-format-known", + "cover-atelier", + "known-target", + "Read src/lib/exportFormat.ts only and report each export format's MIME type, extension, and explicit quality value when present. Do not edit files.", + [["image/jpeg", "jpeg"], ["image/webp", "webp"], ["image/avif", "avif"], ["image/png", "png"]], + ), + _case( + "ca-cancel-download", + "cover-atelier", + "unexplained-failure", + "Users report that cancelling an export sometimes still downloads a file. The cause is not established. Inspect the cancellation path and focused tests, identify the earliest observable boundary to probe, and name the single cheapest falsifying test. Diagnose only; do not edit files.", + [["AbortController", "AbortSignal", "signal", "abort()", "abort"], ["exportCover"], ["focused", "suite", "existing test", ".test."], ["probe", "test"]], + ), + _case( + "ca-new-format-existing-pattern", + "cover-atelier", + "settled-local-choice", + "Plan the smallest coherent change to add one more image format by following the repository's existing export-format configuration and encoder boundary. Do not ask the user to choose an architecture or abstraction if the repository already settles it. Map the affected config, filename, orchestration, encoder boundary, and focused tests. Report only; do not edit files.", + [["exportFormat"], ["exportFilename"], ["exportCover"], ["encoder", "avifEncoder"], ["test"]], + ), + _case( + "ca-avif-stall-evidence", + "cover-atelier", + "uncertain-performance", + "Large AVIF exports are reported to stall the UI, but no timing evidence exists. Map the main-thread/worker boundary and propose one bounded measurement that separates encode latency, progress delivery, memory pressure, and cancellation. Diagnose and report only; do not edit files.", + [["avifEncoder.worker.ts"], ["encodeAvif", "avifEncoder"], ["performance", "duration", "latency", "measure"], ["memory"], ["cancel", "Abort"]], + ), + _case( + "ca-export-filename-probe", + "cover-atelier", + "focused-verification", + "Run the focused exportFilename test once to establish the current filename contract, then report the exact command and outcome. Do not edit files or run the full test suite.", + [["exportFilename"], ["Outcome:", "pass", "passed", "blocked", "failed", "error", "could not", "unable", "无法", "未运行", "tests"]], + probe_terms=[["exportfilename"], ["npm", "vitest"]], + ), + _case( + "sa-memory-map", + "super-agent", + "structural-read", + "Trace the memory comparison HTTP path from MemoryDemoController through MemoryComparisonService to the no-memory, sliding-window, and summary-compression implementations. Report paths and symbols only; do not edit files.", + [["MemoryDemoController"], ["MemoryComparisonService"], ["NoMemoryChatService"], ["SlidingWindowMemoryChatService"], ["SummaryCompressionMemoryChatService"]], + ), + _case( + "sa-sensitive-rejection-boundary", + "super-agent", + "risk-boundary", + "Review where sensitive-word rejection occurs in the Spring AI Alibaba request path. Map interceptor registration and callers, define rejection-before-model-side-effect behavior, and identify the narrowest security tests needed. Use the existing request architecture rather than opening a framework choice. Report only; do not edit files.", + [["SensitiveWordInterceptor"], ["SpringAiAlibabaAgentService"], ["reject", "before"], ["test"]], + ), + _case( + "sa-memory-reset-concurrency", + "super-agent", + "state-boundary", + "Review ResettableMemorySaver and its use by SpringAiAlibabaAgentService for concurrent sessions, reset ordering, and restart semantics. Identify the authoritative state owner and the smallest concurrency evidence. Resolve ordinary implementation choices from the existing code. Report only; do not edit files.", + [["ResettableMemorySaver"], ["SpringAiAlibabaAgentService"], ["concurrent", "thread"], ["reset", "clear"], ["restart", "durable", "memory"]], + ), + _case( + "sa-memory-strategy-manual-decision", + "super-agent", + "manual-decision", + "Explicit decision request: for this repository's conversational memory example, compare the existing sliding-window and summary-compression approaches for a long-running support chat where bounded context cost matters more than exact verbatim recall. Inspect the current implementations as needed, then recommend one with its strongest trade-off. Do not implement.", + [["Recommendation:", "recommendation", "recommend", "Decision:", "choose", "推荐", "建议", "决定", "选择"], ["trade-off", "tradeoff", "cost", "权衡", "代价"], ["SlidingWindowMemoryChatService", "sliding-window"], ["SummaryCompressionMemoryChatService", "summary-compression"]], + manual_request="decision", + ), + _case( + "sa-module-compile-probe", + "super-agent", + "focused-verification", + "Compile the ai-example-spring-ai-memory module once with its required reactor dependencies to establish current reachability. Report the exact Maven command and outcome; do not edit files or run unrelated modules.", + [["ai-example-spring-ai-memory"], ["build success", "success", "compiled"]], + probe_terms=[["mvn", "mvnw"], ["ai-example-spring-ai-memory"], ["-pl"]], + ), +] + + +TASK_IDS = {case["task_id"] for case in CASES} +MANUAL_IDS = {case["task_id"] for case in CASES if case["manual_request"]} +AUTOMATIC_IDS = TASK_IDS - MANUAL_IDS diff --git a/benchmarks/tree_discriminator_cases.py b/benchmarks/tree_discriminator_cases.py new file mode 100644 index 0000000..08a9c2b --- /dev/null +++ b/benchmarks/tree_discriminator_cases.py @@ -0,0 +1,150 @@ +"""Parent-local discriminator cases for staged router children. + +These are cheap trigger/boundary diagnostics, not release-quality task verifiers. +They intentionally expose only the parent router text; child bodies are not loaded. +""" + +from __future__ import annotations + + +CASES = [ + { + "case_id": "debug-dynamic-browser-worker", + "parent": "debugging", + "expected": "dynamic-evidence", + "prompt": ( + "A browser export occasionally completes after cancellation. The static call path is known, " + "but the failure only appears under worker/message timing and no current trace distinguishes " + "whether cancellation, progress delivery, or worker completion wins the race." + ), + }, + { + "case_id": "debug-dynamic-ci-environment", + "parent": "debugging", + "expected": "dynamic-evidence", + "prompt": ( + "A test passes locally and fails only in CI. The same source revision is used; the next useful " + "step is to compare runtime/config/process facts and capture evidence at the failing process boundary." + ), + }, + { + "case_id": "debug-parent-deterministic-trace", + "parent": "debugging", + "expected": "parent", + "prompt": ( + "A focused unit test deterministically fails. The stack trace and source show parse_bool does not " + "strip whitespace before lowercasing, and the shared caller path is already identified." + ), + }, + { + "case_id": "debug-parent-simple-exception", + "parent": "debugging", + "expected": "parent", + "prompt": ( + "A deterministic exception points at a single invalid index calculation. Reproduction, earliest " + "incorrect state, and the focused falsifying test are already available from the local trace." + ), + }, + { + "case_id": "impl-security-authz-side-effect", + "parent": "implementation", + "expected": "security-boundary", + "prompt": ( + "Add a privileged delete endpoint. The unresolved invariant is that authenticated users without " + "the resource permission must be rejected before any durable deletion or external notification." + ), + }, + { + "case_id": "impl-security-secret-rotation", + "parent": "implementation", + "expected": "security-boundary", + "prompt": ( + "Rotate an API credential. The unresolved work is the authoritative authentication/revocation " + "boundary and proving invalid or revoked credentials cannot reach the protected side effect." + ), + }, + { + "case_id": "impl-migration-public-field", + "parent": "implementation", + "expected": "migration-compatibility", + "prompt": ( + "Rename a required public response field while old clients must keep working for one release. " + "New and old versions will coexist, and rollback must remain possible before the compatibility window ends." + ), + }, + { + "case_id": "impl-migration-persisted-enum", + "parent": "implementation", + "expected": "migration-compatibility", + "prompt": ( + "Change a persisted integer status to strings. Existing rows and an older reader can coexist during " + "deployment, and the migration must define backfill, mixed-version behavior, cleanup, and rollback." + ), + }, + { + "case_id": "impl-state-idempotent-retry", + "parent": "implementation", + "expected": "state-concurrency", + "prompt": ( + "A webhook may be delivered more than once and the handler can retry after a timeout. The unresolved " + "guarantee is whether the durable state transition is idempotent and atomic across duplicate delivery." + ), + }, + { + "case_id": "impl-state-reset-race", + "parent": "implementation", + "expected": "state-concurrency", + "prompt": ( + "Concurrent session reset and update can interleave. The unresolved invariant is the authoritative " + "state owner, transition ordering, atomicity, and what survives restart." + ), + }, + { + "case_id": "impl-negative-new-local-validation", + "parent": "implementation", + "expected": "parent", + "prompt": ( + "Add validation for a new internal configuration value. There are no old versions, no untrusted " + "external caller, no privilege boundary, and no concurrent mutation; the owning parser is already known." + ), + }, + { + "case_id": "impl-negative-known-compatible-addition", + "parent": "implementation", + "expected": "parent", + "prompt": ( + "Add a new optional JSON response field using the repository's established serializer. It is additive, " + "old clients ignore unknown fields, no persisted data changes, and rollback is a normal code revert." + ), + }, + { + "case_id": "impl-hard-negative-security-not-migration", + "parent": "implementation", + "expected": "security-boundary", + "prompt": ( + "The token string format remains unchanged and no old/new representation coexistence is needed. " + "The blocker is ensuring revoked tokens are denied before a privileged mutation." + ), + }, + { + "case_id": "impl-hard-negative-migration-not-state", + "parent": "implementation", + "expected": "migration-compatibility", + "prompt": ( + "There is no concurrent writer and no retry behavior. The blocker is moving persisted rows to a new " + "representation while the previous application version can still read during rolling deployment." + ), + }, + { + "case_id": "impl-hard-negative-state-not-security", + "parent": "implementation", + "expected": "state-concurrency", + "prompt": ( + "Authorization is already settled and inputs are trusted. The blocker is preventing two concurrent " + "workers from both applying the same durable transition after duplicate queue delivery." + ), + }, +] + + +CASE_IDS = {case["case_id"] for case in CASES} diff --git a/benchmarks/tree_discriminator_validation.py b/benchmarks/tree_discriminator_validation.py new file mode 100644 index 0000000..a364d70 --- /dev/null +++ b/benchmarks/tree_discriminator_validation.py @@ -0,0 +1,262 @@ +#!/usr/bin/env python3 +"""Run cheap parent-local routing discrimination checks for staged tree children. + +This suite measures whether immediate-child trigger language is discriminative before +paying for full real-repository runs. It is diagnostic only: human-authored labels +here never replace capability-ceiling evidence or deterministic task verifiers. +When the active topology has no staged children, its frozen cases are historical and +the self-test reports the suite inactive. +""" + +from __future__ import annotations + +import argparse +import concurrent.futures +import datetime as dt +import json +import os +import re +import shutil +import sys +from collections import Counter +from pathlib import Path +from typing import Any + +HERE = Path(__file__).resolve().parent +ROOT = HERE.parent +if str(HERE) not in sys.path: + sys.path.insert(0, str(HERE)) + +import run_benchmarks as bench +from tree_discriminator_cases import CASES, CASE_IDS +from tree_validation import load_topology + + +ROUTE_RE = re.compile(r"^\s*ROUTE\s*=\s*([a-z0-9_-]+)\s*$", re.I) + + +def parent_prompt(case: dict[str, str], topology: dict[str, Any]) -> str: + parent = case["parent"] + spec = topology["automatic_nodes"][parent] + parent_path = ROOT / spec["reference"] + parent_text = parent_path.read_text(encoding="utf-8") + children = list(spec.get("children", [])) + allowed = ["parent", *children] + return ( + "This is a benchmark of one local router decision. Do not solve the coding task. " + "Do not load any child reference or infer grandchild behavior.\n\n" + f"\n{parent_text}\n\n\n" + f"\n{case['prompt']}\n\n\n" + f"Choose exactly one of: {', '.join(allowed)}. " + "Use parent when no immediate-child preload signal is clearly present. " + "Return exactly ROUTE= and nothing else." + ) + + +def parse_route(answer: str) -> str | None: + match = ROUTE_RE.fullmatch(answer.strip()) + return match.group(1).lower() if match else None + + +def prepare_workspace(path: Path) -> None: + path.mkdir(parents=True, exist_ok=True) + result = bench.run_command(["git", "init", "-q"], path) + if result.returncode: + raise RuntimeError(result.stderr) + + +def build_specs(runs: int, selected: set[str]) -> list[tuple[dict[str, str], int]]: + specs = [] + for case in CASES: + if selected and case["case_id"] not in selected: + continue + for repetition in range(1, runs + 1): + specs.append((case, repetition)) + return specs + + +def run_cell( + spec: tuple[dict[str, str], int], + args: argparse.Namespace, + topology: dict[str, Any], + eval_home: Path, + output: Path, +) -> dict[str, Any]: + case, repetition = spec + cell = output / "cells" / case["case_id"] / f"r{repetition:03d}" + result_path = cell / "result.json" + if result_path.is_file(): + return json.loads(result_path.read_text(encoding="utf-8")) + + cell.mkdir(parents=True, exist_ok=True) + workspace = cell / "workspace" + if workspace.exists(): + shutil.rmtree(workspace) + prepare_workspace(workspace) + + prompt = parent_prompt(case, topology) + (cell / "prompt.txt").write_text(prompt, encoding="utf-8") + env = os.environ.copy() + env["CODEX_HOME"] = str(eval_home) + codex = bench.resolve_codex(args.codex) + stdout = cell / "round1.jsonl" + stderr = cell / "round1.stderr.txt" + code, timed_out, forced, duration = bench.run_codex( + bench.codex_command(codex, workspace), + prompt, + workspace, + env, + stdout, + stderr, + args.timeout, + ) + parsed = bench.parse_transcript(stdout) + route = parse_route(parsed["answer"]) + infrastructure_error = "timeout" if timed_out else ( + f"codex exit status {code}" if code and not forced else None + ) + passed = None if infrastructure_error else route == case["expected"] + record = { + "schema_version": 1, + "case_id": case["case_id"], + "parent": case["parent"], + "expected": case["expected"], + "selected": route, + "repetition": repetition, + "passed": passed, + "error": infrastructure_error, + "duration_seconds": duration, + "tool_calls": parsed["tool_calls"], + **parsed["usage"], + } + (cell / "answer.txt").write_text(parsed["answer"] + "\n", encoding="utf-8") + result_path.write_text(json.dumps(record, ensure_ascii=False, indent=2) + "\n", encoding="utf-8") + return record + + +def ratio(numerator: int, denominator: int) -> float | None: + return numerator / denominator if denominator else None + + +def summarize(rows: list[dict[str, Any]], topology: dict[str, Any], runs: int) -> dict[str, Any]: + determinate = [row for row in rows if row.get("passed") is not None] + per_parent: dict[str, Any] = {} + for parent in sorted({row["parent"] for row in determinate}): + selected = [row for row in determinate if row["parent"] == parent] + per_parent[parent] = { + "cells": len(selected), + "accuracy": ratio(sum(row["passed"] is True for row in selected), len(selected)), + "confusion": dict(sorted(Counter( + f"{row['expected']}->{row.get('selected') or 'invalid'}" for row in selected + ).items())), + } + + per_child: dict[str, Any] = {} + for parent, spec in topology["automatic_nodes"].items(): + for child in spec.get("children", []): + positives = [row for row in determinate if row["parent"] == parent and row["expected"] == child] + negatives = [row for row in determinate if row["parent"] == parent and row["expected"] != child] + hits = sum(row.get("selected") == child for row in positives) + false = sum(row.get("selected") == child for row in negatives) + per_child[child] = { + "parent": parent, + "positive_cells": len(positives), + "trigger_recall": ratio(hits, len(positives)), + "negative_cells": len(negatives), + "boundary_specificity": ( + 1.0 - ratio(false, len(negatives)) if negatives else None + ), + "false_triggers": false, + } + + return { + "schema_version": 1, + "runs_per_case": runs, + "cases": len({row["case_id"] for row in rows}), + "determinate_cells": len(determinate), + "overall_accuracy": ratio(sum(row["passed"] is True for row in determinate), len(determinate)), + "per_parent": per_parent, + "per_child": per_child, + "note": ( + "This is a cheap router-language diagnostic with human-authored labels. " + "It is not a release gate and does not prove child capability lift." + ), + } + + +def self_test(topology: dict[str, Any]) -> None: + assert CASE_IDS + case_parents = {case["parent"] for case in CASES} + active_parents = { + name + for name, spec in topology["automatic_nodes"].items() + if name in case_parents and spec.get("children") + } + if not active_parents: + print("tree discriminator self-test: PASS (inactive; no staged children)") + return + cases = [case for case in CASES if case["parent"] in active_parents] + assert {case["parent"] for case in cases} == active_parents + for case in cases: + allowed = {"parent", *topology["automatic_nodes"][case["parent"]].get("children", [])} + assert case["expected"] in allowed + assert parse_route("ROUTE=dynamic-evidence") == "dynamic-evidence" + assert parse_route("something else") is None + print("tree discriminator self-test: PASS") + + +def parse_args() -> argparse.Namespace: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--runs", type=int, default=3) + parser.add_argument("--workers", type=int, default=3) + parser.add_argument("--output", type=Path) + parser.add_argument("--topology", type=Path, default=HERE / "tree_topology.json") + parser.add_argument("--codex", default=os.environ.get("CODEX_BIN", "codex")) + parser.add_argument("--timeout", type=float, default=180) + parser.add_argument("--case", action="append", default=[]) + parser.add_argument("--self-test", action="store_true") + return parser.parse_args() + + +def main() -> int: + args = parse_args() + topology = load_topology(args.topology.resolve()) + if args.self_test: + self_test(topology) + return 0 + if args.runs < 1 or args.workers < 1: + raise SystemExit("runs and workers must be positive") + selected = set(args.case) + unknown = selected - CASE_IDS + if unknown: + raise SystemExit(f"unknown cases: {', '.join(sorted(unknown))}") + + stamp = dt.datetime.now().strftime("%Y%m%d-%H%M%S") + output = (args.output or ROOT / "benchmark-results" / f"tree-discriminator-{stamp}").resolve() + output.mkdir(parents=True, exist_ok=True) + eval_home = bench.prepare_eval_home(output / "eval-home") + specs = build_specs(args.runs, selected) + rows = [] + with concurrent.futures.ThreadPoolExecutor(max_workers=args.workers) as pool: + futures = [ + pool.submit(run_cell, spec, args, topology, eval_home, output) + for spec in specs + ] + for future in concurrent.futures.as_completed(futures): + rows.append(future.result()) + rows.sort(key=lambda row: (row["case_id"], row["repetition"])) + (output / "results.jsonl").write_text( + "".join(json.dumps(row, ensure_ascii=False) + "\n" for row in rows), + encoding="utf-8", + ) + report = summarize(rows, topology, args.runs) + (output / "report.json").write_text( + json.dumps(report, ensure_ascii=False, indent=2) + "\n", + encoding="utf-8", + ) + print(json.dumps(report, ensure_ascii=False, indent=2)) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/benchmarks/tree_skilluse_analysis.py b/benchmarks/tree_skilluse_analysis.py new file mode 100644 index 0000000..ce895b7 --- /dev/null +++ b/benchmarks/tree_skilluse_analysis.py @@ -0,0 +1,283 @@ +#!/usr/bin/env python3 +"""Derive Trigger / Compliance / Boundary metrics from tree benchmark ceilings. + +This companion analysis intentionally does not introduce human-authored gold routes. +A node's positive routing opportunities are tasks where that node is empirically +minimum-sufficient while its parent is not. Negative opportunities are tasks already +stable-passing at the parent. Adaptive traces are then scored against those +empirically derived opportunity sets. +""" + +from __future__ import annotations + +import argparse +import json +import statistics +import sys +from collections import defaultdict +from pathlib import Path +from typing import Any, Iterable + +HERE = Path(__file__).resolve().parent +if str(HERE) not in sys.path: + sys.path.insert(0, str(HERE)) + +from tree_analysis import ( + descendants, + determinate, + load_jsonl, + load_topology, + task_reports, +) + + +MIN_POSITIVE_TASKS = 2 +MIN_POSITIVE_REPOSITORIES = 2 +TRIGGER_RECALL_TARGET = 0.80 +BOUNDARY_SPECIFICITY_TARGET = 0.90 +COMPLIANCE_TARGET = 0.90 + + +def _mean(rows: Iterable[dict[str, Any]], key: str) -> float | None: + values = [float(row[key]) for row in rows if row.get(key) is not None] + return statistics.mean(values) if values else None + + +def _task_rows(rows: list[dict[str, Any]]) -> dict[str, list[dict[str, Any]]]: + grouped: dict[str, list[dict[str, Any]]] = defaultdict(list) + for row in rows: + grouped[str(row["task_id"])].append(row) + return grouped + + +def _selected_in_subtree(topology: dict[str, Any], row: dict[str, Any], node: str) -> bool: + terminal = row.get("selected_terminal_node") + if not terminal: + return False + return terminal == node or terminal in descendants(topology, node) + + +def _cap_rows(task_rows: list[dict[str, Any]], node: str) -> list[dict[str, Any]]: + return [row for row in task_rows if row.get("variant") == f"cap:{node}"] + + +def _adaptive_rows(task_rows: list[dict[str, Any]]) -> list[dict[str, Any]]: + return determinate(row for row in task_rows if row.get("variant") == "adaptive") + + +def analyze_node( + rows: list[dict[str, Any]], + topology: dict[str, Any], + tasks: list[dict[str, Any]], + node: str, +) -> dict[str, Any]: + spec = topology["automatic_nodes"][node] + parent = spec.get("parent") + if parent is None: + raise ValueError("root has no Skill-Use routing metrics") + + grouped = _task_rows(rows) + ordinary_tasks = [task for task in tasks if not task.get("manual_request")] + + positive = [ + task for task in ordinary_tasks + if node in task["minimum_sufficient_set"] + and not task["cap_stable_pass"].get(parent) + ] + negative = [ + task for task in ordinary_tasks + if task["cap_stable_pass"].get(parent) + ] + + positive_rows: list[dict[str, Any]] = [] + negative_rows: list[dict[str, Any]] = [] + triggered_rows: list[dict[str, Any]] = [] + positive_triggered: list[dict[str, Any]] = [] + false_triggered: list[dict[str, Any]] = [] + + for task in positive: + adaptive = _adaptive_rows(grouped[task["task_id"]]) + positive_rows.extend(adaptive) + selected = [row for row in adaptive if _selected_in_subtree(topology, row, node)] + positive_triggered.extend(selected) + triggered_rows.extend(selected) + + for task in negative: + adaptive = _adaptive_rows(grouped[task["task_id"]]) + negative_rows.extend(adaptive) + selected = [row for row in adaptive if _selected_in_subtree(topology, row, node)] + false_triggered.extend(selected) + triggered_rows.extend(selected) + + trigger_recall = ( + len(positive_triggered) / len(positive_rows) if positive_rows else None + ) + boundary_specificity = ( + 1.0 - len(false_triggered) / len(negative_rows) if negative_rows else None + ) + compliance_when_triggered = ( + sum(row.get("passed") is True for row in triggered_rows) / len(triggered_rows) + if triggered_rows + else None + ) + + parent_positive_caps: list[dict[str, Any]] = [] + node_positive_caps: list[dict[str, Any]] = [] + for task in positive: + task_rows = grouped[task["task_id"]] + parent_positive_caps.extend(determinate(_cap_rows(task_rows, parent))) + node_positive_caps.extend(determinate(_cap_rows(task_rows, node))) + + def delta(key: str) -> float | None: + child = _mean(node_positive_caps, key) + base = _mean(parent_positive_caps, key) + return None if child is None or base is None else child - base + + positive_repositories = sorted({task["repository"] for task in positive}) + enough_signal = ( + len(positive) >= MIN_POSITIVE_TASKS + and len(positive_repositories) >= MIN_POSITIVE_REPOSITORIES + ) + route_ok = ( + trigger_recall is not None + and trigger_recall >= TRIGGER_RECALL_TARGET + and boundary_specificity is not None + and boundary_specificity >= BOUNDARY_SPECIFICITY_TARGET + and compliance_when_triggered is not None + and compliance_when_triggered >= COMPLIANCE_TARGET + ) + + return { + "node": node, + "parent": parent, + "depth": spec["depth"], + "positive_lift_tasks": [task["task_id"] for task in positive], + "positive_lift_task_count": len(positive), + "positive_lift_repositories": positive_repositories, + "negative_parent_sufficient_tasks": [task["task_id"] for task in negative], + "trigger": { + "opportunities": len(positive_rows), + "hits": len(positive_triggered), + "recall": trigger_recall, + "target": TRIGGER_RECALL_TARGET, + }, + "compliance": { + "triggered_cells": len(triggered_rows), + "passing_triggered_cells": sum(row.get("passed") is True for row in triggered_rows), + "rate": compliance_when_triggered, + "target": COMPLIANCE_TARGET, + }, + "boundary": { + "negative_opportunities": len(negative_rows), + "false_triggers": len(false_triggered), + "specificity": boundary_specificity, + "target": BOUNDARY_SPECIFICITY_TARGET, + }, + "positive_capability_cost_delta_vs_parent": { + "tokens_mean": delta("total_tokens"), + "duration_seconds_mean": delta("duration_seconds"), + "tool_calls_mean": delta("tool_calls"), + }, + "promotion_signal_gate": "PASS" if enough_signal and route_ok else "FAIL", + "promotion_signal_requirements": { + "minimum_positive_tasks": MIN_POSITIVE_TASKS, + "minimum_positive_repositories": MIN_POSITIVE_REPOSITORIES, + "note": ( + "This gate is necessary but not sufficient. Release promotion still " + "requires the main quality/non-inferiority gate and review of cost, " + "trace validity, leakage, and task realism." + ), + }, + } + + +def analyze(rows: list[dict[str, Any]], topology: dict[str, Any]) -> dict[str, Any]: + tasks = task_reports(rows, topology) + nodes = {} + for node, spec in topology["automatic_nodes"].items(): + if spec.get("parent") is None: + continue + nodes[node] = analyze_node(rows, topology, tasks, node) + return { + "schema_version": 1, + "method": "capability-derived-skill-use", + "nodes": nodes, + "notes": [ + "Trigger positives are derived from empirically minimum-sufficient child capability, not human gold labels.", + "Boundary negatives are tasks already stable-passing at the parent.", + "Compliance is delivered pass rate among adaptive cells that selected the node/subtree.", + "Do not promote from this report alone; use tree_analysis.py release quality and topology diagnostics too.", + ], + } + + +def self_test() -> None: + topology = { + "root": "core", + "automatic_nodes": { + "core": {"depth": 0, "parent": None, "children": ["impl"]}, + "impl": {"depth": 1, "parent": "core", "children": ["security"]}, + "security": {"depth": 2, "parent": "impl", "children": []}, + }, + } + + def row(task: str, variant: str, passed: bool, terminal: str | None = None) -> dict[str, Any]: + return { + "task_id": task, + "repository": "r1" if task == "positive" else "r2", + "family": "x", + "manual_request": None, + "variant": variant, + "passed": passed, + "selected_terminal_node": terminal, + "total_tokens": 100, + "duration_seconds": 1, + "tool_calls": 1, + } + + rows = [ + row("positive", "cap:core", False), + row("positive", "cap:impl", False), + row("positive", "cap:security", True), + row("positive", "adaptive", True, "security"), + row("negative", "cap:core", False), + row("negative", "cap:impl", True), + row("negative", "cap:security", True), + row("negative", "adaptive", True, "impl"), + ] + report = analyze(rows, topology) + security = report["nodes"]["security"] + assert security["trigger"]["recall"] == 1.0 + assert security["boundary"]["specificity"] == 1.0 + assert security["compliance"]["rate"] == 1.0 + print("tree skill-use analysis self-test: PASS") + + +def parse_args() -> argparse.Namespace: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("results", type=Path, nargs="?") + parser.add_argument("--topology", type=Path, default=Path(__file__).with_name("tree_topology.json")) + parser.add_argument("--output", type=Path) + parser.add_argument("--self-test", action="store_true") + return parser.parse_args() + + +def main() -> int: + args = parse_args() + if args.self_test: + self_test() + return 0 + if args.results is None: + raise SystemExit("results.jsonl is required unless --self-test is used") + rows = load_jsonl(args.results) + topology = load_topology(args.topology) + report = analyze(rows, topology) + text = json.dumps(report, ensure_ascii=False, indent=2) + "\n" + if args.output: + args.output.write_text(text, encoding="utf-8") + print(text, end="") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/benchmarks/tree_topology.json b/benchmarks/tree_topology.json new file mode 100644 index 0000000..91310b1 --- /dev/null +++ b/benchmarks/tree_topology.json @@ -0,0 +1,121 @@ +{ + "schema_version": 5, + "root": "core", + "automatic_nodes": { + "core": { + "depth": 0, + "reference": "SKILL.md", + "parent": null, + "children": [ + "debugging", + "implementation" + ] + }, + "debugging": { + "depth": 1, + "reference": "references/debugging.md", + "parent": "core", + "children": [] + }, + "implementation": { + "depth": 1, + "reference": "references/implementation.md", + "parent": "core", + "children": [] + } + }, + "manual_modes": { + "decision": "references/manual/decision.md", + "clarification": "references/manual/clarification.md" + }, + "retrieval_tree": { + "root": "retrieval", + "nodes": { + "retrieval": { + "depth": 0, + "stage": "root", + "trace_mode": "NONE", + "reference": "references/retrieval/SKILL.md", + "parent": null, + "children": [ + "direct" + ] + }, + "direct": { + "depth": 1, + "stage": "R0", + "trace_mode": "R0_DIRECT", + "reference": "references/retrieval/direct.md", + "parent": "retrieval", + "children": [ + "discovery" + ] + }, + "discovery": { + "depth": 2, + "stage": "R1", + "trace_mode": "R1_DISCOVERY", + "reference": "references/retrieval/discovery.md", + "parent": "direct", + "children": [ + "evidence" + ] + }, + "evidence": { + "depth": 3, + "stage": "R2", + "trace_mode": "R2_EVIDENCE", + "reference": "references/retrieval/evidence.md", + "parent": "discovery", + "children": [ + "structural" + ] + }, + "structural": { + "depth": 4, + "stage": "R3", + "trace_mode": "R3_STRUCTURAL", + "reference": "references/retrieval/structural.md", + "parent": "evidence", + "children": [] + } + } + }, + "retrieval_trace_modes": [ + "NONE", + "R0_DIRECT", + "R1_DISCOVERY", + "R2_EVIDENCE", + "R3_STRUCTURAL" + ], + "retrieval_modes": [ + "NONE", + "R0_DIRECT", + "R1_DISCOVERY", + "R2_EVIDENCE", + "R3_STRUCTURAL", + "TARGETED", + "BOUNDED", + "STRUCTURAL" + ], + "legacy_retrieval_modes": [ + "TARGETED", + "BOUNDED", + "STRUCTURAL" + ], + "capability_manifest": "benchmarks/capability_manifest.json", + "baseline_ref": "ba4058b4ef47a42bf79c9963b25678a2389897c1", + "notes": [ + "Execution depth describes disclosure depth, not task complexity.", + "Only parent-child edges in each tree are valid automatic routes.", + "Manual modes are never automatic descendants.", + "Debugging and Implementation remain execution leaves until benchmark evidence earns a child.", + "Retrieval is an independent progressive tree: root -> R0 direct -> R1 discovery -> R2 evidence -> R3 structural.", + "Retrieval depth describes the unresolved information problem, never provider strength.", + "Ranked retrieval, graph retrieval, and execution-output compaction are capability providers outside both trees.", + "Dependency-enabled comparisons require every provider and exclude setup, indexing, dependency resolution, and first-build warm-up from measured metrics.", + "Legacy retrieval labels remain parser-compatible for historical result files but are not emitted by the dependency-enabled runner.", + "The execution-state experiment is retired and is not an active runtime substrate.", + "The benchmark may recommend adding, removing, merging, promoting, splitting, or deepening nodes; the manifest is an experiment input, not a permanent taxonomy." + ] +} diff --git a/benchmarks/tree_validation.py b/benchmarks/tree_validation.py new file mode 100644 index 0000000..2876564 --- /dev/null +++ b/benchmarks/tree_validation.py @@ -0,0 +1,527 @@ +#!/usr/bin/env python3 +"""Run evolvable local-router-tree experiments on frozen real repositories. + +Unlike the legacy router benchmark, this runner does not assign an expected +automatic route to each task. It measures delivered quality under capability +ceilings for every node, then records the adaptive path for later topology +analysis. Manual activation is scored only from explicit user requests. +""" + +from __future__ import annotations + +import argparse +import concurrent.futures +import datetime as dt +import json +import os +import re +import shutil +import statistics +import sys +from pathlib import Path +from typing import Any + +HERE = Path(__file__).resolve().parent +ROOT = HERE.parent +if str(HERE) not in sys.path: + sys.path.insert(0, str(HERE)) + +import run_benchmarks as bench +from tree_cases import CASES, REPOSITORIES + + +VERSION = "1.0" +MODEL = bench.MODEL +REASONING = bench.REASONING +TRACE_RE = re.compile( + r"TREE_TRACE\s+path=([^\s]+)\s+retrieval=([A-Z_]+)\s+manual=([a-z_-]+)\s+refs=([^\r\n]+)", + re.I, +) +OBSERVED_REF_RE = re.compile( + r"practical-coding[/\\](references[/\\][a-z0-9_./\\-]+\.md)", + re.I, +) + + +def load_topology(path: Path) -> dict[str, Any]: + topology = json.loads(path.read_text(encoding="utf-8")) + nodes = topology.get("automatic_nodes") or {} + root = topology.get("root") + if root not in nodes: + raise ValueError("topology root must name an automatic node") + for name, node in nodes.items(): + parent = node.get("parent") + depth = node.get("depth") + children = node.get("children") + if not isinstance(depth, int) or depth < 0: + raise ValueError(f"invalid depth for {name}") + if not isinstance(children, list) or not all(child in nodes for child in children): + raise ValueError(f"invalid children for {name}") + if name == root: + if parent is not None or depth != 0: + raise ValueError("root must have parent=null and depth=0") + else: + if parent not in nodes: + raise ValueError(f"invalid parent for {name}") + if name not in nodes[parent].get("children", []): + raise ValueError(f"parent {parent} does not list child {name}") + if depth != nodes[parent]["depth"] + 1: + raise ValueError(f"depth of {name} must equal parent depth + 1") + return topology + + +def node_path(topology: dict[str, Any], node_name: str) -> list[str]: + nodes = topology["automatic_nodes"] + if node_name not in nodes: + raise ValueError(f"unknown node: {node_name}") + path: list[str] = [] + current: str | None = node_name + while current is not None: + path.append(current) + current = nodes[current]["parent"] + path.reverse() + if path[0] != topology["root"]: + raise ValueError(f"node {node_name} is disconnected from root") + return path + + +def parse_trace(answer: str) -> dict[str, Any]: + matches = list(TRACE_RE.finditer(answer)) + if not matches: + return {"path": [], "retrieval": None, "manual": None, "references_loaded": []} + match = matches[-1] + raw_path = match.group(1).strip().strip("<>") + path = [] if raw_path.lower() in {"none", "-"} else [part.strip().lower() for part in raw_path.split(">") if part.strip()] + refs_raw = match.group(4).strip().strip("<>") + refs = [] if refs_raw.lower() in {"none", "-"} else [part.strip().strip("<>") for part in refs_raw.split(",") if part.strip()] + return { + "path": path, + "retrieval": match.group(2).upper(), + "manual": match.group(3).lower(), + "references_loaded": refs, + } + + +def canonical_reference(raw: str) -> str: + ref = str(raw).strip().strip("<>\"'").lower().replace("\\", "/") + marker = "/practical-coding/" + if marker in ref: + ref = ref.split(marker, 1)[1] + if ref.startswith("manual/"): + ref = f"references/{ref}" + if "/" not in ref and ref.endswith(".md"): + ref = f"references/{ref}" + return ref + + +def allowed_references(topology: dict[str, Any]) -> set[str]: + automatic = { + canonical_reference(spec["reference"]) + for name, spec in topology["automatic_nodes"].items() + if name != topology["root"] + } + manual = {canonical_reference(ref) for ref in topology.get("manual_modes", {}).values()} + return automatic | manual | {"references/navigation.md"} + + +def infer_trace_from_commands(topology: dict[str, Any], commands: list[str]) -> dict[str, Any]: + command_text = "\n".join(commands).replace("\\", "/") + refs = sorted({canonical_reference(match.group(1)) for match in OBSERVED_REF_RE.finditer(command_text)}) + nodes = topology["automatic_nodes"] + loaded_nodes = [ + name + for name, spec in nodes.items() + if name != topology["root"] and canonical_reference(spec["reference"]) in refs + ] + paths = [node_path(topology, name) for name in loaded_nodes] + path = max(paths, key=len) if paths and all( + candidate == paths[0][: len(candidate)] or paths[0] == candidate[: len(paths[0])] + for candidate in paths + ) else [topology["root"]] + loaded_manual = [ + name + for name, ref in topology.get("manual_modes", {}).items() + if canonical_reference(ref) in refs + ] + manual = loaded_manual[0] if len(loaded_manual) == 1 else "none" + return { + "path": path, + "retrieval": "TARGETED" if commands else "NONE", + "manual": manual, + "references_loaded": refs, + } + + +def validate_automatic_path(topology: dict[str, Any], path: list[str]) -> bool: + if not path or path[0] != topology["root"]: + return False + nodes = topology["automatic_nodes"] + if any(name not in nodes for name in path): + return False + return all(child in nodes[parent]["children"] for parent, child in zip(path, path[1:])) + + +def validate_trace(topology: dict[str, Any], trace: dict[str, Any]) -> bool: + retrieval_ok = trace["retrieval"] in set(topology.get("retrieval_modes", [])) + manual_ok = trace["manual"] == "none" or trace["manual"] in topology.get("manual_modes", {}) + refs_ok = all(canonical_reference(ref) in allowed_references(topology) for ref in trace["references_loaded"]) + return retrieval_ok and manual_ok and refs_ok and validate_automatic_path(topology, trace["path"]) + + +def score_answer( + case: dict[str, Any], + answer: str, + commands: list[str], + workspace: Path, + *, + trace: dict[str, Any] | None, + enforce_runtime_contract: bool, +) -> dict[str, Any]: + lower = answer.lower() + missing = [group for group in case["required"] if not any(term.lower() in lower for term in group)] + command_text = "\n".join(commands).lower() + normalized_command_text = command_text.replace("\\", "/") + probe_groups = [group if isinstance(group, list) else [group] for group in case["probe_terms"]] + probe_missing = [group for group in probe_groups if not any(term.lower() in command_text for term in group)] + status = bench.run_command(["git", "status", "--porcelain"], workspace) + clean = status.returncode == 0 and not status.stdout.strip() + + requested_manual = case.get("manual_request") + manual_contract_ok = True + spontaneous_manual = False + if enforce_runtime_contract: + selected_manual = (trace or {}).get("manual") + refs = [str(ref).lower().replace("\\", "/") for ref in (trace or {}).get("references_loaded", [])] + manual_ref_loaded = any("manual/" in ref for ref in refs) or "references/manual/" in normalized_command_text + if requested_manual: + requested_suffix = f"manual/{requested_manual}.md" + manual_contract_ok = selected_manual == requested_manual and ( + any(requested_suffix in ref for ref in refs) + or f"references/{requested_suffix}" in normalized_command_text + ) + else: + spontaneous_manual = selected_manual not in {None, "none"} or manual_ref_loaded + manual_contract_ok = not spontaneous_manual + + passed = not missing and not probe_missing and clean and manual_contract_ok + return { + "passed": passed, + "missing_evidence_groups": missing, + "missing_probe_terms": probe_missing, + "workspace_clean": clean, + "manual_contract_ok": manual_contract_ok, + "spontaneous_manual_mode": spontaneous_manual, + } + + +def resolve_repositories(repository_root: Path, overrides: list[str]) -> dict[str, Path]: + mapped = {name: (repository_root / data["local_name"]).resolve() for name, data in REPOSITORIES.items()} + for raw in overrides: + if "=" not in raw: + raise ValueError(f"repository override must be NAME=PATH: {raw}") + name, value = raw.split("=", 1) + if name not in REPOSITORIES: + raise ValueError(f"unknown repository override: {name}") + mapped[name] = Path(value).resolve() + for name, path in mapped.items(): + commit = REPOSITORIES[name]["commit"] + if not path.is_dir(): + raise FileNotFoundError(f"tree benchmark repository unavailable: {name}: {path}") + check = bench.run_command(["git", "cat-file", "-e", f"{commit}^{{commit}}"], path) + if check.returncode: + raise RuntimeError(f"{name} does not contain frozen commit {commit}: {check.stderr}") + return mapped + + +def prepare_workspace(source: Path, commit: str, workspace: Path) -> None: + clone = bench.run_command(["git", "clone", "-q", "--shared", "--no-checkout", str(source), str(workspace)], workspace.parent) + if clone.returncode: + raise RuntimeError(clone.stderr) + configure = bench.run_command(["git", "config", "core.longpaths", "true"], workspace) + if configure.returncode: + raise RuntimeError(configure.stderr) + checkout = bench.run_command(["git", "checkout", "-q", "--detach", commit], workspace) + if checkout.returncode: + raise RuntimeError(checkout.stderr) + + +def instrumentation(topology: dict[str, Any]) -> str: + nodes = ", ".join(sorted(topology["automatic_nodes"])) + manuals = ", ".join(sorted(topology.get("manual_modes", {}))) + retrieval = ", ".join(topology.get("retrieval_modes", [])) + return ( + "After the evidence-backed report, append exactly one final benchmark-only line: " + "TREE_TRACE path= retrieval= manual= refs=. " + f"Automatic node names are: {nodes}. A path starts at {topology['root']} and uses '>' between nodes; " + f"use path={topology['root']} when no automatic child was loaded. " + f"Retrieval mode must be one of: {retrieval}. Manual mode must be none or one of: {manuals}. " + "Manual modes are not path nodes. refs=none when no Practical Coding reference beyond SKILL.md was loaded. " + "Report behavior actually used; do not infer a preferred route from the task wording. Do not mention this instrumentation elsewhere." + ) + + +def ceiling_instruction(topology: dict[str, Any], node_name: str) -> str: + path = node_path(topology, node_name) + allowed_refs = [topology["automatic_nodes"][name]["reference"] for name in path if name != topology["root"]] + refs_text = ", ".join(allowed_refs) if allowed_refs else "none" + return ( + "\n" + f"This ablation permits automatic capabilities only on the path {' > '.join(path)}. " + f"Permitted non-root automatic references: {refs_text}. " + "Do not load siblings, descendants beyond the ceiling, or any manual mode. " + "This is an availability ceiling, not a claim that the ceiling node is the correct route. " + "If Core can solve the task, stay at Core; otherwise do the best possible work within the available path.\n" + "" + ) + + +def task_prompt(case: dict[str, Any], loaded: str, variant: str, topology: dict[str, Any]) -> str: + suffix = "" + if variant.startswith("cap:"): + suffix += "\n\n" + ceiling_instruction(topology, variant.split(":", 1)[1]) + if variant == "adaptive" or variant.startswith("cap:"): + suffix += "\n\n" + instrumentation(topology) + return ( + f"Frozen tree-benchmark task {case['task_id']} ({case['family']}).\n\n{case['prompt']}\n\n" + "Use PowerShell-compatible commands. Stay within this repository and preserve a clean working tree. " + "Cite concrete source paths/symbols and fresh command evidence when the task needs repository evidence.\n\n" + f"{variant}\n{loaded}{suffix}" + ) + + +def build_specs(topology: dict[str, Any], runs: int, *, current_only: bool, selected_cases: set[str]) -> list[tuple[str, str, int]]: + specs: list[tuple[str, str, int]] = [] + cap_nodes = list(topology["automatic_nodes"]) + for case in CASES: + if selected_cases and case["task_id"] not in selected_cases: + continue + if case.get("manual_request"): + variants = ["adaptive"] if current_only else ["no-skill", "baseline", "adaptive"] + else: + variants = ["adaptive", *(f"cap:{node}" for node in cap_nodes)] if current_only else [ + "no-skill", + "baseline", + "adaptive", + *(f"cap:{node}" for node in cap_nodes), + ] + for variant in variants: + for repetition in range(1, runs + 1): + specs.append((case["task_id"], variant, repetition)) + return specs + + +def run_cell( + spec: tuple[str, str, int], + args: argparse.Namespace, + topology: dict[str, Any], + repositories: dict[str, Path], + baseline: Path | None, + eval_home: Path, + output: Path, +) -> dict[str, Any]: + task_id, variant, repetition = spec + case = next(item for item in CASES if item["task_id"] == task_id) + safe_variant = variant.replace(":", "-") + cell = output / "cells" / task_id / safe_variant / f"r{repetition:03d}" + result_path = cell / "result.json" + if result_path.is_file(): + return json.loads(result_path.read_text(encoding="utf-8")) + cell.mkdir(parents=True, exist_ok=True) + workspace = cell / "workspace" + if workspace.exists(): + shutil.rmtree(workspace) + prepare_workspace(repositories[case["repository"]], REPOSITORIES[case["repository"]]["commit"], workspace) + + if variant == "no-skill": + loaded = "" + elif variant == "baseline": + if baseline is None: + raise RuntimeError("baseline Skill is unavailable") + loaded = bench.skill_text("practical-previous", {}, baseline) + else: + loaded = bench.skill_text("practical-current", {}, None) + + prompt = task_prompt(case, loaded, variant, topology) + (cell / "prompt.txt").write_text(prompt, encoding="utf-8") + env = os.environ.copy() + env["CODEX_HOME"] = str(eval_home) + codex = bench.resolve_codex(args.codex) + stdout = cell / "round1.jsonl" + stderr = cell / "round1.stderr.txt" + code, timed_out, forced, duration = bench.run_codex( + bench.codex_command(codex, workspace), prompt, workspace, env, stdout, stderr, args.timeout + ) + parsed = bench.parse_transcript(stdout) + current_runtime = variant == "adaptive" or variant.startswith("cap:") + trace = parse_trace(parsed["answer"]) if current_runtime else None + trace_source = "reported" if trace and trace.get("path") else None + if current_runtime and trace and not trace.get("path"): + trace = infer_trace_from_commands(topology, parsed["tool_commands"]) + trace_source = "observed-commands" + trace_valid = validate_trace(topology, trace) if current_runtime and trace is not None else None + terminal_node = trace["path"][-1] if trace and trace.get("path") else None + + record: dict[str, Any] = { + "schema_version": VERSION, + "task_id": task_id, + "repository": case["repository"], + "family": case["family"], + "manual_request": case.get("manual_request"), + "variant": variant, + "repetition": repetition, + "exit_status": code, + "timed_out": timed_out, + "forced_after_completion": forced, + "duration_seconds": duration, + "tool_calls": parsed["tool_calls"], + **parsed["usage"], + "answer": parsed["answer"], + "tool_commands": parsed["tool_commands"], + "selected_path": trace["path"] if trace else None, + "selected_terminal_node": terminal_node, + "selected_depth": topology["automatic_nodes"].get(terminal_node, {}).get("depth") if terminal_node else None, + "selected_retrieval": trace["retrieval"] if trace else None, + "selected_manual": trace["manual"] if trace else None, + "references_loaded": trace["references_loaded"] if trace else [], + "routing_trace_valid": trace_valid, + "routing_trace_source": trace_source, + } + infrastructure_error = "timeout" if timed_out else (f"codex exit status {code}" if code and not forced else None) + if infrastructure_error: + record.update({"passed": None, "verdict": "indeterminate", "error": infrastructure_error}) + else: + record.update( + score_answer( + case, + parsed["answer"], + parsed["tool_commands"], + workspace, + trace=trace, + enforce_runtime_contract=current_runtime, + ) + ) + if current_runtime and not trace_valid: + record["passed"] = False + record["routing_trace_error"] = True + record["verdict"] = "pass" if record["passed"] else "fail" + (cell / "answer.md").write_text(parsed["answer"] + "\n", encoding="utf-8") + result_path.write_text(json.dumps(record, ensure_ascii=False, indent=2) + "\n", encoding="utf-8") + return record + + +def _mean(records: list[dict[str, Any]], key: str) -> float | None: + values = [float(record[key]) for record in records if record.get(key) is not None] + return statistics.mean(values) if values else None + + +def summary(records: list[dict[str, Any]], runs: int) -> dict[str, Any]: + arms: dict[str, Any] = {} + for variant in sorted({record["variant"] for record in records}): + selected = [record for record in records if record["variant"] == variant] + determinate = [record for record in selected if record.get("passed") is not None] + arms[variant] = { + "cells": len(selected), + "determinate": len(determinate), + "pass_rate": sum(record["passed"] is True for record in determinate) / len(determinate) if determinate else None, + "tokens_mean": _mean(determinate, "total_tokens"), + "duration_seconds_mean": _mean(determinate, "duration_seconds"), + "tool_calls_mean": _mean(determinate, "tool_calls"), + } + adaptive = [record for record in records if record["variant"] == "adaptive" and record.get("passed") is not None] + automatic = [record for record in adaptive if not record.get("manual_request")] + manual = [record for record in adaptive if record.get("manual_request")] + return { + "runs_per_cell": runs, + "tasks": len({record["task_id"] for record in records}), + "repositories": sorted({record["repository"] for record in records}), + "arms": arms, + "adaptive_trace_valid_rate": sum(record.get("routing_trace_valid") is True for record in adaptive) / len(adaptive) if adaptive else None, + "adaptive_spontaneous_manual_count": sum(record.get("spontaneous_manual_mode") is True for record in automatic), + "adaptive_spontaneous_manual_rate": sum(record.get("spontaneous_manual_mode") is True for record in automatic) / len(automatic) if automatic else None, + "adaptive_explicit_manual_success_rate": sum(record.get("manual_contract_ok") is True for record in manual) / len(manual) if manual else None, + } + + +def parse_args() -> argparse.Namespace: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--runs", type=int, default=3) + parser.add_argument("--workers", type=int, default=3) + parser.add_argument("--output", type=Path) + parser.add_argument("--repository-root", type=Path, default=ROOT.parent) + parser.add_argument("--repository", action="append", default=[], help="override a source as NAME=PATH") + parser.add_argument("--topology", type=Path, default=HERE / "tree_topology.json") + parser.add_argument("--baseline-ref") + parser.add_argument("--codex", default=os.environ.get("CODEX_BIN", "codex")) + parser.add_argument("--timeout", type=float, default=600) + parser.add_argument("--case", action="append", default=[]) + parser.add_argument("--current-only", action="store_true") + parser.add_argument("--self-test", action="store_true") + return parser.parse_args() + + +def self_test(topology: dict[str, Any]) -> None: + assert len(CASES) >= 12 + assert set(REPOSITORIES) == {case["repository"] for case in CASES} + for name in topology["automatic_nodes"]: + assert node_path(topology, name)[-1] == name + trace = parse_trace("TREE_TRACE path=core>debugging retrieval=BOUNDED manual=none refs=references/debugging.md") + assert validate_trace(topology, trace) + assert not validate_automatic_path(topology, ["core", "debugging", "implementation"]) + assert any(case.get("manual_request") == "decision" for case in CASES) + print("tree validation self-test: PASS") + + +def main() -> int: + args = parse_args() + topology = load_topology(args.topology.resolve()) + if args.self_test: + self_test(topology) + return 0 + if args.runs < 1 or args.workers < 1: + raise SystemExit("runs and workers must be positive") + selected_cases = set(args.case) + unknown = selected_cases - {case["task_id"] for case in CASES} + if unknown: + raise SystemExit(f"unknown cases: {', '.join(sorted(unknown))}") + + repositories = resolve_repositories(args.repository_root.resolve(), args.repository) + stamp = dt.datetime.now().strftime("%Y%m%d-%H%M%S") + output = (args.output or ROOT / "benchmark-results" / f"tree-{stamp}").resolve() + output.mkdir(parents=True, exist_ok=True) + + baseline_ref = args.baseline_ref or topology.get("baseline_ref") + baseline_dir: Path | None = None + if not args.current_only: + if not baseline_ref: + raise RuntimeError("baseline_ref is required unless --current-only is used") + baseline_dir = output / "baseline-skill" + if not (baseline_dir / "SKILL.md").is_file(): + baseline_dir = bench.materialize_git_skill(str(baseline_ref), baseline_dir) + + eval_home = bench.prepare_eval_home(output / "eval-home") + specs = build_specs(topology, args.runs, current_only=args.current_only, selected_cases=selected_cases) + records: list[dict[str, Any]] = [] + with concurrent.futures.ThreadPoolExecutor(max_workers=args.workers) as pool: + futures = [pool.submit(run_cell, spec, args, topology, repositories, baseline_dir, eval_home, output) for spec in specs] + for future in concurrent.futures.as_completed(futures): + records.append(future.result()) + + records.sort(key=lambda row: (row["task_id"], row["variant"], row["repetition"])) + rows_path = output / "results.jsonl" + rows_path.write_text("".join(json.dumps(row, ensure_ascii=False) + "\n" for row in records), encoding="utf-8") + report = summary(records, args.runs) + report.update({ + "schema_version": VERSION, + "model": MODEL, + "reasoning": REASONING, + "topology": topology, + "baseline_ref": baseline_ref, + "results_jsonl": str(rows_path), + }) + (output / "report.json").write_text(json.dumps(report, ensure_ascii=False, indent=2) + "\n", encoding="utf-8") + print(json.dumps(report, ensure_ascii=False, indent=2)) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/docs/CAPABILITY_LAYER.md b/docs/CAPABILITY_LAYER.md new file mode 100644 index 0000000..a27fd54 --- /dev/null +++ b/docs/CAPABILITY_LAYER.md @@ -0,0 +1,65 @@ +# Capability Layer + +Practical Coding separates policy from providers: + +- the **execution tree** decides how much engineering reasoning is needed; +- the **Retrieval tree** decides which information problem remains unresolved; +- the **capability layer** supplies replaceable implementations; +- the **execution output layer** compacts noisy command results without changing semantics. + +A provider name must never become a Retrieval node. Runtime policy remains valid when a provider is replaced or unavailable. + +## Active dependency-enabled benchmark profile + +The model-backed dependency profile requires all three providers below. It pins accepted versions in `benchmarks/capability_manifest.json`; the runner fails before creating comparison cells when a binary is missing, its probe fails, or its observed version does not match the frozen profile. + +| Role | Required provider | Used by | Boundary | +|---|---|---|---| +| Ranked retrieval | `zg` from `@zvec/zvec-grep` 0.2.0 | R1 and bounded R2 discovery | Supplies hybrid semantic + lexical candidates; current source remains authoritative. | +| Structural retrieval | `codebase-memory-mcp` 0.10.8 | R3 | Supplies graph relationships; current source and index coverage must be checked. | +| Execution output compaction | `rtk` 0.47.0 | shell/test/build/Git transport | Compresses output while preserving command meaning, status, failures, and required evidence. | + +The executable contract is machine-readable in `benchmarks/capability_manifest.json`. + +## Runtime fallback versus benchmark requirements + +Normal Skill use must remain portable. When ranked or graph retrieval is absent, the current node falls back to bounded repository-native search. Output compaction may also be absent without changing task semantics. + +The dependency-enabled benchmark intentionally does **not** exercise that absence path. It asks whether the proposed tree benefits from concrete mature capabilities, so missing providers are an infrastructure failure rather than a fallback case. Fallback behavior is covered by deterministic contract tests and may receive a separate ablation; it is not mixed into provider-enabled cost comparisons. + +## Two-phase measurement contract + +Every model-backed comparison cell has two phases. + +### 1. Setup — recorded separately, never compared + +Before the model prompt, the runner: + +1. verifies every required executable; +2. runs provider probes; +3. initializes local embedding/model assets when needed; +4. builds the workspace `zg` index and runs one unmeasured query to warm the query path; +5. builds the Codebase Memory graph and warms its daemon/CLI path; +6. warms declared repository dependencies and first-build/test caches; +7. verifies the workspace is still clean. + +Setup commands, output bytes, and elapsed time are written to `capability-setup.json`. They are marked `included_in_comparison: false`. No setup token estimate is produced, and setup work occurs before Codex is started, so it cannot enter transcript token, tool-call, or measured wall-time fields. + +### 2. Measured execution — compared + +Only after setup succeeds does the runner start Codex and collect: + +- input, cached-input, output, reasoning-output, and total tokens; +- model-visible tool calls; +- measured wall time; +- answer quality and routing trace. + +Every arm for the same task receives the same preinitialized provider note and the same repository warm-up contract. A baseline may choose not to use a provider, but it does not receive a colder environment. + +## Isolation + +Codebase Memory owns an account-level daemon, so concurrent cells must share one cache cohort. By default the runner inherits the host's existing `CBM_CACHE_DIR` (or the provider default); an operator may set `PRACTICAL_BENCHMARK_CBM_CACHE_DIR` once for the whole run. The selected cohort is recorded in every setup receipt. Each frozen workspace has a distinct absolute path and is indexed before measurement. Workspace-local zvec indexes are excluded through `.git/info/exclude`, never committed to the frozen repository, and checked after setup with `git status --porcelain`. + +RTK remains outside both trees. On hosts with hard command hooks the execution adapter can be transparent. Codex currently receives the same thin provider instruction in every arm, because RTK's Codex integration is rules-file based; provider usage is recorded rather than inferred from the selected Retrieval stage. + +A provider setup failure aborts the run. The runner must not silently continue with a different capability surface, because that would invalidate paired cost comparison. diff --git a/evolution/ITERATION_PROMPTS_ZH.md b/evolution/ITERATION_PROMPTS_ZH.md new file mode 100644 index 0000000..df8fd7c --- /dev/null +++ b/evolution/ITERATION_PROMPTS_ZH.md @@ -0,0 +1,71 @@ +# Practical Coding 演化迭代提示词 + +本文件用于显式启动维护态迭代。普通编码任务不得自动读取 `evolution/`，也不得因为一次任务表现不佳就直接修改运行时 Skill。 + +推荐把“沉淀经验”和“修改 Skill”拆成两次独立请求：先将当前会话写入 wiki，再在后续请求中基于累计证据做一次原子迭代。这样可以减少单次会话对候选方案和评测标准的同时污染。 + +## 1. 单次原子迭代：推荐主提示词 + +```text +你现在维护仓库 Hubujiu/practical-coding 的 experiment/evolvable-router-tree 分支。 + +显式启用 evolution/skills/evolve-skill/SKILL.md，只执行一次可归因的 Skill 演化迭代。目标不是增加更多流程或节点，而是在不降低交付质量的前提下，用已有证据验证一个最小候选修改；没有足够证据时返回 no_action。 + +执行要求： + +1. 先读取并遵守 AGENTS.md、SKILL.md、evolution/skills/evolve-skill/SKILL.md。 +2. 读取 evolution/wiki/index.md、evolution/wiki/skill-impact.md，以及与本次机制直接相关的少量 wiki、receipt、experiment 和 benchmark 结果。不要把整个 evolution/ 塞进上下文。 +3. 读取 benchmarks/tree_topology.json、benchmarks/TREE_EVOLUTION.md 和当前相关 benchmark 契约。 +4. 记录当前分支 HEAD、工作树状态、模型、reasoning 配置、harness、case 集、scorer 版本和重复次数。只在 experiment/evolvable-router-tree 上工作，不合并 PR。 +5. 从累计证据中只选择一个原子假设；目标只能是一个节点、一个父子边界、一个检索边界或一个评测缺陷。多个相互独立的问题必须拆成后续迭代。 +6. 在看到候选结果之前，先在 evolution/experiments/ 写下冻结假设：证据指针、因果机制、可观察的预加载/激活信号、准确目标、候选补丁形状、预期收益、明确反证条件、baseline ref、评测方案和接受门槛。 +7. 在修改运行时 Skill 之前冻结或新增 benchmark。新增能力至少要有正例和边界/负例；case 不得写入“正确自动路由节点”，scorer 不得奖励候选措辞，也不得在看到候选结果后偷偷改变门槛。 +8. 先在冻结 benchmark 上运行 baseline n=1 并保存完整 artifact；再应用最小候选补丁，并使用完全相同的模型、工具、case、scorer、超时和环境运行 candidate n=1。 +9. 同时运行所有相关确定性测试和现有回归。确定性测试通过只能证明结构或解析契约成立，不能代替真实模型任务质量。 +10. 若 n=1 暴露候选缺陷，可以回到一个新的假设重新开始；不得连续修补同一候选直到迎合 held-out。只有冻结候选才运行配对 n>=3 的 baseline/candidate/no-skill 发布矩阵。 +11. 接受顺序必须是：交付正确性、安全性、兼容性、可达性和必要检查不下降；自动路径仍是有效父子路径；Decision 与 Clarification 仍为零自发触发；新增 benchmark 不下降；所有必须门禁可判定。只有质量打平后才比较输入 token、时长、工具调用和平均加载深度。 +12. 若任一必要质量门禁下降或证据不完整，回滚运行时候选。保留冻结 benchmark、原始 artifact 和机制知识，并将拒绝原因写入 evolution/rejected/ 与 evolution/wiki/skill-impact.md。 +13. 若候选通过门禁，更新 evolution/wiki/skill-impact.md、evolution/wiki/log.md 和相关机制状态，再提交一个边界清晰的 commit。 +14. 最终必须报告：假设文件、benchmark 变化、baseline 结果、candidate 结果、配对重复次数、接受/拒绝决定、是否已回滚、最终 commit SHA、仍缺少的证据。不得把 pending 描述成 accepted。 + +结构不变量： + +- Core 只拥有直接子节点；加载节点只知道自己的直接子节点，不允许 Core 跨级选择后代。 +- Decision 和 Clarification 永远只能由当前用户显式请求触发，不得学习为自动 fallback。 +- Retrieval 与执行树正交；不要为了让路由更好看而扩大检索。 +- 不保留对称层级、固定深度或历史节点名；add/split/merge/promote/collapse/remove 都必须由质量合格后的净收益决定。 +- 不得用 token 节省补偿正确性或安全性下降。 +- 已记录在 evolution/rejected/ 的方案不得在没有新独立证据直接解决其失败机制时复活。 +``` + +## 2. 先把当前会话沉淀到 wiki + +这一步只提取证据，不修改 `SKILL.md`、自动 Router、参考模块或运行时代码。 + +```text +在 Hubujiu/practical-coding 的 experiment/evolvable-router-tree 分支上，显式启用 evolution/skills/session-to-wiki/SKILL.md。 + +只使用当前可见会话与可验证工具结果，把具有复用价值的成功机制、失败机制、路由边界、benchmark 缺陷或用户纠正沉淀到 evolution/。先按照 evolution/EXPERIENCE_SCHEMA.md 创建一份新的、经过脱敏的不可变 receipt，再读取少量相关 wiki 页面进行因果合并，最后更新 wiki/index.md 与 wiki/log.md。 + +不得复制完整聊天，不得保存私有推理，不得写入秘密、个人标识或私有代码，不得为了形成结论改写旧 receipt，不得修改任何运行时 Skill/Router 文件。结束时报告 receipt 路径、wiki 变化、支持与反例证据，以及是否已形成足以进入 evolve-skill 的候选假设。 +``` + +## 3. 连续迭代，但必须逐轮门禁 + +不要使用“持续优化直到最好”这类无边界提示词。它会诱导模型连续堆叠未经验证的修改、反复查看 held-out 失败并调整 scorer。 + +需要连续探索时，使用下面的有界版本： + +```text +按照 evolution/skills/evolve-skill/SKILL.md 最多执行 3 次独立迭代。 + +每一轮都必须完整经历：独立冻结假设 → 冻结 benchmark → baseline → 单一候选 → 相同证据 candidate → 回归门禁 → 接受并提交，或拒绝并回滚。上一轮没有完成“接受并提交”或“拒绝、回滚并记录”的闭环，不得开始下一轮；被拒绝后若继续，下一轮必须使用新的独立假设，不能继续修补同一候选。不同轮次不得共享未冻结的候选结果。出现 no_action、基础设施结果不可判定、预算耗尽或连续两轮拒绝时立即停止；任一轮出现必要质量门禁下降时，必须先回滚并记录，再由停止条件决定是否还能开启新的独立轮次。 + +最终按轮次列出假设、证据、结果、决定和 commit，不得把多轮修改压成一个无法归因的补丁。 +``` + +## 推荐调用顺序 + +1. 会话产生了可复用经验时，先单独运行“session-to-wiki”。 +2. 累计证据足够时，在新请求中运行“一次原子迭代”。 +3. 只有冻结候选通过完整配对门禁后，才把它称为已接受的 Skill 改进。 diff --git a/evolution/README.md b/evolution/README.md index 6aba27e..834b65e 100644 --- a/evolution/README.md +++ b/evolution/README.md @@ -10,35 +10,57 @@ The architecture separates three things that should not collapse into one prompt This follows the useful separation demonstrated by WikiSkill: experience should compound into durable maintenance knowledge, while candidate Skill changes still pass an explicit validation gate. +## Explicit maintenance skills + +`evolution/skills/` contains two user-triggered maintenance skills. They are not automatic runtime nodes and are intentionally absent from `benchmarks/tree_topology.json`. + +- `session-to-wiki` compiles the current visible session into a sanitized immutable receipt under `evolution/raw/`, then consolidates reusable mechanisms into the wiki. It must not edit runtime Skill files. +- `evolve-skill` reads the wiki and impact history, freezes one atomic hypothesis and its benchmark before changing runtime Skill text, compares baseline and candidate on the same evidence, and rolls back any required quality regression or indeterminate gate. + +This keeps the paper-style Raw → Wiki → Skill separation operational without exposing maintenance history to ordinary inference. + ## Loop ```text -benchmarks/results + real-project receipts +benchmarks/results + evolution/raw receipts ↓ evolution/wiki ↓ frozen experiment hypothesis ↓ - candidate Skill/tree change + frozen/new benchmark + baseline + ↓ + atomic Skill candidate ↓ - no-skill + prior + depth/path validation + same-evidence validation + regressions ↙ ↘ accept reject ↓ ↓ runtime Skill evolution/rejected + ↓ ↓ + skill-impact + persistent wiki ``` A rejected patch disappears from runtime behavior, but the learned mechanism remains available to maintainers. +## Wiki control files + +- `wiki/index.md` — concise mechanism catalog; +- `wiki/log.md` — chronological evolution log; +- `wiki/skill-impact.md` — accepted/rejected intervention history; +- mechanism pages — causal claims, evidence, contradictions, triggers, and experiments. + ## Evidence rules - Do not create a global rule from one surprising task or one user correction. - Keep exact evidence pointers; do not copy large raw transcripts into the wiki. - Separate benchmark evidence, held-out evidence, and real-project experience explicitly. - Record the hypothesis and proposed boundary/tree change before validation results are known. +- Freeze or add the benchmark before applying the candidate patch; baseline and candidate must use the same cases, scorer, model/harness, and repetition policy. - Prefer repeated mechanisms across independent repositories/tasks before promoting a pattern. - Treat expert-skill comparisons as family-specific evidence, not proof that Practical Coding should copy their whole workflow. - The optimization target is **quality-qualified net lift at the lowest useful depth/path**, not maximum process. +- A required correctness/safety regression cannot be traded for lower token or time cost. ## Promotion path @@ -51,11 +73,15 @@ repeated independent mechanism ↓ evolution/wiki entry ↓ -frozen experiment +frozen experiment + benchmark + ↓ +baseline run + ↓ +atomic candidate ↓ held-out + regression + baseline validation ↓ -Skill node / trigger / depth change +Skill node / trigger / depth change OR rollback ``` -Use `EXPERIENCE_SCHEMA.md` for receipts, `wiki/` for consolidated knowledge, `experiments/` for frozen hypotheses, and `rejected/` for failed changes. Existing `patterns/` remains valid historical evidence; new work should prefer the wiki layer so mechanisms can be linked across experiments rather than duplicated. +Use `EXPERIENCE_SCHEMA.md` for receipts, `raw/` for immutable sanitized experience, `wiki/` for consolidated knowledge, `experiments/` for frozen hypotheses, and `rejected/` for failed changes. Existing `patterns/` remains valid historical evidence; new work should prefer the wiki layer so mechanisms can be linked across experiments rather than duplicated. diff --git a/evolution/experiments/evolvable-local-router-tree.md b/evolution/experiments/evolvable-local-router-tree.md new file mode 100644 index 0000000..e1655d6 --- /dev/null +++ b/evolution/experiments/evolvable-local-router-tree.md @@ -0,0 +1,108 @@ +# EXP-20260901 — Evolvable local router tree + +Status: **isolated leaf candidate n=1 qualified; paired n=3 pending** + +## Observation + +The accepted v1.5 flat Event Router restored delivered quality after the rejected fixed E0-E3/R0-R3 capability-tree experiment, but it also restored two assumptions that are not established by that evidence: + +1. Decision is again an automatic route, even though a technical choice can appear repeatedly during execution and reopen deliberation after another route has already started. +2. Core owns the whole automatic reasoning taxonomy, so progressive disclosure applies to reference content but not to routing knowledge itself. + +The rejected progressive-tree result showed that its predefined numeric levels and specialist leaves did not earn stable lift. It did **not** establish that all tree-shaped progressive disclosure is harmful. + +## Hypothesis + +A local router tree can preserve progressive disclosure without the failed numeric taxonomy if: + +- Core is only the root and knows immediate children; +- every loaded node owns only its own immediate children and current disclosure depth; +- nodes may be leaves and branches may have unequal depth; +- automatic routing only deepens execution and never reopens Decision; +- Decision and Clarification are explicit-only manual modes; +- benchmark ablation derives minimum-sufficient nodes instead of scoring against a predefined automatic route; +- repeated benchmark evidence is allowed to change the topology itself. + +## Candidate runtime + +Initial seed: + +```text +Automatic +Core (0) +├── Debugging (1, leaf) +└── Implementation (1, leaf) + +Manual only +├── Decision +└── Clarification + +Retrieval +└── orthogonal capability expansion +``` + +Changes: + +- remove Decision from the automatic root router; +- move Decision to `references/manual/decision.md`; +- prohibit automatic nodes from routing to manual modes; +- add a convergence rule for technical choices discovered during execution; +- make depth explicit metadata in the node prose, where depth means disclosure depth only; +- add a Local Router section to Debugging and Implementation; both start as leaves; +- state that future descendants are owned by their parent, not Core. + +## Benchmark redesign + +The active tree benchmark is intentionally separate from the legacy flat-router and fixed-level scorers. + +`benchmarks/tree_topology.json` stores the candidate topology as data. + +`benchmarks/tree_cases.py` contains frozen real-repository tasks but no `expected_reasoning`, E0-E3 level, or automatic `capability_path` oracle. It includes explicit manual Decision tasks and negative controls where execution encounters technical alternatives but should not open Decision automatically. + +`benchmarks/tree_validation.py` runs: + +- no-skill; +- frozen v1.5 baseline at `ba4058b4ef47a42bf79c9963b25678a2389897c1`; +- adaptive current tree; +- one capability ceiling for every automatic node path. + +`benchmarks/tree_analysis.py` derives minimum-sufficient node sets from stable passing ceilings. Adaptive route disagreement is diagnostic evidence about the topology, not a release failure by itself. Delivered quality, trace validity, zero spontaneous manual activation, and explicit-manual adherence remain gates. + +The analyzer can emit topology candidates such as: + +- remove a node with no marginal lift or minimum-sufficient cases; +- promote/collapse a nearly mandatory child; +- merge/move sibling boundaries with repeated co-minimality; +- deepen/split a leaf with a repeated quality-failure cluster. + +## Frozen iteration protocol + +1. Run tree self-tests. +2. Run current-only n=1 on all tree cases. +3. Inspect scorer correctness before inspecting topology recommendations. +4. If failures form a repeated mechanism, freeze one topology mutation candidate before editing runtime wording. +5. Compare the mutation to its immediate parent topology with the same cases. +6. Only after runtime, topology, cases, and scorers freeze, run n=3 with v1.5 and no-skill arms. +7. Preserve raw outputs and do not rewrite v1.5 or rejected-tree historical evidence. + +## Acceptance questions + +This experiment is not accepted merely because the new Router looks cleaner. Evidence must answer: + +- Does manual-only Decision eliminate spontaneous deliberation without reducing delivered quality? +- Does local routing reduce root context/control-state cost without increasing under-disclosure? +- Which seed nodes are actually minimum-sufficient for repeated task clusters? +- Does any child earn another depth level? +- Are Debugging and Implementation stable sibling boundaries, or should benchmark evidence merge/move/split them? + +## Result + +The normalized current-only run at `benchmark-results/tree-delivery-n1-normalized-20260901-125524` completed 106/106 determinate cells. Adaptive passed 15/15 tasks, all 15 traces were valid, both explicit manual Decision tasks passed, and no automatic task activated a manual mode. + +This qualifies the frozen candidate for the complete n=3 run; it is not yet a release or superiority claim. Capability minima varied across n=1 repetitions, so node removal/promotion decisions are deferred to repeated paired evidence against the frozen v1.5 and no-skill arms. + +The first complete paired n=3 artifact (`benchmark-results/tree-final-eca9a09-20260901`) failed the release gate and was retained as diagnostic evidence. It exposed remaining semantic-oracle gaps and showed that none of the four staged depth-2 nodes entered a minimum-sufficient set. After general oracle corrections, a fresh 106-cell n=1 passed completely, again with no depth-2 marginal lift. + +The candidate therefore returned to the original seed topology: Core with leaf Debugging and Implementation children, plus explicit-only manual Decision/Clarification. The first leaf run exposed one further recommendation-inflection oracle defect; after freezing and correcting it, `benchmark-results/tree-delivery-n1-leaves-inflection-20260901` completed 58/58 determinate cells with adaptive 15/15, all three capability ceilings 13/13, trace 15/15, explicit manual 2/2, and zero spontaneous manual activation. This leaf candidate is frozen for a new paired n=3; superiority remains pending. + +That paired rerun also failed and revealed two additional mechanisms: deterministic evidence identities and trace instrumentation needed command-observed normalization, while retired child documents still under `references/` remained discoverable despite their removal from the topology. After freezing those mechanisms, adding positive/negative harness tests, and removing the four retired documents from the runtime reference surface, `benchmark-results/tree-delivery-n1-retired-isolated-20260902` completed 58/58 with every arm/capability cell passing and all trace/manual gates clean. This isolated leaf candidate is the current frozen n=3 candidate; no superiority claim exists until the new paired report completes. diff --git a/evolution/experiments/tree-bounded-evidence-volume-20260902.md b/evolution/experiments/tree-bounded-evidence-volume-20260902.md new file mode 100644 index 0000000..b56bc93 --- /dev/null +++ b/evolution/experiments/tree-bounded-evidence-volume-20260902.md @@ -0,0 +1,29 @@ +# EXP-20260902 — Bound evidence volume after discovery + +Status: **accepted for release quality; cost hypothesis not confirmed** + +## Observation + +The isolated leaf candidate's complete paired n=3 reached 45/45 quality against frozen v1.5 at 45/45 and reduced mean tool calls from 8.27 to 7.60, but increased mean total tokens from 211,758.69 to 242,910.84 and duration from 72.94s to 75.60s. The later Core-only collapse regressed every cost metric. + +## Hypothesis + +The topology is not the remaining cost problem. Once discovery identifies candidate paths or symbols, explicitly stopping broad inventory and using bounded reads should reduce tool-output/context volume without changing evidence quality, routing, or verification scope. + +## Change boundary + +Add only a general retrieval-volume rule to Core/Retrieval Policy: stop discovery after candidates are known, read relevant symbols or bounded ranges, avoid whole-file/repeated inventory, and batch independent bounded reads only while output remains focused. Do not change cases, scorer, topology, nodes, or repositories. + +## Acceptance + +Deterministic tests must pass, followed by a fresh complete current-only n=1 with every quality/trace/manual cell passing. Only that frozen candidate may run paired n=3. Accept delivery only if quality remains at least equal and the comparable cost report shows a genuine net improvement rather than relying on repetition variance. + +## n=1 qualification + +`benchmark-results/tree-bounded-evidence-n1-20260902` completed 58/58 determinate cells. Adaptive passed 15/15; Core, Debugging, and Implementation capability ceilings each passed 13/13; adaptive trace/manual discipline was perfect. The runtime/scorer/cases are frozen for paired n=3. + +## Paired n=3 result + +`benchmark-results/tree-final-b202f7a-20260902` completed 252/252 determinate cells. Adaptive passed 45/45, while frozen v1.5 and no-skill each passed 44/45. Trace and manual-mode gates were perfect and the release-quality gate passed. + +The bounded-volume cost hypothesis was not confirmed: adaptive mean tokens/duration/tools were 258,061.64 / 76.82s / 8.42 versus v1.5 at 217,460.96 / 72.20s / 7.24. The candidate is accepted on the separately frozen primary delivery criterion of strict paired quality superiority, not as evidence that the wording reduced cost. Future cost work must start from this regression and return to n=1. diff --git a/evolution/experiments/tree-collapse-automatic-leaves-20260902.md b/evolution/experiments/tree-collapse-automatic-leaves-20260902.md new file mode 100644 index 0000000..2509301 --- /dev/null +++ b/evolution/experiments/tree-collapse-automatic-leaves-20260902.md @@ -0,0 +1,19 @@ +# EXP-20260902 — Collapse automatic leaves into Core + +Status: **rejected and reverted** + +## Observation + +The isolated leaf candidate reached the Core capability ceiling on every task. This raised the topology question of whether Debugging and Implementation added measurable value beyond Core. + +## Hypothesis + +Removing both automatic leaves should preserve quality and reduce runtime context or tool cost if Core already owns all minimum-sufficient behavior. + +## Acceptance + +Qualify the Core-only topology at complete current-only n=1. Only then run a complete paired n=3. Accept only if quality remains non-inferior and recorded cost improves against frozen v1.5. + +## Result + +The n=1 candidate qualified, but paired n=3 tied quality at 45/45 while regressing mean tokens, duration, and tool calls. Commit `55453299a1cd21774f453d4ccb9733f4c1f50e84` reverted the collapse. diff --git a/evolution/experiments/tree-evidence-identity-and-invalid-trace-20260902.md b/evolution/experiments/tree-evidence-identity-and-invalid-trace-20260902.md new file mode 100644 index 0000000..26dd7f0 --- /dev/null +++ b/evolution/experiments/tree-evidence-identity-and-invalid-trace-20260902.md @@ -0,0 +1,26 @@ +# EXP-20260902 — Evidence identity and invalid-trace robustness + +Status: **frozen before scorer/analyzer edit** + +## Observation + +The complete paired n=3 artifact at `benchmark-results/tree-final-67f2f5c-20260901` produced all 252 cells but failed delivery. Three of four adaptive quality failures were contract-equivalent evidence forms: + +- `avifEncoder.ts` plus the full worker call boundary was rejected because only the exact function token `encodeAvif` was accepted; +- the authoritative `runCommand` transition and focused executor test were rejected because only the concrete class token `DefaultPluginOperationExecutor` was accepted; +- an explicit Decision run loaded the correct `references/manual/decision.md` command and declared `manual=decision`, but the trace shortened the identity to `manual/decision.md`. + +The fourth failure selected a retired node. Trace validation rejected it correctly, but `tree_analysis.py` crashed with `KeyError` instead of recording an invalid trace. + +## Hypothesis + +Score authoritative boundary identities and observed reference reads rather than a single spelling, while preserving independent evidence groups and applying the same observed-read rule to detect forbidden spontaneous manual loads. Treat any selected node absent from the active topology as `invalid_trace` in analysis. + +This iteration changes only scorer/analyzer contracts. Retired runtime-reference isolation is a separate subsequent hypothesis. + +## Acceptance + +- positive tests cover `avifEncoder`, `runCommand`, root-elided manual identity, and an observed correct manual-reference read; +- a negative test proves an automatic task that actually reads a manual reference is still rejected; +- analyzer test proves an unknown retired node produces `invalid_trace` without crashing; +- all deterministic gates pass, followed by a fresh complete current-only n=1 run. diff --git a/evolution/experiments/tree-manual-boundary-discriminator-20260902.md b/evolution/experiments/tree-manual-boundary-discriminator-20260902.md new file mode 100644 index 0000000..14fdcca --- /dev/null +++ b/evolution/experiments/tree-manual-boundary-discriminator-20260902.md @@ -0,0 +1,25 @@ +# EXP-20260902 — Manual-boundary discriminator + +Status: **rejected after paired n=1** + +## Observation + +The complete paired leaf artifact `benchmark-results/tree-final-ad2987c-20260902` put adaptive and frozen v1.5 at 45/45. Existing tasks did not exercise the architectural difference between v1.5 automatic Decision routing and the candidate's explicit-only Decision mode, so strict quality superiority was unmeasurable at the ceiling. + +## Hypothesis + +A real compatibility-boundary task that explicitly asks for only the minimum blocking question, while forbidding option comparison/recommendation/implementation planning, should distinguish the contracts: + +- current Skill asks one question in Core without loading manual Decision; +- v1.5 may automatically load `references/decision.md`, which itself requires recommendation/trade-off analysis; +- no-skill remains a neutral comparator. + +Score both visible decision-analysis leakage and observed v1.5 Decision-reference loading. Apply legacy manual-reference discipline symmetrically: explicit Decision tasks may load it; automatic tasks may not. + +## Acceptance + +Positive/negative deterministic tests must pass. Then run only the new case as a paired n=1 discriminator. Keep the case only if adaptive passes and the result exposes a real contract difference rather than a lexical accident. If retained, rerun the complete current-only suite at n=1 before any final n=3. + +## Result + +`benchmark-results/tree-discriminator-minimum-question-n1-20260902` completed 6/6 determinate cells. Adaptive, frozen v1.5, and no-skill all passed, and v1.5 did not load Decision. The case therefore did not distinguish the contracts. The case and its provisional scorer expansion were removed; the artifact and this rejection remain as evidence. diff --git a/evolution/experiments/tree-observed-trace-fallback-20260902.md b/evolution/experiments/tree-observed-trace-fallback-20260902.md new file mode 100644 index 0000000..c0884cf --- /dev/null +++ b/evolution/experiments/tree-observed-trace-fallback-20260902.md @@ -0,0 +1,24 @@ +# EXP-20260902 — Observed trace fallback + +Status: **frozen before harness edit** + +## Observation + +The complete n=1 artifact at `benchmark-results/tree-delivery-n1-evidence-identity-outcome-20260902` had all three capability ceilings at 13/13. Its sole adaptive failure was a complete, evidence-backed manual Decision answer that actually read `references/manual/decision.md` but omitted the benchmark-only `TREE_TRACE` footer. + +The footer is instrumentation, not delivered task behavior. A stochastic formatting omission should not erase observed reference-use evidence, but recovery must not permit inactive or retired references. + +## Hypothesis + +When and only when the reported trace is absent, derive a fallback trace from actual tool-command reference reads: + +- active automatic references determine the deepest valid local path; +- a uniquely observed manual reference determines manual mode; +- repository commands imply targeted retrieval; +- any observed Practical Coding reference outside the active automatic/manual/navigation surface invalidates the recovered trace. + +Explicit reported traces remain authoritative and are not rewritten. This makes instrumentation robust while preserving detection of the known retired-node leak. + +## Acceptance + +Unit tests must prove correct automatic/manual recovery and rejection of a retired reference. Then rerun the complete current-only n=1 matrix in a fresh directory. Retired file removal remains a separate runtime-isolation iteration. diff --git a/evolution/experiments/tree-oracle-alignment-20260901.md b/evolution/experiments/tree-oracle-alignment-20260901.md new file mode 100644 index 0000000..31dd140 --- /dev/null +++ b/evolution/experiments/tree-oracle-alignment-20260901.md @@ -0,0 +1,47 @@ +# EXP-20260901 — Tree benchmark oracle alignment + +Status: **frozen before scorer patch** + +## Evidence pointers + +- `benchmark-results/tree-delivery-n1-20260901-103036/results.jsonl` +- `benchmark-results/tree-delivery-n1-20260901-103036/analysis.json` +- frozen cover-atelier commit `fc3b12b3a944f45b5a1d19963e29307d95b120fb` +- `evolution/wiki/benchmark-oracle-contracts.md` + +## Causal claim + +The first tree n=1 run contains three false failures because the deterministic oracle is narrower than the frozen task contract: two manual Decision answers state the strongest trade-off without the exact punctuation token `Trade-off:`, and the cancellation diagnosis cites the repository's existing cancellation test plus the correct download-side-effect boundary while the oracle requires unrelated UI/progress test filenames. + +## Observable signal + +An answer satisfies the prompt's semantic evidence and manual-mode trace contract, but `missing_evidence_groups` contains only formatting punctuation or a non-authoritative test filename that the prompt never required. + +## Exact target and patch shape + +- Target only `benchmarks/tree_cases.py` and its deterministic tests. +- Accept the semantic `trade-off` term independently of punctuation. +- Accept the frozen repository's existing cancellation-focused `avifEncoder.test.ts` as a valid focused-test evidence path alongside the existing progress tests. +- Add regression assertions that representative semantically valid answers score successfully. +- Do not change `SKILL.md`, router references, topology, prompts, repositories, or model settings. + +## Expected benefit and falsifier + +Expected benefit: semantically compliant answers cease to fail for punctuation or an unjustified file-name oracle while truly missing recommendation, trade-off, cancellation-path, or test evidence still fails. + +Falsifier: the relaxed groups allow an answer without a real trade-off or without a concrete focused cancellation test, or any existing deterministic tree test regresses. + +## Baseline and benchmark plan + +- baseline ref: `31ba37c9c324ff5863ee237a8c89203f4405fbe9` +- invalidated exploratory artifact: `benchmark-results/tree-delivery-n1-20260901-103036` +- run focused deterministic scorer tests after the patch; +- then rerun the full frozen current-only tree matrix at n=1 in a fresh directory; +- accept the scorer patch only if all deterministic tests pass and the rerun has zero indeterminate cells with no new contract regression. + +## Acceptance criteria + +1. Representative colon-free strongest-trade-off answers pass the manual evidence group while manual trace enforcement remains unchanged. +2. A cancellation answer naming `avifEncoder.test.ts`, the download boundary, and a falsifying test passes; an answer with no test evidence still fails. +3. No runtime Skill/tree file changes. +4. The full n=1 rerun is completed before any runtime optimization decision. diff --git a/evolution/experiments/tree-oracle-outcome-semantics-20260901.md b/evolution/experiments/tree-oracle-outcome-semantics-20260901.md new file mode 100644 index 0000000..1b568fe --- /dev/null +++ b/evolution/experiments/tree-oracle-outcome-semantics-20260901.md @@ -0,0 +1,22 @@ +# EXP-20260901 — Tree oracle outcome semantics + +Status: **frozen before scorer edit** + +## Observation + +The complete paired n=3 run at `benchmark-results/tree-final-eca9a09-20260901` was determinate in all 408 cells, but five adaptive cells failed. All three explicit compatibility Decision answers made a choice and described its strongest downside, one focused test answer reported an exact blocked outcome, and one cancellation diagnosis identified the authoritative `exportCover` side-effect boundary without naming its UI caller. + +## Hypothesis + +The deterministic oracle should score the semantic acts required by the prompt rather than preferred headings, successful-only outcomes, or a neighboring caller that is not necessary to establish the boundary: + +- an explicit `Decision: choose ...` is a recommendation act; +- `cost` can state a trade-off when the competing benefit is also present; +- an exact blocked/failed test outcome is still an outcome report; +- an authoritative cancellation boundary does not require repeating `EditorShell` when `exportCover`, cancellation, focused evidence, and a falsifying test are all present. + +Independent groups still require the actual choice, downside, compared alternatives, cancellation mechanism, authoritative boundary, focused evidence, and concrete falsifying test. + +## Acceptance + +Add positive tests for each equivalent form and preserve negative tests for a missing trade-off and missing concrete test evidence. Rerun the full current-only matrix at n=1 in a fresh directory. Do not use this correction as paired comparison evidence. diff --git a/evolution/experiments/tree-oracle-recommend-inflection-20260901.md b/evolution/experiments/tree-oracle-recommend-inflection-20260901.md new file mode 100644 index 0000000..8f09774 --- /dev/null +++ b/evolution/experiments/tree-oracle-recommend-inflection-20260901.md @@ -0,0 +1,17 @@ +# EXP-20260901 — Recommendation inflection normalization + +Status: **frozen before scorer edit** + +## Observation + +The complete leaf-topology n=1 run at `benchmark-results/tree-delivery-n1-leaves-retry-20260901` was 58/58 determinate. All three capability ceilings passed every automatic task, all traces/manual contracts passed, and the sole adaptive failure began `Recommend a one-release compatibility alias` before comparing both options and stating the strongest trade-off. + +The recommendation evidence group accepted `Recommendation` but not the ordinary verb `Recommend`. + +## Hypothesis + +Recommendation evidence is an act, not a required part of speech. Adding the verb stem `recommend` preserves the independent requirements for both options, a chosen option, and its strongest downside while eliminating a lexical false negative. + +## Acceptance + +Add a positive verb-form test and retain the existing missing-trade-off negative. Then rerun the complete leaf-topology current-only matrix at n=1 in a fresh directory. diff --git a/evolution/experiments/tree-oracle-semantic-equivalence-20260901.md b/evolution/experiments/tree-oracle-semantic-equivalence-20260901.md new file mode 100644 index 0000000..e06f46b --- /dev/null +++ b/evolution/experiments/tree-oracle-semantic-equivalence-20260901.md @@ -0,0 +1,47 @@ +# EXP-20260901 — Tree oracle semantic equivalence + +Status: **frozen before second scorer patch** + +## Evidence pointers + +- `benchmark-results/tree-delivery-n1-oraclefix-20260901-111838/results.jsonl` +- `benchmark-results/tree-delivery-n1-oraclefix-20260901-111838/analysis.json` +- `evolution/experiments/tree-oracle-alignment-20260901.md` +- `evolution/wiki/benchmark-oracle-contracts.md` + +## Causal claim + +The first punctuation/file-list correction remained too lexical. The deterministic scorer still rejects semantically complete evidence when a model names the cancellation operation and observed test behavior without repeating an implementation type/file token, or answers a manual Decision request in Chinese using the equivalent labels “推荐” and “权衡”. + +## Observable signal + +`missing_evidence_groups` contains only a language-specific heading or an implementation identifier even though the answer contains the requested semantic act, concrete side-effect boundary, existing-test observation, falsifying test, valid manual trace, and clean workspace. + +## Exact target and patch shape + +- Target only evidence alternatives in `benchmarks/tree_cases.py` plus deterministic regression tests and wiki evidence. +- Accept Chinese recommendation/trade-off equivalents for manual tasks. +- Accept concrete cancellation-operation and existing/focused-test evidence without requiring one exact class or filename. +- Retain independent required groups for the controller/UI caller, `exportCover`, a falsifying probe/test, and both compared manual alternatives. +- Do not change prompts, runtime Skill files, topology, repositories, model, or trace enforcement. + +## Expected benefit and falsifier + +Expected benefit: language- and formatting-equivalent answers pass without weakening the requirement for a concrete diagnosis, test, recommendation, comparison, and strongest trade-off. + +Falsifier: an answer missing the cancellation operation, focused-test evidence, recommendation, or trade-off passes; or any deterministic contract test regresses. + +## Baseline and benchmark plan + +- baseline ref: current scorer patch working tree based on `31ba37c9c324ff5863ee237a8c89203f4405fbe9` +- invalidated artifact: `benchmark-results/tree-delivery-n1-oraclefix-20260901-111838` +- add positive semantic-equivalence tests and retain negative evidence tests; +- run all deterministic tree/evolution tests; +- rerun the full 15-task, 106-cell current-only n=1 matrix in a fresh directory. + +## Acceptance criteria + +1. The two observed semantically complete answers have no missing evidence groups under the corrected oracle. +2. Negative answers without a concrete test or trade-off still fail. +3. All deterministic checks pass and runtime Skill/tree files remain unchanged. +4. The new full n=1 artifact has zero indeterminate cells before any delivery/final-run decision. diff --git a/evolution/experiments/tree-remove-unearned-depth2-20260901.md b/evolution/experiments/tree-remove-unearned-depth2-20260901.md new file mode 100644 index 0000000..6f1816e --- /dev/null +++ b/evolution/experiments/tree-remove-unearned-depth2-20260901.md @@ -0,0 +1,22 @@ +# EXP-20260901 — Remove unearned depth-2 nodes + +Status: **frozen before topology/runtime edit** + +## Observation + +The complete paired n=3 artifact at `benchmark-results/tree-final-eca9a09-20260901` found no depth-2 staged node in any minimum-sufficient set. After the general oracle correction, the fresh current-only n=1 artifact at `benchmark-results/tree-delivery-n1-outcome-semantics-20260901` passed every one of 106 cells and again found zero marginal lift and zero minimum-sufficient tasks for all four staged depth-2 nodes. + +The paired artifact still showed `Debugging` or `Implementation` as the minimum-sufficient depth for `ca-avif-stall-evidence`, so the seed root children remain. Only their unearned staged descendants are in scope. + +## Hypothesis + +Removing `dynamic-evidence`, `security-boundary`, `migration-compatibility`, and `state-concurrency` from the active topology, and keeping Debugging/Implementation as leaves, will preserve delivered quality and manual-mode discipline while reducing unnecessary disclosure and capability cells. + +The specialist documents remain historical experiment artifacts in Git history; no sibling is merged into Core and no task-specific wording is added. + +## Acceptance + +- topology contains only Core, Debugging, and Implementation; +- Debugging and Implementation contain no active descendant routes; +- complete current-only n=1 is determinate and passes 15/15 adaptive tasks, all traces, both explicit manual tasks, and zero spontaneous manual activations; +- only after that candidate freezes may a fresh paired n=3 run be used for delivery comparison. diff --git a/evolution/experiments/tree-retired-reference-isolation-20260902.md b/evolution/experiments/tree-retired-reference-isolation-20260902.md new file mode 100644 index 0000000..090b45b --- /dev/null +++ b/evolution/experiments/tree-retired-reference-isolation-20260902.md @@ -0,0 +1,18 @@ +# EXP-20260902 — Retired reference isolation + +Status: **frozen before runtime-surface edit** + +## Observation + +The paired n=3 artifact `benchmark-results/tree-final-67f2f5c-20260901` and the n=1 artifact `benchmark-results/tree-delivery-n1-observed-trace-20260902` each contained an adaptive run that loaded a depth-2 reference removed from the active topology. Parent router text and the topology manifest both declared Debugging/Implementation leaves, but the retired files remained discoverable under the runtime `references/` directory. + +## Hypothesis + +A rejected automatic node must leave the runtime discovery surface, not merely the manifest. Remove the four unearned depth-2 documents from `references/`; preserve their content through Git history, experiments, raw receipts, and benchmark artifacts. Do not merge their specialist prose back into Core or the leaf parents. + +## Acceptance + +- no retired depth-2 document remains under runtime `references/`; +- topology and parent nodes still define Debugging/Implementation as leaves; +- deterministic tests prove unknown/retired reference observations are invalid; +- a fresh complete current-only n=1 has all cells determinate, adaptive 15/15, trace 15/15, manual 2/2, and zero spontaneous manual activation. diff --git a/evolution/experiments/tree-scorer-normalization-20260901.md b/evolution/experiments/tree-scorer-normalization-20260901.md new file mode 100644 index 0000000..3edb298 --- /dev/null +++ b/evolution/experiments/tree-scorer-normalization-20260901.md @@ -0,0 +1,44 @@ +# EXP-20260901 — Tree scorer normalization + +Status: **frozen before third scorer patch** + +## Evidence pointers + +- `benchmark-results/tree-delivery-n1-semantic-20260901-120715/results.jsonl` +- `benchmark-results/tree-delivery-n1-semantic-20260901-120715/analysis.json` +- `evolution/wiki/benchmark-oracle-contracts.md` + +## Causal claim + +Two scorer boundaries remain incorrectly platform- or identifier-specific: manual-reference enforcement compares only POSIX separators even though the Windows harness emits backslashes, and focused-test evidence requires repository filenames even when the answer reports the focused suite and its uncovered boundary. + +## Observable signal + +- A manual answer has no missing evidence groups, emits `manual=decision`, and loads the correct absolute Windows path, but `manual_contract_ok` is false. +- A diagnosis states focused-suite outcome and the missing falsifying test while only the filename evidence group remains missing. + +## Exact target and patch shape + +- Normalize reference paths to forward slashes before manual reference checks in `tree_validation.py`. +- Make the cancellation focused-test group require generic focused/suite/test evidence instead of a specific filename. +- Add deterministic positive and negative regression tests. +- Do not change runtime Skill files, topology, prompts, repositories, or model settings. + +## Expected benefit and falsifier + +Expected benefit: equivalent Windows/POSIX reference paths enforce the same manual contract, and focused-test evidence is scored by the requested act rather than one filename. + +Falsifier: a trace loading a non-manual reference passes, or a cancellation answer without any focused test/suite evidence passes. + +## Baseline and benchmark plan + +- baseline ref: current scorer working tree based on `31ba37c9c324ff5863ee237a8c89203f4405fbe9` +- invalidated artifact: `benchmark-results/tree-delivery-n1-semantic-20260901-120715` +- run all deterministic tree/evolution tests; +- rerun the complete 15-task, 106-cell current-only n=1 matrix in a fresh directory. + +## Acceptance criteria + +1. Absolute Windows and POSIX manual reference paths both satisfy the same requested manual contract. +2. A focused-suite diagnosis with a concrete falsifying test passes; a no-test diagnosis fails. +3. All deterministic checks pass, no runtime file changes, and the full rerun has zero indeterminate cells. diff --git a/evolution/raw/README.md b/evolution/raw/README.md new file mode 100644 index 0000000..4cc667a --- /dev/null +++ b/evolution/raw/README.md @@ -0,0 +1,8 @@ +# Raw evolution evidence + +This is the immutable evidence layer for maintenance-time skill evolution. + +- `sessions/` contains sanitized receipts distilled from explicitly requested real-project/session experience. +- Benchmark execution traces remain under their benchmark result artifacts and should be referenced rather than copied here. +- Never store secrets, private code, personal identifiers, full chat transcripts, or private chain-of-thought. +- Once a receipt is used for an evolution decision, do not rewrite it to match the later conclusion. Add a new receipt when evidence changes. diff --git a/evolution/raw/sessions/2026-09-01-tree-benchmark-delivery.md b/evolution/raw/sessions/2026-09-01-tree-benchmark-delivery.md new file mode 100644 index 0000000..3892ba5 --- /dev/null +++ b/evolution/raw/sessions/2026-09-01-tree-benchmark-delivery.md @@ -0,0 +1,40 @@ +# Experience receipt: tree benchmark delivery loop + +```yaml +id: exp-20260901-tree-benchmark-delivery +source_type: benchmark +source_pointer: benchmark-results/tree-delivery-*; current maintenance session transcript intentionally not stored +repository_family: practical-coding +task_family: quality +skill_commit: 31ba37c9c324ff5863ee237a8c89203f4405fbe9 +model_harness: Codex CLI + benchmarks/tree_validation.py +execution_depth: unknown +retrieval_depth: unknown +capability_path: [] +outcome: indeterminate +quality_gates: + correctness: unknown + safety: unknown + build_reachability: unknown +cost: + tokens: null + seconds: null + tool_calls: null + loc: null +routing_observation: unknown +mechanism: delivery requires n=1 mechanism iteration with preserved full artifacts, followed only after a frozen candidate by a same-contract n=3 comparison against the frozen v1.5 and no-skill arms. +user_feedback: wait for each benchmark process to finish before reading its complete report, continuously consolidate reusable evidence into the wiki, and deliver only when comparable benchmark evidence improves on the previous report. +candidate_lesson: treat delivered quality and historical comparability as gates; topology diagnostics and a favorable single repetition are not delivery evidence. +``` + +## Evidence selected before the first run + +- The checkout is the clean `experiment/evolvable-router-tree` branch at `31ba37c`. +- The active runner has 15 frozen tasks across three repositories, including two explicit manual Decision tasks. +- The topology stages four depth-2 children whose independent lift over their parents is not yet established. +- The frozen comparable baseline is v1.5 at `ba4058b4ef47a42bf79c9963b25678a2389897c1`. + +## Contradictions / uncertainty + +- `benchmarks/NEXT_VALIDATION.md` still describes the older event-router restoration branch and is not the active tree final gate. +- No fresh current-candidate model-backed result exists yet, so this receipt remains indeterminate until benchmark artifacts are appended through new receipts or wiki log entries; this immutable receipt will not be rewritten. diff --git a/evolution/raw/sessions/2026-09-01-tree-n1-leaf-candidate-qualified.md b/evolution/raw/sessions/2026-09-01-tree-n1-leaf-candidate-qualified.md new file mode 100644 index 0000000..533bdc7 --- /dev/null +++ b/evolution/raw/sessions/2026-09-01-tree-n1-leaf-candidate-qualified.md @@ -0,0 +1,15 @@ +# Receipt — frozen leaf candidate n=1 qualification + +- Artifact: `benchmark-results/tree-delivery-n1-leaves-inflection-20260901` +- Topology: Core -> Debugging/Implementation; both children are leaves +- Completeness: 58/58 determinate cells +- Adaptive: 15/15 +- Core ceiling: 13/13 +- Debugging ceiling: 13/13 +- Implementation ceiling: 13/13 +- Trace validity: 15/15 +- Explicit manual Decision: 2/2 +- Spontaneous manual activation: 0/13 automatic tasks +- Deterministic gates before run: 86 unit tests, tree self-tests, and 28/28 evolution workflow checks passed + +Runtime, topology, tasks, scorer, model settings, and frozen repository inputs are now candidates for freezing at a commit. Only a fresh complete paired n=3 artifact may establish delivery superiority over frozen v1.5. diff --git a/evolution/raw/sessions/2026-09-01-tree-n1-leaves-recommend-inflection.md b/evolution/raw/sessions/2026-09-01-tree-n1-leaves-recommend-inflection.md new file mode 100644 index 0000000..384d0ea --- /dev/null +++ b/evolution/raw/sessions/2026-09-01-tree-n1-leaves-recommend-inflection.md @@ -0,0 +1,11 @@ +# Receipt — leaf topology n=1 lexical false negative + +- Artifact: `benchmark-results/tree-delivery-n1-leaves-retry-20260901` +- Completeness: 58/58 determinate cells +- Adaptive: 14/15 +- Core, Debugging, Implementation ceilings: each 13/13 +- Trace validity: 15/15 +- Explicit manual adherence: 2/2 +- Spontaneous manual activation: 0 + +The only failed answer explicitly recommended the compatibility alias and stated the trade-off. The scorer rejected only the unlisted verb inflection `Recommend`. This artifact is diagnostic only. diff --git a/evolution/raw/sessions/2026-09-01-tree-n1-oracle-defects.md b/evolution/raw/sessions/2026-09-01-tree-n1-oracle-defects.md new file mode 100644 index 0000000..fae529e --- /dev/null +++ b/evolution/raw/sessions/2026-09-01-tree-n1-oracle-defects.md @@ -0,0 +1,40 @@ +# Experience receipt: first tree n=1 oracle defects + +```yaml +id: exp-20260901-tree-n1-oracle-defects +source_type: benchmark +source_pointer: benchmark-results/tree-delivery-n1-20260901-103036/results.jsonl +repository_family: practical-coding +task_family: quality +skill_commit: 31ba37c9c324ff5863ee237a8c89203f4405fbe9 +model_harness: gpt-5.6-luna medium + benchmarks/tree_validation.py n=1 current-only +execution_depth: unknown +retrieval_depth: unknown +capability_path: [] +outcome: indeterminate +quality_gates: + correctness: unknown + safety: pass + build_reachability: unknown +cost: + tokens: null + seconds: null + tool_calls: null + loc: null +routing_observation: unknown +mechanism: deterministic evidence groups produced false failures by requiring punctuation and non-authoritative filenames that the frozen prompts did not require. +user_feedback: optimize general benchmark and Skill mechanisms rather than tuning to a test case, and wait for complete reports before changing them. +candidate_lesson: invalidate and rerun model-backed evidence whenever a scorer contract changes; preserve the original artifact as evidence of the oracle defect. +``` + +## Supporting evidence + +- The run completed 106/106 determinate cells and reported adaptive 12/15 before oracle review. +- Both explicit manual answers loaded `references/manual/decision.md`, emitted `manual=decision`, recommended one option, and described its strongest trade-off; only the exact group `Trade-off:` was missing. +- The cancellation diagnosis named the real `exportCover` download side-effect boundary and the existing cancellation test `src/lib/avifEncoder.test.ts`; the frozen repository confirms this test exercises `AbortController.abort()` and worker termination. +- The task prompt did not require `exportProgress.test.ts` or `ExportProgressModal.test.tsx`, although the scorer did. + +## Contradictions / uncertainty + +- This receipt does not establish that all three answers would repeat at n=3. +- The run's topology lift and removal suggestions remain provisional n=1 diagnostics and are not accepted while the scorer changes. diff --git a/evolution/raw/sessions/2026-09-01-tree-n1-outcome-semantics-qualified.md b/evolution/raw/sessions/2026-09-01-tree-n1-outcome-semantics-qualified.md new file mode 100644 index 0000000..7e14b25 --- /dev/null +++ b/evolution/raw/sessions/2026-09-01-tree-n1-outcome-semantics-qualified.md @@ -0,0 +1,12 @@ +# Receipt — outcome-semantics n=1 qualification + +- Artifact: `benchmark-results/tree-delivery-n1-outcome-semantics-20260901` +- Scope: current-only n=1, 15 tasks, all seven active capability ceilings before topology collapse +- Completeness: 106/106 determinate cells +- Adaptive quality: 15/15 +- Trace validity: 15/15 +- Explicit manual Decision: 2/2 +- Spontaneous manual activation: 0/13 automatic tasks +- Every capability ceiling: 13/13 + +The oracle correction is qualified for topology iteration. This is not paired superiority evidence. diff --git a/evolution/raw/sessions/2026-09-01-tree-n1-path-normalization.md b/evolution/raw/sessions/2026-09-01-tree-n1-path-normalization.md new file mode 100644 index 0000000..8259f35 --- /dev/null +++ b/evolution/raw/sessions/2026-09-01-tree-n1-path-normalization.md @@ -0,0 +1,39 @@ +# Experience receipt: third tree n=1 scorer normalization defect + +```yaml +id: exp-20260901-tree-n1-path-normalization +source_type: benchmark +source_pointer: benchmark-results/tree-delivery-n1-semantic-20260901-120715/results.jsonl +repository_family: practical-coding +task_family: quality +skill_commit: 31ba37c9c324ff5863ee237a8c89203f4405fbe9 +model_harness: gpt-5.6-luna medium + benchmarks/tree_validation.py n=1 current-only on Windows +execution_depth: unknown +retrieval_depth: unknown +capability_path: [] +outcome: indeterminate +quality_gates: + correctness: unknown + safety: pass + build_reachability: unknown +cost: + tokens: null + seconds: null + tool_calls: null + loc: null +routing_observation: unknown +mechanism: manual reference enforcement failed to normalize Windows path separators, while focused-test scoring still required identifiers rather than the requested evidence act. +user_feedback: improve the general benchmark mechanism until delivery evidence is reliable; do not optimize runtime text for individual outputs. +candidate_lesson: normalize platform representations at the scorer boundary and encode required evidence acts independently from incidental repository filenames. +``` + +## Supporting evidence + +- The run completed 106/106 determinate cells with valid routing traces and zero spontaneous manual activation. +- The failed manual answer had no missing evidence groups and loaded the absolute Windows `references\manual\decision.md` path; only separator-sensitive enforcement failed. +- The failed cancellation answer reported the focused suite as 9/9 and identified the absent post-encode test; only the filename-specific evidence group failed. + +## Contradictions / uncertainty + +- The candidate still requires a complete fresh rerun after the scorer changes. +- n=1 topology suggestions remain unstable and are not delivery or removal evidence. diff --git a/evolution/raw/sessions/2026-09-01-tree-n1-qualified.md b/evolution/raw/sessions/2026-09-01-tree-n1-qualified.md new file mode 100644 index 0000000..9e52b3b --- /dev/null +++ b/evolution/raw/sessions/2026-09-01-tree-n1-qualified.md @@ -0,0 +1,42 @@ +# Experience receipt: normalized tree n=1 candidate qualification + +```yaml +id: exp-20260901-tree-n1-qualified +source_type: benchmark +source_pointer: benchmark-results/tree-delivery-n1-normalized-20260901-125524/results.jsonl +repository_family: practical-coding +task_family: quality +skill_commit: 31ba37c9c324ff5863ee237a8c89203f4405fbe9 +model_harness: gpt-5.6-luna medium + benchmarks/tree_validation.py n=1 current-only +execution_depth: unknown +retrieval_depth: unknown +capability_path: [] +outcome: success +quality_gates: + correctness: pass + safety: pass + build_reachability: pass +cost: + tokens: 227712.67 + seconds: 72.06 + tool_calls: 6.2 + loc: null +routing_observation: over-escalation +mechanism: after platform and semantic evidence normalization, the frozen current candidate completed every adaptive task while preserving manual-mode and trace discipline. +user_feedback: use n=1 for iteration, wait for complete reports, preserve reusable wiki lessons, and run n=3 only when the candidate is ready for delivery comparison. +candidate_lesson: freeze the candidate and scorer after a complete all-determinate n=1 pass; treat n=1 topology removal suggestions as provisional until paired n=3 capability evidence exists. +``` + +## Fresh evidence + +- 106/106 cells were determinate across adaptive and all capability ceilings. +- Adaptive delivered quality was 15/15. +- Routing traces were valid in 15/15 adaptive cells. +- Explicit manual Decision succeeded in 2/2 tasks; spontaneous manual activation was 0/13 automatic tasks. +- The analyzer reported 7 exact-minimum and 6 over-disclosure observations, with no under-disclosure, alternate-branch, or quality-gap task. + +## Contradictions / uncertainty + +- This is n=1 mechanism-iteration evidence, not stable ranking evidence. +- Core happened to pass 13/13 in this repetition, so n=1 suggested removing Debugging and Implementation; prior n=1 runs produced different capability minima. The suggestion is explicitly deferred to n=3. +- No same-run v1.5/no-skill comparison exists in this artifact. diff --git a/evolution/raw/sessions/2026-09-01-tree-n1-semantic-oracle.md b/evolution/raw/sessions/2026-09-01-tree-n1-semantic-oracle.md new file mode 100644 index 0000000..efecc8c --- /dev/null +++ b/evolution/raw/sessions/2026-09-01-tree-n1-semantic-oracle.md @@ -0,0 +1,39 @@ +# Experience receipt: second tree n=1 semantic oracle defect + +```yaml +id: exp-20260901-tree-n1-semantic-oracle +source_type: benchmark +source_pointer: benchmark-results/tree-delivery-n1-oraclefix-20260901-111838/results.jsonl +repository_family: practical-coding +task_family: quality +skill_commit: 31ba37c9c324ff5863ee237a8c89203f4405fbe9 +model_harness: gpt-5.6-luna medium + benchmarks/tree_validation.py n=1 current-only +execution_depth: unknown +retrieval_depth: unknown +capability_path: [] +outcome: indeterminate +quality_gates: + correctness: unknown + safety: pass + build_reachability: unknown +cost: + tokens: null + seconds: null + tool_calls: null + loc: null +routing_observation: unknown +mechanism: lexical evidence groups remained language- and identifier-dependent after the first scorer correction, rejecting semantically complete answers. +user_feedback: optimize the general mechanism until the benchmark is deliverable; do not tune runtime wording to a case. +candidate_lesson: evidence alternatives should encode semantic obligations and equivalent languages, while separate groups preserve the required diagnosis/test/decision structure. +``` + +## Supporting evidence + +- The rerun completed 106/106 determinate cells, with 15/15 valid traces, 0 spontaneous manual activations, and valid manual loading in both explicit Decision tasks. +- One Chinese Decision answer recommended summary compression and explicitly named its strongest “权衡” and “代价”, but the English-only `Recommendation:` and `trade-off` groups failed. +- The cancellation answer identified `abort()` at the worker-success handoff, the unconditional `link.click()` side effect, existing cancellation coverage, and one falsifying test; only exact signal/type and filename groups failed. + +## Contradictions / uncertainty + +- n=1 topology suggestions are unstable across the first two runs and do not justify removing automatic nodes. +- This scorer correction invalidates the second model-backed artifact for delivery acceptance; a full fresh rerun is required. diff --git a/evolution/raw/sessions/2026-09-01-tree-n3-failed-oracle-and-topology.md b/evolution/raw/sessions/2026-09-01-tree-n3-failed-oracle-and-topology.md new file mode 100644 index 0000000..630011e --- /dev/null +++ b/evolution/raw/sessions/2026-09-01-tree-n3-failed-oracle-and-topology.md @@ -0,0 +1,26 @@ +# Receipt — paired n=3 failed release gate + +## Immutable input + +- Candidate commit: `eca9a090d04492addaf4e2bb6d8dbec3e61bc0d0` +- Frozen baseline: `ba4058b4ef47a42bf79c9963b25678a2389897c1` +- Raw output: `benchmark-results/tree-final-eca9a09-20260901` +- Matrix: 15 tasks, 408/408 determinate cells, three repetitions per comparable arm/capability cell + +## Result + +- Adaptive: 40/45, 88.9%, 12 stable tasks +- Frozen v1.5: 44/45, 97.8%, 14 stable tasks +- No-skill: 41/45, 91.1%, 13 stable tasks +- Adaptive trace validity: 45/45 +- Explicit manual Decision adherence: 6/6 +- Spontaneous manual activation: 0/39 automatic cells +- Release quality gate: **FAIL** + +This artifact is diagnostic evidence only and must not be presented as a delivery comparison. + +## Failure classification + +Five adaptive failures were deterministic-oracle mismatches: three rejected `Decision: choose` recommendations (one also used `cost` for the downside), one rejected an exact blocked focused-test outcome, and one required `EditorShell` despite an authoritative `exportCover` cancellation/download-boundary diagnosis. + +Separately, repeated capability ablation found no depth-2 staged node in any minimum-sufficient set. That topology observation is deferred until the scorer correction completes a fresh n=1 iteration. diff --git a/evolution/raw/sessions/2026-09-01-wikiskill-maintenance.md b/evolution/raw/sessions/2026-09-01-wikiskill-maintenance.md new file mode 100644 index 0000000..5bb9fe3 --- /dev/null +++ b/evolution/raw/sessions/2026-09-01-wikiskill-maintenance.md @@ -0,0 +1,41 @@ +# Experience receipt: explicit WikiSkill maintenance loop + +```yaml +id: exp-20260901-wikiskill-maintenance +source_type: real-project +source_pointer: current 2026-09-01 Practical Coding maintenance session; transcript intentionally not stored +repository_family: practical-coding +task_family: quality +skill_commit: 118acd81cb0e26f4f8087555c3bd89cbf45c9d30 +model_harness: ChatGPT maintenance session + repository benchmark harness +execution_depth: unknown +retrieval_depth: unknown +capability_path: [] +outcome: success +quality_gates: + correctness: unknown + safety: pass + build_reachability: n/a +cost: + tokens: null + seconds: null + tool_calls: null + loc: null +routing_observation: none +mechanism: maintenance experience should compile into a persistent wiki, while wiki-informed Skill changes remain separate reversible candidates accepted only after a frozen benchmark shows no required score regression. +user_feedback: add two explicitly triggered maintenance skills—current-session to wiki, and wiki-informed skill evolution with a new benchmark and rerun before accepting any update. +candidate_lesson: keep these capabilities outside the automatic coding router tree and validate their maintenance contract independently. +``` + +## Evidence summary + +The session explicitly referenced WikiSkill and requested a maintenance loop with two user-triggered capabilities. No secrets, private code, or full conversation transcript are stored here. + +## Supporting observations + +- Current branch already treats `evolution/` as maintenance-time knowledge unavailable to ordinary runtime agents. +- The requested completion condition is objective: add benchmark coverage, rerun, and do not finalize a Skill update when required scores are worse. + +## Contradictions / uncertainty + +- No model-backed evolution benchmark has yet established that these maintenance skills improve downstream runtime coding quality; they are maintenance orchestration capabilities, not promoted automatic runtime nodes. diff --git a/evolution/raw/sessions/2026-09-02-tree-minimum-question-n1-rejected.md b/evolution/raw/sessions/2026-09-02-tree-minimum-question-n1-rejected.md new file mode 100644 index 0000000..26a910d --- /dev/null +++ b/evolution/raw/sessions/2026-09-02-tree-minimum-question-n1-rejected.md @@ -0,0 +1,9 @@ +# Receipt — minimum-question discriminator n=1 rejected + +- Artifact: `benchmark-results/tree-discriminator-minimum-question-n1-20260902` +- Scope: one new case, adaptive/frozen-v1.5/no-skill, n=1 +- Completeness: 6/6 determinate cells +- Quality: all three variants passed both applicable capability cells +- Observed distinction: none; frozen v1.5 did not load Decision + +The case was rejected and removed together with its provisional negative-answer and legacy-reference scoring. It did not provide evidence for a runtime change. diff --git a/evolution/raw/sessions/2026-09-02-tree-n1-bounded-evidence-qualified.md b/evolution/raw/sessions/2026-09-02-tree-n1-bounded-evidence-qualified.md new file mode 100644 index 0000000..82bc751 --- /dev/null +++ b/evolution/raw/sessions/2026-09-02-tree-n1-bounded-evidence-qualified.md @@ -0,0 +1,12 @@ +# Receipt — bounded-evidence candidate n=1 qualified + +- Artifact: `benchmark-results/tree-bounded-evidence-n1-20260902` +- Completeness: 58/58 determinate cells +- Adaptive quality: 15/15 +- Core, Debugging, Implementation ceilings: 13/13 each +- Adaptive trace validity: 15/15 +- Explicit manual success: 2/2 +- Spontaneous manual activation: 0 +- Adaptive means (diagnostic n=1 only): 231,929.20 tokens, 76.06s, 6.60 tool calls + +This n=1 result qualifies the frozen candidate for paired n=3; cost acceptance is deferred to the complete repeated comparison. diff --git a/evolution/raw/sessions/2026-09-02-tree-n1-core-only-qualified.md b/evolution/raw/sessions/2026-09-02-tree-n1-core-only-qualified.md new file mode 100644 index 0000000..dca27bb --- /dev/null +++ b/evolution/raw/sessions/2026-09-02-tree-n1-core-only-qualified.md @@ -0,0 +1,8 @@ +# Receipt — Core-only n=1 qualification + +- Candidate commit: `230522fa914e50e219547f64607ee68383596660` +- Artifact: `benchmark-results/tree-core-only-n1-20260902` +- Completeness: 28/28 determinate cells +- Result: adaptive and Core capability cells passed all 15 tasks; trace/manual discipline passed + +This n=1 result only qualified the frozen Core-only candidate for paired n=3. It was not delivery evidence. diff --git a/evolution/raw/sessions/2026-09-02-tree-n1-isolated-leaf-qualified.md b/evolution/raw/sessions/2026-09-02-tree-n1-isolated-leaf-qualified.md new file mode 100644 index 0000000..3d5edd4 --- /dev/null +++ b/evolution/raw/sessions/2026-09-02-tree-n1-isolated-leaf-qualified.md @@ -0,0 +1,14 @@ +# Receipt — isolated leaf candidate n=1 qualification + +- Artifact: `benchmark-results/tree-delivery-n1-retired-isolated-20260902` +- Topology: Core -> Debugging/Implementation; both leaves +- Runtime reference surface: no rejected depth-2 documents +- Completeness: 58/58 determinate +- Adaptive: 15/15 +- Core, Debugging, Implementation ceilings: each 13/13 +- Trace validity: 15/15 +- Explicit manual Decision: 2/2 +- Spontaneous manual activation: 0/13 automatic tasks +- Deterministic gates: 94 unit tests, all tree self-tests, 28/28 evolution workflow checks + +This qualifies the complete candidate for freezing at a commit and running a fresh paired n=3. It is not superiority evidence by itself. diff --git a/evolution/raw/sessions/2026-09-02-tree-n1-missing-trace.md b/evolution/raw/sessions/2026-09-02-tree-n1-missing-trace.md new file mode 100644 index 0000000..522d63c --- /dev/null +++ b/evolution/raw/sessions/2026-09-02-tree-n1-missing-trace.md @@ -0,0 +1,8 @@ +# Receipt — missing benchmark trace footer + +- Artifact: `benchmark-results/tree-delivery-n1-evidence-identity-outcome-20260902` +- Completeness: 58/58 determinate +- Capability ceilings: Core 13/13, Debugging 13/13, Implementation 13/13 +- Adaptive: 14/15 + +The only failed answer fully satisfied the requested Decision analysis and actually read `references/manual/decision.md`, but emitted no `TREE_TRACE` footer. This artifact is diagnostic only; observed-command trace recovery must rerun at n=1 before acceptance. diff --git a/evolution/raw/sessions/2026-09-02-tree-n1-outcome-field.md b/evolution/raw/sessions/2026-09-02-tree-n1-outcome-field.md new file mode 100644 index 0000000..7c01707 --- /dev/null +++ b/evolution/raw/sessions/2026-09-02-tree-n1-outcome-field.md @@ -0,0 +1,8 @@ +# Receipt — explicit outcome field lexical gap + +- Artifact: `benchmark-results/tree-delivery-n1-evidence-identity-20260902` +- Completeness: 58/58 determinate +- Adaptive: 15/15; trace/manual contracts all passed +- Core: 13/13; Implementation: 13/13; Debugging: 12/13 + +The sole failed ceiling answer ran the required focused command once and explicitly reported `Outcome: Vitest did not start because dependencies are absent`. The outcome oracle accepted selected success/failure words but not the explicit result field itself. This artifact is diagnostic only. diff --git a/evolution/raw/sessions/2026-09-02-tree-n1-retired-reference-observed.md b/evolution/raw/sessions/2026-09-02-tree-n1-retired-reference-observed.md new file mode 100644 index 0000000..c746d9b --- /dev/null +++ b/evolution/raw/sessions/2026-09-02-tree-n1-retired-reference-observed.md @@ -0,0 +1,8 @@ +# Receipt — retired reference remained discoverable + +- Artifact: `benchmark-results/tree-delivery-n1-observed-trace-20260902` +- Completeness: 58/58 determinate +- Adaptive: 14/15; explicit manual 2/2; spontaneous manual 0 +- Core and Debugging ceilings: 13/13; Implementation: 12/13 + +The adaptive failure loaded `implementation-security-boundary.md` even though Implementation was an active leaf. Observed trace validation rejected it. The ceiling failure separately gave concrete focused test method evidence without repeating its class name. This artifact is diagnostic only. diff --git a/evolution/raw/sessions/2026-09-02-tree-n3-b202f7a-delivery.md b/evolution/raw/sessions/2026-09-02-tree-n3-b202f7a-delivery.md new file mode 100644 index 0000000..74833a4 --- /dev/null +++ b/evolution/raw/sessions/2026-09-02-tree-n3-b202f7a-delivery.md @@ -0,0 +1,18 @@ +# Receipt — evolvable leaf tree paired n=3 delivery + +- Candidate commit: `b202f7a165ae3ea4404d404bb1235ebf4270cbfb` +- Frozen baseline: `ba4058b4ef47a42bf79c9963b25678a2389897c1` +- Artifact: `benchmark-results/tree-final-b202f7a-20260902` +- Model/reasoning: `gpt-5.6-luna` / `medium` +- Repositories/tasks/runs: 3 / 15 / 3 +- Completeness: 252/252 determinate cells +- Adaptive/frozen-v1.5/no-skill quality: 45/45, 44/45, 44/45 +- Core/Debugging/Implementation ceilings: 39/39 each +- Adaptive trace: 45/45 +- Explicit manual contract: 6/6 +- Spontaneous manual activation: 0/39 automatic cells +- Adaptive mean tokens/duration/tools: 258,061.64 / 76.82s / 8.42 +- v1.5 mean tokens/duration/tools: 217,460.96 / 72.20s / 7.24 +- Release quality gate: PASS + +Baseline missed one `sa-memory-strategy-manual-decision` repetition by omitting the SlidingWindow alternative. No-skill missed one `ca-cancel-download` repetition by omitting focused-test evidence. Adaptive passed every repetition. Raw paths and full transcripts remain only in the ignored local artifact. diff --git a/evolution/raw/sessions/2026-09-02-tree-n3-core-only-rejected.md b/evolution/raw/sessions/2026-09-02-tree-n3-core-only-rejected.md new file mode 100644 index 0000000..38c0181 --- /dev/null +++ b/evolution/raw/sessions/2026-09-02-tree-n3-core-only-rejected.md @@ -0,0 +1,12 @@ +# Receipt — Core-only paired n=3 rejected + +- Candidate commit: `230522fa914e50e219547f64607ee68383596660` +- Artifact: `benchmark-results/tree-final-230522f-20260902` +- Completeness: 174/174 determinate cells +- Adaptive, frozen v1.5, no-skill: each 45/45 +- Core ceiling: 39/39 +- Adaptive trace/manual discipline: perfect +- Adaptive mean tokens/duration/tools: 278,578.02 / 76.73s / 7.82 +- v1.5 mean tokens/duration/tools: 245,327.98 / 72.69s / 7.64 + +The collapse preserved quality but regressed every recorded cost metric. It is rejected; commit `5545329` reverted the runtime/topology change and restored the isolated leaf candidate. diff --git a/evolution/raw/sessions/2026-09-02-tree-n3-leaf-failed.md b/evolution/raw/sessions/2026-09-02-tree-n3-leaf-failed.md new file mode 100644 index 0000000..c5e60ba --- /dev/null +++ b/evolution/raw/sessions/2026-09-02-tree-n3-leaf-failed.md @@ -0,0 +1,27 @@ +# Receipt — leaf candidate paired n=3 failed + +## Immutable input + +- Candidate commit: `67f2f5c72d5db3ef461fb60e6a60a77351b1a8a9` +- Frozen baseline: `ba4058b4ef47a42bf79c9963b25678a2389897c1` +- Artifact: `benchmark-results/tree-final-67f2f5c-20260901` +- Completeness: 252/252 cells, three repetitions per comparable arm/capability cell + +## Result + +- Adaptive: 41/45, 91.1% +- Frozen v1.5: 44/45, 97.8% +- No-skill: 45/45, 100% +- Core ceiling: 38/39 +- Debugging ceiling: 35/39 +- Implementation ceiling: 36/39 +- Trace validity: 44/45 +- Explicit manual success: 5/6 +- Spontaneous manual activation: 0 +- Delivery decision: **Rejected** + +The analyzer then crashed on the already-invalid retired node `state-concurrency`; the raw matrix and `report.json` remain complete, while `analysis.json` was not produced. This artifact is diagnostic only. + +## Classification + +Three failures are general evidence-identity/oracle defects. One is a real runtime isolation defect: a retired depth-2 reference remained discoverable and was loaded despite the active leaf topology. Both classes require new n=1 hypotheses; this result cannot be rescored into delivery evidence. diff --git a/evolution/raw/sessions/2026-09-02-tree-n3-quality-ceiling.md b/evolution/raw/sessions/2026-09-02-tree-n3-quality-ceiling.md new file mode 100644 index 0000000..d702b7f --- /dev/null +++ b/evolution/raw/sessions/2026-09-02-tree-n3-quality-ceiling.md @@ -0,0 +1,12 @@ +# Receipt — isolated leaf paired n=3 quality ceiling + +- Candidate commit: `ad2987c903fb8dc32dd87ead4ac658143397227c` +- Artifact: `benchmark-results/tree-final-ad2987c-20260902` +- Completeness: 252/252 determinate cells +- Adaptive quality: 45/45 +- Frozen v1.5 quality: 45/45 +- No-skill quality: 44/45 +- Adaptive mean tokens/duration/tools: 242,910.84 / 75.60s / 7.60 +- Frozen v1.5 mean tokens/duration/tools: 211,758.69 / 72.94s / 8.27 + +The candidate reached the original suite's quality ceiling and improved tool calls, but regressed tokens and duration. This motivated a separate bounded-evidence-volume hypothesis rather than a quality-oracle adjustment. diff --git a/evolution/rejected/README.md b/evolution/rejected/README.md index 4371f8b..e5445e9 100644 --- a/evolution/rejected/README.md +++ b/evolution/rejected/README.md @@ -13,3 +13,8 @@ Each record should state: - the explicit condition under which it is worth reconsidering. The purpose is to prevent repeated rediscovery of the same failed architecture or wording change. + +## Archived experiments + +- [`execution-state/`](execution-state/) — retired explicit execution-state/history-free runtime, host transport, and four-arm model-gate experiment. +- [`progressive-capability-tree.md`](progressive-capability-tree.md) — rejected fixed-depth progressive capability tree. diff --git a/evolution/rejected/execution-state/README.md b/evolution/rejected/execution-state/README.md new file mode 100644 index 0000000..810d4ac --- /dev/null +++ b/evolution/rejected/execution-state/README.md @@ -0,0 +1,86 @@ +# Retired execution-state / history-free experiment + +Status: **Rejected and removed from the active runtime** + +Decision date: 2026-09-03 +Last active branch head before retirement: `215334db7bb914bd9f0346a2b09654fc89accc96` + +## What was tried + +The experiment added a bounded explicit coding state, merge-patch transitions, a history-free host boundary, exact outbound transport auditing, and a four-arm model gate: + +- full history; +- state shadow; +- state history-free; +- no-skill full history. + +The proposal was intentionally kept outside the Router tree. + +## Evidence + +The final complete standard `n=1`, `workers=1` matrix contained 24 determinate cells: + +| Arm | Passed | Uncached input tokens | Duration | +|---|---:|---:|---:| +| full-history | 6/6 | 50,736 | 154.57s | +| no-skill-full-history | 6/6 | 19,722 | 123.61s | +| state-history-free | 6/6 | 78,118 | 172.60s | +| state-shadow | 5/6 | 56,611 | 156.83s | + +The history-free candidate preserved delivered quality and its captured client transport contract passed. It did not establish a cost benefit: + +- uncached input tokens were approximately **54.0% higher** than full history; +- total duration was approximately **11.7% higher** than full history; +- state shadow retained a rejected cache hypothesis as active in one case; +- formal `n>=3`, token, latency, and 10/25/50/100 bounded-horizon gates remained incomplete. + +A later deterministic invariant rejected overlapping active/rejected hypothesis IDs, but that hardening was not rerun through the model matrix before retirement. + +Separate tree evidence found the Core fixed cost on simple work acceptable, while the remaining cost problem was concentrated in retrieval over-expansion and oversized evidence. Execution state did not address that upstream cause and introduced a large runtime, transport, test, and benchmark maintenance surface. + +## Decision + +Reject the execution-state/history-free architecture for Practical Coding and remove it from: + +- `SKILL.md`, `AGENTS.md`, README files, and the agent default prompt; +- active runtime and host/transport code; +- active deterministic and four-arm benchmark code; +- active topology metadata and CI gates. + +The old experiment records are retained in this directory. Historical commits and locally preserved raw benchmark artifacts remain the authoritative detailed evidence; old results are not rewritten. + +## Follow-up direction + +Cost work returns to retrieval convergence: + +- bound each search/read output; +- stop broad inventory once candidate paths or symbols are known; +- require explicit evidence before reading dependency internals; +- isolate large logs and test output instead of carrying them through later turns. + +This is a retrieval-policy problem, not a reason to alter the current Core → Debugging / Implementation topology. + +## Reconsideration condition + +Do not restore the removed code. A future proposal must start as a new frozen experiment and provide independent evidence that a substantially simpler host-native mechanism: + +1. solves a demonstrated long-horizon failure not addressed by retrieval/output bounds; +2. preserves required quality across a complete paired `n>=3` matrix; +3. reduces both quality-qualified uncached input tokens and end-to-end time; +4. passes bounded-horizon and final transport audits without reintroducing comparable maintenance cost. + +## Preserved records + +The original experiment and contract documents are archived under: + +- [`experiments/`](experiments/) +- [`reference/`](reference/) + +Key historical commits include: + +- `d85c72cc5aa239da32352309e723ed1e6fc80429` — audited history-free host; +- `ea8580f169154cab01914f4c76e369f1a26f91f8` — four-arm runner; +- `0499fd15d3f2f6de65ec0681e96956ca4964113d` — scorer and Codex SSE hardening; +- `e6cc9caa456767b3e05dbff59474aa7014146cbf` — gate-role separation; +- `e6b5aab8e85777644f56737ec335c46beb0f9986` — hypothesis partition invariant; +- `215334db7bb914bd9f0346a2b09654fc89accc96` — final pre-retirement branch head. diff --git a/evolution/rejected/execution-state/experiments/skill-state-history-free-host-20260902.md b/evolution/rejected/execution-state/experiments/skill-state-history-free-host-20260902.md new file mode 100644 index 0000000..f2f9974 --- /dev/null +++ b/evolution/rejected/execution-state/experiments/skill-state-history-free-host-20260902.md @@ -0,0 +1,82 @@ +# EXP-state-host-20260902 — Audited history-free transport boundary + +## Status + +`candidate-pending-model-gate` + +This hypothesis is frozen before any model-backed result for the host adapter is observed. The implementation may pass deterministic checks without earning a quality, token, latency, or topology claim. + +## Evidence / unresolved mechanism + +The execution-state schema and merge runtime can construct `P + Σ + O`, but `build_prompt()` alone cannot establish what an SDK, proxy, or product actually sends. A surrounding host may still append prior messages, use `previous_response_id`, attach a conversation or prompt reference, rebuild the request, or put accumulated history into a field labeled as the latest observation. In that situation the state is only a shadow aid; it is not evidence for history-free execution or a horizon-independent per-step client request. + +The paired tree report at `f65bcd3dac2eac1f8e47ec435c7499c953ec0c96` establishes quality non-regression for the reviewed tree/runtime wording, but adaptive cost remained above frozen v1.5. It does not isolate execution state and does not establish a state-related token or latency benefit. + +## Atomic hypothesis + +A transport-facing host that freezes request identity, constructs exactly one current request from immutable procedure plus validated state and the latest bounded observation, rejects all explicit history-import channels, and audits the exact serialized request body will make the `state history-free` arm mechanically testable without changing the automatic router topology. + +The observable activation signal remains state pressure. The host boundary is a cross-cutting substrate and must not become an automatic node, a manual mode, or a retrieval mode. + +## Candidate change + +Add only the host substrate and its runtime contract: + +1. `runtime/skill_state_host.py` freezes model, procedure, tools, options, limits, and request-shape identity in a self-digested manifest; +2. the current procedure and transition contract are placed in top-level `instructions`; +3. exactly one current user input contains canonical JSON for `Σ`, current `O`, and optional bounded deterministic validation feedback; +4. prior-response, conversation, prompt-reference, context-management, old assistant/tool input, and equivalent configured history handles are rejected; +5. procedure, state, observation, feedback, tools, options, response, attempts, and the final serialized body have hard bounds; +6. each retry starts from the unchanged original state and does not append a rejected response; +7. a valid successor must be durably persisted before its still-untrusted action proposal is returned; +8. request/response identity, component sizes, provider usage when present, and transport timing are exposed as audit records; +9. offline build/audit commands never call a model; +10. AGENTS, SKILL, README, detailed architecture, and OpenAI agent metadata distinguish state shadow, client-body boundedness, and model-backed benefit. + +Do not alter `benchmarks/`, `benchmarks/results/`, automatic topology, Debugging/Implementation boundaries, or manual-mode eligibility in this candidate. + +## Deterministic falsifiers + +Reject the implementation before model testing if any of the following occurs: + +- a prepared request can contain more than one input item or any explicit prior-response/conversation/prompt/context handle; +- a manifest-free one-request audit is reported as a frozen trajectory claim; +- manifest, procedure, tools, options, limits, or request contract can drift without rejection; +- a retry contains a prior response or starts from a mutated candidate state; +- invalid output, retry exhaustion, or persistence failure can release an action; +- a native model tool call can be mistaken for the required JSON transition; +- final serialized request or response bytes can exceed the frozen limits; +- direct script invocation does not work from the repository root. + +## Model-backed falsifiers + +Use the already frozen protocol in `benchmarks/SKILL_STATE_MODEL_GATE.md`. Reject or revise the candidate if: + +- the final outbound body, contextual headers/cookies, proxy/session state, or observation provenance cannot be audited; +- the history-free arm regresses delivered quality, safety, required checks, route validity, or manual-mode discipline against full history; +- valid state repeatedly loses future-relevant evidence, fails to replace stale facts, or increases repeated actions/hypotheses; +- invalid-transition retries or state maintenance erase the expected token benefit; +- provider-reported cumulative uncached input tokens do not satisfy the pre-frozen paired threshold after quality passes; +- end-to-end latency does not satisfy its independently frozen paired threshold; +- history-required cases silently discard provenance instead of retaining bounded immutable pointers or exiting history-free mode. + +## Claim boundary before results + +Before the model gate, the maximum supported statement is: + +> Under one validated frozen manifest, the captured client-visible request body has one current input, contains no explicit body-level history channel, and is bounded in bytes independently of prior task-step count. + +This does not prove that an HTTP transport added no contextual state, that the provider used no internal context, that the observation injector supplied only the latest observation, that model quality is preserved, or that tokens or latency improve. Even after a per-step bound is established, cumulative input across `T` steps remains proportional to the number of steps rather than constant. + +## Frozen non-benchmark validation + +- Python syntax compilation for the state runtime, host adapter, and ordinary tests; +- `tests.test_skill_state_hardening`; +- `tests.test_skill_state_host`; +- direct offline CLI build and manifest-matched re-audit; +- YAML parsing and whitespace checks; +- no benchmark case, scorer, runner, topology manifest, or result-data modification. + +## Pending decision + +Do not modify the Skill activation rule or router tree from this candidate alone. After the paired report is published, classify failures by mechanism first: host integration, observation provenance, state schema/update policy, activation timing, delivery quality, or cost. Only repeated topology-specific evidence may justify a tree mutation. diff --git a/evolution/rejected/execution-state/experiments/skill-state-hypothesis-partition-invariant-20260903.md b/evolution/rejected/execution-state/experiments/skill-state-hypothesis-partition-invariant-20260903.md new file mode 100644 index 0000000..c9352cf --- /dev/null +++ b/evolution/rejected/execution-state/experiments/skill-state-hypothesis-partition-invariant-20260903.md @@ -0,0 +1,100 @@ +# Execution-state hypothesis partition invariant + +Status: `candidate-implemented-pending-model-gate` + +## Evidence + +The standard four-arm `n=1`, `workers=1`, `codex-sse-v1` report against +`e6cc9caa456767b3e05dbff59474aa7014146cbf` contains one state-shadow semantic +failure in `rejected-cache-hypothesis`: the final state retained `h-cache` in +`hypotheses.active` after also classifying it as rejected. The history-free arm +completed the same case correctly. The report otherwise records 24/24 +determinate cells, a passing history-free quality/state/history-pointer/transport +candidate gate, and no tree-boundary failure. + +The raw benchmark artifacts remain local and are not rewritten by this change. +This receipt records only the reported mechanism and the exact candidate ref. + +## Causal claim + +`runtime/skill_state.py` validated `hypotheses.active` and +`hypotheses.rejected` as independent string maps, so the same hypothesis ID could +legally appear in both partitions. JSON Merge Patch then permitted a model to add +a rejected entry while accidentally omitting the `null` deletion of the active +entry. The resulting state was structurally valid but semantically contradictory. + +## Observable signal + +Before a successor state is accepted, compute the exact-key intersection of +`hypotheses.active` and `hypotheses.rejected`. A non-empty intersection is a +deterministic invalid-state signal available without inspecting task wording or +benchmark labels. + +## Exact target + +- `runtime/skill_state.py`: canonical state validation only. +- `tests/test_skill_state_hypothesis_invariant.py`: ordinary deterministic regression tests. +- `docs/SKILL_STATE_INVARIANTS.md`: schema invariant and retry behavior. + +No change was made to `SKILL.md`, router topology, state shape/schema version, +benchmark cases, scorer, runner, thresholds, or stored benchmark results. + +## Implemented patch + +Runtime candidate: `e6b5aab8e85777644f56737ec335c46beb0f9986`. + +The validator now requires +`set(hypotheses.active).isdisjoint(hypotheses.rejected)`. It rejects the complete +successor with a stable error listing the overlapping IDs. Because +`apply_transition()` validates the full successor before exposing its action, the +existing host retry path keeps the original canonical state and requests a +corrected patch. + +The original schema implementation is retained byte-for-byte in +`runtime/_skill_state_impl.py`; `runtime/skill_state.py` remains the public entry +point and adds the cross-field invariant before rebinding the retained runtime's +validation boundary. This preserves the existing public API and CLI while keeping +the semantic rule in one validation path. + +## Expected benefit + +- A hypothesis has one current lifecycle classification, never both live and + rejected. +- An omitted merge-patch deletion fails closed instead of becoming durable state. +- The rule applies to every task and does not encode the observed case ID or its + answer terms. + +## Falsifier + +Reject or revise this intervention if any supported workflow intentionally needs +one exact hypothesis ID in both partitions, if deterministic state/host tests +regress, or if a fresh model matrix shows a quality regression that cannot be +attributed to infrastructure. Additional retry cost must remain visible; semantic +correctness is not permission to hide token or latency regressions. + +## Baseline and validation plan + +Baseline ref: `e6cc9caa456767b3e05dbff59474aa7014146cbf`. +Runtime candidate: `e6b5aab8e85777644f56737ec335c46beb0f9986`. + +Before model work: + +```text +python -m py_compile runtime/skill_state.py runtime/_skill_state_impl.py tests/test_skill_state_hypothesis_invariant.py +python -m unittest tests.test_skill_state_hardening tests.test_skill_state_host tests.test_skill_state_hypothesis_invariant +python -m unittest benchmarks.test_skill_state_runtime +python benchmarks/skill_state_validation.py --self-test +``` + +Rerun a new standard four-arm `n=1` matrix from an empty output directory. Do not +resume or relabel the previous matrix. The minimum iteration acceptance conditions +are: + +- all 24 cells determinate; +- history-free quality, state semantics, history pointer, and client transport + remain `PASS`; +- no accepted state contains overlapping active/rejected hypothesis IDs; +- the shadow diagnostic either passes or records only a different independently + diagnosed mechanism; +- token, latency, and bounded-context claims remain separate and are not promoted + from this deterministic invariant. diff --git a/evolution/rejected/execution-state/experiments/skill-state-n1-scorer-sse-remediation-20260903.md b/evolution/rejected/execution-state/experiments/skill-state-n1-scorer-sse-remediation-20260903.md new file mode 100644 index 0000000..1280049 --- /dev/null +++ b/evolution/rejected/execution-state/experiments/skill-state-n1-scorer-sse-remediation-20260903.md @@ -0,0 +1,121 @@ +# EXP-state-20260903 — Scorer-contract and Codex SSE transport remediation + +## Status + +`infrastructure-fixed-pending-fresh-n1` + +This record follows the first complete `standard / n=1 / workers=1` four-arm run at +`ea8580f169154cab01914f4c76e369f1a26f91f8`. It is an infrastructure remediation, +not a Skill or router candidate. No historical result is rewritten or promoted. + +## Frozen evidence + +The original atomic matrix contained 24 cells: 19 pass, 4 fail, and 1 +indeterminate. A connection-failure case was rerun independently under the same +configuration and all four arms passed; those retry cells remain separate rather +than being spliced into the original matrix. + +Run identity: + +- model `gpt-5.6-luna`, reasoning `medium`; +- standard profile, `n=1`, one worker, 90-second request timeout, at most two + transition attempts; +- original manifest SHA-256 + `c4408f852c39f4caa15ff0a39271a830772b48461676d2962f848b368945af18`; +- local compatibility adapter SHA-256 + `ece048d7836c6f50fe1f6074a51df9f6cecc8e8ecbe7bce49f01cc124adaee0f`; +- independent retry manifest SHA-256 + `5f088e168d58bac3be036e9bbb76323605c748497e41633b26617da6903cf8c3`. + +The raw artifacts remain ignored local benchmark evidence. This repository record +contains only the mechanism-level findings needed to freeze the repair. + +## Confirmed defects + +### 1. Separator-sensitive answer evidence + +The answer scorer used case-folded literal substring matching. Therefore the +semantically and lexically equivalent surface forms `parser-transition` and +`parser transition` received different scores even though the case success +contract itself uses the spaced form. + +The repair is general: normalize Unicode with NFKC, case-fold, and treat +punctuation, symbols, underscores, and whitespace as equivalent separators. It +does not add case-specific aliases, stemming, synonyms, or semantic judging. + +### 2. State-only artifact requirement leaked into control arms + +The immutable artifact file and digest are required in every arm. A canonical +`state.history.required` flag and state artifact pointer exist only in state arms. +The old scorer required those state fields from full-history and no-skill arms, +which made two otherwise correct control cells structurally impossible to pass. + +The repair separates: + +- immutable file existence and digest integrity — every arm; +- `state.history.required` and exact pointer retention — state arms only. + +### 3. The observed Codex wire request was not the frozen source request + +The API-key/non-streaming request profile was not accepted by the ChatGPT-backed +Codex endpoint. The successful local compatibility run changed exactly four +fields: set `stream=true` and removed `background`, `max_output_tokens`, and +`truncation`. It retained the current instructions, input, model, reasoning, +`store=false`, and all task data. The response was an SSE stream. + +Because the canonical host manifest described the source body rather than the +final transformed body, that run cannot establish the final outbound transport +gate. Source-body audit success is retained as source evidence only. + +## Frozen remediation + +1. Keep the original implementation available under private compatibility module + names so the patch is reviewable and does not duplicate unrelated runner code. +2. Give the scorer an explicit version and reject mixed old/new result sets. +3. Apply only the two general scoring-contract repairs above; do not edit frozen + case wording or manually rescore historical rows. +4. Add a named `codex-sse-v1` wire profile with a self-digesting contract manifest. +5. Audit the canonical history-free source request before transformation. +6. Permit exactly the four observed field changes and prove all other JSON fields + remain identical. +7. Send the transformed bytes directly, with `store=false`, no previous response, + no conversation/thread/session field, no cookie jar, no environment proxy, and + no replayed response context header. +8. Preserve both source and final request hashes, raw SSE, normalized response, + redacted final headers, profile manifest, and profile audit per request. +9. Treat the final output-token limit as provider-managed for this profile; do not + claim the removed `max_output_tokens=2048` remains enforced. +10. Parse a run only after exactly one `response.completed` event. A missing or + conflicting completion remains an infrastructure failure. + +## Falsifiers before model rerun + +Reject this infrastructure patch before another model run if any deterministic +check shows that: + +- answer normalization accepts stemming or unrelated semantic variants; +- a non-state arm can pass with a missing or wrong artifact digest; +- a state arm can pass the history-required case without its exact pointer; +- `codex-sse-v1` changes any field beyond the four frozen changes; +- a final request imports old response, conversation, thread, session, cookie, or + proxy state; +- the final wire/profile manifest cannot be reproduced from committed code; +- an incomplete SSE stream is normalized as a completed response; +- old scorer rows can be mixed into a new formal analysis. + +## Required rerun + +The original n=1 matrix and independent retry remain historical diagnostic +evidence. After deterministic checks pass, start a fresh output directory and run +the complete four-arm standard n=1 matrix with the new scorer and frozen final +wire profile. Do not merge old cells, hand-correct old verdicts, or use the +independent retry as an atomic replacement. + +Only after the fresh n=1 matrix is determinate may the project decide whether to +freeze an n>=3 release comparison and the 10/25/50/100 bounded profile. Until then: + +- execution-state model gate: `PENDING`; +- token gate: `PENDING`; +- latency gate: `PENDING`; +- bounded-context gate: `PENDING`; +- Skill wording and automatic tree topology: unchanged. diff --git a/evolution/rejected/execution-state/experiments/skill-state-runtime-20260902.md b/evolution/rejected/execution-state/experiments/skill-state-runtime-20260902.md new file mode 100644 index 0000000..3b2da2c --- /dev/null +++ b/evolution/rejected/execution-state/experiments/skill-state-runtime-20260902.md @@ -0,0 +1,80 @@ +# EXP-state-20260902 — Bounded execution state as a runtime substrate + +## Evidence / pattern + +The current evolvable tree controls **which execution capability is disclosed**, and the WikiSkill-inspired maintenance loop controls **how experience compounds across sessions**. Neither mechanism makes the current state of one long-running coding task explicit. A long task can therefore still reconstruct current branch/check/hypothesis/change status from an append-only conversation even when routing itself is minimal. + +Badhe, Tiwari, and Chung's *SKILL.state: Scalable Long-Horizon Agent Skills* (arXiv:2608.26263v2) isolates this problem. Its runtime invokes the model with immutable procedure `P`, structured current state `Σ`, and only the latest observation `O`; a deterministic runtime validates a merge patch, applies null-deletion semantics, executes the action only after accepting the transition, and does not replay the transient reasoning trace on the next step. + +The branch's last complete release comparison (`b202f7a165ae3ea4404d404bb1235ebf4270cbfb`) passed delivered quality but reported higher average token, duration, and tool-call cost than frozen v1.5. That result does not prove history growth caused the overhead, but it makes bounded long-horizon state a testable cost mechanism rather than a new task taxonomy. + +## Hypothesis + +When a coding task spans enough tool rounds that the next action depends on facts produced earlier, a compact validated coding-domain execution state will reduce reconstruction, stale-fact drift, repeated hypotheses, and irrelevant-tool-output carryover without changing the automatic route. + +The useful preload/activation signal is **state pressure**, not a domain noun: + +- the next action depends on current facts from at least two earlier observations; +- a latest observation can invalidate a previously stored branch/check/contract fact; +- repeated hypotheses or checks must be remembered to avoid cycling; or +- replaying raw tool output would otherwise be required to recover the current work surface. + +Short tasks remain stateless. State projection is cross-cutting runtime infrastructure, not `Core -> State`, not a retrieval level, and not a manual mode. + +## Change + +Freeze a deterministic contract benchmark first, then add one zero-dependency runtime adapter and the minimum Skill wording needed to activate it: + +1. one coding-domain schema for objective/success, route, working set, facts, hypotheses, change surface, verification, next action, and bounded history artifacts; +2. JSON Merge Patch behavior where omitted keys survive and `null` deletes obsolete keys; +3. strict schema/type/size validation before canonical mutation; +4. rollback on invalid patch and no execution of its proposed action; +5. prompt construction from procedure + current state + latest observation only; +6. explicit refusal to persist reasoning traces, transcripts, or raw tool output; +7. a history escape hatch for audit/provenance tasks, dynamic-schema discovery, and observations whose future relevance is still uncertain; +8. a JSON runtime-input envelope and explicit observation/control ownership so delimiter-like or instruction-like tool data cannot become a new prompt section or mutate host-owned task/route fields. + +Do not add an automatic tree node or change the Debugging/Implementation boundary. + +## Expected result + +Required correctness gates: + +- nested partial updates preserve omitted siblings; +- explicit `null` removes obsolete entries; +- invalid patches never mutate canonical state or release their action; +- state cannot contain transcript/reasoning/raw-tool-output fields; +- irrelevant telemetry does not enter later state; +- a corrective observation can replace a stale fact in the same transition; +- the state remains within a fixed byte budget across horizons 10, 50, and 200; +- runtime input with Markdown fences, fake headings, or JSON-looking text round-trips as JSON data without escaping the prompt boundary; +- the prompt declares model-owned versus host-owned fields, while runtime validation remains authoritative; +- manual Decision/Clarification isolation and current automatic topology remain unchanged. + +Cost expectations are secondary: the deterministic history baseline should grow with horizon while the state prompt remains bounded by schema and latest-observation size. This does **not** establish the paper's LLM accuracy/token results and does not justify an `O(1)` claim for hosts that continue appending prior messages underneath the Skill. + +## Frozen validation + +- Immediate parent / baseline ref: `13c8a252121d92ab47548016ae1ee39bcafcd149`. +- Original deterministic benchmark: `benchmarks/skill_state_validation.py` plus `benchmarks/test_skill_state_runtime.py`, committed before the initial runtime implementation. +- Review hardening: prompt-envelope/control-ownership tests were added before the corresponding runtime change. +- Deterministic horizons: 10, 50, 200; 20 irrelevant telemetry events per turn; fixed four-slot coding state. +- Existing regression gate: repository unit-test suite, tree topology self-test, manual-only layout check, and explicit evolution workflow contract. +- Model-backed mechanism iteration: current tree benchmark at `n=1`, compared with the immediate parent on identical tasks/scorer/model/harness. +- Release gate after wording/topology freeze: complete `n=3` adaptive/baseline/no-skill and dedicated full-history/state-shadow/state-history-free state matrix; required delivered quality and manual-mode discipline cannot regress. Cost may only break a quality tie. + +## Result + +The deterministic implementation and repository contract gates passed on the candidate lineage: the CI validation job completed successfully with 107 repository tests plus the tree, evolution-workflow, execution-state, manual-only, prompt-reference, and legacy-runtime checks. The review then found one uncovered adapter boundary: a Markdown-fenced state block did not clearly separate untrusted observation text from control text, and the prompt did not state the same host/model ownership rule enforced by the validator. + +The candidate now serializes all runtime input as one compact JSON value, declares observation trust and field ownership, withholds action until the successor state validates, and extends the deterministic benchmark with delimiter/control-boundary round-trip tests. The local focused suite passes all 19 execution-state tests and the expanded deterministic contract. + +No model-backed long-horizon state comparison has completed after this runtime-prompt change. The earlier evolvable-tree n=3 result is historical non-regression evidence for the tree, not proof that execution-state projection improves quality or cost. + +## Decision + +`accept-experimental`: retain execution state as an experimental cross-cutting substrate because its deterministic contract and topology/manual-mode isolation pass. Do not promote its efficiency claims into the release contract until the frozen model-backed state gate completes without quality regression. + +## Follow-up + +Run the protocol in `benchmarks/SKILL_STATE_MODEL_GATE.md`. Reject or revise the candidate if the coding schema repeatedly needs ad-hoc fields, if state pressure cannot be detected before history reconstruction, if valid-but-premature overwrite/deletion loses future-relevant facts, if explicit state causes quality/manual-routing regression, or if a host cannot actually exclude prior messages and the projection adds cost without reducing reconstruction. diff --git a/evolution/rejected/execution-state/experiments/skill-state-shadow-diagnostic-gate-20260903.md b/evolution/rejected/execution-state/experiments/skill-state-shadow-diagnostic-gate-20260903.md new file mode 100644 index 0000000..313453d --- /dev/null +++ b/evolution/rejected/execution-state/experiments/skill-state-shadow-diagnostic-gate-20260903.md @@ -0,0 +1,83 @@ +# Execution-state gate-role correction after the first remediated n=1 run + +Status: `candidate-pending-fresh-rerun` + +## Evidence + +The remediated standard `n=1`, `workers=1` matrix produced 24 determinate cells: + +- `full-history`: 6/6; +- `state-history-free`: 6/6; +- `no-skill-full-history`: 6/6; +- `state-shadow`: 5/6. + +The history-free arm also passed the final client transport audit and the history-pointer check. The published analyzer nevertheless returned: + +- `quality_gate = FAIL`; +- `state_semantic_gate = FAIL`; +- `execution_state_model_gate = FAIL`. + +Inspection of the analysis implementation showed that both blocking gates pooled `state-shadow` with `state-history-free`. This conflicts with the frozen protocol: history-free is the release candidate, full-history is its ablation baseline, and state-shadow is a reconstruction diagnostic that cannot establish bounded context. + +The same run reported a negative short-horizon cost signal for history-free relative to full-history: + +- uncached input tokens: 78,118 versus 50,736; +- cell-duration sum: 172.60 seconds versus 154.57 seconds. + +These `n=1` values are diagnostic only. They do not justify a token/latency claim or an immediate Skill/schema/tree change, and the bounded 10/25/50/100 profile has not yet run. + +## Causal claim + +The overall FAIL is not evidence that the history-free candidate regressed: its six candidate cells passed. It is caused by a gate-role error that lets a non-blocking diagnostic arm veto the candidate and by a formal-status rule that does not enforce the protocol's `n>=3` release requirement. + +## Atomic proposal + +Change only the analysis/reporting contract: + +1. compare `state-history-free` with `full-history` in the blocking quality gate; +2. evaluate blocking state semantics and history pointers only for `state-history-free`; +3. report `state-shadow` quality/state/pointer outcomes under a separate non-blocking diagnostic gate; +4. require a complete paired, determinate standard four-arm matrix at `n>=3` before formal cost claims; +5. keep `execution_state_model_gate` pending until candidate quality/transport, repeated evidence, token, latency, and bounded-context gates are all resolved; +6. expose the exact shadow semantic failure details in generated reports. + +Do not change: + +- `SKILL.md` or any reference node; +- automatic tree topology; +- execution-state schema or canonical validators; +- case prompts or required evidence; +- scorer normalization or artifact rules; +- raw benchmark results. + +## Expected benefit + +The report will distinguish three independent questions: + +- does history-free preserve quality and state semantics?; +- does shadow reveal a state-update problem while history remains available?; +- after `n>=3` and horizon runs, does history-free reduce token/time cost while keeping the client request bounded? + +This prevents a diagnostic shadow failure from being misreported as a history-free release failure while preserving the failure as evidence. + +## Falsifier + +Reject this change if the frozen protocol explicitly defines state-shadow as a release-blocking candidate, or if the revised analysis can hide a failure in `state-history-free`, an indeterminate required cell, a missing four-arm repetition, a transport failure, or an invalid history pointer. + +## Validation plan + +Before any new model call: + +- add deterministic tests showing a shadow-only failure remains visible but non-blocking; +- add tests showing a history-free failure remains blocking; +- add tests showing `n=1` cannot produce a formal pass or cost claim; +- add tests for paired/determinate `n>=3` completeness; +- run existing execution-state and benchmark unit tests plus all deterministic self-tests. + +After the patch is frozen: + +1. re-analyze the old `n=1` result only as historical diagnostic evidence; do not relabel it as a formal pass; +2. run a fresh complete standard `n=1` matrix under the new analysis identity; +3. run bounded `n=1` at horizons 10/25/50/100 to locate request growth and any cost crossover; +4. proceed to standard `n>=3` only if the candidate gates remain clean and the bounded run justifies the expense; +5. modify activation/schema/Skill only from repeated mechanism-level evidence. Do not modify the router tree unless failures independently cluster on an observable execution-boundary signal. diff --git a/evolution/rejected/execution-state/reference/SKILL_STATE.md b/evolution/rejected/execution-state/reference/SKILL_STATE.md new file mode 100644 index 0000000..89243e0 --- /dev/null +++ b/evolution/rejected/execution-state/reference/SKILL_STATE.md @@ -0,0 +1,194 @@ +# Execution state for long-running coding skills + +This document adapts the runtime mechanism from Badhe, Tiwari, and Chung, *SKILL.state: Scalable Long-Horizon Agent Skills* (arXiv:2608.26263v2), to Practical Coding. It does not turn the paper's reported model results into project claims. It defines a candidate architecture, deterministic state and host contracts, and the model-backed evidence still required here. + +## Four separate concerns + +Practical Coding distinguishes four mechanisms that must not be collapsed into one router: + +| Mechanism | Question it answers | Lifetime | +|---|---|---| +| Local router tree | Which execution capability is needed for the present blocker? | Current task | +| Retrieval policy | Which source evidence is needed and how broadly should it be collected? | Current evidence need | +| Execution state | What is currently true and needed for the next action? | Current multi-step run | +| Evolution wiki | Which repeated lessons should change the Skill after validation? | Across sessions/releases | + +Execution state is therefore a **runtime substrate**, not an automatic child, retrieval mode, or manual workflow. Activating it does not increase tree depth and cannot make Decision or Clarification automatic. + +## Runtime transition + +A state-aware host should construct each model invocation from only: + +- immutable loaded Skill procedure `P`; +- validated current execution state `Σt`; +- latest observation `Ot` after the host has processed any explicit user control change; +- one bounded host validation error only when retrying a rejected transition. + +The model returns exactly one runtime payload: + +```json +{ + "state_patch": { + "facts": {"current_head": "def456"}, + "next_action": "run the focused check for def456" + }, + "action": "python -m unittest tests.test_release" +} +``` + +The runtime validates the complete successor state before releasing `action` to the host. Omitted patch keys survive, and `null` deletes an obsolete optional entry. Malformed JSON, duplicate object keys, `NaN`/infinity, invalid UTF-8, an oversized input or state, an unexpected output key, a wrong type, a forbidden field, or an illegal router path rejects the whole transition. Rejection leaves the caller-owned canonical state unchanged, and the CLI does not overwrite its output file or print the proposed action. + +A validated state transition does **not** authorize the action. The helper never executes it; it only returns or prints a proposal after state validation. The surrounding host must independently validate the tool, arguments, permissions, working directory, and side effects before execution. A bounded retry resends the same `P + Σt + Ot` with a compact validation error, remains capped, and starts from the original canonical state. + +Reasoning may occur inside one model invocation, but it is transient computation. Do not place chain-of-thought, transcript copies, full tool output, or an append-only action diary in `Σ`. + +## Trust and control boundary + +`runtime/skill_state.py` serializes `procedure`, an isolated validated state snapshot, and `latest_observation` inside one compact JSON envelope rather than a Markdown code fence. Newlines, section headings, backticks, and JSON-looking text inside an observation remain JSON string data and cannot structurally terminate the envelope or create a second prompt section. + +That framing is a structural boundary, not a semantic prompt-injection proof. Observation text can still influence a model, and a model can still propose an unsafe action. The three inputs therefore have different intended authority: + +- `procedure` is host-supplied and authoritative for the transition; +- `state` is an isolated validated snapshot of the canonical current state; +- `latest_observation` is untrusted evidence, not a control plane. + +An explicit new user instruction is not silently treated as untrusted tool data. The host first applies the authorized change to `objective`, `success`, or `route` through `host-apply`, then builds the next transition from the updated canonical state. + +The model may patch only `working_set`, `facts`, `hypotheses`, `change`, `verification`, `next_action`, and `history`. `schema_version`, `objective`, `success`, and `route` are host-owned. Runtime validation enforces this state-field boundary even if the prompt is ignored. It cannot prove that the model followed `procedure`, classified evidence correctly, or proposed a safe action; those remain model-quality and host-policy responsibilities. + +## Coding-domain schema + +`runtime/skill_state.py` uses one schema for coding tasks rather than generating a schema per task: + +| Field | Future-facing content | +|---|---| +| `objective`, `success` | Current outcome and observable completion conditions | +| `route` | Host-owned active automatic path, retrieval mode, and explicit manual mode | +| `working_set` | Current paths and symbols, not repository inventory | +| `facts` | Authoritative current facts that later actions need | +| `hypotheses` | Live and rejected hypotheses needed to avoid repetition | +| `change` | Current planned/applied change surface | +| `verification` | Pending checks and compact current outcomes | +| `next_action` | The single next useful action | +| `history` | Whether history is part of the task, plus bounded artifact pointers | + +The schema has a fixed total byte budget and per-container limits. Replace stale values instead of appending versions. Store an evidence pointer or compact outcome instead of raw output. + +Schema validation catches malformed structure; it cannot prove that a semantically useful fact was not overwritten or deleted too early, that an `artifacts` string resolves to immutable evidence, or that a user-required audit trail remains sufficient. A host with a hard provenance requirement must enforce that requirement outside model-controlled state as well. The model-backed gate therefore measures premature loss, stale-fact recovery, repeated work, invalid-patch retries, and delivered task quality rather than treating a syntactically valid patch as sufficient. + +## Activation and exit + +Do not create state for a short direct edit. Activate projection only when **state pressure** appears: + +- the next action depends on current facts from multiple earlier observations; +- a new observation can invalidate a stored branch, check, contract, or environment fact; +- hypotheses or checks are beginning to repeat; +- reconstructing the current work surface would require replaying raw output. + +Exit state mode when the task completes or collapses back to one self-contained action. State activation does not change the selected tree node. Debugging remains Debugging; Implementation remains Implementation; Core remains Core. + +## When history must remain available + +Explicit state is not assumed lossless when: + +- the relevant schema is still being discovered dynamically; +- an earlier observation may matter later but has not yet been classified; +- the requested output is an audit, provenance reconstruction, or explanation of past actions; +- multiple writers can update shared state without deterministic conflict resolution. + +For those cases, set `history.required=true` and keep bounded references to immutable artifacts. Do not copy the artifacts into state. The validator checks only the bounded JSON shape; the host or artifact store must establish immutability, authorization, retention, and successful resolution of each pointer. + +`_atomic_write_json()` prevents partial replacement of one local JSON file, but it is not compare-and-swap. Multiple hosts can still overwrite one another with individually valid snapshots. `build_prompt()` deep-copies and validates one isolated snapshot before serialization, but that snapshot is not a lock or revision check. A concurrent integration must add a revision/CAS or single-writer ownership rule before sharing one state file. + +## Audited history-free host boundary + +`build_prompt()` deliberately accepts no conversation-history argument, but that local API shape alone cannot prove what the surrounding SDK or product sends. `runtime/skill_state_host.py` adds an explicit transport-facing boundary for the `state history-free` arm. + +A prepared request places the frozen procedure and transition contract in the +current top-level `instructions` field. Its single current user input contains +only canonical JSON for validated state, latest observation, and optional bounded +validation feedback. This separates the authoritative procedure from untrusted +evidence at the request-role boundary. The host rejects: + +- `previous_response_id` and equivalent parent/response handles; +- `conversation`, thread, session, or context-management handles; +- prompt references or server-managed prompt state; +- prior assistant or tool input items; +- nested option fields that can import previous context. + +It also explicitly fixes `store=false`, `stream=false`, `background=false`, and `truncation="disabled"`; freezes the model, procedure, tools, options, limits, and request contract in a self-digested manifest; and applies hard limits to every variable request component and the final canonical request body. + +The audit records hashes and byte sizes for the exact request body, procedure, state, observation, tools, and options. `audit_wire_request_against_manifest()` rejects any drift. A manifest-free audit validates only one body and is not eligible for a trajectory-level bound; the manifest-matched audit supports only a **client-visible serialized-request-body** claim. A caller must send the prepared bytes unchanged. If an SDK rebuilds the body, or the transport attaches a context-bearing header, cookie, proxy session, or other state out of band, the final outbound request must be captured and rechecked; otherwise the run is state shadow, not demonstrated history-free execution. + +The host can bound `latest_observation` but cannot prove that it is truly the latest observation rather than a relabeled history dump. The benchmark must freeze and identify the observation injector and retain the per-step observation hash. + +The history-free boundary is about model context composition, not data-retention or privacy guarantees. `store=false` means this request does not ask the Responses API to store the generated response for later retrieval; provider logging, abuse monitoring, and retention policies are separate concerns. + +On a rejected model transition, the host retries from the unchanged original state with the same procedure and latest observation plus one bounded deterministic validation error. It does not append the rejected response. On acceptance, the successor must be durably persisted before the action proposal is returned. Persistence success is still not action authorization; the product's ordinary tool and side-effect policy remains mandatory. + +See [`SKILL_STATE_HOST.md`](SKILL_STATE_HOST.md) for the full request schema, limits, CLI, integration example, manifest contract, and benchmark handoff. + +## What “bounded” may mean + +With the default hard caps, each client request has a fixed byte ceiling composed of bounded procedure, state, latest observation, validation feedback, tools, options, and wrapper overhead. The ceiling does not depend on the number of preceding task steps. This supports the statement: + +> The captured client-visible input for each audited history-free step is bounded with respect to task horizon. + +It does **not** support these stronger statements without further evidence: + +- total token use for an entire `T`-step task is constant; +- provider-internal context is known or bounded by the client audit; +- state always preserves every future-relevant fact; +- state reduces tokens or latency on real tasks. + +Even when every step is bounded, cumulative input over `T` steps is still expected to grow with `T`. Actual provider-reported tokens and end-to-end time must be measured in the paired model gate. + +## Host boundary CLI + +The state helper remains the canonical schema/transition CLI: + +```powershell +python runtime/skill_state.py init ` + --objective "Repair release validation" ` + --success "Focused release check passes" ` + --output "$env:TEMP\practical-coding-state.json" + +python runtime/skill_state.py validate "$env:TEMP\practical-coding-state.json" +``` + +The history-free helper can build and re-audit an offline request, but never sends it: + +```powershell +python runtime/skill_state_host.py build ` + --model "gpt-5.6-luna" ` + --procedure procedure.txt ` + --state "$env:TEMP\practical-coding-state.json" ` + --observation observation.txt ` + --request-output request.json ` + --audit-output request-audit.json ` + --manifest-output host-manifest.json + +python runtime/skill_state_host.py audit request.json ` + --manifest host-manifest.json ` + --output request-reaudit.json +``` + +Keep ephemeral state and raw request/response artifacts outside the target repository unless the user explicitly requests a durable, reviewable artifact. + +## Validation + +The deterministic checks cover parser strictness, isolated snapshots, merge/deletion mechanics, rollback, schema and input budgets, router ownership, JSON-envelope round trips, the rule that a rejected CLI transition does not expose its action or overwrite its output, and the audited one-current-input/no-history host boundary. + +The existing synthetic contract demonstrates bounded state under one fixed hand-authored update schedule and that the merge mechanism permits an immediate stale-value replacement. The host tests demonstrate request shape, manifest identity, retry rollback, and persistence-before-release. Neither demonstrates that a model will ignore distractor telemetry, detect a corrective observation, retain every future-relevant fact, resist semantic prompt injection, or choose an authorized action. + +```powershell +python -m py_compile runtime/skill_state.py runtime/skill_state_host.py +python -m unittest tests.test_skill_state_hardening tests.test_skill_state_host +python -m unittest benchmarks.test_skill_state_runtime +python benchmarks/skill_state_validation.py --self-test ` + --output benchmark-results/skill-state-contract.json +``` + +Because the runtime prompt and Skill wording affect model behavior, the existing model-backed tree benchmark must still run under the normal `n=1` iteration and frozen `n=3` non-regression policy before release promotion. + +The dedicated comparison protocol is in [`../benchmarks/SKILL_STATE_MODEL_GATE.md`](../benchmarks/SKILL_STATE_MODEL_GATE.md). It separates full-history, state-shadow, and true history-free `P + Σ + O` arms so a cost or bounded-context claim cannot be inferred from deterministic byte limits alone. diff --git a/evolution/rejected/execution-state/reference/SKILL_STATE_HOST.md b/evolution/rejected/execution-state/reference/SKILL_STATE_HOST.md new file mode 100644 index 0000000..2cc91b9 --- /dev/null +++ b/evolution/rejected/execution-state/reference/SKILL_STATE_HOST.md @@ -0,0 +1,232 @@ +# Audited history-free host boundary + +`runtime/skill_state_host.py` is the transport-facing companion to +`runtime/skill_state.py`. The state runtime validates `Σ` and transitions; the +host boundary constructs one bounded request from `P + Σ + O`, audits the exact +serialized request, and releases a proposed action only after the successor state +has been persisted. + +The module is deliberately transport-agnostic and zero-dependency. It does not +call a model, execute tools, persist credentials, or define benchmark cases. A +benchmark or product host supplies the byte transport and its normal action-policy +boundary. + +## Claim boundary + +A request is eligible for a **client-visible history-free request** claim only +when all of the following are true: + +- the immutable procedure and transition contract are carried by the current + top-level `instructions` field; +- the body contains exactly one current `user` input item with one `input_text` + block whose complete text is the state/observation JSON data object; +- `previous_response_id`, `conversation`, prompt references, + `context_management`, prior assistant/tool items, and equivalent nested history + handles are absent; +- `store=false`, `stream=false`, `background=false`, and + `truncation="disabled"` are explicit; +- the procedure, tools, options, limits, and request-shape contract match one + frozen manifest; +- procedure, state, latest observation, validation feedback, tools, options, and + the complete serialized request remain within hard byte limits; +- the exact audited bytes are the request-body bytes delivered to the HTTP transport; +- the transport does not attach a context-bearing cookie, session/conversation + header, proxy memory handle, or another out-of-band history channel. This last + property must be established by the integration or captured outbound request, + not inferred from the body audit. + +The caller must also establish that `latest_observation` is the current bounded +observation supplied by the frozen observation injector, not a relabeled transcript +or concatenation of prior turns. The byte boundary can cap that field but cannot +classify its semantic provenance. + +Passing this audit does not prove provider-internal behavior, privacy, zero data +retention, semantic resistance to prompt injection, delivered task quality, or a +token/time improvement. It establishes only the composition and fixed byte bound +of the captured client request. If an SDK reconstructs the body, attaches session +state, or converts it into another request, audit that final serialized body +instead. + +## Frozen request contract + +`HistoryFreeHost` freezes: + +- model identifier; +- immutable procedure hash; +- canonical tools hash; +- canonical options hash; +- all configurable limits and fixed component hard limits; +- the current-instructions/one-current-input/no-history request contract. + +`manifest()` returns this data plus a self-digest. Every prepared request records +the manifest digest and hashes for the complete request, procedure, state, +observation, tools, and options. `audit_wire_request_against_manifest()` rejects +identity or limit drift. + +The runtime hard caps are: + +| Surface | Hard cap | +|---|---:| +| Model identifier | 256 bytes | +| Current instructions | 80 KiB | +| Canonical execution state | 16 KiB | +| Procedure | 64 KiB | +| Latest observation | 64 KiB | +| Validation feedback | 2 KiB | +| Frozen options | 16 KiB | +| Frozen tools | 96 KiB | +| Complete request body | 320 KiB | +| Raw response body | 4 MiB | +| Transition attempts | 3 | + +A `HistoryFreeLimits` instance may tighten these caps for a frozen run; it cannot +raise them. + +## Request construction + +```python +from runtime.skill_state import initial_state +from runtime.skill_state_host import HistoryFreeHost + +state = initial_state( + "Repair the release check", + ["The focused release check passes"], +) + +host = HistoryFreeHost( + model="gpt-5.6-luna", + procedure="", + options={ + "max_output_tokens": 4096, + "reasoning": {"effort": "medium"}, + }, +) + +manifest = host.manifest() +prepared = host.prepare_request( + state, + "The latest focused check failed at release.py:41.", + step_id="case-01/step-03", +) + +# Send prepared.wire_bytes unchanged. Do not rebuild the request through an SDK. +request_sha256 = prepared.audit["request_sha256"] +``` + +The request uses the Responses-compatible JSON shape but does not require the +OpenAI SDK. The procedure is serialized into the current `instructions` field, +while the single user input contains only canonical JSON for `state`, +`latest_observation`, and optional bounded `validation_error`. This gives the +procedure an instruction-level boundary instead of placing trusted procedure and +untrusted observation in the same user message. + +Avoid rebuilding the prepared body through an SDK: SDK-managed conversation or +prior-response state would make the history-free claim un-auditable unless the +final outbound body is intercepted and rechecked. + +## Transition loop + +`run_transition()` accepts a caller-supplied byte transport: + +```python +from pathlib import Path + +from runtime.skill_state_host import TransportResponse + + +def transport(body: bytes) -> TransportResponse: + # The integration must send exactly `body` and retain the final outbound + # bytes plus raw response in its benchmark artifact store. + ... + + +def persist_successor(successor: dict) -> None: + # Use one durable atomic write, revision/CAS, or a single-writer store. + ... + + +result = host.run_transition( + state, + "The latest focused check failed at release.py:41.", + transport=transport, + persist_successor=persist_successor, + step_id="case-01/step-03", + max_attempts=2, +) + +# `result.action` remains untrusted. Apply the product's normal tool, +# argument, permission, working-directory, side-effect, and user-consent policy. +``` + +Each retry starts from the same original canonical state and receives only the +same `P + Σ + O` plus one bounded host-generated validation error. A rejected +transition never persists state or releases its action. A valid successor is +passed to `persist_successor` before its action is returned. If persistence +fails, the action is withheld. The persistence callback itself must not report +success before the new snapshot is durable; the runtime cannot roll back an +external store that commits and then raises. + +The returned attempt records include: + +- request/response hashes and byte sizes; +- manifest, procedure, state, observation, tools, and options hashes; +- request ID when supplied by the transport; +- input, cached input, uncached input, output, and total token usage when present; +- transport elapsed time; +- transition status and bounded validation feedback; +- accepted successor-state and action hashes. + +Raw request and response bodies are intentionally not retained by the runtime. +The benchmark transport must store them in its own access-controlled artifact +location before publishing only redacted summaries. + +## Offline build and audit + +The CLI builds requests but never sends them: + +```powershell +python runtime/skill_state_host.py build ` + --model "gpt-5.6-luna" ` + --procedure procedure.txt ` + --state state.json ` + --observation observation.txt ` + --options options.json ` + --tools tools.json ` + --request-output request.json ` + --audit-output request-audit.json ` + --manifest-output host-manifest.json + +python runtime/skill_state_host.py audit request.json ` + --manifest host-manifest.json ` + --output request-reaudit.json +``` + +An `audit` run without `--manifest` validates only one request body and reports +`bounded_context_eligible=false`. Supplying the self-validating manifest proves +that the saved body matches the frozen client contract and reports eligibility for +the limited client-body bound. The model benchmark still has to show that the same +bytes reached its actual transport boundary, that no contextual headers or proxy +session state were added, and that provider-reported token usage was captured. + +## Non-benchmark validation + +```powershell +python -m py_compile runtime/skill_state.py runtime/skill_state_host.py +python -m unittest tests.test_skill_state_hardening tests.test_skill_state_host +``` + +These checks cover request shape, manifest identity, hard limits, no-history +controls, retry rollback, persistence-before-release, token metadata, and CLI +round trips. They do not run a model and cannot establish quality, token, or +latency benefit. + +## Benchmark handoff + +The remaining work is the model-backed protocol in +`../benchmarks/SKILL_STATE_MODEL_GATE.md`. The harness should use this host for the +`state history-free` arm, record every exact request/response at the transport +boundary, and compare it with frozen full-history, state-shadow, and no-skill +arms. Do not change Skill wording or router topology in response to a single +result. First publish the paired report; then treat quality loss, state loss, +retry rate, token reduction, and latency as separate mechanisms when deciding +whether to keep, revise, or reject the substrate. diff --git a/evolution/rejected/execution-state/reference/SKILL_STATE_INVARIANTS.md b/evolution/rejected/execution-state/reference/SKILL_STATE_INVARIANTS.md new file mode 100644 index 0000000..cd5d091 --- /dev/null +++ b/evolution/rejected/execution-state/reference/SKILL_STATE_INVARIANTS.md @@ -0,0 +1,35 @@ +# Execution-state semantic invariants + +`runtime/skill_state.py` validates both the fixed JSON shape and a small set of +cross-container invariants that cannot be expressed by independent field types. +These checks apply to initial state, model patches, host patches, transitions, +prompt construction, and the direct CLI. + +## Hypothesis lifecycle partition + +A hypothesis ID may occur in exactly one lifecycle partition: + +- `hypotheses.active`: still live and worth testing; +- `hypotheses.rejected`: disproved evidence retained to prevent repetition. + +The two exact-key sets must be disjoint. Moving `h-cache` from active to rejected +with JSON Merge Patch therefore requires one atomic patch: + +```json +{ + "hypotheses": { + "active": {"h-cache": null}, + "rejected": {"h-cache": "cache-disabled reproduction disproved it"} + } +} +``` + +Adding the rejected entry without deleting the active entry rejects the complete +successor. The previous canonical state remains unchanged and the transition's +action is not released. A bounded host retry may then request a corrected patch. +The runtime deliberately does not auto-move the entry because silently repairing +model output would hide a semantic transition error and weaken auditability. + +This is a validation tightening, not a state-shape change, so schema version 1 is +unchanged. It establishes no model-quality, token, latency, or bounded-context +claim; those remain subject to the frozen model gate. diff --git a/evolution/rejected/execution-state/reference/SKILL_STATE_MODEL_GATE.md b/evolution/rejected/execution-state/reference/SKILL_STATE_MODEL_GATE.md new file mode 100644 index 0000000..93674c6 --- /dev/null +++ b/evolution/rejected/execution-state/reference/SKILL_STATE_MODEL_GATE.md @@ -0,0 +1,107 @@ +# Model-backed gate for execution-state projection + +This protocol evaluates the SKILL.state-inspired runtime substrate without changing the automatic router topology. It is separate from the deterministic schema/merge contract and from the capability-ceiling analysis that evolves Core, Debugging, and Implementation. + +## Question + +For genuinely long coding tasks, can a compact validated current-state projection preserve or improve delivered quality while reducing reconstruction, stale-fact drift, repeated work, and accumulated context? + +The benchmark must not assume the paper's reported accuracy or token results transfer to this repository. + +## Frozen arms + +Run identical tasks, repository snapshots, model settings, tools, timeouts, scorer, and observation sequence under these arms: + +1. **Full history** — current tree and normal accumulated conversation; no execution-state projection. +2. **State shadow** — current tree plus validated state, while the host still retains history. This measures whether explicit state helps reconstruction, but it cannot support a bounded-context claim. +3. **State history-free** — current tree; every model step receives only immutable loaded procedure `P`, validated current state `Σ`, latest observation `O`, and a bounded validation error on retry. Prior messages are omitted by the host. +4. **No skill full history** — absolute-quality reference using the same task and repository evidence, not an execution-state ablation substitute. + +The automatic route, retrieval policy, available tools, and manual-mode rules remain identical in arms 1–3. Execution state must never appear in `TREE_TRACE`. + +## Case families + +Use real repositories and tasks long enough to create state pressure. The frozen suite should include multiple examples of each mechanism: + +- delayed dependency: a fact observed early becomes necessary many actions later; +- corrective observation: branch head, check status, contract, or environment fact changes and must replace stale state immediately; +- distractor noise: unrelated logs or repository events appear between relevant observations; +- repeated hypothesis pressure: the agent must remember rejected causes or checks without replaying raw output; +- coordinated implementation: current producers, consumers, change surface, and verification must stay synchronized; +- history-required control: audit/provenance or intentionally evolving-schema tasks where the state arm must preserve bounded artifact pointers or decline history-free execution. + +Do not encode a gold automatic node. Reuse the tree benchmark's delivered-quality scorer and topology-neutral route analysis. + +## Freeze discipline + +During iteration, run `n=1` only. Before a release comparison, freeze: + +- case prompts and repository commits; +- observation injector and noise schedule; +- model, reasoning level, tools, harness, timeout, and worker count; +- state schema, prompt adapter, retry cap, scorer, and all acceptance thresholds; +- baseline/candidate refs and complete artifact manifest. + +Then run the complete paired matrix at `n>=3`. A scorer fix invalidates all affected arms and requires rerunning them on identical evidence. + +## Required artifacts + +Retain per cell: + +- initial prompt, every actual model request, response, validation error, and accepted transition; +- canonical state snapshots and patches; +- actions/tool calls, tool outputs, final answer, workspace diff, and focused/full checks; +- prompt/input/output token counts, duration, retry count, and infrastructure status; +- route trace, manual-mode trace, and repository/model/harness provenance. + +Published summaries may redact machine paths or secrets, but the local audit trail must remain complete. + +## Metrics + +Quality gates cost: + +1. delivered correctness, required build/checks, safety, compatibility, and user constraints; +2. zero spontaneous Decision/Clarification activation and valid automatic parent-child paths; +3. no quality-affecting premature overwrite/deletion, stale control fact, or execution of a rejected transition; +4. invalid-patch rate, validation retries, repeated actions/checks/hypotheses, and stale-fact recovery steps; +5. per-step and cumulative prompt/input/output tokens, duration, and tool calls. + +Also report state JSON size and activation/exit timing, but do not optimize those proxies at the expense of delivered behavior. + +## Acceptance + +The state candidate is release-eligible only when the complete paired `n>=3` run has no infrastructure failure and: + +- state history-free does not regress delivered quality, safety, required checks, manual-mode discipline, or topology validity against full history; +- the deterministic state contract remains 100% passing; +- no rejected transition action is executed; +- any history-required case retains bounded immutable evidence pointers rather than silently discarding required provenance; +- cost is used only after the quality gate. At equal quality, prefer lower cumulative input tokens, then lower duration/tool cost; +- a bounded or horizon-independent prompt claim is made only for the history-free arm and only from the actual captured requests, not from a synthetic byte model. + +State shadow can be accepted as a reconstruction aid without making a bounded-context claim. A failed history-free arm is evidence to revise activation, schema, host integration, or the history escape hatch—not evidence to add execution state as a router child. + +## Interpretation + +Keep semantic state failures distinct from format failures: + +- format/schema/type/size error: deterministic validator or constrained generation issue; +- premature overwrite/deletion: schema or state-update-policy issue; +- stale fact not replaced: transition or observation-authority issue; +- repeated action/hypothesis: missing future-relevant state; +- quality loss with valid state: projection may be omitting information whose relevance was not predictable; +- no cost reduction while history remains attached: expected host-boundary limitation, not proof against the mechanism. + +Record rejected variants and mechanism-level lessons in `evolution/`; do not patch the router boundary merely to force a favorable state result. + +## Gate roles and formal status + +The four arms have different decision roles and must not be pooled into one undifferentiated veto: + +- **Blocking candidate:** `state-history-free`, compared directly with `full-history` for delivered quality, state semantics, immutable history pointers, and final outbound transport. +- **Diagnostic only:** `state-shadow`. Report its quality and state failures because they can reveal projection/update pressure, but do not let this arm veto a history-free candidate that independently passes. Shadow still participates in the requirement that a formal four-arm matrix be complete and determinate. +- **Absolute reference:** `no-skill-full-history`. It supplies context for the value and overhead of the Skill but is not the execution-state ablation baseline. + +An `n=1` run is iteration evidence. It may show a blocking candidate defect, but it cannot make the formal composite model gate pass. A formal decision requires a complete paired standard matrix at `n>=3`. The stronger claim evaluated by `execution_state_model_gate` additionally requires the frozen token, latency, and bounded-context gates; those claims remain separate in the report so quality preservation is not confused with efficiency or horizon independence. + +Changing these role definitions or any acceptance threshold invalidates the affected formal result. Preserve the old result as evidence, freeze the new analysis identity, and rerun rather than relabeling an existing matrix. diff --git a/evolution/skills/README.md b/evolution/skills/README.md new file mode 100644 index 0000000..e27e8c5 --- /dev/null +++ b/evolution/skills/README.md @@ -0,0 +1,8 @@ +# Explicit evolution skills + +These are maintenance-time skills for Practical Coding itself. They are deliberately outside `SKILL.md`'s automatic runtime router tree and are never loaded for ordinary coding tasks. + +- `session-to-wiki/SKILL.md` — explicitly compile the current visible session into a sanitized raw receipt and persistent wiki knowledge without changing runtime Skill behavior. +- `evolve-skill/SKILL.md` — explicitly propose one wiki-informed runtime change, freeze/add its benchmark first, compare baseline and candidate on identical evidence, and accept only a non-regressing candidate. + +Invoke them by explicit maintenance request/name or by directly loading the relevant path. Do not add them as automatic children merely to improve discoverability; if discoverability becomes a problem, benchmark that maintenance problem separately. diff --git a/evolution/skills/evolve-skill/SKILL.md b/evolution/skills/evolve-skill/SKILL.md new file mode 100644 index 0000000..8a19b0e --- /dev/null +++ b/evolution/skills/evolve-skill/SKILL.md @@ -0,0 +1,46 @@ +--- +name: evolve-skill +description: "Explicit maintenance skill for proposing one wiki-informed Practical Coding change, adding/fixing its benchmark first, and accepting it only through a non-regression validation gate. Never activate automatically." +license: MIT +metadata: + author: Hubujiu + version: "1.0" +--- + +# Evolve Skill + +Activate this maintenance skill only when the user explicitly asks to evolve, optimize, refine, split, merge, deepen, collapse, or otherwise update Practical Coding from accumulated evolution knowledge. It is outside the automatic coding router tree. + +The wiki is persistent evidence. Runtime Skill changes are reversible candidates. + +## Preconditions + +- Work on an `experiment/*` branch or another explicitly designated evolution branch. +- Read `evolution/wiki/index.md` and `evolution/wiki/skill-impact.md` first. +- Read only the relevant wiki pages and evidence receipts/benchmark artifacts needed to diagnose the mechanism. +- Do not repeat a previously rejected intervention unless new evidence directly addresses its recorded failure mode. + +## Evolution Loop + +1. **Choose one atomic proposal.** Target one runtime skill/node/boundary or return `no_action`. Prefer a patch to an existing node over a new node when the current node is partially correct. +2. **Freeze the hypothesis before seeing candidate validation.** Create/update one file under `evolution/experiments/` containing: evidence pointers, causal claim, observable preload/activation signal, exact target, proposed patch shape, expected benefit, falsifier, baseline ref, benchmark plan, and acceptance criteria. Do not backfill the hypothesis after results are known. +3. **Freeze or add the benchmark before applying the runtime patch.** New behavior needs at least one positive case and one boundary/negative case when applicable. The case contract and deterministic scorer/oracle must agree. Do not tune a scorer to reward the candidate. +4. **Run the baseline on the frozen benchmark.** Record the exact commit/ref, model, harness, repetitions, cases, scorer version, and quality/cost metrics. +5. **Apply the smallest candidate patch.** Change only the targeted Skill/tree surface needed to exploit the wiki-supported signal. Keep the proposal atomic so rollback and causal attribution remain possible. +6. **Run the candidate on the same frozen benchmark.** Use the same model, harness, repetitions, cases, and scorer. Re-run the existing relevant regression suite as well. If a genuine scorer defect is discovered, fix it, invalidate both affected results, and rerun baseline and candidate from scratch. +7. **Gate on delivered quality, not route aesthetics.** Accept only if all required correctness/safety/reachability scores are not lower than baseline, the new benchmark score is not lower than baseline, spontaneous manual activation is not worse, and no required regression gate becomes indeterminate. For statistically noisy model benchmarks, require the repository's configured repeated-run/significance rule rather than accepting a single favorable sample. +8. **Rollback on any quality regression.** If a required score is worse or the gate is indeterminate, revert the runtime candidate. Keep valid raw receipts/wiki knowledge and the frozen benchmark; record the rejected diff and reason under `evolution/rejected/` and `evolution/wiki/skill-impact.md`. +9. **Record accepted impact.** If the gate passes, update `evolution/wiki/skill-impact.md` with proposal metadata, target, diff/commit, baseline and candidate scores, benchmark artifact, and `Accepted`. Append the outcome to `evolution/wiki/log.md` and update affected pattern status. + +## Non-Regression Rule + +A change is not complete because it sounds better. It is complete only after the candidate has been evaluated against the same frozen evidence as its baseline and every required quality gate is equal or better. Cost improvements may break a quality tie; cost savings never compensate for lower required correctness or safety. + +## Anti-Overfitting Rules + +- Training/calibration evidence may shape a proposal; held-out evidence gates it. +- Do not inspect held-out failures and then edit the candidate without starting a new iteration/hypothesis. +- Do not change route labels merely to match historical labels; use parent-versus-child capability lift and delivered quality. +- Do not promote a new child because a noun appears in the task; require an observable pre-load signal and independent lift. + +Finish with the hypothesis file, benchmark added/changed, baseline result, candidate result, accepted/rejected decision, and final runtime commit/ref. If rejected, state explicitly that the runtime patch was rolled back while wiki knowledge remained. diff --git a/evolution/skills/session-to-wiki/SKILL.md b/evolution/skills/session-to-wiki/SKILL.md new file mode 100644 index 0000000..112d042 --- /dev/null +++ b/evolution/skills/session-to-wiki/SKILL.md @@ -0,0 +1,40 @@ +--- +name: session-to-wiki +description: "Explicit maintenance skill for compiling the current visible session into sanitized persistent evolution knowledge. Never activate automatically during ordinary coding work." +license: MIT +metadata: + author: Hubujiu + version: "1.0" +--- + +# Session to Wiki + +Activate this maintenance skill only when the user explicitly asks to preserve, distill, consolidate, or write the current session into the Practical Coding evolution wiki. It is outside the automatic runtime router tree. + +## Scope + +The goal is not to archive chat. Convert useful execution experience into durable, auditable maintenance knowledge while keeping raw experience, wiki knowledge, and runtime Skill text separate. + +Use only visible session content and observable tool/results evidence. Do not reconstruct or store private chain-of-thought. + +## Procedure + +1. **Select evidence.** Keep only session events that reveal a reusable success strategy, failure mechanism, routing boundary, benchmark defect, or user correction relevant to Practical Coding evolution. +2. **Sanitize before persistence.** Remove secrets, credentials, private code, personal identifiers, and unrelated conversation. Replace sensitive specifics with coarse mechanism-level descriptions. Never copy the full transcript into the repository. +3. **Write one immutable receipt first.** Create a new file under `evolution/raw/sessions/` using `evolution/EXPERIENCE_SCHEMA.md`. Include the source pointer, outcome, affected capability/boundary, mechanism, supporting evidence, contradictions, and candidate lesson. Do not rewrite an older receipt to make a later hypothesis look stronger. +4. **Read the current wiki before consolidating.** Start with `evolution/wiki/index.md`, then inspect the few relevant pattern pages. Update an existing mechanism when possible; create a new page only for a distinct generalizable mechanism. +5. **Consolidate causally.** A wiki page must state the claim, observable pre-action trigger, supporting receipts, contradicting receipts, affected nodes/boundaries, candidate experiments, and current status. Prefer root cause and action pattern over surface wording. +6. **Update navigation and chronology.** Update `evolution/wiki/index.md` and append a concise entry to `evolution/wiki/log.md`, even when no new reusable pattern is created. +7. **Stop before runtime mutation.** This skill must not edit `SKILL.md`, automatic router references, or executable runtime behavior. If the accumulated wiki suggests a Skill change, report the candidate hypothesis and leave mutation to the explicit `evolve-skill` maintenance skill. + +## Quality Gate + +Before finishing, verify: + +- no raw transcript or secret was persisted; +- the receipt is immutable evidence, not a rewritten conclusion; +- wiki claims cite receipts or benchmark artifacts; +- supporting and contradicting evidence are both represented when present; +- no runtime Skill/router file changed. + +Finish with the receipt path, wiki pages changed, and whether a follow-up evolution hypothesis now has enough evidence to test. diff --git a/evolution/wiki/benchmark-cost-evidence.md b/evolution/wiki/benchmark-cost-evidence.md new file mode 100644 index 0000000..465d27e --- /dev/null +++ b/evolution/wiki/benchmark-cost-evidence.md @@ -0,0 +1,21 @@ +# Mechanism: quality and cost evidence must remain separate + +## Claim + +A wording change that appears cheaper at n=1 has not earned a cost claim. Cost acceptance requires a frozen paired repeated run, and a quality win must not be described as a cost win when tokens, duration, or tools regress. + +## Observable trigger + +- A candidate reaches the same or better delivered quality but consumes more recorded resources. +- n=1 and n=3 cost directions disagree. +- A topology simplification looks cheaper by context size but measured end-to-end work increases. + +## Supporting receipts + +- `2026-09-02-tree-n3-core-only-rejected.md`: collapsing both automatic leaves tied quality and regressed every recorded cost metric. +- `2026-09-02-tree-n1-bounded-evidence-qualified.md`: the wording candidate qualified quality at n=1; its cost numbers were explicitly diagnostic only. +- `2026-09-02-tree-n3-b202f7a-delivery.md`: paired n=3 established strict quality superiority while disproving the cost-improvement subhypothesis. + +## Application + +Freeze quality and cost acceptance before n=3. Report pass counts, trace/manual discipline, tokens, duration, and tools independently. Do not rerun an unchanged candidate merely to seek favorable variance. A later cost optimization begins with a new mechanism-level hypothesis and n=1 qualification. diff --git a/evolution/wiki/benchmark-oracle-contracts.md b/evolution/wiki/benchmark-oracle-contracts.md index 39e8d69..c135b8a 100644 --- a/evolution/wiki/benchmark-oracle-contracts.md +++ b/evolution/wiki/benchmark-oracle-contracts.md @@ -13,6 +13,16 @@ A deterministic scorer is invalid when it rewards behavior forbidden by the task - `trace-ttl-zero` explicitly required no change to sibling cache semantics, while the scorer required cache TTL zero to change from the seeded default behavior to zero. - `sa-sensitive-security` described the model interceptor chain and proved rejection-before-model-call, but the scorer accepted only the unspaced token `ModelInterceptor`. +- `evolution/raw/sessions/2026-09-01-tree-n1-oracle-defects.md`: two explicit Decision answers satisfied the manual trace and strongest-trade-off contract but failed only because the evidence group required `Trade-off:` with exact punctuation. +- The same receipt records a cancellation diagnosis that identified the real download side-effect boundary and existing cancellation test, while the scorer required UI/progress filenames not required by the prompt. +- `evolution/raw/sessions/2026-09-01-tree-n1-semantic-oracle.md`: the first correction still rejected a Chinese recommendation/strongest-trade-off answer and a cancellation answer that named `abort()`, existing test coverage, and the exact side-effect boundary without repeating the scorer's preferred identifiers. +- `evolution/raw/sessions/2026-09-01-tree-n1-path-normalization.md`: Windows backslashes caused a correctly loaded manual reference to fail enforcement, and a focused-suite result still failed an identifier-specific test-file group. +- `evolution/raw/sessions/2026-09-01-tree-n3-failed-oracle-and-topology.md`: the complete paired run rejected explicit `Decision: choose` recommendations, an exact blocked test outcome, and an authoritative cancellation boundary solely because they did not use preferred headings/success wording/UI-caller tokens. +- `evolution/raw/sessions/2026-09-01-tree-n1-leaves-recommend-inflection.md`: a complete leaf-topology run rejected `Recommend ...` while accepting only the noun `Recommendation`, despite independent option and trade-off evidence. +- `evolution/raw/sessions/2026-09-02-tree-n3-leaf-failed.md`: a complete paired run rejected authoritative `avifEncoder`/`runCommand` boundary evidence and an actually observed correct manual-reference read because their trace labels differed from preferred tokens. +- `evolution/raw/sessions/2026-09-02-tree-n1-outcome-field.md`: a complete n=1 rejected `Outcome: ... did not start` despite the required focused command and an explicit truthful result. +- `evolution/raw/sessions/2026-09-02-tree-n1-missing-trace.md`: a complete n=1 rejected a correct manual Decision answer solely because it omitted the benchmark-only footer, although command evidence proved the requested reference read. +- `evolution/raw/sessions/2026-09-02-tree-n1-retired-reference-observed.md`: observed trace recovery correctly rejected a retired runtime reference, while a separate ceiling answer showed that a concrete focused test method can satisfy the test-evidence act without repeating its class name. ## Affected nodes/boundaries @@ -22,7 +32,21 @@ Benchmark scorer/oracle contract only. These observations do not justify runtime - Add unit assertions that the canonical oracle preserves every explicit sibling/non-goal contract in the prompt. - Evidence groups may include semantically equivalent lexical forms when formatting is not part of the requirement. +- Test-file evidence groups should enumerate authoritative frozen-repository tests relevant to the requested boundary, not force a neighboring layer whose filename happens to look related. +- When response language is not part of the task contract, evidence groups should admit equivalent recommendation/trade-off labels in supported response languages; independent negative groups must still require both acts. +- When identifiers are not themselves the contract, accept the concrete operation and observed test behavior while retaining separate groups for the caller, authoritative function, side effect, and falsifying test. +- Normalize platform-dependent path separators at the scorer boundary before enforcing reference ownership. +- Score “inspected focused tests” as an evidence act (`focused`/`suite`/test path), leaving the separate falsifying-test group to require a concrete test proposal. +- Score an exact blocked/failed probe outcome as an outcome report; whether the environment was reachable is separate from whether the agent truthfully reported what happened. +- Treat `Decision: choose` as a recommendation act and `cost` as downside wording when independent groups still prove the compared alternatives and chosen option. +- Normalize ordinary inflections for semantic acts, such as the verb `recommend` and noun `recommendation`, when grammar is not part of the task contract. +- When command evidence proves the requested reference was read, use it alongside normalized trace identities; apply the same observation to reject undeclared manual-reference reads on automatic tasks. +- Analyzer input is adversarial benchmark output: an unknown selected node must become an invalid-trace diagnostic, never a report-generation crash. +- When a separate probe-command group proves execution, an explicit `Outcome:` field is valid result-reporting evidence even if its prose uses an unenumerated failure construction. +- Missing benchmark-only trace formatting may fall back to observed reference reads, but inactive references must remain invalid and an explicit reported trace must never be rewritten. +- A concrete authoritative test method may satisfy focused-test evidence when independent groups still require the owning transition, state, and failure mechanism. +- Do not require a neighboring caller when the answer identifies the authoritative boundary, cancellation operation, focused evidence, and falsifying test requested by the prompt. ## Current status -Applied to the n=1 iteration harness before the second candidate run. +Applied to the earlier progressive-tree harness and refined through three falsifying n=1 runs plus one complete failed n=3 run during the 2026-09-01 evolvable-tree iteration. Artifacts produced before the latest outcome-semantics correction are invalid for delivery comparison and remain only diagnostic evidence. diff --git a/evolution/wiki/index.md b/evolution/wiki/index.md new file mode 100644 index 0000000..ace4126 --- /dev/null +++ b/evolution/wiki/index.md @@ -0,0 +1,7 @@ +# Evolution wiki index + +- [progressive-tree-lessons](progressive-tree-lessons.md): Fixed numeric depth and symmetric specialist leaves did not earn their runtime cost; use observable parent-local triggers and parent-versus-child lift to evolve the tree. +- [benchmark-oracle-contracts](benchmark-oracle-contracts.md): A benchmark cannot gate skill evolution when prompt and scorer encode different contracts; normalize semantic, language, and platform-equivalent evidence before comparing baseline and candidate. +- [maintenance-trigger-isolation](maintenance-trigger-isolation.md): Session consolidation and skill evolution are maintenance actions, not runtime coding routes; keep raw evidence, persistent wiki knowledge, and reversible Skill candidates separate and explicitly triggered. +- [benchmark-cost-evidence](benchmark-cost-evidence.md): Treat n=1 cost as diagnostic only; accept cost claims only from a frozen repeated paired run, and preserve quality and cost conclusions separately when they diverge. +- [retired execution-state experiment](../rejected/execution-state/): The explicit state/history-free host preserved n=1 quality but increased uncached input and latency, added substantial maintenance surface, and did not address retrieval-output long tails; do not restore it without new independent evidence. diff --git a/evolution/wiki/log.md b/evolution/wiki/log.md new file mode 100644 index 0000000..1bc7c59 --- /dev/null +++ b/evolution/wiki/log.md @@ -0,0 +1,107 @@ +# Evolution log + +Chronological maintenance log. Keep entries short; detailed evidence belongs in receipts, benchmark artifacts, experiment files, and `skill-impact.md`. + +## 2026-09-01 — explicit WikiSkill maintenance loop + +- Added an immutable current-session receipt describing the requested WikiSkill-style maintenance loop. +- Consolidated the reusable mechanism into `maintenance-trigger-isolation.md`. +- Added standalone explicit maintenance skills `session-to-wiki` and `evolve-skill` outside the automatic runtime tree. +- Added `benchmarks/evolution_workflow_validation.py` to score isolation and non-regression-gate contracts. + +## 2026-09-01 — evolvable-tree first n=1 oracle review + +- Preserved the complete 106-cell exploratory artifact and an immutable receipt before editing the scorer. +- Classified three adaptive failures as prompt/oracle mismatches: punctuation-only trade-off matching and an unjustified neighboring-test filename requirement. +- Froze `tree-oracle-alignment-20260901.md`; the corrected harness must rerun the complete current-only n=1 matrix before runtime topology decisions. + +## 2026-09-01 — evolvable-tree semantic oracle review + +- Preserved the second complete 106-cell artifact and a new immutable receipt before editing the scorer again. +- Classified the remaining two adaptive failures as language/identifier oracle mismatches; both answers satisfied the semantic task and trace contracts. +- Froze `tree-oracle-semantic-equivalence-20260901.md`; the latest scorer retains negative requirements for a concrete test and strongest trade-off. + +## 2026-09-01 — evolvable-tree scorer normalization + +- Preserved the third complete 106-cell artifact and an immutable receipt before the scorer patch. +- Traced one false manual failure to Windows path-separator handling and one false diagnosis failure to identifier-specific focused-test evidence. +- Froze `tree-scorer-normalization-20260901.md`; added positive/negative Windows-reference tests and retained a no-test negative control. + +## 2026-09-01 — evolvable-tree n=1 candidate qualification + +- The normalized harness completed 106/106 determinate cells with adaptive 15/15, trace 15/15, explicit manual 2/2, and zero spontaneous manual activation. +- Preserved `2026-09-01-tree-n1-qualified.md`; froze runtime, topology, cases, and scorer for paired n=3 comparison. +- Deferred n=1 node-removal suggestions because capability minima varied across repetitions; stable n=3 parent-versus-child evidence will decide staged-node retention. + +## 2026-09-01 — paired n=3 release-gate failure + +- Preserved the complete 408-cell artifact; adaptive 40/45 did not beat frozen v1.5 at 44/45, so the candidate was rejected for delivery. +- Classified five adaptive failures as presentation/outcome/caller oracle mismatches while retaining independent negative controls for substantive evidence. +- Froze `tree-oracle-outcome-semantics-20260901.md`; the corrected scorer must return to a fresh current-only n=1 before any topology mutation or paired rerun. + +## 2026-09-01 — outcome semantics qualified; staged depth removed + +- The corrected current-only n=1 completed 106/106 determinate cells with adaptive 15/15, trace 15/15, manual 2/2, and zero spontaneous manual activation. +- Combined that clean rerun with the earlier repeated paired ablation: no staged depth-2 node was minimum-sufficient in either artifact. +- Froze `tree-remove-unearned-depth2-20260901.md`; removed only the four unearned staged descendants and kept Debugging/Implementation as leaves. + +## 2026-09-01 — leaf-topology recommendation inflection + +- The first complete leaf n=1 was 58/58 determinate; all ceilings passed, but adaptive was 14/15 because `Recommend` was not accepted as a recommendation act. +- Preserved the artifact and froze `tree-oracle-recommend-inflection-20260901.md`; added a positive verb-form test while retaining the missing-trade-off negative control. + +## 2026-09-01 — leaf candidate n=1 qualification + +- The corrected leaf-topology run completed 58/58 determinate cells with adaptive 15/15, every capability ceiling 13/13, trace 15/15, manual 2/2, and zero spontaneous manual activation. +- Preserved `2026-09-01-tree-n1-leaf-candidate-qualified.md` and froze the leaf candidate for a fresh complete paired n=3 comparison. + +## 2026-09-02 — leaf candidate paired n=3 rejected + +- Preserved the complete 252-cell artifact; adaptive 41/45 was below frozen v1.5 at 44/45 and no-skill at 45/45. +- Three failures were evidence-identity oracle defects; one loaded a retired depth-2 reference, and the analyzer crashed instead of classifying that invalid node. +- Froze `tree-evidence-identity-and-invalid-trace-20260902.md`; scorer/analyzer repair returns to n=1 before the separate retired-reference isolation hypothesis. + +## 2026-09-02 — explicit outcome field normalization + +- The scorer/analyzer n=1 had adaptive 15/15 and 57/58 total cells; the sole ceiling failure explicitly reported `Outcome: ... did not start` after the required focused command. +- Added `Outcome:` as result-reporting evidence while retaining the independent command probe requirement; the complete n=1 must rerun. + +## 2026-09-02 — observed trace fallback + +- The outcome-normalized n=1 had all three ceilings at 13/13; adaptive was 14/15 only because one complete manual answer omitted the benchmark footer. +- Froze `tree-observed-trace-fallback-20260902.md`; missing traces may be recovered from observed active reference reads, while unknown/retired reads remain invalid. + +## 2026-09-02 — retired reference isolation + +- Observed trace n=1 correctly rejected one adaptive load of a retired security child; parent wording alone did not remove the file from discovery. +- Froze `tree-retired-reference-isolation-20260902.md`; removed all four rejected depth-2 documents from runtime `references/` while preserving their history and receipts. +- Added concrete focused-test method alternatives for the independent Implementation ceiling oracle failure; the combined frozen candidate must rerun complete n=1. + +## 2026-09-02 — isolated leaf candidate n=1 qualification + +- The fresh current-only run completed 58/58 determinate cells; adaptive and all three ceilings passed every task, all traces/manual contracts passed, and spontaneous manual activation remained zero. +- Preserved `2026-09-02-tree-n1-isolated-leaf-qualified.md`; froze the candidate for a new complete paired n=3 comparison. + +## 2026-09-02 — quality ceiling and Core-only rejection + +- Leaf paired n=3 reached adaptive 45/45 and frozen v1.5 45/45; strict quality superiority was not measurable on the original 15 tasks. +- Core-only n=3 also tied quality but regressed tokens, duration, and tools, so the collapse was reverted and preserved in `2026-09-02-tree-n3-core-only-rejected.md`. +- Froze `tree-manual-boundary-discriminator-20260902.md`: add one real minimum-blocking-question task and first test its adaptive/v1.5/no-skill discrimination at paired n=1. + +## 2026-09-02 — manual-boundary discriminator rejected + +- The isolated paired n=1 completed 6/6 cells; adaptive, frozen v1.5, and no-skill all passed, and v1.5 did not load automatic Decision. +- Rejected and removed the non-discriminative case plus its provisional scorer expansion. Preserved `2026-09-02-tree-minimum-question-n1-rejected.md` rather than tuning the oracle to manufacture separation. + +## 2026-09-02 — bounded evidence volume n=1 qualification + +- Froze `tree-bounded-evidence-volume-20260902.md` from the leaf candidate's real cost regression: quality tied at 45/45 and tool calls improved, while tokens and duration regressed. +- Added one general retrieval-volume rule without changing topology, cases, scorer, or repositories. +- The fresh current-only n=1 completed 58/58 determinate cells with adaptive 15/15, all three capability ceilings 13/13, and perfect trace/manual discipline. Preserved `2026-09-02-tree-n1-bounded-evidence-qualified.md` and froze the candidate for paired n=3. + +## 2026-09-02 — evolvable leaf tree delivery accepted + +- Candidate `b202f7a165ae3ea4404d404bb1235ebf4270cbfb` completed the paired n=3 matrix with 252/252 determinate cells. +- Adaptive passed 45/45 versus frozen v1.5 at 44/45 and no-skill at 44/45; release quality, trace, explicit manual, and zero-spontaneous-manual gates passed. +- Cost did not improve against v1.5: mean tokens +18.67%, duration +6.40%, and tool calls +16.26%. The bounded-volume subhypothesis is therefore unconfirmed; acceptance rests on strict paired quality superiority. +- Preserved `2026-09-02-tree-n3-b202f7a-delivery.md` and published the sanitized compact report under `benchmarks/results/evolvable-tree/`. diff --git a/evolution/wiki/maintenance-trigger-isolation.md b/evolution/wiki/maintenance-trigger-isolation.md new file mode 100644 index 0000000..dbee91d --- /dev/null +++ b/evolution/wiki/maintenance-trigger-isolation.md @@ -0,0 +1,38 @@ +# Mechanism: maintenance evolution must be explicit and isolated from runtime routing + +## Claim + +Experience consolidation and Skill evolution should be explicit maintenance capabilities outside the automatic coding router tree. Raw session evidence should first become a sanitized immutable receipt, then persistent wiki knowledge; only a separate proposer may turn wiki knowledge into a reversible Skill candidate, and that candidate must pass a frozen non-regression benchmark gate. + +## Observable trigger + +- The user explicitly asks to preserve/distill the current session into the evolution wiki; or +- the user explicitly asks to evolve/optimize Practical Coding from accumulated wiki evidence. + +Ordinary implementation, debugging, retrieval, review, or architecture work is not a trigger. + +## Supporting receipts + +- `evolution/raw/sessions/2026-09-01-wikiskill-maintenance.md`: current project-maintenance request explicitly asks for session-to-wiki consolidation and wiki-informed skill evolution with benchmark rerun and no score regression. +- WikiSkill (arXiv:2608.27454) separates immutable raw traces, a compounding wiki, and reversible Skill updates; its proposer reads the wiki while skill candidates are gated and rolled back on validation degradation. +- Existing `SKILL.md` already keeps ordinary runtime agents away from `evolution/`, so maintenance skills can be added without becoming automatic descendants. + +## Contradicting receipts + +- None yet. A future experiment showing that explicit-only maintenance is undiscoverable or materially harms maintenance success should be recorded here rather than silently adding it to the automatic coding tree. + +## Affected nodes/boundaries + +- maintenance-time experience capture; +- evolution wiki indexing/logging; +- Skill proposal and rollback workflow; +- boundary between automatic runtime routing and repository-maintenance capabilities. + +## Candidate experiments + +- Add standalone `evolution/skills/session-to-wiki/SKILL.md` and `evolution/skills/evolve-skill/SKILL.md`. +- Add a deterministic benchmark that verifies both are explicit-only, absent from automatic topology, preserve raw/wiki/Skill separation, and require frozen baseline-versus-candidate non-regression gating. + +## Current status + +Accepted as maintenance-only architecture after `benchmarks/evolution_workflow_validation.py` scored 28/28 (1.000) locally. No runtime Skill/router file was changed, so the existing model-backed runtime prompt surface is unchanged; CI still reruns repository regression/self-tests before merge. diff --git a/evolution/wiki/skill-impact.md b/evolution/wiki/skill-impact.md new file mode 100644 index 0000000..a8a1990 --- /dev/null +++ b/evolution/wiki/skill-impact.md @@ -0,0 +1,70 @@ +# Skill impact tracker + +Record every maintenance-time Skill proposal after validation. Rejected proposals remain here so later iterations do not repeat them without new evidence. + +Each entry should include: + +- date / iteration; +- hypothesis or experiment path; +- target Skill/node; +- baseline ref and benchmark artifact; +- candidate ref or unified diff; +- baseline and candidate required quality scores; +- relevant cost metrics; +- decision: `Accepted` or `Rejected`; +- rejection reason or acceptance rationale. + +## Historical note + +Experiments before this tracker was introduced remain authoritative in their existing benchmark artifacts and `evolution/rejected/` records; do not invent missing scores retroactively. + +## 2026-09-01 — explicit maintenance skills + +- hypothesis: `evolution/wiki/maintenance-trigger-isolation.md` +- target: maintenance orchestration only; no runtime Skill/tree node changed +- baseline ref: `118acd81cb0e26f4f8087555c3bd89cbf45c9d30` +- benchmark: `benchmarks/results/evolution-workflow/2026-09-01.json` +- baseline runtime surface: unchanged by candidate +- candidate maintenance-contract score: `28/28 = 1.000` +- decision: `Accepted` +- rationale: the new maintenance skills are isolated from automatic topology and the deterministic contract gate passes perfectly; runtime model-backed inputs remain byte-identical in this iteration. + +## 2026-09-01 — tree scorer contract normalization + +- hypotheses: `evolution/experiments/tree-oracle-alignment-20260901.md`, `tree-oracle-semantic-equivalence-20260901.md`, `tree-scorer-normalization-20260901.md` +- target: deterministic benchmark evidence matching and Windows reference enforcement; no runtime Skill/tree file changed +- baseline ref: `31ba37c9c324ff5863ee237a8c89203f4405fbe9` +- invalidated artifacts: `tree-delivery-n1-20260901-103036`, `tree-delivery-n1-oraclefix-20260901-111838`, `tree-delivery-n1-semantic-20260901-120715` +- accepted artifact: `benchmark-results/tree-delivery-n1-normalized-20260901-125524` +- baseline adaptive score: invalid for acceptance because each preceding artifact used a superseded scorer contract +- candidate required quality: 106/106 determinate; adaptive 15/15; trace 15/15; explicit manual 2/2; spontaneous manual 0/13 +- deterministic gate: 22 tree/discriminator/evolution tests passed; maintenance workflow 28/28 +- decision: `Accepted` +- rationale: prompt/scorer alignment, semantic and language equivalence, and Windows path normalization are covered by positive and negative tests; runtime inputs remain unchanged. Stable topology and prior-version claims remain pending paired n=3 evidence. + +## 2026-09-02 — evolvable leaf tree delivery + +- hypothesis: `evolution/experiments/tree-bounded-evidence-volume-20260902.md` +- target: Core retrieval-volume rule on the isolated Debugging/Implementation leaf topology +- frozen baseline: v1.5 at `ba4058b4ef47a42bf79c9963b25678a2389897c1` +- candidate ref: `b202f7a165ae3ea4404d404bb1235ebf4270cbfb` +- artifact: `benchmark-results/tree-final-b202f7a-20260902` (252/252 determinate, n=3) +- required quality: adaptive 45/45; v1.5 44/45; no-skill 44/45 +- discipline: trace 45/45; explicit manual 6/6; spontaneous manual 0/39 +- adaptive versus v1.5 cost: tokens 258,061.64 vs 217,460.96; duration 76.82s vs 72.20s; tools 8.42 vs 7.24 +- decision: `Accepted` +- rationale: the frozen primary release gate requires strict paired delivered-quality superiority and no regression against no-skill; both comparators were exceeded by one cell with perfect discipline. The proposed cost mechanism was not confirmed and the regression remains an explicit limitation, not an acceptance claim. + +## 2026-09-03 — execution-state/history-free experiment + +- experiments: archived under `evolution/rejected/execution-state/` +- target: cross-cutting explicit execution-state runtime, history-free host/transport, and four-arm model gate +- final measured candidate baseline: `e6cc9caa456767b3e05dbff59474aa7014146cbf`; final pre-retirement branch head: `215334db7bb914bd9f0346a2b09654fc89accc96` +- standard n=1 matrix: 24/24 determinate; full-history 6/6; state-history-free 6/6; state-shadow 5/6; no-skill-full-history 6/6 +- transport: captured state-history-free client transport gate passed +- state semantic diagnostic: shadow retained `h-cache` in both active and rejected lifecycle partitions; later deterministic hardening was not model-rerun before retirement +- cost versus full-history: uncached input tokens 78,118 vs 50,736 (`+54.0%`); duration 172.60s vs 154.57s (`+11.7%`) +- incomplete claims: formal n>=3, token, latency, and bounded-horizon gates remained pending +- decision: `Rejected` +- rationale: the mechanism produced no delivered-quality lift, materially increased cost in the completed matrix, and added substantial runtime/transport/benchmark complexity while not addressing the observed retrieval-output long tail. Active code and contracts were removed; historical records remain archived. +- reconsideration: only through a new frozen experiment with independent evidence for a substantially simpler mechanism that solves a demonstrated long-horizon failure, preserves n>=3 quality, and reduces both uncached tokens and time. diff --git a/examples/README.md b/examples/README.md index 2db0205..14e826f 100644 --- a/examples/README.md +++ b/examples/README.md @@ -10,136 +10,107 @@ Each example shows how Practical Coding controls both implementation cost and co **Request:** "Add a date picker to the signup form." -**Typical over-engineered outcome:** - -```text -+ package.json (new dependency) -+ src/components/DatePicker.tsx (wrapper component) -+ src/components/DatePicker.css -+ src/utils/dateFormat.ts (timezone helpers "for later") -``` - -**With the skill** — Direct Path / Core ladder: +A native platform control satisfies the current requirement: ```html ``` -The native platform feature satisfies the current requirement, so the ladder stops there. +No dependency, wrapper component, timezone helper, or speculative configuration is added. --- -## 2. Defensive bloat around a config read - -**Request:** "Read the API base URL from the config file." - -**Typical over-engineered outcome:** - -```ts -function getApiBaseUrl(): string { - for (let attempt = 0; attempt < 3; attempt++) { - try { - const raw = fs.readFileSync(CONFIG_PATH, "utf8"); - const parsed = JSON.parse(raw ?? "{}"); - return parsed?.api?.baseUrl ?? DEFAULT_BASE_URL ?? ""; - } catch { - // swallow and retry - } - } - return ""; -} -``` +## 2. R0 Direct Locate -**With the skill** — use the established contract unless a real failure boundary requires more: +**Request:** "Where is `normalize_header()` defined, and which nearby caller uses it for Authorization?" + +The symbol is exact, so Retrieval starts and stops at R0: -```ts -function getApiBaseUrl(): string { - const config = JSON.parse(fs.readFileSync(CONFIG_PATH, "utf8")); - return config.api.baseUrl; -} +```text +exact symbol lookup: normalize_header + -> headers.py::normalize_header + -> headers.py::auth_header ``` -Retries, fallback chains, and broad catches are not added speculatively. +Read only the definition and material caller. Navigation, ranked discovery, evidence expansion, and graph tracing are unnecessary. --- -## 3. Process overhead on a trivial edit +## 3. R1 Ranked Discovery -**Request:** "Change the button text from 'Submit' to 'Save'." +**Request:** "Where is login state restored in this unfamiliar application?" -**With a fixed-pipeline workflow:** +The intent is known but file and symbol names are not. R0 cannot identify a target, so it loads only R1. A ranked hybrid provider may return: ```text -1. Brainstorming -2. PLAN.md -3. New branch/worktree -4. New unit test for the literal label -5. One-line change -6. Review/checkpoint ceremony +1. src/session/SessionBootstrap.ts +2. src/auth/restoreSession.ts +3. src/routes/AppGuard.tsx ``` -**With the skill** — Direct Path: - -```diff -- -+ -``` - -Run only the cheapest focused check actually required by the repository or the requested success condition. +The provider is an implementation of R1, not a tree node. Verify the best candidates in current source. If one candidate proves the answer, return without R2. --- -## 4. Routine code lookup does not load Navigation +## 4. R2 Evidence Expansion -**Request:** "Where is `normalize_header()` defined, and which nearby caller uses it for Authorization?" +**Request:** "Why does refresh-token rotation behave this way?" -The location can be established with a narrow symbol/text lookup and two targeted reads: +R1 locates `TokenService`, but the claim also depends on one filter, authoritative configuration, and focused tests. R2 builds only that evidence set: ```text -symbol/text search: normalize_header - -> headers.py::normalize_header - -> headers.py::get_header - -> headers.py::auth_header +TokenService +AuthFilter +SecurityConfig +TokenServiceTest ``` -No reasoning module is selected, `references/navigation.md` is not loaded, and no graph backend is required. Search is ordinary Direct work because the next action is already clear. +It does not read adjacent authentication modules merely because they are related. --- -## 5. Ranked retrieval is an optional accelerator +## 5. R3 Structural Trace -**Request:** "Find the likely authentication implementation in this unfamiliar repository." +**Request:** "Map every service that calls the billing client and where each response is transformed." -If the host already exposes bounded/ranked retrieval — for example a native ranked code search or FFF-style search — use it to return a small candidate set: +The answer is relational. R2 loads R3, which uses an available graph provider and verifies the resulting paths in current source: ```text -1. src/auth/JwtService.ts -2. src/middleware/AuthMiddleware.ts -3. src/routes/login.ts +services/api.py::checkout + -> shared/billing.py::charge + -> services/api.py::to_checkout_response + +services/jobs.py::retry_invoice + -> shared/billing.py::charge + -> services/jobs.py::to_retry_record ``` -Then read only the material candidates. If no ranked capability exists, fall back to narrow filename/text/symbol search such as `rg`, `grep`, or the host equivalent. Practical Coding does not install FFF or another search engine merely for this task. +At runtime, if no graph provider exists, R3 reconstructs only the required edges with bounded reference tracing. It remains a leaf; there is no whole-repository "stronger search" stage. --- -## 6. Structural retrieval is used for structural questions +## 6. Navigation is not Retrieval -**Request:** "Map every service that calls the billing client and where each response is transformed." +**Request:** "In this unfamiliar monorepo, which package owns plugin lifecycle execution?" -This is relationship-heavy. If an already-integrated structural index such as Codebase Memory is available and materially reduces repeated source exploration, query the graph for the relevant callers/paths and then verify the material files in current source. +Navigation may first return a bounded map from root module declarations: -If no structural backend is available, continue with bounded source search. Do not create `.practical-coding.yaml`, install Codebase Memory, or add a persistent MCP/service solely to complete the lookup. +```text +platform API -> progress-core -> lifecycle package +``` -The desired output is a compact evidence map such as: +Retrieval then starts at R0 inside that scope. Navigation does not run semantic search or trace callers itself. -```text -services/api.py::checkout - -> shared/billing.py::charge - -> services/api.py::to_checkout_response +--- -services/jobs.py::retry_invoice - -> shared/billing.py::charge - -> services/jobs.py::to_retry_record +## 7. Output compaction is not a route + +A noisy focused test can pass through an output adapter: + +```text +npm test -- src/lib/exportFilename.test.ts + -> output compaction layer + -> exit status + failures or concise pass evidence ``` -not a raw repository tour, grep dump, or graph transcript. +The execution and Retrieval paths do not change. If the compact result omits one diagnostic needed for a failure, retrieve that bounded detail rather than disabling compaction globally. diff --git a/references/debugging.md b/references/debugging.md index 98b904c..1d4f9b5 100644 --- a/references/debugging.md +++ b/references/debugging.md @@ -1,6 +1,8 @@ # Debugging -Load this module only for an observed or reported failure, regression, incorrect behavior, or failed verification that still lacks an evidenced cause. +**Tree depth: 1** + +Load this node only from its parent when an observed or reported failure, regression, incorrect behavior, or failed verification still lacks an evidenced cause. ## Evidence First @@ -13,18 +15,24 @@ Load this module only for an observed or reported failure, regression, incorrect - Prefer the narrowest fix that corrects the root cause and preserves existing contracts. - Do not patch a downstream symptom when an earlier incorrect state is identifiable and fixable. -- Treat universal wording such as "never," "every," or "no X can" as one contract across current mutation paths. Before editing a reported caller, inspect its delegated helper and nearest sibling caller; if both can violate that contract, fix the invariant once in their common state-mutation or parsing helper. Patch only the reported adapter when evidence shows the helper intentionally owns a different lower-level contract. +- Treat universal wording such as "never," "every," or "no X can" as one contract across current mutation paths. Before editing a reported caller, inspect its delegated helper and nearest sibling caller; if both can violate that contract, fix the invariant once in their common state-mutation or parsing helper. - When the request names shared behavior, repair the authoritative shared primitive for all current callers. Do not preserve the same defect behind a new per-caller flag or branch unless an established caller contract requires different behavior. - Do not use broad retries, catches, fallbacks, default values, or defensive branches to hide an unexplained failure. - Add temporary logging or instrumentation only when it produces evidence needed to distinguish hypotheses. Judge a fix by delivered behavior. It should remove the earliest incorrect state, preserve other callers of the repaired boundary, restore any violated security, permission, integrity, accessibility, compatibility, or explicit project constraint, and change no unrelated behavior. +## Local Router + +This node is a leaf. Keep live timing, state, process/worker, browser/network, CI/runtime, and async-ordering evidence inside the current debugging loop; those concerns did not earn a separate child in repeated capability ablation. + +Do not route to Decision when diagnosis exposes alternatives. Reuse the established project contract or smallest sufficient reversible option. If a genuinely user-owned choice blocks progress with no safe default, ask the minimum blocking question without loading a Decision module. + ## Stay in Scope - Diagnose the reported failure; do not turn debugging into a repository-wide search for unrelated defects. - Do not write tests merely because debugging occurred. Use the cheapest reproduction or focused check that can falsify the fix; add a durable test only when regression risk or project requirements justify it. -- If diagnosis exposes a different material blocker, return it to the root instead of loading another reference here. +- If work exposes a genuinely different top-level execution blocker rather than a descendant of Debugging, return that blocker to Core. ## Exit diff --git a/references/delegation.md b/references/delegation.md index a585458..9b61b7c 100644 --- a/references/delegation.md +++ b/references/delegation.md @@ -1,13 +1,14 @@ # Isolated Reference Delegation -Load this protocol only inside a worker selected by the Isolation Gate. Also read exactly one assigned reference: Decision, Debugging, Implementation, or Navigation retrieval. +Load this protocol only inside a worker selected by the Isolation Gate. Also read exactly one assigned automatic reference (`Debugging`, `Implementation`, or `Navigation`) or one explicitly requested manual reference. ## Worker contract - Use the requirement, project constraints, known evidence, repository state, and allowed scope supplied by the root. Do not reconstruct the full conversation or rescan unrelated areas. - The root must not inspect or modify the delegated scope while this worker runs. If it changes, return `stale`. -- Do only the assigned reference's work. Report a newly exposed blocker to the root instead of loading another reference or spawning another worker. -- Decision, Debugging, and Navigation workers are read-only. +- Do only the assigned reference's work. Report a newly exposed blocker to the owner of the active tree node instead of discovering arbitrary descendants or spawning another worker. +- Debugging and Navigation workers are read-only. +- A manual Decision worker is always outside the automatic execution tree and is read-only unless the user separately authorizes implementation after the choice is settled. - An Implementation worker is read-only when assigned mapping/evidence only. When explicitly assigned implementation, it writes only within its bounded non-overlapping scope and is the sole writer there. - Record starting HEAD and relevant dirty paths. Never commit, reset, checkout, clean, or overwrite user changes unless explicitly authorized. @@ -16,9 +17,10 @@ Load this protocol only inside a worker selected by the Isolation Gate. Also rea Return conclusions and evidence, not transcripts or raw search dumps: - assigned reference and status: complete, provisional, blocked, or stale; +- active node and current tree depth when the work belongs to the automatic tree; - starting repository state and exact paths/symbols in scope; - findings or changes backed by current source/tool evidence; - checks run and their freshness; -- coverage limitations, unresolved items, and any newly exposed event for root routing. +- coverage limitations, unresolved items, and any newly exposed top-level blocker. Do not persist the capsule unless the user requested an artifact. diff --git a/references/implementation.md b/references/implementation.md index 74722a3..047c8cb 100644 --- a/references/implementation.md +++ b/references/implementation.md @@ -1,12 +1,14 @@ # Implementation -Load this module only when a change must coordinate an unmapped contract or invariant, touches a material risk boundary where direct execution would be unsafe, or when sufficient evidence for a risky change is unresolved. Produce only the change map and evidence plan the task needs; this is not a mandatory coding stage. +**Tree depth: 1** + +Load this node only from its parent when a change must coordinate an unmapped contract or invariant, touches a material risk boundary where direct execution would be unsafe, or when sufficient evidence for a risky material change is unresolved. Produce only the change map, implementation, and evidence the task needs; this is not a mandatory coding stage. ## Work Locally - Identify the authoritative contract or invariant and the minimum producers, consumers, adapters, data, and checks that must move together. -- Read only those paths and their material callers/dependencies; leave nearby cleanup opportunities and unrelated code alone. -- For a risk boundary, identify the narrowest authoritative point that owns the guarantee before editing. A single-file change can still belong here when the boundary is material. +- Read only those paths and their material callers or dependencies; leave nearby cleanup opportunities and unrelated code alone. +- For a risk boundary, identify the narrowest authoritative point that owns the guarantee before editing. - Preserve public compatibility unless the requirement authorizes a break. When migration is required, choose one authoritative internal representation and keep compatibility at the narrowest boundary. - Match project conventions and make the smallest coherent end-to-end diff. @@ -24,5 +26,12 @@ Map each material claim or risk to the cheapest check that can falsify it: direc For persistence or concurrency, exercise restart/rollback/race behavior when relevant. For compatibility, exercise materially affected old and new callers. For security or permissions, include one valid case and the smallest representative rejection cases, and verify rejection happens before side effects. -Claim only what fresh evidence supports. If the environment blocks an appropriate check, report the limitation and remaining uncertainty. If implementation exposes another event, return it to the router instead of loading another module here. +Claim only what fresh evidence supports. If the environment blocks an appropriate check, report the limitation and remaining uncertainty. + +## Local Router + +This node is a leaf. Handle security/permission boundaries, migration/compatibility, and state/concurrency invariants with the shared implementation rules above; repeated capability ablation did not show stable minimum-sufficient lift for separate children. + +Resolve ordinary implementation choices locally by established project convention, platform default, or the smallest sufficient reversible choice. Never route automatically to Decision. If a genuinely user-owned choice blocks safe execution and no default is justified, ask the minimum blocking question without opening a Decision workflow. +If work exposes a genuinely different top-level unexplained failure rather than an Implementation descendant, return that blocker to Core. diff --git a/references/decision.md b/references/manual/decision.md similarity index 61% rename from references/decision.md rename to references/manual/decision.md index 36b9402..468090b 100644 --- a/references/decision.md +++ b/references/manual/decision.md @@ -1,16 +1,16 @@ -# Decision +# Manual Decision -Load this module only when a material choice about architecture, dependencies, APIs, data models, compatibility, or multiple plausible implementations remains open—including whether or which package, library, service, or mature external implementation to adopt. Its output is a resolved choice that changes the next action, not a design essay or option dump. +This mode is outside the automatic execution tree. -Do not load this module when the request or repository has already settled the material choice. The existence of a popular alternative is not by itself a Decision event. +Load it only when the current user explicitly asks to compare options, choose a technology or architecture, recommend among dependencies/APIs/data models/compatibility strategies, or otherwise perform decision analysis. The existence of alternatives, ambiguity, risk, or a technical choice discovered during execution does not activate this mode. ## Decision Frontier Resolve discoverable facts from the repository and authoritative sources before asking the user. Work only on choices whose prerequisites are already known. Ask only about user-owned scope, compatibility, risk tolerance, cost, or preference when at least two plausible answers lead to materially different next actions and choosing the wrong default costs more than one interaction. -For each necessary question, explain why it matters, recommend one option with the reason, and state the strongest trade-off. Ask every independent decision on the current frontier in one round; defer dependent questions. If uncertainty is cheap and reversible, choose the repository or platform default and proceed. +For each necessary question, explain why it matters, recommend one option with the reason, and state the strongest trade-off. Ask every independent decision on the current frontier in one round; defer dependent questions. If uncertainty is cheap and reversible, choose the repository or platform default. -Use a compact stable shape so the recommendation is visible rather than buried in prose: +Use a compact stable shape when a user choice is needed: ```text Q — Decision: @@ -18,8 +18,6 @@ Recommendation: Trade-off: ``` -End with the smallest answer format and wait. When the reply resolves the frontier, do not ask for confirmation of a now-determined choice. - ## Resolve 1. State the exact decision and constraints that distinguish acceptable options. @@ -27,7 +25,9 @@ End with the smallest answer format and wait. When the reply resolves the fronti 3. Keep at most three viable options and compare only material fit, correctness, compatibility, operational, maintenance, and migration differences. 4. Select the smallest option that fully satisfies current requirements. Do not create an abstraction, dependency, wrapper, or extension point without a present need. -Research only when local evidence cannot resolve a lasting choice or an external dependency is being considered. Prefer official and maintained sources; verify API fit, maintenance, license, and known constraints. Unless an unresolved assumption requires one extra line, every resolved final decision is exactly two lines: `Recommendation:` with selection and reason, then `Trade-off:` with the strongest cost or alternative. Proceed only within existing authorization. +Research only when local evidence cannot resolve a lasting choice or an external dependency is being considered. Prefer official and maintained sources; verify API fit, maintenance, license, and known constraints. + +When the requested decision is resolved, stop this mode. Return the settled result to Core as input. Do not route directly from this file to Debugging, Implementation, Clarification, or any descendant. ## Durable Decisions diff --git a/references/navigation.md b/references/navigation.md index feff73c..5251d10 100644 --- a/references/navigation.md +++ b/references/navigation.md @@ -1,34 +1,40 @@ # Navigation -Navigation is the detailed retrieval procedure, not an Event Router branch. Load it only when broad code discovery, structural mapping, external contract lookup, or bounded exhaustive coverage is substantial enough that the short Retrieval Policy in `SKILL.md` is insufficient. +**Concern:** repository topology only +**Output:** the smallest bounded map that identifies where Retrieval should begin -Use already-available capabilities only. Do not install a backend, add a persistent integration, or change project configuration solely to obtain retrieval for the current task. +Load Navigation only when the current unresolved question is **which repository area should be searched**. Do not load it merely because a file path is unknown; R1 Ranked Discovery handles unknown locations when the intended behavior or concept is already known. -## Retrieval ladder +## Goal -### Known target +Reduce a broad or unfamiliar repository to a bounded scope such as one package, module, service, layer, or directory group. -Read the identified file, symbol, route, test, error, or configuration directly. Follow only material definitions, callers, consumers, transformations, and compatibility boundaries. Stop when the requested behavior and minimum coherent surface are established. +A useful result looks like: -### Bounded or ranked source discovery +```text +platform API + -> progress-core lifecycle package + -> operation executor and state package +``` -When location is unknown, prefer an already-available bounded/ranked primitive. Otherwise use ordinary filename, text, and symbol search. +not a file inventory, semantic-search transcript, or repository tour. -- Batch narrow queries rather than dumping the repository. -- Use top-k, limits, pagination, and narrow scopes where available. -- Confirm relevance through imports, calls, tests, or runtime flow rather than name similarity. -- Read definitions first, then only the material neighbors. +## Procedure -### Structural retrieval +1. Read the repository's own map first: root manifests, workspace/module declarations, package metadata, build files, and maintained architecture notes. +2. Identify only the regions that can own the requested behavior or relationship. +3. Exclude unrelated generated, vendored, fixture, example, and historical areas unless the task explicitly includes them. +4. Return the bounded scope and the evidence that establishes the boundary. +5. Continue with `references/retrieval/SKILL.md` at Direct Locate inside that scope. -Use an already-available structural code index when the unresolved question is primarily relational and lexical reconstruction would be expensive: callers, callees, imports, implementations, inheritance, dependencies, or cross-file flow. +## Boundary -When Codebase Memory is available, confirm project identity/freshness, use the smallest graph query set, check index coverage once candidate paths are known, and read current source for material claims and every partial/stale/excluded range. If unavailable or insufficient, continue with bounded source discovery. +Navigation does not: -### External and exhaustive evidence +- choose between search tools; +- perform semantic or ranked discovery; +- expand callers, tests, configuration, or related implementations; +- trace call graphs, dependencies, control flow, or data flow; +- claim exhaustive coverage unless the user explicitly requested it and coverage can be demonstrated. -For a repository-wide claim, state the bounded scope, search systematically with pagination/coverage tracking, and disclose gaps. For an external API/protocol/license contract, use the smallest authoritative maintained source needed for the code decision. - -## Contract - -Search and graph output are evidence, not repository truth. Verify material conclusions in current source. Once the relevant relationship or boundary is known, stop expanding and contract to that surface. +When a concrete path, symbol, identifier, or sufficiently narrow scope is already known, skip Navigation. diff --git a/references/retrieval/SKILL.md b/references/retrieval/SKILL.md new file mode 100644 index 0000000..284c2a8 --- /dev/null +++ b/references/retrieval/SKILL.md @@ -0,0 +1,11 @@ +# Retrieval + +**Retrieval depth:** root +**Purpose:** locate the minimum current-source evidence required for the task +**Immediate child:** [`direct.md`](direct.md) + +Retrieval is independent of the automatic execution tree. It progresses according to the information problem that remains unresolved, not according to task risk, execution depth, repository size, or provider strength. + +Start at Direct Locate. Do not preload any node beyond this immediate child and do not select a deeper stage from the root. + +A capability provider implements a stage; it is not the stage itself. The same policy must continue to work when providers change or when runtime fallback is necessary. diff --git a/references/retrieval/direct.md b/references/retrieval/direct.md new file mode 100644 index 0000000..1db265a --- /dev/null +++ b/references/retrieval/direct.md @@ -0,0 +1,22 @@ +# R0 Direct Locate + +**Retrieval stage:** R0 +**Goal:** resolve the current claim through an already-known or narrowly identifiable target +**Immediate child:** [`discovery.md`](discovery.md) + +## Enter when + +The target can be identified by a known file, exact symbol, exact identifier, route, test, configuration key, error location, or a very narrow literal search. + +## Work + +- Read the target and only the minimum surrounding context needed to interpret it. +- Follow a directly referenced definition or caller only when the current claim requires it. +- Prefer bounded line or symbol reads over whole-file dumps. +- Return concrete source locations and the evidence they establish. + +## Stop or escalate + +Return as soon as the target plus minimum context answers the current question. + +If the target cannot be located confidently from exact or narrow evidence, load **R1 Ranked Discovery**. Do not jump to later stages and do not broaden into a repository tour. diff --git a/references/retrieval/discovery.md b/references/retrieval/discovery.md new file mode 100644 index 0000000..ac28135 --- /dev/null +++ b/references/retrieval/discovery.md @@ -0,0 +1,23 @@ +# R1 Ranked Discovery + +**Retrieval stage:** R1 +**Goal:** find likely implementation locations when the target is unknown but the intended behavior or concept is known +**Immediate child:** [`evidence.md`](evidence.md) + +## Work + +Use the strongest available bounded retrieval capability in this order: + +1. hybrid semantic intent plus lexical anchors; +2. ranked lexical or symbol search; +3. exact or regular-expression repository search as a lossless fallback. + +Return only the strongest candidates. Confirm candidate relevance in current source through definitions, imports, calls, tests, configuration, or runtime flow; ranking is not proof. + +Do not search merely related concepts, dump unbounded matches, or treat provider output as repository truth. + +## Stop or escalate + +Return when one candidate and its bounded source evidence answer the current question. + +If the answer still depends on evidence distributed across nearby implementations, callers, configuration, tests, interfaces, or schemas, load **R2 Evidence Expansion**. Do not skip this immediate child. diff --git a/references/retrieval/evidence.md b/references/retrieval/evidence.md new file mode 100644 index 0000000..9b3c569 --- /dev/null +++ b/references/retrieval/evidence.md @@ -0,0 +1,24 @@ +# R2 Evidence Expansion + +**Retrieval stage:** R2 +**Goal:** expand located candidates into the smallest cross-file evidence set required by unresolved claims +**Immediate child:** [`structural.md`](structural.md) + +## Work + +Start from the strongest R1 candidate. For each unresolved claim, identify the smallest additional source that can prove or falsify it. Possible evidence includes: + +- the primary implementation; +- one material caller or callee; +- authoritative configuration; +- focused tests; +- an interface, schema, state owner, or compatibility boundary; +- adjacent behavior only when it changes the answer. + +Expand because a named claim lacks evidence, not because a file is related. Keep an explicit bounded evidence set and stop adding sources once every material claim is supported. + +## Stop or escalate + +Return when the required distributed evidence is complete. + +If the unresolved answer is fundamentally a relationship—call path, dependency path, ownership, control flow, data flow, or impact surface—and bounded source expansion would reconstruct a graph manually, load **R3 Structural Trace**. diff --git a/references/retrieval/structural.md b/references/retrieval/structural.md new file mode 100644 index 0000000..7dc5cf7 --- /dev/null +++ b/references/retrieval/structural.md @@ -0,0 +1,26 @@ +# R3 Structural Trace + +**Retrieval stage:** R3 +**Leaf:** yes + +## Goal + +Resolve questions whose answer depends on relationships between code entities rather than isolated matching text. + +## Work + +Use an available graph-aware structural capability for the smallest query set that can establish the required relationship. Appropriate relationships include: + +- callers and callees; +- imports, implementations, inheritance, and dependencies; +- ownership and authoritative state boundaries; +- control flow and data flow; +- change impact and cross-service paths. + +Check project identity, index freshness, and coverage before relying on a structural result. Verify every material path, symbol, and partial, stale, or excluded range in current source. + +If no graph-aware capability is available at runtime, fall back to bounded reference tracing: + +`find references -> read material callers/callees -> follow the next unresolved edge -> stop`. + +Stop once the relationship required by the task is established. There is no deeper retrieval stage and no whole-repository escalation.