Skip to content

experiment: benchmark-driven evolvable local router tree - #6

Draft
Hubujiu wants to merge 57 commits into
experiment/progressive-laddersfrom
experiment/evolvable-router-tree
Draft

experiment: benchmark-driven evolvable local router tree#6
Hubujiu wants to merge 57 commits into
experiment/progressive-laddersfrom
experiment/evolvable-router-tree

Conversation

@Hubujiu

@Hubujiu Hubujiu commented Aug 31, 2026

Copy link
Copy Markdown
Owner

Why

This experiment keeps progressive disclosure but removes two assumptions from the v1.5 flat Event Router:

  • Decision is no longer an automatic route. Automatic execution must converge toward resolving the current blocker rather than reopening deliberation whenever another technical choice appears.
  • Core no longer owns the whole future routing taxonomy. Core is only the root; every loaded node owns only its immediate children and current disclosure depth.

This does not restore the rejected fixed E0-E3/R0-R3 taxonomy. The old experiment predefined numeric levels and specialist leaves and then scored the model against them. This PR makes topology itself an experiment variable.

Runtime candidate

Automatic
Core (depth 0)
├── Debugging (depth 1, current leaf)
└── Implementation (depth 1, current leaf)

Manual only
├── Decision
└── Clarification

Retrieval
└── orthogonal capability expansion

Execution state
└── orthogonal long-task runtime substrate

Key runtime changes:

  • move references/decision.md to references/manual/decision.md;
  • remove Decision from the automatic root router;
  • prohibit automatic nodes from routing to Decision/Clarification;
  • add a convergence rule for technical choices discovered during execution;
  • add local Router contracts to Debugging and Implementation; both start as leaves;
  • make future descendants owned by their parent, not Core.

Paper-derived execution state

This branch now adapts the mechanism from SKILL.state: Scalable Long-Horizon Agent Skills (arXiv:2608.26263v2) as a cross-cutting runtime substrate rather than another Router node:

  • each state-aware step is built from immutable procedure P, validated current state Σ, and latest observation O;
  • the model returns only state_patch plus one proposed action;
  • the runtime applies JSON Merge Patch semantics on a copy, validates the complete successor, and releases the action only after acceptance;
  • invalid transitions preserve canonical state and their actions must not execute;
  • runtime input is one JSON envelope, so backticks, fake headings, and instruction-like tool output remain data;
  • procedure is authoritative, latest_observation is untrusted evidence, and host-owned task/router fields cannot be changed by model patches;
  • audit/provenance, evolving-schema, and concurrency-sensitive tasks retain a bounded history escape hatch instead of assuming projection is always lossless.

Execution state does not increase tree depth, does not enter TREE_TRACE, and cannot make Decision or Clarification automatic.

Benchmark redesign

The tree benchmark is intentionally separate from the legacy flat-router/fixed-level scorer:

  • benchmarks/tree_topology.json: topology manifest; node names, parent/child edges and depth are data, not scorer constants.
  • benchmarks/tree_cases.py: real-repository cases with no expected automatic route, E0-E3 label, or fixed capability path.
  • benchmarks/tree_validation.py: no-skill + frozen v1.5 baseline + adaptive candidate + one root-to-node capability ceiling per automatic node.
  • benchmarks/tree_analysis.py: derives minimum-sufficient node sets from stable ceiling results, then reports over/under disclosure, alternate branches, quality gaps, node marginal lift, sibling ambiguity, and topology-change candidates.
  • benchmarks/TREE_EVOLUTION.md: add/split/merge/promote/collapse/remove rules.

Automatic route exactness is no longer a release gate. Delivered quality, valid parent-child traces, zero spontaneous manual activation, and explicit-manual adherence are gates. Route disagreement is evidence that the topology boundary may be wrong.

Execution-state mechanics have their own deterministic contract in benchmarks/skill_state_validation.py. Model-level evidence is frozen separately in benchmarks/SKILL_STATE_MODEL_GATE.md with four arms: full history, state shadow, true history-free P + Σ + O, and no-skill full history. Cost is considered only after delivered-quality, safety, topology, and manual-mode gates pass.

Historical evidence

The v1.5 flat Event Router remains the frozen baseline at ba4058b4ef47a42bf79c9963b25678a2389897c1. Existing progressive-validation artifacts, v1.5 results, and rejected-tree evidence are preserved rather than rewritten.

The paper's reported accuracy, token, and bounded-context results are background evidence only; they are not treated as results for this repository. A horizon-independent prompt claim is permitted only if captured host requests actually omit prior messages and contain only bounded P + Σ + O inputs.

Validation state

Current head: a7ea30f4383c122e5dc07a6115d843307278a5ff.

  • GitHub Validate skill: PASS.
  • skills-ref validation: PASS.
  • Repository suite: 109 tests PASS.
  • Tree topology self-test: PASS.
  • Explicit evolution-workflow contract: 28/28 PASS.
  • Execution-state runtime contract: PASS; focused execution-state suite: 19 tests PASS.
  • Manual-only Decision/Clarification layout: PASS.
  • Legacy local graph runtime exclusion: PASS.
  • Fresh model-backed tree n=1 / frozen n>=3 matrix: pending.
  • Dedicated full-history/state-shadow/state-history-free execution-state matrix: pending.

This PR remains draft until the fresh model-backed gates are reviewed. Final topology is explicitly not assumed to remain Core -> {Debugging, Implementation}; benchmark evidence may merge, split, remove, promote, or deepen nodes. Execution state remains experimental until its frozen n>=3 model gate shows no delivered-quality regression.

Hubujiu and others added 30 commits August 31, 2026 09:52
Add explicit session-to-wiki and wiki-informed evolve-skill maintenance capabilities, persistent raw/wiki controls, and a deterministic non-regression workflow benchmark. CI passes without modifying the automatic runtime router tree.
@Hubujiu
Hubujiu force-pushed the experiment/evolvable-router-tree branch from b99b2a2 to a7ea30f Compare September 2, 2026 03:51
Hubujiu and others added 18 commits September 1, 2026 21:35
Reject ambiguous JSON and unsafe runtime inputs, isolate state snapshots before merge and prompt rendering, clarify host-side action authorization, and add non-benchmark regression coverage.
- add progressive R0-R3 Retrieval tree with local child routing
- move zvec-grep, Codebase Memory, and RTK into a replaceable capability layer
- require versioned providers for dependency-enabled benchmark runs
- exclude provider indexes, dependency resolution, and first-build warm-up from measured comparisons
- add independent Retrieval ceilings, trace validation, and minimum-stage analysis
- keep the established Retrieval Policy section boundary while retaining the new progressive tree
- repair JSON escaping for pinned provider version probes
Preserve the rewritten retrieval policy while retaining the established lowercase contract phrase used by the deterministic harness.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant