experiment: benchmark-driven evolvable local router tree - #6
Draft
Hubujiu wants to merge 57 commits into
Draft
Conversation
Add explicit session-to-wiki and wiki-informed evolve-skill maintenance capabilities, persistent raw/wiki controls, and a deterministic non-regression workflow benchmark. CI passes without modifying the automatic runtime router tree.
This reverts commit 230522f.
Hubujiu
force-pushed
the
experiment/evolvable-router-tree
branch
from
September 2, 2026 03:51
b99b2a2 to
a7ea30f
Compare
Reject ambiguous JSON and unsafe runtime inputs, isolate state snapshots before merge and prompt rendering, clarify host-side action authorization, and add non-benchmark regression coverage.
- add progressive R0-R3 Retrieval tree with local child routing - move zvec-grep, Codebase Memory, and RTK into a replaceable capability layer - require versioned providers for dependency-enabled benchmark runs - exclude provider indexes, dependency resolution, and first-build warm-up from measured comparisons - add independent Retrieval ceilings, trace validation, and minimum-stage analysis
- keep the established Retrieval Policy section boundary while retaining the new progressive tree - repair JSON escaping for pinned provider version probes
Preserve the rewritten retrieval policy while retaining the established lowercase contract phrase used by the deterministic harness.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
This experiment keeps progressive disclosure but removes two assumptions from the v1.5 flat Event Router:
This does not restore the rejected fixed E0-E3/R0-R3 taxonomy. The old experiment predefined numeric levels and specialist leaves and then scored the model against them. This PR makes topology itself an experiment variable.
Runtime candidate
Key runtime changes:
references/decision.mdtoreferences/manual/decision.md;Paper-derived execution state
This branch now adapts the mechanism from SKILL.state: Scalable Long-Horizon Agent Skills (arXiv:2608.26263v2) as a cross-cutting runtime substrate rather than another Router node:
P, validated current stateΣ, and latest observationO;state_patchplus one proposedaction;procedureis authoritative,latest_observationis untrusted evidence, and host-owned task/router fields cannot be changed by model patches;Execution state does not increase tree depth, does not enter
TREE_TRACE, and cannot make Decision or Clarification automatic.Benchmark redesign
The tree benchmark is intentionally separate from the legacy flat-router/fixed-level scorer:
benchmarks/tree_topology.json: topology manifest; node names, parent/child edges and depth are data, not scorer constants.benchmarks/tree_cases.py: real-repository cases with no expected automatic route, E0-E3 label, or fixed capability path.benchmarks/tree_validation.py: no-skill + frozen v1.5 baseline + adaptive candidate + one root-to-node capability ceiling per automatic node.benchmarks/tree_analysis.py: derives minimum-sufficient node sets from stable ceiling results, then reports over/under disclosure, alternate branches, quality gaps, node marginal lift, sibling ambiguity, and topology-change candidates.benchmarks/TREE_EVOLUTION.md: add/split/merge/promote/collapse/remove rules.Automatic route exactness is no longer a release gate. Delivered quality, valid parent-child traces, zero spontaneous manual activation, and explicit-manual adherence are gates. Route disagreement is evidence that the topology boundary may be wrong.
Execution-state mechanics have their own deterministic contract in
benchmarks/skill_state_validation.py. Model-level evidence is frozen separately inbenchmarks/SKILL_STATE_MODEL_GATE.mdwith four arms: full history, state shadow, true history-freeP + Σ + O, and no-skill full history. Cost is considered only after delivered-quality, safety, topology, and manual-mode gates pass.Historical evidence
The v1.5 flat Event Router remains the frozen baseline at
ba4058b4ef47a42bf79c9963b25678a2389897c1. Existing progressive-validation artifacts, v1.5 results, and rejected-tree evidence are preserved rather than rewritten.The paper's reported accuracy, token, and bounded-context results are background evidence only; they are not treated as results for this repository. A horizon-independent prompt claim is permitted only if captured host requests actually omit prior messages and contain only bounded
P + Σ + Oinputs.Validation state
Current head:
a7ea30f4383c122e5dc07a6115d843307278a5ff.Validate skill: PASS.skills-refvalidation: PASS.n=1/ frozenn>=3matrix: pending.This PR remains draft until the fresh model-backed gates are reviewed. Final topology is explicitly not assumed to remain
Core -> {Debugging, Implementation}; benchmark evidence may merge, split, remove, promote, or deepen nodes. Execution state remains experimental until its frozenn>=3model gate shows no delivered-quality regression.