Skip to content

phase8_prereg.md

Latest commit

 

History

History
130 lines (111 loc) · 7.67 KB

phase8_prereg.md

File metadata and controls

130 lines (111 loc) · 7.67 KB
Raw

Phase 8 pre-registration: does equilibration earn its complexity?

Committed to git BEFORE any contest call runs. This document is the analysis contract; deviations must be documented as amendments in a follow-up commit, not silently applied.

Question

On a domain where the bare contestant model demonstrably fails, does the substrate's equilibration-based standing improve a small model's answers over (a) plain retrieval and (b) the same retrieval re-ranked by explicit vote-count standing WITHOUT equilibration?

Primary contrast: arm D256 vs arm C, identical retrieval, identical context format, differing only in how standing is computed (equilibrated net_score vs raw vote counting). This isolates the geometry.

Past flaws → guards (each guard is a hard gate, not a note)

# Past flaw Guard in phase 8
1 Phase 6: substrate arm silently received empty context (world never wired) Per-row invariant: arms B/C/D must have ≥3 artifacts with non-None payloads; any violation ABORTS the run. Probe metadata persisted per row.
2 v3: locate→render returned labels, not knowledge Context = full payload text. Assert mean context length in [3000, cap]; persist every prompt verbatim; sample contexts logged.
3 Phase 7: saturated domain (bare model already knew toolz), so no headroom Calibration gate: contest proceeds only if bare-model mean < 2.5/5 on the selected question set. Otherwise STOP and report.
4 v3: oracle = same model as a contestant (self-preference bias) Two graders (Opus 4.8, Sonnet 5), both ≠ contestant (Haiku). Blind, randomized answer labels, grade against a code-grounded key the grader writes first. Report inter-grader agreement.
5 Selection/regression bias risk when picking "hard" questions Questions selected on a CALIBRATION run of bare Haiku; the contest re-runs the bare arm FRESH on the selected set.
6 Context-length confound between arms Same char budget (6000), same k=5, same prompt template for B/C/D; assert per-row context lengths within 10% band across arms.

Amendment 2 (2026-07-03, before any calibration/contest call): guard #6's 10% band applies strictly within {C, D256, D64}, the arms of the primary contrast, which share an identical format. Arm B intentionally omits the claims section (that omission IS the B-vs-C treatment), so a cross-arm char band including B either aborts spuriously or forces truncating C/D payloads to B's length, distorting the primary contrast. B instead gets its own floor: per-row context ≥ 2500 chars and the same [3000, 6000] mean window. B−C remains a secondary contrast and its content asymmetry is by design. | 7 | Non-reproducibility / mid-run peeking | Every LLM call cached to jsonl keyed by prompt hash; analysis script is pure over persisted artifacts and runs only after all rows exist. |

Corpus

  • Source: extracted Claude Code work units for the autonet repo (work_units_autonet.jsonl in the substrate_experiment dir), the real session-trace corpus.
  • Amendment 1 (2026-07-03, before any calibration/contest call): work_units_autonet.jsonl turned out to hold only 17 units, too few for meaningful retrieval (k=5 of 17). Corpus source amended to: fresh extraction (same extract_sessions.py pipeline) from the C--code-autonet and C--code-world-model session transcripts as of 2026-07-03, merged with the legacy 17 and deduplicated by uid. All other sampling rules unchanged. Questions and judge claims are regenerated against the amended sample; nothing downstream had run.
  • Amendment 3 (2026-07-03, before any calibration/contest call): the amended extraction yielded only 44 units (the world-model project transcripts were deleted from disk after May; main-session-only autonet extraction is small). Corpus source amended to additionally include (a) subagent-session work units (pipeline default excludes them as "operational helpers", but they carry real implementation knowledge and the legacy world-model extraction included them; yields ~194 autonet units), and (b) the surviving May 2026 world-model extraction (13 units; source transcripts no longer exist). Merge, dedupe by uid, same deterministic 200-cap sampling. Known limitation: one near-duplicate pair whose outcome drifted between extractions survives uid-dedupe.
  • Sample: 200 units, deterministic (sort by sha256 of the unit's JSON, take first 200). Cap chosen for substrate build tractability (~15 s/unit); reported as a limitation.
  • Judge claims: one Sonnet 5 call per unit produces 2 to 3 structured sub-claims (axes: correctness, simplicity, robustness), each PRO or CON with honest CONs required. Cached; the SAME cache feeds arms C and D so verdict content is identical across them.

Questions

  • 40 questions generated by Opus with access to the sampled corpus + the autonet substrate code (nodes/common/world_model_substrate/, world_model/generalized/). Categories: mechanism, architectural, code-reference, tradeoff, gotcha (8 each). Must be answerable from corpus/code content.
  • Calibration: bare Haiku answers all 40; both graders score; select the 25 with lowest mean bare score as the contest set (ties broken by question id hash). Gate #3 applies to this set.

Arms (contestant: Haiku via bridge, one completion per question per arm)

  • A bare: no context. (Fresh run on the selected set; guard #5.)
  • B rag: top-5 artifacts by cosine (dim 256), payload text as context.
  • C ledger: same top-15 candidates as B, re-ranked by cosine × (1 + tanh(vote_standing)), top-5 kept. vote_standing = Σ(+1 per PRO judge claim, −1 per CON) + outcome signal. NO world, NO equilibration. Context = payloads + judge-claim texts as verdicts.
  • D256: full two-plane path. World seeded with the 200 work-unit claims + judge sub-claims (per two_plane_inference.md ingestion), equilibrated; standing = Σ net_score(PRO) − Σ net_score(CON); re-rank identical in form to C. Same embedder/dim as B/C so retrieval candidates are identical. Context format identical to C.
  • D64: as D256 with embedding dim 64 (secondary: answers POST_AUTONET question "should embedding_dim drop?").

Grading

Per question: grader receives the question + relevant code files, writes a reference key, then scores the five blinded, order-randomized answers 1 to 5 on correctness / completeness / reference-accuracy (overall = mean). Both graders grade everything. Metric = mean of the two graders. Report Pearson r and mean absolute disagreement between graders; if r < 0.4 the run is declared ungradeable (stop, revise rubric, amend).

Analysis (pre-registered)

  • Paired per question. Contrasts, Holm-corrected ttest_rel + bootstrap 95% CIs: primary D256−C; secondary B−A, C−B, D64−D256.
  • Decision rule: equilibration is retained iff D256 > C at Holm-corrected p < 0.05 AND mean difference ≥ 0.25 (5-pt scale). Otherwise the gate in the course-of-action stands: mint pricing moves toward explicit debate standing (economics change ratified by the user), equilibration demoted to experimental kernel.
  • Expansion rule: if the primary CI spans 0 with |mean diff| < 0.3, expand once to 50 questions (new questions, same protocol), then final.
  • No other subgroup analyses are confirmatory; anything else reported is labeled exploratory.

Artifacts (all under substrate_experiment/phase8/)

corpus_sample.jsonl, judge_cache.jsonl, questions.jsonl, calibration.jsonl, selected_questions.jsonl, contest_rows.jsonl (one row per question with all arms' prompts, answers, probe metadata), grades.jsonl, aggregate8.json, llm_cache/ (all raw calls), run.log. Analysis in analyze.py, pure over these files.