Correct false and stale claims across the READMEs (plus research-loop hardening)

.gitignore

@@ -40,11 +40,45 @@ SmolLM2-134(base)/results/lm_eval/
 # Claude Code state (skills, settings, scheduled tasks, session caches)
 .claude/
 
-# Research-loop working tree — churning state (ledger.json, loop_state.json),
-# generated artifacts, concurrent-session output, manuscript packages. Kept
-# local only; already-committed hardening files (ledger.py, sentinel tests,
-# eval_stats.py, CI) stay tracked. Durable ledger backup: research/backup_ledger.sh.
-research/
+# Research-loop working tree. Durable SOURCE is tracked by rule; churning STATE
+# and generated artifacts stay local-only.
+#
+# The previous blanket `research/` made backup depend on accident of history: a
+# file was safe only if it happened to be committed BEFORE that line was added.
+# Everything written after it — ~70 load-bearing, unit-tested .py/.sh files
+# including posttrain_losses.py, scaling_ladder.py, boot_resume.sh, thermal_log.py,
+# cce_*.py and 11 test modules — had NO backup anywhere, while their siblings
+# (ledger.py, sentinel tests, eval_stats.py, CI) stayed tracked. It also silently
+# overrode research/.gitignore, whose own stated policy is to "keep the durable
+# record tracked; exclude only transient/runtime files". Source now follows that
+# policy again; state does not.
+#
+# The DURABLE RECORD is now tracked too (2026-07-22 decision): ledger.json is the
+# truth store and the runs/*.md + briefs/ + digests/ are the human-readable record —
+# a one-disk truth store was the audit's #3 risk, and a branch switch already
+# destroyed a ledger.json once (recovered from a git blob). Git history IS the
+# off-box backup; commit the record before any branch switch. The single-generation
+# .bak and the dated backups/ snapshots stay local (git history is the real trail).
+#
+# Deliberately still local-only: loop_state.json (churns every loop wake — a stale
+# tracked copy is worse than none), generated non-record artifacts (pulse/, radar/,
+# provenance/), prepared datasets, manuscript build output, checkpoints/logs, and
+# harness-search's machine-GENERATED candidates (search output, not authored source).
+research/**
+# re-include directories so git descends and the rules below can match
+!research/**/
+!research/**/*.py
+!research/**/*.sh
+# the durable record (small text; the truth store + its human-readable trail)
+!research/ledger/ledger.json
+!research/ledger/runs/**/*.md
+!research/briefs/**/*.md
+!research/digests/**/*.md
+# ...but never the churn/backup copies of the ledger
+research/ledger/ledger.json.bak
+research/ledger/backups/
+research/harness_search/archive/
+research/harness_search/targets/*/candidates/
 
 # Skills showcase (Claude skill .md files + demo site) — kept local only
 skills_showcase/

HybridSSM-0.2B/.gitignore

@@ -0,0 +1,10 @@
+# HybridSSM-0.2B: commit code + docs only; exclude training artifacts
+*.pkl
+*.pkl.tmp
+*.log
+*.stdout
+__pycache__/
+*.pyc
+*.thermal.log
+probe*.log
+sentinel_kill_*.json

HybridSSM-0.2B/experiments/2026-07-19_hybrid-ssm-0.2b_build/ARCHITECTURE.md

@@ -0,0 +1,55 @@
+# HybridSSM-0.2B — architecture design (novel from-scratch build, JAX/Flax)
+
+**Purpose:** study the attention-vs-efficient-mixer composition (brief `hybrid-attention-rethink`,
+arXiv 2606.15378) on a single GB10, on this repo's evidence standard. Novel design → **no bit-exact
+oracle**; the verify gate is numerical cross-check vs an independent reference at ~1e-2 (§C14/JAX).
+Framework **JAX/Flax** (user-confirmed 2026-07-19; installed + verified, `associative_scan` runs).
+
+## Config (base arm)
+
+| field | value | why |
+|---|---|---|
+| d_model | 768 | ~146M non-embed at L=24 (paper S4 104M < this < S5 477M) |
+| n_layers | 24 | 1:1 interleave → 12 full-attention + 12 efficient-mixer |
+| layer pattern | `[full, eff, full, eff, …]` | paper's 1:1 main setting (1:3 ≈ same val loss is a later arm) |
+| full-attn | GQA n_heads=12, n_kv=4, head_dim=64, RoPE (θ=1e4) | Qwen3-family attention, shrunk |
+| efficient mixer | **Mamba-2-style selective SSM** via `associative_scan` (diagonal linear recurrence + input/gate proj) | JAX-native scan; the SWA-128 and GatedDeltaNet variants are ablation arms |
+| MLP | SwiGLU, intermediate 2048 | Qwen3 recipe |
+| norm | RMSNorm (eps 1e-6), pre-norm | Qwen3 recipe |
+| vocab / tokenizer | **151,936 (Qwen3-0.6B-Base)** | reuses the validated text-lm-v2 data + eval pipeline; BPB-comparable to the 596M study. Report NON-EMBED params (paper convention). |
+| tied embedding | yes | 117M embed counted once |
+| CE | **chunked** over vocab (152k > 64k, §C1) | never materialize (N,152k) logits — the box-crash vector |
+| seq_len (pretrain) | **4096** (probe decides; 16K is the paper's, memory-tight here) | emergence-speed + relative-hybrid comparison is visible at 4K; long-context/NoPE finding needs a later ctx-extension arm |
+| optimizer | Muon (2D weights) + AdamW (1D/embed) — JAX port of `normuon.py` | paper uses Muon; repo has the PyTorch impl to port + cross-check |
+| precision | bf16 compute, fp32 master/optimizer state | Qwen3 recipe |
+
+## The toggles that make it a STUDY (single-variable ablation matrix)
+
+Each is one flag on the base arm; iso-FLOP where the flag changes params (≤5%, §C18); ≥3 seeds; BPB
+CIs on wikitext-2 + code (text-lm-v2) + a long-context retrieval probe (RULER-NIAH-style):
+
+1. **mixer type** on the efficient layers: `ssm` (Mamba-2) vs `swa128` (sliding-window attn, w=128) vs
+   `full` (all-attention control = the dense baseline, comparable to the 596M study) vs `none` (all-SSM).
+2. **attention fraction / placement**: 1:1 vs 1:3 (one full-attn per three efficient) — the "how much
+   attention does a hybrid need" curve.
+3. **NoPE-on-full-attention** (the paper's headline design knob): RoPE vs NoPE on the full-attn layers of
+   the SWA-128 hybrid — predicted long-context gain, ~zero short-context cost.
+4. **token-budget ladder** (the emergence-speed instrument, mirroring the scaling-persistence study): score
+   each hybrid at increasing budgets → does the efficient-mixer choice affect emergence SPEED but converge?
+
+Headline object = the **emergence-speed curve** (quality vs tokens per hybrid) + a box-scale validation of
+NoPE-on-full-attn. This is the same *shape* as "the disappearing win" — a coherent next chapter.
+
+## Files (novel design → its own folder, not canonical)
+
+- `model.py` — the JAX/Flax hybrid model (implemented from blank).
+- `ssm.py` — the selective-SSM mixer (associative_scan) + the SWA mixer.
+- `muon_jax.py` — JAX Muon (ported + cross-checked vs `normuon.py`).
+- `verify.py` — numerical cross-check vs an independent reference (~1e-2) + shape/dtype grid.
+- `train.py` — training loop (chunked CE, safe guards, ckpt/resume for the recovery chain).
+
+## Guards (§C1)
+
+`import jax_safe_env` BEFORE `import jax` (PREALLOCATE=false, MEM_FRACTION=0.5). Chunked CE for the 152k
+vocab. sentinel preflight before any GPU work; sentinel watch + the hardened `boot_resume.sh` recovery
+chain + thermal kill beside any unattended trainer.

HybridSSM-0.2B/experiments/2026-07-19_hybrid-ssm-0.2b_build/BUILD_STATUS.md

@@ -0,0 +1,109 @@
+# HybridSSM-0.2B — build status (updated 2026-07-20 04:15 UTC)
+
+**Novel from-scratch hybrid attention-SSM LM in JAX/Flax is BUILT, VERIFIED, SMOKE-PASSED — and the
+first pretrain arm (`ssm_base_s0`) is IN FLIGHT on real data.**
+
+## Build phase — done ✓ (2026-07-19, all correctness-gated)
+
+- **Architecture** (`ARCHITECTURE.md`): d=768, 24 layers (1:1 full-attn:efficient interleave), GQA 12/4,
+  SwiGLU, RMSNorm, RoPE↔NoPE toggle, Qwen3 152k tokenizer, chunked CE. Design doc estimated ~146M
+  non-embed; the built model reports **189.1M non-embed / 305.8M total** (`[build]` line of every run
+  log) — tied embedding = 151,936 × 768 = 116.7M counted once.
+- **Implementation** (`ssm.py`, `model.py`): SelectiveSSM (Mamba-2-style diagonal scan via
+  `associative_scan`) + SlidingWindowAttention + GQA attention + the full hybrid decoder. Written from blank.
+- **Verify gate** (`verify.py` → **PASS**): SSM parallel-scan == sequential reference (max|Δ|=2.4e-7);
+  chunked CE == naive CE (|Δ|=4.8e-5, never materializes the 152k logits); param count sane; all 8
+  ablation toggles forward-finite; forward deterministic. ⚠️ **See "Open gate gap" below — this PASS
+  predates the `nn.remat` memory fix and has not been re-run since.**
+- **Smoke** (`train.py --smoke` → **PASS**, all variants): SSM / SWA-128+NoPE / 1:3-attention each overfit
+  a fixed batch 8.8 → ~0.003 loss (forward+backward+AdamW+chunked-CE all learn), grad norms healthy
+  (33 → 0.03), checkpoint save→reload exact (max|Δ|=0.0 — recovery-chain ready). Smoke used synthetic data.
+- **Fit probes on real data** (`probe.log` / `probe2.log` / `probe3.log`, 15 / 12 / 30 steps):
+  step-0 loss 12.4317 / 12.4312 / 12.4312 ≈ ln(151936)=11.93 + init noise, and 30 steps moves 12.43 → 8.42.
+
+## Pretrain arm `ssm_base_s0` — COMPLETE (2026-07-20 15:04 UTC)
+
+Exited cleanly on its own: `[done] 21156 steps · final loss=3.8149`, `arm_ssm_base_s0.done`
+written, final checkpoint saved, and the sentinel disarmed itself (`watched pid 3164922
+exited on its own; disarming (no kill)`). Wall clock **990 min / 16.5 h** for the final
+process — this excludes the killed first attempt, whose start time is not on disk, so
+total GPU time is a lower bound.
+
+| result | value |
+|---|---|
+| final train loss | **3.8149** |
+| best val loss | **3.7839** @ step 19,200 (final eval 3.9245 @ 20,800 — noisy tail) |
+| eval-harness `text-lm-v2` | PPL wikitext2_val **133.4628**, code_py **5142.6426** (`self_floor=true`; corpora pinned `wikitext-2-raw-v1:validation@b08601e`, `codeparrot-clean-valid@4db92d2`) |
+| verdict | **directional** — n=1 seed, no comparand, no iso-FLOP match (§C17/§C18/§C25) |
+
+**Verify gate CLOSED.** `verify.py` was re-run 2026-07-20 17:46 against the post-`nn.remat`
+`model.py` (last modified 07-19 22:27) — `verify.log`, 6/6 PASS, exit 0: scan-vs-reference
+max|Δ|=2.38e-07, chunked-vs-naive CE |Δ|=4.77e-05, param count, forward finite/deterministic,
+all 8 toggle combos finite.
+
+### ⚠️ Budget overshoot — a resume bug, and it matters for the ladder
+
+`train_hybrid.py:129` is `for s in range(start_step, start_step + steps)`. A **resumed** run
+therefore repeats the FULL step budget from the resume point instead of finishing the
+original one. This arm resumed at step 400, so it ran **21,156 steps = 173,309,952 tokens
+against a declared budget of 170,034,304 (+1.93%)**, wrapping ~1.9% into a second epoch.
+
+Harmless at n=1, but it silently breaks **§C18 iso-FLOP** across arms: any arm that crashes
+and resumes gets *more* compute than one that doesn't, scaling with the resume point — a
+resume at step 5,000 would be **+24%**, far past the 5% tolerance, and nothing in the logs
+would flag it. **Fix to `range(start_step, steps)` before running the ladder.**
+
+## Pretrain arm `ssm_base_s0` — configuration as launched
+
+Ledger run `2026-07-19_hybrid-ssm-0.2b_pretrain-ssm-base-s0` (type=ablation, status=running,
+lifecycle_stage=architecture, framework=jax, technique `hybrid-attention-rethink`).
+
+| field | value | source |
+|---|---|---|
+| data | FineWeb-Edu sample-10BT, Qwen3-0.6B-Base tokenizer, **170,034,304 train + 300,000 val** tokens, seed 0 | `tokcache_170034304_300000_seed0_Qwen3-0.6B-Base.pt`, built by `Qwen3-0.6B/builds/2026-06-08_reproduce-faithful_qwen3-0.6b/train_qwen3.py:151` |
+| config as launched | seq **2048**, batch **4**, 20,756 steps × 8,192 tok/step, AdamW lr 3e-3, warmup 200 | live cmdline of PID 3164922 |
+| trainer / watchdog | PID 3164922 (`train_hybrid.py`) · sentinel PID 3167084 | `pgrep`, `sentinel.log` |
+| progress @ 04:15Z | **step 7,480 / 20,756 (36.0%)** — 61.3M of 170.0M tokens | `run_ssm_base_s0.log` |
+| loss | step-0 12.4317 → train ~4.88; val 6.6844@400 → 6.2845@1200 → **4.9020@7200** (best) | `run_ssm_base_s0.log` |
+| grad norm | 0.28–0.31, stable | `run_ssm_base_s0.log` |
+| throughput | ~1,247 steps/h ≈ **2,837 tok/s** (measured over 7,080 steps / 5.68 h since resume) | derived from log + process start |
+| ETA | ≈ **2026-07-20 14:54 UTC** (13,276 steps remaining) | same |
+| memory | pool 37–41%, rss 11.6 GiB, GPU 66–69 °C / SoC 72–74 °C | `sentinel.log` heartbeats |
+
+**Deviations from `ARCHITECTURE.md`, recorded honestly:** the design doc specifies seq_len 4096 and
+Muon(2D)+AdamW(1D); this arm runs **seq 2048 with plain AdamW**. The JAX Muon port (`muon_jax.py`) is not
+written yet — AdamW-vs-Muon is itself a planned arm, and every arm in the ladder must use the same
+optimizer for the comparison to hold, so the ladder's baseline optimizer is now AdamW unless re-based.
+
+### Incident + recovery (the run survived a real kill)
+
+First launch was killed by the sentinel at **step 580, 2026-07-19T16:58:48Z** — pool usage 81.3% ≥ the
+0.80 kill line (MemAvailable 22.4 GiB / 119.7 GiB; SSM scan + chunked CE under autodiff held ~61 GB);
+GPU 58 °C, no thermal component (`sentinel_kill_step580_2026-07-19.json`). Fix: **`nn.remat` on the
+decoder block** (`model.py:129`, `BlockR = nn.remat(Block)`) + batch 8 → 4 → allocation 61.5 GB → 16.6 GB,
+pool 81% → ~40%. Resumed from the step-400 checkpoint at 22:34:29 and has run clean since.
+This was a *manual* recovery behind a config change, i.e. the §C5/S1-4a "not safe to auto-resume at the
+same config" path — `loop_state.auto_resumes` correctly stayed at 0.
+
+## Gate gap — CLOSED 2026-07-20
+
+For the record, since it was flagged as blocking while the arm ran: `verify.py`'s original PASS
+(2026-07-19 12:52) predated the 22:27 `nn.remat` change, so for the whole run the gate was stale
+against the model actually training. It was re-run **2026-07-20 17:46**, after the arm finished
+(GPU work — §C4.5 forbids co-running it beside a live trainer), and captured to `verify.log`:
+**6/6 PASS, exit 0**. `nn.remat` is confirmed value-preserving here, as expected.
+
+## Next
+
+1. **Close the verify gap** (above) + write `verify.log`, so the artifact set is self-evidencing.
+2. **Score the finished arm** via `/eval-harness` (BPB on wikitext-2 + code, text-lm-v2, `suite_version`
+   stamped) → write `verdict.json`. A single arm is a *baseline datum*, not a win: no cross-arm claim
+   exists until the ladder has ≥3 seeds and iso-FLOP-matched comparands (§C17/§C18), so the terminal
+   verdict for this arm caps at `directional` at best (§C25).
+3. **The emergence-speed ladder** (the study): mixer type × attention fraction × NoPE, each scored at
+   increasing token budgets → does the efficient-mixer choice affect emergence SPEED but converge? Plus
+   the NoPE-on-full-attn validation. Multi-day; drive arms as they complete.
+4. **Recovery chain**: the user still pastes the `@reboot bash research/boot_resume.sh` cron line
+   (§C4.2 — the agent never auto-installs cron).
+5. **Downstream lifecycle** (long-context retrieval probe → data / mid-training / SFT), each to a §C25
+   terminal verdict — the whole-lifecycle finish line for the new model.