Record: 11L + Score-Every-Epoch LoRA TTT 5ep (3-seed mean val_bpb=0.8173)

[minh-stakc]

Summary

• 3-seed mean val_bpb: 0.8173 (std 0.0023)
• Pre-TTT sliding window: ~1.126 BPB
• TTT improvement: -0.309 BPB
• Built on PR Record: 11L EMA + GPTQ-lite + warmdown3500 + QAT@0.15 (val_bpb=1.1233) #414 base architecture

Results

Seed	Pre-TTT BPB	Post-TTT BPB	Artifact
42	1.1264	0.8186	17.13 MB*
1337	1.1260	0.8187	17.13 MB*
7	1.1275	0.8146	17.13 MB*
Mean	1.1266	0.8173
Std	0.0008	0.0023

*Artifact exceeds 16MB on B200 without FlashAttention 3. Requires 8xH100 validation.

Key Innovation: Score-Every-Epoch Multi-Scale LoRA TTT

Per-document LoRA adaptation where each epoch re-scores all chunks with progressively better-adapted weights. Only the final epoch's scores contribute to BPB.

• LM-head LoRA rank-16 (2x LR)
• V projections rank-8 (1.5x LR), Q projections rank-8 (0.5x LR)
• Per-block bias tuning (3x LR)
• Cosine LR decay across total TTT steps
• Post-TTT temperature T=0.98

Architecture (PR #414 stack)

11L, d=512, 8H/4KV GQA, MLP 3x, XSA4, Partial RoPE, LN Scale, EMA(0.997), GPTQ-lite int6 + zstd-22, SmearGate, BigramHash(2048), VE128, Muon WD=0.04

Credits

• Base: PR Record: 11L EMA + GPTQ-lite + warmdown3500 + QAT@0.15 (val_bpb=1.1233) #414 (@signalrush)
• TTT: PR [record bpb=1.195] sliding window + LoRA TTT #77 (@samacqua), PR Record: PROTEUS v8 — 11L INT6 + LoRA TTT 5ep cosine (mean val_bpb=0.7853, 4 seeds) #568 (@MatoTeziTanka), PR Record: DeepQuant V10b — 11L INT6 + 8ep LoRA TTT (val_bpb=0.6430) #596 (DeepQuant)
• Per-layer LR: PR Record: Cosine TTT scheduling with per-layer lr — mean val_bpb=1.0970 (3 seeds) #481 (@mrdavtan)

Test plan

• 3-seed validation (mean 0.8173, std 0.0023)
• 8xH100 artifact size verification
• Eval time under 600s on 8xH100
• 8-epoch TTT experiments (in progress, targeting sub-0.7)

[valerio-oai]

Per-document LoRA adaptation where each epoch re-scores all chunks with progressively better-adapted weights. Only the final epoch's scores contribute to BPB.

This is equivalent to training on test, this is an invalid submission, closing this PR for now.