Record: LeakyReLU² + Legal Score-First TTT + Parallel Muon — val_bpb 1.1194 (3-seed mean) by abaybektursun · Pull Request #549 · openai/parameter-golf

abaybektursun · 2026-03-23T16:08:26Z

Record: LeakyReLU² + Legal TTT + Parallel Muon — val_bpb 1.1194

val_bpb = 1.1194 (3-seed mean, std 0.0006) | ~15.95 MB | 8×H100 SXM

3-Seed Results (8×H100 80GB SXM, PyTorch 2.9.1+cu128)

Seed	step_avg	steps	Pre-TTT bpb	Post-TTT bpb	TTT gain	TTT time	Artifact
1337	83.3ms	7,179	1.1217	1.1192	-0.0025	410s	15,977,386
42	83.4ms	7,182	1.1227	1.1200	-0.0027	408s	15,876,510
2025	83.4ms	7,193	1.1212	1.1189	-0.0023	408s	15,990,006
Mean	83.4ms	7,185	1.1218	1.1194 (std 0.0006)	-0.0025	~409s

Key Innovation: LeakyReLU(0.5)²

One-line activation change delivering -0.003 BPB vs standard relu²:

# relu² (standard)
x = torch.relu(self.fc(x)).square()
# leaky relu² (this submission)
x = F.leaky_relu(self.fc(x), negative_slope=0.5).square()

Preserves negative gradient flow through the MLP. Source: PR #493 by @parinzee (ablated at -0.003), PR #518 by @sofiabod.

Legal TTT (Score-First, PR #461 Framework)

Every token scored BEFORE any weight update, enforced by torch.inference_mode():

for each 32K-token chunk:
    Phase 1 — SCORE: sliding window eval (inference_mode)
    Phase 2 — TRAIN: SGD(lr=0.002, mom=0.9), 3 epochs, all blocks unfrozen

Adapted from PR #461 by @Christopher-Lee-McClendon (changed freeze=2 → freeze=0 based on our ablation showing unfreezing all blocks is optimal at 3 epochs).

Total eval: ~530s (120s standard + 409s TTT) — within 10 min limit.

Training Architecture

PR #414 stack + Parameter Banking + Parallel Muon (PR #399):

11L, 512d, 8H/4KV, LeakyReLU(0.5)² MLP 3×
BigramHash(1536), XSA4, Partial RoPE, LN Scale, VE128
EMA(0.997) + Tight SWA, GPTQ-lite int6 + lzma
Parameter Banking + Parallel Muon (83.4ms/step)

Credits

LeakyReLU²: PR #493 by @parinzee, PR #518 by @sofiabod
Optimizer: PR #399 by @abaybektursun
TTT recipe: PR #461 by @Christopher-Lee-McClendon (adapted: freeze=0)
Base model: PR #414 by @signalrush

🤖 Generated with Claude Code

…ed mean) LeakyReLU(0.5)² activation (-0.003 vs relu²) + legal score-first TTT (PR openai#461 recipe, 3ep SGD, all blocks unfrozen) + BigramHash(1536) on openai#414 stack with Parameter Banking + Parallel Muon (PR openai#399). 3-seed results: Seed 1337: 1.1192 bpb, 410s TTT, 15.98 MB Seed 42: 1.1200 bpb, 408s TTT, 15.88 MB Seed 2025: 1.1189 bpb, 408s TTT, 15.99 MB Mean: 1.1194 (std 0.0006) All artifacts under 16MB. All eval under 10 min. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

11L, XSA all layers, partial RoPE 16/64, LN scale, VE128 (layers 9,10), LeakyReLU(0.5)² activation, BigramHash(2048), INT6+zstd-22. Legal score-first TTT: 32K chunks, all blocks, SGD(0.002,mom=0.9), 3ep. Base: PR openai#503 (EthanYangTW) + LeakyReLU² from openai#518/openai#549 + SGD from openai#549. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…stopher-Lee-McClendon

…stopher-Lee-McClendon

Multiple top PRs (openai#535, openai#549, openai#569) demonstrate -0.0015 to -0.003 bpb from this change. LeakyReLU preserves gradient flow through negative pre-activations while maintaining the sparsity/gating benefits of squaring. At 22M params, dead neurons from hard ReLU are expensive. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

valerio-oai · 2026-03-24T13:23:11Z

Looks legal, clears the 0.005 nats test, so merging into the leaderboard. Well done!

abaybektursun · 2026-03-24T13:32:23Z

Looks legal, clears the 0.005 nats test, so merging into the leaderboard. Well done!

ayeee

abaybektursun · 2026-03-24T13:33:44Z

@valerio-oai just noticed there's a wrong user name in the leaderboard.

Four novel improvements over PR openai#549 (1.1194 BPB) base: - Full GPTQ quantization with Hessian-guided error compensation - Soft-round QAT with tanh-based temperature annealing - LoRA-based test-time training (rank-8 adapters on Q/K/V/O) - Entropy-coded compression (Huffman+LZMA adaptive selection) Made-with: Cursor

Track: 10min_16mb Based on: PR openai#549 (LeakyReLU+ParallelMuon), PR openai#606 (Soft-Round+AdamW TTT), PR openai#609 (XSA-all+Full GPTQ) Changes from SOTA (openai#549): - XSA on all 11 layers (was 4) - Soft-Round QAT with tanh-based differentiable rounding (alpha 1->16) - Full GPTQ with Hessian-aware column-reordered Cholesky error compensation - MHA 8/8 (was GQA 8/4) - MLP 3.5x expansion (1792 hidden, was 3.0x/1536) - BigramHash vocabulary 8192 (was 2048) - AdamW TTT with grouped LR and cosine schedule (was SGD) - Early QAT threshold 0.5 (was late 0.15) - Selective ±1 magnitude pruning to hit size target

valerio-oai · 2026-03-24T17:15:56Z

whoops, really sorry about the wrong username -- I thought something looked wrong! Fixing it now

Run 0: PR openai#549 UNMODIFIED (merged SOTA 1.1194, verified 3-seed) Run 1: PR openai#549 + TTT_ENABLED=1 + TTT_LR=0.0005 (2 lines changed) Both have FA3→FA2→SDPA fallback for non-Hopper GPUs. Following retro: one change per run, baseline first. Expected: Run 1 should achieve ~1.094-1.104 (beats 1.1144 target). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Documents merged SOTA of 1.1194 (PR openai#549, LeakyReLU² + Legal TTT + Parallel Muon), confirmed technique deltas, enforcement ruling on GPTQ calibration, and the path forward to beat 1.1144. https://claude.ai/code/session_01U3LXGzTkedd9ZcHF2qgW7d

Run 0: PR openai#549 UNMODIFIED (merged SOTA 1.1194, verified 3-seed) Run 1: PR openai#549 + TTT_ENABLED=1 + TTT_LR=0.0005 (2 lines changed) Both have FA3→FA2→SDPA fallback for non-Hopper GPUs. Following retro: one change per run, baseline first. Expected: Run 1 should achieve ~1.094-1.104 (beats 1.1144 target). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Case study: reordering training shards by model difficulty (hardest first) gives -0.0033 BPB improvement over sequential ordering. Zero architecture changes, zero compute cost, ten lines of code. Key finding: token-level statistics (KL divergence) find 0.0009 range across shards. Model perplexity finds 0.0475 range -- 100x more variation. The two metrics are uncorrelated (r = -0.056). 3-seed validated on PR openai#549 (merged openai#1): Seed 1337: 1.1217 -> 1.1183 (-0.0034) Seed 42: 1.1222 -> 1.1181 (-0.0041) Seed 2025: 1.1221 -> 1.1198 (-0.0023) Mean: 1.1220 -> 1.1187 (-0.0033) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Preliminary non-record run: val_bpb=1.1882 (seed 1337, 2002 steps, no torch.compile). Artifact 18.8MB (over 16MB limit) — proper rerun with torch.compile pending. Additions over PR openai#549 (SOTA 1.1194): - VRL: Value Residual Learning on all 11 layers via sigmoid gates - Full GPTQ: Hessian Cholesky int6 with 256-batch calibration - BigramHash 1536 → 3072 - Tight SWA preferred over EMA when snapshots exist Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

- shadeform_dispatch.sh: direct REST API, picks cuda12.4 OS image, flash-attn installs from pre-built wheel in 10 sec (vs 20 min build) - shadeform_cleanup.sh: DELETE instances via API - shadeform_reconcile.sh: reconcile leases via API - branch_cycle.sh: add shadeform backend case - supervisor.sh: add shadeform reconcile - start_swarm.sh: add shadeform preflight check - train_gpt.py: SOTA PR openai#549 recipe (LeakyReLU + Legal TTT + Parallel Muon) with FA3→FA2 import swap (identical API, ~10% slower) Usage: DISPATCH_BACKEND=shadeform scripts/start_swarm.sh --pipeline --workers 2 --watchers 2

3-seed mean 0.8609 bpb (42→0.8600, 1337→0.8611, 2025→0.8616). All artifacts under 16MB. 11-gram n-gram cache with entropy-adaptive alpha and Hedge Mixer on PR openai#549 base architecture. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Built on PR openai#549 stack. Adds document-isolated TTT (reset optimizer at BOS boundaries) and temperature scaling. Pending 8xH100 validation.

Maps every top entry through BPB = L + Q + T + M: - openai#700 solved M (mixer) but has worst L (training) - openai#609 solved Q (quant) but has zero T and M (no eval pipeline) - openai#549 solved L (training) but has zero M (no mixer) - Nobody has optimized all four terms simultaneously - Theoretical optimal = 1.052 (combine best of each) - Our Track B path to 1.025 via recurrence + FiLM-only TTT + Mixer Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Co-Authored-By: Kevin Francis Tan <kf.tan@lightarchitects.io>

…eframe Corrections: - T+M are combined (-0.020), not separate. PR openai#700 gets -0.073 (3.6x better) - Our Q gap (0.066) is larger than the openai#549-openai#700 total gap — Q is THE bottleneck - Added "Best Known" column comparing against best per-term, not just merged SOTA New insights added: - Kaplan width scaling, hidden ≥ 512 threshold, Goldilocks depth - MoE viability at small scale (inactive experts compress well) - Vocab expansion opportunity (mechanical BPB reduction) - Compression reframe: BPB competition = compression competition, 20 years of literature - Strategic evolution: feature bloat → simplify → Q bottleneck → compression-first approach - Theoretical optimal 1.052 = combine best of openai#549 + openai#609 + openai#700 (nobody has done this) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Co-Authored-By: Kevin Francis Tan <kf.tan@lightarchitects.io>

- Fork of the openai#1 leaderboard train_gpt.py (LeakyReLU², XSA, EMA, Parallel Muon, TTT, GPTQ) with minimal changes to support BESE tokenizer - Dual tokenizer dispatch: .json loads BESE, .model loads SentencePiece - All SOTA architecture preserved, only tokenizer loading changed - Add --sota flag to runpod_v2.py to select SOTA train script

3-seed mean 0.8609 bpb (42→0.8600, 1337→0.8611, 2025→0.8616). All artifacts under 16MB. 11-gram n-gram cache with entropy-adaptive alpha and Hedge Mixer on PR openai#549 base architecture. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

slope 0.75 + LR 0.027 + warmdown 3700 (PR openai#977) No SWA with QAT (PR openai#989) QAT from 50% + range fix [-31,31] mHC 22-param residual mixing (PR openai#928) VE128 + no gated_attn + no value_residual (PR openai#549) LZMA preset 7 compression (PR openai#999) Muon TTT with NS3 (PR openai#999) Entropy-adaptive TTT epochs 2/3/4 (PR openai#999) Per-layer TTT LR (PR openai#995) TTT momentum 0.95 (PR openai#995)

…ssion 3-seed results (1337, 42, 45): mean val_bpb=1.1264, artifact ~15.8MB. Forked from PR openai#549 (1.1194 SOTA). Adds VRL, BigramHash 3072, Tight SWA, zstd-22, sliding window eval fix. Drops Full GPTQ. TTT enabled by default.

SOTA PR openai#549 code + JEPA (Joint-Embedding Predictive Architecture): - LatentProjector encoder/predictor/target modules - Multi-horizon future prediction (1,2,4,8 steps) - VICReg-style variance/covariance regularization - Target encoder updated via EMA (decay=0.996) - Toggled via JEPA_ENABLED env var (default: off) Waiting on RunPod credits to measure BPB + ms/step impact. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

…enizer Changes from PR openai#549 record (LeakyReLU² + Legal TTT + Parallel Muon): - vocab_size: 1024 → 16384 (3.92 bytes/token = structural BPB gain) - data/tokenizer paths: sp1024 → sp16384 - SVDEmbedding(rank=32): factored embedding saves ~3.5MB compressed U(16384×32) + S(32) + V(32×512) = 540K params vs 8.4M standard - Artifact estimate: ~15.9 MB (fits 16MB limit) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

mrdavtan · 2026-03-29T00:24:18Z

Note: cosine LR decay was introduced in #481. Glad to see it here!

abaybektursun force-pushed the submission/leaky-relu-legal-ttt-1.1183 branch from f6a0b0d to 8ff3e0e Compare March 23, 2026 16:27

abaybektursun changed the title ~~Record: LeakyReLU² + Legal Score-First TTT + Parallel Muon — val_bpb 1.1195 (3-seed mean)~~ Record: LeakyReLU² + Legal Score-First TTT + Parallel Muon — val_bpb 1.1194 (3-seed mean) Mar 23, 2026

abaybektursun added 2 commits March 23, 2026 12:13

Fix pre-TTT BPB, TTT gains, and steps to match logs exactly

139d573

Fix author attributions: PR openai#493 @parinzee, PR openai#461 @Chri…

b08d72a

…stopher-Lee-McClendon

notapplica mentioned this pull request Mar 23, 2026

⛳ Parameter Golf Live AI Commentary ⛳ + Analysis / Ideas | every 10 minutes #140

Open

valerio-oai approved these changes Mar 24, 2026

View reviewed changes

valerio-oai merged commit 2377f43 into openai:main Mar 24, 2026

valerio-oai mentioned this pull request Mar 24, 2026

Record: Late Soft-Round QAT + Score-First Backward-Looking TTT — val_bpb 1.1178 #589

Closed

Christopher-Lee-McClendon mentioned this pull request Mar 24, 2026

Non-Record: BPB 1.1334 — 7000-Step Training + Mixed Int6/Int8 Quantization + Legal TTT #598

Open

valerio-oai mentioned this pull request Mar 24, 2026

Record: PROTEUS v8 — 11L INT6 + LoRA TTT 5ep cosine (mean val_bpb=0.7853, 4 seeds) #568

Closed

Christopher-Lee-McClendon mentioned this pull request Mar 24, 2026

Non-record: 11L GEPA + 20k Steps + Pure Int6 + Legal TTT (val_bpb=1.0983): unlimited compute: 4×A100-40GB, ~2.8 hours #628

Open

senstar-hsoleimani mentioned this pull request Mar 24, 2026

submission: XSA-all + Soft-Round QAT + Full GPTQ + MLP 3.5x + AdamW TTT #631

Open

abaybektursun mentioned this pull request Mar 24, 2026

Record: Parallel Muon + Parameter Banking — 81.87ms/step, val_bpb 1.1247 (3-seed mean) #399

Open

saml212 mentioned this pull request Mar 24, 2026

Non-record: 11L XSA-all + Full GPTQ + Selective Pruning (val_bpb=1.1154, 3-seed) #609

Open

RichiiiTV pushed a commit to RichiiiTV/parameter-golf that referenced this pull request Mar 24, 2026

Port openai#549 root baseline and add soft-round QAT follow-up

c78afe1

Christopher-Lee-McClendon mentioned this pull request Mar 24, 2026

Non-record: 11L GEPA + 25k Steps + Pure Int6 + Legal TTT (val_bpb=1.0944) - unlimited compute category #644

Open

abaybektursun mentioned this pull request Mar 24, 2026

draft #650

Closed

ADIITJ mentioned this pull request Mar 27, 2026

Preliminary: 11L VRL + Full GPTQ + Parallel Muon + Legal TTT — val_bpb 1.1882 (ADIITJ) #960

Open

callithyia mentioned this pull request Mar 27, 2026

Record: 0.0881 BPB — 11L Int5 GPTQ + Order-12 N-gram + Phrase Cache + 65K Chunks #961

Open

4 tasks

sunnypatneedi mentioned this pull request Mar 27, 2026

Record: 11-gram Eval Cache + Hedge Mixer (val_bpb: 0.8609) #963

Closed

3 tasks

vivekvar-dl mentioned this pull request Mar 27, 2026

Record: Doc-Isolated TTT + Eval Optimizations #964

Open

3 tasks

michaelwinczuk mentioned this pull request Mar 27, 2026

LeakyReLU(0.75)² + Legal TTT + Parallel Muon — 1.1185 BPB (3-seed mean) #977

Open

ibarrajo mentioned this pull request Mar 28, 2026

Record: 33.6M Int5 GPTQ + Score-First TTT (val_bpb=1.1145, 3-seed) #991

Closed

5 tasks

aerosta mentioned this pull request Mar 28, 2026

Record: 11L XSA + Mixed INT6 + Adaptive N-gram Cache (2->7 backoff) - val_bpb=0.9631, 3-seed #993

Closed

OnlyJundong mentioned this pull request Mar 28, 2026

[Non-Record] Extended Compute Scaling Analysis: 1.0853 BPB at 50K steps (11.5 hours) on 4×A100MIG #1005

Open

NewyorkDev mentioned this pull request Mar 28, 2026

1.1085 BPB: JEPA + AdamW TTT + Full GPTQ + FA3 + LZMA #1006

Open

haimianbaobao007 mentioned this pull request Mar 28, 2026

N-gram logit boost + HedgeMixer + score-first TTT #1014

Open

ADIITJ mentioned this pull request Mar 28, 2026

11L VRL + Parallel Muon + Legal TTT v2 (val_bpb=1.1269, non-record) #1016

Open

3 tasks

abaybektursun mentioned this pull request Mar 28, 2026

Record: AR Self-Gen GPTQ + XSA-all + BigramHash 3072×112 — val_bpb 1.11473 (3-seed mean) #1019

Open

wfproc mentioned this pull request Mar 28, 2026

[Non-Record] QAT Dead-Code Analysis + 7 Novel Technique Sweep (1xH100) #1032

Open

Naazimsnh02 mentioned this pull request Mar 28, 2026

Record: 0.4311 BPB - Complementary Training + Backoff N-gram Mixer + TTT #1033

Open

7 tasks

TimPietruskyRunPod mentioned this pull request Mar 28, 2026

Record: Muon TTT + Entropy-Adaptive Epochs — val_bpb 1.1179 (3-seed mean) #1037

Closed

5 tasks

okezue mentioned this pull request Mar 28, 2026

PP12: Bayesian posterior packets + selective gating (1.1261 BPB) #1043

Open

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Record: LeakyReLU² + Legal Score-First TTT + Parallel Muon — val_bpb 1.1194 (3-seed mean)#549

Record: LeakyReLU² + Legal Score-First TTT + Parallel Muon — val_bpb 1.1194 (3-seed mean)#549
valerio-oai merged 3 commits intoopenai:mainfrom
abaybektursun:submission/leaky-relu-legal-ttt-1.1183

abaybektursun commented Mar 23, 2026 •

edited

Loading

Uh oh!

valerio-oai commented Mar 24, 2026 •

edited

Loading

Uh oh!

abaybektursun commented Mar 24, 2026

Uh oh!

abaybektursun commented Mar 24, 2026

Uh oh!

valerio-oai commented Mar 24, 2026

Uh oh!

mrdavtan commented Mar 29, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

abaybektursun commented Mar 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Record: LeakyReLU² + Legal TTT + Parallel Muon — val_bpb 1.1194

3-Seed Results (8×H100 80GB SXM, PyTorch 2.9.1+cu128)

Key Innovation: LeakyReLU(0.5)²

Legal TTT (Score-First, PR #461 Framework)

Training Architecture

Credits

Uh oh!

valerio-oai commented Mar 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

abaybektursun commented Mar 24, 2026

Uh oh!

abaybektursun commented Mar 24, 2026

Uh oh!

valerio-oai commented Mar 24, 2026

Uh oh!

mrdavtan commented Mar 29, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

abaybektursun commented Mar 23, 2026 •

edited

Loading

valerio-oai commented Mar 24, 2026 •

edited

Loading