Skip to content

Record: Score-First TTT + N-gram Backoff (3-seed mean val_bpb=0.9581)#761

Open
Asukabot0 wants to merge 14 commits intoopenai:mainfrom
Asukabot0:submission/score-first-ttt-ngram-0.9581
Open

Record: Score-First TTT + N-gram Backoff (3-seed mean val_bpb=0.9581)#761
Asukabot0 wants to merge 14 commits intoopenai:mainfrom
Asukabot0:submission/score-first-ttt-ngram-0.9581

Conversation

@Asukabot0
Copy link
Copy Markdown

@Asukabot0 Asukabot0 commented Mar 25, 2026

Record: Score-First TTT + Multi-Order N-gram Backoff (val_bpb=0.9581)

3-seed mean val_bpb: 0.9581 (std=0.0005) | ~15.7 MB artifact | 8xH100 SXM

Results

Seed Sliding BPB (s64) Artifact Steps ms/step TTT time Total eval
1337 0.9576 15,721,728 6409 93.63 107.0s ~303s
42 0.9581 15,702,393 6403 93.73 107.9s ~255s
7 0.9585 15,768,158 6407 93.65 105.2s ~251s
Mean 0.9581 ~6406 ~93.67 ~106.7s ~270s

Architecture

  • 11L, 512d, GQA (8H/4KV), MLP 3x, U-Net skip connections
  • LeakyReLU(0.5)^2: preserves negative gradient flow
  • XSA on all 11 layers: removes self-position bias
  • Value Residual (VR): layer 0 V output mixed via sigmoid gates
  • Gated Attention (GA): per-head sigmoid gates
  • SmearGate + OrthoInit, BigramHash(4096), Partial RoPE (16/64), LN Scale
  • EMA(0.997), warmdown=3000, int6 per-row + zstd-16

Eval-Time Techniques

Score-First TTT (compliant with Issue #677)

  • Process val data in sequential 131K-token chunks
  • Phase 1: Score chunk under inference_mode (forward only)
  • Phase 2: Train on scored tokens with AdamW (lr=0.0001, 4 epochs)
  • Freeze first 2 blocks, grad clip 1.0
  • Each token scored BEFORE model trains on it

Multi-Order N-gram Backoff + Entropy-Adaptive Alpha

  • Orders 2-7: highest order first, cascade on miss
  • Entropy-adaptive: alpha = 0.05 + 0.55 * sigmoid(2 * (H - 4.0))
  • Fixed formula, no oracle selection, no target-aware gating
  • Backward-looking: cache built from already-scored tokens only

Compliance

  • Score-first TTT: tokens scored under inference_mode before training
  • N-gram cache: backward-looking, entropy-based mixing (not target-aware)
  • GPTQ: not used (naive int6 per-row quantization)
  • All training within 600s, all eval within 600s
  • No training data accessed at eval time

Reproduction

python3 data/cached_challenge_fineweb.py --variant sp1024
SEED=1337 TTT_ENABLED=1 NGRAM_CACHE=1 \
  torchrun --standalone --nproc_per_node=8 train_gpt.py

Credits

Asukabot0 and others added 14 commits March 25, 2026 03:35
@Asukabot0 @claude
Record: 11L XSA-all + LeakyReLU(0.5)² + VR + GA (val_bpb=1.1164)
af58387 
Non-TTT submission: XSA on all 11 layers, LeakyReLU(0.5)², Value Residual,
Gated Attention. Single-GPU 7500-step result, pending 8xH100 3-seed validation.
Artifact 15.94MB (zstd-21). Requesting compute grant.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@Asukabot0 @claude
Fix all default values to match p17 config
320bdb3 
12 defaults were inherited from old PR#398 base and didn't match
the actual p17 experiment config:
- WARMDOWN_ITERS: 1200 -> 3500
- MATRIX_LR: 0.04 -> 0.025
- SCALAR_LR: 0.04 -> 0.025
- TIED_EMBED_LR: 0.05 -> 0.035
- SWA_ENABLED: 1 -> 0
- XSA_LAST_N: 0 -> 11
- LEAKY_RELU: 0 -> 1
- MUON_MOMENTUM: 0.95 -> 0.99
- MUON_MOMENTUM_WARMUP_START: 0.85 -> 0.92
- MUON_MOMENTUM_WARMUP_STEPS: 500 -> 1500
- TTT_ENABLED: 1 -> 0
- ZSTD_LEVEL: 22 -> 21 (configurable via env var)

Now the code runs p17 config with zero env vars needed.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@Asukabot0 @claude
Fix DDP 3x slowdown: replace find_unused_parameters with static_graph
b64318c 
find_unused_parameters=True was enabled for VR+GA (layer 0's vr_lambda
is unused when v0=None). This forces DDP to scan the entire autograd
graph every backward pass, causing ~3x slowdown on 8xH100 (288ms vs
expected ~87ms/step).

static_graph=True only checks once on first iteration then caches,
which is much more efficient with torch.compile.

This only affects multi-GPU runs (single GPU doesn't use DDP).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@Asukabot0 @claude
p21: auto-downgrade quant + warmdown=3000 + n-gram auto-enable
4755684 
Three changes for 8xH100 3-seed submission:
- Artifact auto-downgrade: try int6+zstd [16,1,17,2], fall back to
  int5 middle layers (L2-8) if still over 16MB
- Warmdown default 3000 (was 1200): 46.5% ratio on 8xH100 matches
  single-GPU 47%, fixes v9's 54% over-warmdown
- 5-gram eval cache auto-enabled on multi-GPU (world_size>1),
  alpha=0.20, order=5

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@Asukabot0 @claude
Fix: progressive int5 fallback — one layer at a time from center
73b5e68 
Instead of downgrading all middle layers (L2-8) to int5 at once
(wasting 2.1MB and +0.014 BPB), now downgrades one layer at a time
expanding outward from center (L5→L6→L4→L7→...).

Tested: single layer (L5) saves ~290KB, enough to fit most seeds.
BPB penalty reduced from ~0.014 to ~0.002.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@Asukabot0 @claude
Add n-gram parameter sweep script for 8xH100
b72167f 
Train 1 seed, then sweep alpha=[0.10-0.30] and order=[3-7]
using EVAL_ONLY mode. Each eval ~3min on 8xH100.
Total sweep time: ~10min train + 9×3min eval = ~37min.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@Asukabot0 @claude
Optimal n-gram params: alpha=0.40 order=7 (8xH100 sweep)
c46ecee 
Best from 20-point grid search on 8xH100:
  alpha=0.40 order=7 → 1.0336 BPB (vs 1.0517 at alpha=0.20 order=5)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@Asukabot0 @claude
Multi-order n-gram backoff + entropy-adaptive alpha
7d65721 
Two eval-time improvements (no retraining needed):

1. Multi-order backoff (orders 2-7): When 7-gram has no cache hit,
   falls back to 6/5/4/3/2-gram. Dramatically increases cache hit rate
   on 8xH100 where per-GPU cache is sparse. PR openai#702 reports -0.018 BPB.

2. Entropy-adaptive alpha: alpha = 0.05 + 0.55 * sigmoid(2*(H-4.0))
   Model uncertain → trust n-gram more. Model confident → keep LM.
   Compliant: alpha depends only on model's own distribution.

Both configurable via env vars (NGRAM_ENTROPY=0 to disable adaptive).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@Asukabot0 @claude
Score-first TTT + DDP static_graph fix for our_experiment
a96341a 
1. Rewrite ttt_adapt() to score-first pattern (Issue openai#677 compliant):
   - Process val data in sequential chunks (TTT_CHUNK_TOKENS=131072)
   - Phase 1: score chunk under inference_mode (forward only)
   - Phase 2: train on scored tokens with AdamW (K epochs)
   - Each token scored BEFORE model trains on it

2. Switch TTT optimizer from SGD to AdamW (lr=0.0001, wd=0.0)
   - PR openai#700 showed AdamW >> SGD for TTT
   - Default 4 epochs, freeze first 2 blocks

3. Fix DDP find_unused_parameters → static_graph=True
   - Same 3x slowdown fix as submission directory

4. TTT defaults: disabled by default (TTT_ENABLED=0)
   - Enable with TTT_ENABLED=1 for TTT+n-gram combined eval

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@Asukabot0 @claude
Fix all training defaults in our_experiment to match p17 config
b92b5bf 
10 defaults were wrong (inherited from old PR#398 base):
- MATRIX_LR: 0.04 -> 0.025
- SCALAR_LR: 0.04 -> 0.025
- TIED_EMBED_LR: 0.05 -> 0.035
- SWA_ENABLED: 1 -> 0
- XSA_LAST_N: 0 -> 11
- LEAKY_RELU: 0 -> 1
- MUON_MOMENTUM: 0.95 -> 0.99
- MUON_MOMENTUM_WARMUP_START: 0.85 -> 0.92
- MUON_MOMENTUM_WARMUP_STEPS: 500 -> 1500

Previous PR openai#727 runs worked because env vars were passed manually.
After cloud restart, defaults kicked in producing wrong model.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@Asukabot0 @claude
Aggressive SGD TTT: LR=1.0, 20 epochs, all blocks unfrozen
435e798 
Inspired by PR openai#757 which found SGD LR=1.0 gives 16x better TTT gain
than conventional LR=0.002. Key changes:

- TTT_OPTIMIZER env var: "sgd" (default) or "adamw"
- Default LR: 0.0001 -> 1.0 (SGD)
- Default epochs: 4 -> 20
- Default freeze_blocks: 2 -> 0 (all unfrozen)

PR openai#757 showed: freeze=0 + high LR converges fine, extra capacity
absorbs aggressive learning rate. 20ep × ~16s = ~320s on 8xH100.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@Asukabot0 @claude
LeakyReLU slope 0.5 -> 0.9 (Issue openai#140 sweep: -0.013 BPP)
443f565 
@MatoTeziTanka's 7-point sweep showed monotonic improvement with
higher slopes. 0.9 beats 0.5 by 0.013 BPP + 200 more steps (less
dead activation = faster per step).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@Asukabot0 @claude
Revert to p23 config for submission (3-seed mean=0.9581)
fe4d387 
Defaults now match the exact config that produced the verified results:
- TTT: AdamW lr=0.0001, 4 epochs, freeze_blocks=2
- LeakyReLU slope: 0.5
- Score-first TTT (Issue openai#677 compliant)

3-seed results: 0.9576/0.9581/0.9585 (mean=0.9581, std=0.0005)
All artifacts <16MB, all eval <600s.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@Asukabot0 @claude
Record: Score-First TTT + Multi-Order N-gram Backoff (3-seed mean val…
6827973 
…_bpb=0.9581)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@Asukabot0 Asukabot0 mentioned this pull request Mar 25, 2026
Closed
pablinga19 added a commit to pablinga19/parameter-golf that referenced this pull request Mar 26, 2026
@pablinga19 @claude
fix hash alignment, add rANS codec, kill Hadamard
10503f4 
- hash order now matches PR openai#761 (primes[0] -> oldest token)
- rANS codec: perfect roundtrip, near-Shannon compression
- Hadamard tested and killed (hurts per-row quant)
- warmup bounds checked
- integration guide for train_gpt.py

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
pablinga19 added a commit to pablinga19/parameter-golf that referenced this pull request Mar 27, 2026
@pablinga19 @claude
add competitive entry based on PR openai#761
ecc7dcf 
proven 0.9581 BPB entry with full SOTA stack:
11L XSA-all, LeakyReLU(0.9)², VR, GA, EMA, score-first TTT,
multi-order n-gram backoff. ready to deploy on 8xH100.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
pablinga19 added a commit to pablinga19/parameter-golf that referenced this pull request Mar 27, 2026
@pablinga19 @claude
innovate: orders 2-12, warm cache, complementary training
e76449d 
three innovations on top of PR openai#761 base:
1. extend n-gram to orders 2-12 (was 2-7) with 14 primes
2. warm cache: load pre-computed tables from artifact at startup
3. complementary training: down-weight bigram-easy tokens so neural
   model focuses on what the cache can't predict

all controlled by env vars (NGRAM_ORDER, WARM_CACHE, COMP_WEIGHT).
set COMP_WEIGHT=0 to disable complementary training.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@Asukabot0