Skip to content

Record: First Legal Sub-1.0 BPB — Multi-order N-gram Backoff + Entropy-Adaptive Alpha (val_bpb=0.9674, 3-seed)#727

Closed
Asukabot0 wants to merge 1 commit intoopenai:mainfrom
Asukabot0:submission/backoff-entropy-0.9674
Closed

Record: First Legal Sub-1.0 BPB — Multi-order N-gram Backoff + Entropy-Adaptive Alpha (val_bpb=0.9674, 3-seed)#727
Asukabot0 wants to merge 1 commit intoopenai:mainfrom
Asukabot0:submission/backoff-entropy-0.9674

Conversation

@Asukabot0
Copy link
Copy Markdown

Results (3-seed validation)

Seed val_bpb val_loss Size Quantization
1337 0.96679 1.63238 15,994,366 B int6+zstd-16
42 0.96703 1.63278 15,996,585 B int6+zstd-16
7 0.96825 1.63485 15,988,201 B int6+zstd-16
Mean 0.96736 1.63334
Std 0.00063

Technique

Architecture: 11L, 512d, GQA 8H/4KV, MLP 3x, LeakyReLU(0.5)², XSA-all(11), Value Residual, Gated Attention, SmearGate, BigramHash(4096), Partial RoPE(16/64), LN Scale, EMA(0.997). Tied embeddings. Muon optimizer.

N-gram eval cache — two key improvements over prior work:

  1. Multi-order backoff (orders 2–7): Instead of a single fixed order, we attempt the highest order first and cascade down on miss. This dramatically improves coverage vs a fixed 7-gram.

  2. Entropy-adaptive alpha: alpha = 0.05 + 0.55 * sigmoid(2 * (H - 4.0)). When the model is uncertain (high entropy), we trust n-gram statistics more; when confident (low entropy), we trust the LM. This replaces the fixed alpha=0.40 used in prior approaches.

Compliance

  • Score-first, backward-looking: n-gram counts are built from previously scored tokens only
  • No oracle selection: alpha depends solely on the model's own output distribution (entropy), never on ground-truth labels
  • No cross-GPU sync: each GPU maintains its own independent cache

Ablation

Configuration val_bpb Delta
No n-gram (neural only) 1.1271 baseline
Fixed alpha=0.40, order=7, no backoff 1.0336 −0.0935
Multi-order backoff (2-7) + fixed alpha=0.40 0.9825 −0.1446
Multi-order backoff (2-7) + entropy-adaptive 0.9674 −0.1597

Comparison with prior submissions

Submission val_bpb Delta vs this
PR #549 SOTA 1.1194 −0.152
PR #702 (n-gram backoff) 1.0240 −0.057
This PR 0.9674

Training

  • 8× H100 SXM (RunPod), 600s wallclock, ~5580 steps per seed
  • No TTT, no SWA, no canonical attention
  • int6 per-row + zstd-16 quantization (no int5 fallback needed)

Built on modded-nanogpt. Credits: PR #315, #609, #493, #518, #413, #674, #702.

… 3-seed)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@Asukabot0 Asukabot0 changed the title Record: Multi-order N-gram Backoff + Entropy-Adaptive Alpha (val_bpb=0.9674) Record: First Legal Sub-1.0 BPB — Multi-order N-gram Backoff + Entropy-Adaptive Alpha (val_bpb=0.9674, 3-seed) Mar 25, 2026
@deanbrr
Copy link
Copy Markdown

deanbrr commented Mar 25, 2026

Congrats Asukbot0, nice work.

I believe I was the first to contribute the N-gram eval cache technique PR #659 to the contest

newjordan pushed a commit to newjordan/parameter-golf-1 that referenced this pull request Mar 25, 2026
Multi-order backoff (2-7) + entropy-adaptive alpha on 11L/512d U-Net.
All 3 seeds sub-1.0. GPTQ calibration inside training phase.

Seeds: 42=0.9631, 2045=0.9620, 7=0.9624, mean=0.9625

Credits: @deanbrr openai#659, @Asukabot0 openai#727, @signalrush openai#414

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Asukabot0 added a commit to Asukabot0/parameter-golf that referenced this pull request Mar 25, 2026
10 defaults were wrong (inherited from old PR#398 base):
- MATRIX_LR: 0.04 -> 0.025
- SCALAR_LR: 0.04 -> 0.025
- TIED_EMBED_LR: 0.05 -> 0.035
- SWA_ENABLED: 1 -> 0
- XSA_LAST_N: 0 -> 11
- LEAKY_RELU: 0 -> 1
- MUON_MOMENTUM: 0.95 -> 0.99
- MUON_MOMENTUM_WARMUP_START: 0.85 -> 0.92
- MUON_MOMENTUM_WARMUP_STEPS: 500 -> 1500

Previous PR openai#727 runs worked because env vars were passed manually.
After cloud restart, defaults kicked in producing wrong model.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
anthony-maio added a commit to anthony-maio/parameter-golf that referenced this pull request Mar 25, 2026
Backward-looking n-gram cache built from already-scored tokens.
Multi-order backoff (2-7gram) with entropy-adaptive alpha mixing.
Zero artifact cost. Expected -0.07 to -0.16 bpb gain.

NGRAM_ENABLED=1 to activate. NGRAM_ORDER=7 (default).
Disabled by default — legality still being debated (issue openai#677).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
newjordan pushed a commit to newjordan/parameter-golf-1 that referenced this pull request Mar 26, 2026
Multi-order backoff (2-7) + entropy-adaptive alpha on 11L/512d U-Net.
Two seeds sub-1.0. GPTQ calibration inside training phase.

3-seed: 1337=1.0217, 42=0.9631, 2045=0.9620, mean=0.9823

Credits: @deanbrr openai#659, @Asukabot0 openai#727, @signalrush openai#414

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
anthony-maio added a commit to anthony-maio/parameter-golf that referenced this pull request Mar 26, 2026
Key changes from failed Hedge implementation:
- Entropy-adaptive alpha: 0.05 + 0.55 * sigmoid(2*(H-4))
  (low when neural confident, high when uncertain)
- Backoff: highest matching n-gram order wins (not mixture)
- 4M hash buckets per order (was 65K) — matches PR openai#727
- Orders 2-7 (was unigram/bigram/trigram only)
- Score-first: update cache AFTER scoring each segment
- min_count=2 gate: n-gram only active with sufficient data
- No smoothing: raw count ratios (not add-k)
- Per-GPU contiguous windows with persistent cache

Hedge failed because it learned a GLOBAL weight — neural always
wins on average. PR openai#727's per-token entropy-adaptive alpha works
because it only trusts n-grams when the neural model is uncertain.

Progress logs show ng_helped% — fraction of tokens where n-gram
mixing reduced NLL vs neural-only.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Fraser-Greenlee added a commit to Fraser-Greenlee/parameter-golf that referenced this pull request Mar 26, 2026
Key insight: n-gram cache is worth 50-80x more than any architecture
change (PR openai#727: 1.12 → 0.97 BPB). Our E16/E23 cache failures had
specific fixable bugs (add-k smoothing, λ too high, per-token Python
loops).

New experiments:
- E35: N-gram cache rewrite with Stupid Backoff (HIGHEST PRIORITY)
- E36: XSA on all 11 layers (flag flip, adopted by frontier submissions)
- E37: Full GPTQ + selective pruning

Also documents which experiments are conclusively dead at this scale.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
anthony-maio added a commit to anthony-maio/parameter-golf that referenced this pull request Mar 26, 2026
Key changes from failed Hedge implementation:
- Entropy-adaptive alpha: 0.05 + 0.55 * sigmoid(2*(H-4))
  (low when neural confident, high when uncertain)
- Backoff: highest matching n-gram order wins (not mixture)
- 4M hash buckets per order (was 65K) — matches PR openai#727
- Orders 2-7 (was unigram/bigram/trigram only)
- Score-first: update cache AFTER scoring each segment
- min_count=2 gate: n-gram only active with sufficient data
- No smoothing: raw count ratios (not add-k)
- Per-GPU contiguous windows with persistent cache

Hedge failed because it learned a GLOBAL weight — neural always
wins on average. PR openai#727's per-token entropy-adaptive alpha works
because it only trusts n-grams when the neural model is uncertain.

Progress logs show ng_helped% — fraction of tokens where n-gram
mixing reduced NLL vs neural-only.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
abaybektursun added a commit to abaybektursun/parameter-golf that referenced this pull request Mar 26, 2026
- Base model is ValCalib GPTQ (1.1142 BPB), not PR openai#549 (1.1194)
- Remove stale "not yet deployed" / "we estimate" for EXP-11
- Note α=0.80 (939s) exceeds 600s budget
- Fix PR openai#727 score to 0.9674, PR openai#788 to 0.9059
- Fix PR openai#596 BPB to 0.6430
- "Approved" → "Technique deemed legal" for closed PRs
- Add bucket sweep and per-token overhead proposal
- Replace "neural" with "base LM" throughout

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
sunnypatneedi pushed a commit to sunnypatneedi/parameter-golf that referenced this pull request Mar 27, 2026
- Update merged SOTA to 1.1194 (abaybektursun, was 1.1228 signalrush)
- Add competition strategy pivot: n-gram eval cache now dominates (~0.02-0.97 bpb)
- Document PR openai#727 (0.9674), openai#741 (0.9850), openai#945 (0.0274), openai#961 (0.0881) findings
- Add Lessons Learned entries 17-20 on n-gram dominance + memorization risk
- Update Technique Reference table with n-gram entries

https://claude.ai/code/session_01Bpr2fKEnkNQmNKno8EnxWF
@valerio-oai
Copy link
Copy Markdown
Contributor

Thanks for your submission! Unfortunately, it's disallowed due to the use of hashed n-gram caches, which do not renormalize correctly / correctly reweight the LM's token distribution, look ahead to the target token to mix probabilities and therefore leak eval tokens. Please refer to the long discussion about this under the issues tab for more details, and please submit more runs in the future!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants