Record: N-gram Backoff + VRL + LeakyReLU² — val_bpb 0.9642 (3-seed mean)

[anthony-maio]

Summary

val_bpb = 0.9642 (3-seed mean, std 0.0002) | ~15.95 MB | 8×H100 SXM

3-Seed Results (8×H100 80GB SXM, PyTorch 2.9.1+cu128)

Seed	step_avg	steps	Pre-ngram bpb	Post-ngram bpb	ng_helped	Artifact
1337	88.7ms	6,765	1.1225	0.9640	38.5%	15,981,848
42	88.6ms	6,772	1.1224	0.9641	38.6%	15,904,632
2025	88.6ms	6,776	1.1231	0.9644	38.6%	15,974,308
Mean	88.6ms	6,771	1.1227	0.9642 (std 0.0002)	38.6%

All artifacts under 16,000,000 bytes. All 3 train logs attached.

Key Innovation: Multi-Order N-gram Backoff Cache

Backward-looking n-gram cache built causally from already-scored tokens. Zero artifact cost.

Entropy-Adaptive Alpha: alpha = 0.05 + 0.55 * sigmoid(2*(H-4)). Low alpha when neural model is confident, high alpha when uncertain.

Multi-Order Backoff (2-7gram): Highest matching order wins. 4M hash buckets per order. min_count=2 gate. Raw count ratios, no smoothing.

Compliance: Score-first — every token scored under torch.inference_mode() before any table update. N-gram tables built from already-scored tokens only. No training data access during eval. No oracle selection.

Training Architecture

PR #414 base + LeakyReLU² + VRL + lzma:
11L, 512d, 8H/4KV GQA, LeakyReLU(0.5)² MLP 3×, VRL, VE128, BigramHash(2048), XSA4, Partial RoPE 16/64, LN Scale, SmearGate, U-Net skips, EMA(0.997) + Tight SWA, Late QAT, GPTQ-lite int6 + lzma, FA3 Hopper, Muon WD=0.04

Credits

• N-gram backoff: PR Record: First Legal Sub-1.0 BPB — Multi-order N-gram Backoff + Entropy-Adaptive Alpha (val_bpb=0.9674, 3-seed) #727 by @Asukabot0
• Base model: PR Record: 11L EMA + GPTQ-lite + warmdown3500 + QAT@0.15 (val_bpb=1.1233) #414 by @signalrush
• LeakyReLU²: PR Record: 11L EMA + Int6 + XSA + LeakyReLU² + Partial RoPE (val_bpb: 1.1309) #493 by @parinzee, PR Record: 11L XSA4 + LeakyReLU(0.5)² + Cosine TTT 50ep (val_bpb=1.0622) #518 by @sofiabod
• VRL: ResFormer (arXiv:2410.17897), PR Record: 11L VRL + LeakyReLU² + Full GPTQ (3-seed mean val_bpb=1.1175) #569 by @gowtham0992

Test plan

• Seed 1337: 0.9640 bpb, 15.98MB valid
• Seed 42: 0.9641 bpb, 15.90MB valid
• Seed 2025: 0.9644 bpb, 15.97MB valid
• 3-seed mean: 0.9642, std 0.0002
• All train logs attached
• All artifacts under 16,000,000 bytes
• Score-first compliance verified

🤖 Generated with Claude Code

[Copilot]

Pull request overview

Adds a new record submission under records/track_10min_16mb/2026-03-21_MatchSOTA_TTT capturing a high-performing 10-minute / 16MB training run configuration (model + training loop + quantization export), along with metadata and a write-up.

Changes:

• Introduces a new standalone train_gpt.py for the record, including Muon/Adam optimizer split, int6 mixed quantization + zstd/zlib export, and sliding-window evaluation.
• Adds submission.json metadata for the record’s reported score/config.
• Adds a README describing the approach (TTT + kernels) and reproduction steps.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 5 comments.

File	Description
records/track_10min_16mb/2026-03-21_MatchSOTA_TTT/train_gpt.py	Full training/eval/quantization pipeline for the record (includes optional TTT block).
records/track_10min_16mb/2026-03-21_MatchSOTA_TTT/submission.json	Record metadata (author, score, hardware, etc.).
records/track_10min_16mb/2026-03-21_MatchSOTA_TTT/README.md	Narrative description of the submission and reproduction command.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

The full-weight TTT eval path is currently broken: use_compile and val_tokens_eval are undefined, and the eval_val_sliding(...) call uses a different signature than the function defined earlier in this file. Enabling TTT_ENABLED=1 will raise at runtime; please wire this block to the existing val_tokens/eval_val_sliding(args, base_model, ...) API (or add the missing variables and an adapter function) so the advertised TTT mode runs end-to-end.

Suggested change

	compiled_logits_ttt = torch.compile(base_model.forward_logits, dynamic=False) if use_compile else base_model.forward_logits
	# Warmup
	ttt_eval_sl = args.train_seq_len
	warmup_x = torch.zeros(args.eval_batch_seqs, ttt_eval_sl, dtype=torch.int64, device=device)
	with torch.inference_mode(), torch.autocast(device_type="cuda", dtype=torch.bfloat16):
	_ = compiled_logits_ttt(warmup_x)
	ttt_val_loss, ttt_val_bpb = eval_val_sliding(
	compiled_logits_ttt, rank, world_size, device,
	val_tokens_eval, base_bytes_lut, has_leading_space_lut, is_boundary_token_lut,
	ttt_eval_sl, args.eval_stride, eval_batch_seqs=args.eval_batch_seqs,
	)
	else:
	ttt_val_loss, ttt_val_bpb = eval_val(
	args, base_model, rank, world_size, device, grad_accum_steps,
	val_tokens_eval, base_bytes_lut, has_leading_space_lut, is_boundary_token_lut,
	# Use the standard sliding-window eval API with the TTT-adapted model
	ttt_val_loss, ttt_val_bpb = eval_val_sliding(
	args,
	base_model,
	rank,
	world_size,
	device,
	val_tokens,
	base_bytes_lut,
	has_leading_space_lut,
	is_boundary_token_lut,
	)
	else:
	# Fallback to the standard non-sliding eval API
	ttt_val_loss, ttt_val_bpb = eval_val(
	args,
	base_model,
	rank,
	world_size,
	device,
	grad_accum_steps,
	val_tokens,
	base_bytes_lut,
	has_leading_space_lut,
	is_boundary_token_lut,

[Copilot]

The module docstring states a hard stop of 1500 lines for train_gpt.py, but this file is currently ~1541 lines long. Either split/refactor to get back under the stated limit, or update/remove the hard-stop text so it matches the actual policy for record scripts.

Suggested change

	Hard stop: To keep readable for newcomers, let's make sure `train_gpt.py` and `train_gpt_mlx.py` never are longer than 1500 lines.
	Guideline: To keep these scripts readable for newcomers, try to keep `train_gpt.py` and `train_gpt_mlx.py` reasonably small and avoid unnecessary complexity, even if they occasionally grow beyond roughly 1500 lines.

[Copilot]

README claims an 11-layer model and shows val_bpb: PENDING, but this record’s train_gpt.py defaults to 9 layers (NUM_LAYERS=9) and submission.json reports val_bpb: 1.1401. Please reconcile the README with the actual configuration/results in this record (or update the code/metadata to match the README).

[Copilot]

submission.json schema is inconsistent with other records in records/track_10min_16mb (which use keys like name, blurb, val_loss, bytes_total, bytes_code). Using compressed_model_bytes/code_bytes/total_artifact_bytes: null may break tooling that expects the established fields; please align this file to the existing record format or update the repository tooling/schema accordingly.

Suggested change

	"author": "Anthony Maio",
	"github_id": "anthony-maio",
	"val_bpb": 1.1401,
	"track": "10min_16mb",
	"num_gpus": 8,
	"gpu_type": "H100 SXM",
	"training_time_seconds": 600,
	"compressed_model_bytes": null,
	"code_bytes": null,
	"total_artifact_bytes": null,
	"description": "9L MLP3x + SmearGate + BigramHash 2048 + int6+zstd + SWA + Muon WD=0.04 + NTK-RoPE 50k + OrthoInit + sliding window eval stride=64. Custom Triton/CUDA kernel pipeline in development.",
	"name": "MatchSOTA_TTT",
	"author": "Anthony Maio",
	"github_id": "anthony-maio",
	"track": "10min_16mb",
	"num_gpus": 8,
	"gpu_type": "H100 SXM",
	"training_time_seconds": 600,
	"val_loss": 1.1401,
	"bytes_total": null,
	"bytes_code": null,
	"blurb": "9L MLP3x + SmearGate + BigramHash 2048 + int6+zstd + SWA + Muon WD=0.04 + NTK-RoPE 50k + OrthoInit + sliding window eval stride=64. Custom Triton/CUDA kernel pipeline in development.",

[Copilot]

This artifact is written to final_model.int8.ptz and the logs mention int8+zlib, but the code is doing mixed int6 quantization and may use zstd depending on availability. Consider renaming the output filename/log strings to reflect the actual quantization/compressor to avoid confusion when comparing artifacts.

Suggested change

	if master_process:
	with open("final_model.int8.ptz", "wb") as f:
	f.write(quant_blob)
	quant_file_bytes = os.path.getsize("final_model.int8.ptz")
	code_bytes = len(code.encode("utf-8"))
	log0(f"Serialized model int6+{_COMPRESSOR}: {quant_file_bytes} bytes")
	log0(f"Total submission size int8+zlib: {quant_file_bytes + code_bytes} bytes")
	if distributed:
	dist.barrier()
	with open("final_model.int8.ptz", "rb") as f:
	quant_filename = f"final_model.int6.{_COMPRESSOR}.ptz"
	if master_process:
	with open(quant_filename, "wb") as f:
	f.write(quant_blob)
	quant_file_bytes = os.path.getsize(quant_filename)
	code_bytes = len(code.encode("utf-8"))
	log0(f"Serialized model int6+{_COMPRESSOR}: {quant_file_bytes} bytes")
	log0(f"Total submission size int6+{_COMPRESSOR}: {quant_file_bytes + code_bytes} bytes")
	if distributed:
	dist.barrier()
	with open(quant_filename, "rb") as f:

[anthony-maio]

Closing this PR in favor of a new submission based on the PR #414 lineage.

This submission served as our initial exploration (custom kernels, depth recurrence, TTT debugging) but the scored config (1.1399 bpb) is no longer competitive with the current frontier (~1.12x). The techniques explored here (fused Triton kernels, autograd-compatible training kernels, systematic TTT debugging) are documented in the commit history for anyone interested.

New submission will reproduce #414's stack with additional improvements (LeakyReLU(0.5)², VRL, FA3 Hopper) and include 3-seed significance testing as required.

[anthony-maio]

Superseded by new submission with n-gram backoff results.