Record: int5 GPTQ + Soft-Round QAT (3-seed mean 1.1162)

[EthanYangTW]

Summary

• 3-seed mean val_bpb = 1.1162 (std 0.0006)
• int5 GPTQ quantization ([-15,15], 31 levels) with Hessian-aware error compensation enables 33.6M params in 16MB
• Soft-Round QAT: differentiable tanh-based rounding (alpha 1→16) replacing STE for better training quality at zero cost
• Legal score-first AdamW TTT with cosine LR decay across chunks

Results

Seed	TTT BPB	Artifact
1337	1.1155	15,822,078 bytes
42	1.1163	15,415,405 bytes
7	1.1167	15,368,627 bytes
Mean	1.1162
Std	0.0006

Key Innovations

1. int5 quantization — 31 unique values stored as int8, ~0.46 bytes/param after zstd (vs int6's ~0.58). Enables 33.6M param model under 16MB.
2. GPTQ error compensation — Hessian-aware column reordering + Cholesky error redistribution. 256-sample calibration.
3. Soft-Round QAT — s_α(y) = floor(y) + 0.5 * tanh(α·r) / tanh(α/2) + 0.5 with alpha annealing 1→16 during QAT steps.
4. 33.6M param model — MHA 8/8, BigramHash 8192, MLP 3.5x (1792), XSA all 11 layers.
5. Early QAT 0.5 + EMA 0.997 + FA3 Hopper

Architecture

• 11L, model_dim=512, 8H/8KV (MHA), MLP 3.5x relu²
• XSA on all 11 layers, Partial RoPE 16/64, LN Scale
• SmearGate + OrthoInit, BigramHash 8192, Shared VE128
• Tight SWA (every 50), Muon lr=0.025, WD=0.04

TTT (Legal, Score-First)

• 131072-token chunks, score first then adapt
• AdamW (lr=0.0001, wd=0.0), 3 epochs/chunk, cosine LR
• Last 2 blocks + norms + lm_head unfrozen (~5.8M / 33.6M)
• Every token scored BEFORE any gradient update using it

Test plan

• 3-seed validation (seeds 1337, 42, 7)
• All artifacts under 16,000,000 bytes
• Training completes under 600s on 8xH100 SXM
• Evaluation completes under 600s
• Score-first TTT (legal per rulings val-only 10min record (val_bpb:1.1111) #44, Add TTT (Test-Time Training) submission: 1.1767 BPB #152, SOTA Attempt: Paid prefix (val_bpb=1.0238) #168)

[Copilot]

Pull request overview

Adds a new 10-minute / 16MB record submission folder capturing a 33.6M-parameter model that uses int5 GPTQ quantization, Soft-Round QAT, and “legal score-first” test-time training (TTT), along with the logs and metadata to reproduce the reported 3-seed mean val_bpb.

Changes:

• Adds a full training/eval script implementing int5 GPTQ, Soft-Round QAT, and score-first TTT.
• Adds per-seed training logs documenting results, artifact sizes, and run configuration.
• Adds submission metadata (submission.json) and a README describing the method and reproduction command.

Reviewed changes

Copilot reviewed 2 out of 6 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
records/track_10min_16mb/2026-03-24_Int5GPTQ_33M_LegalTTT/train_gpt.py	New end-to-end training + GPTQ quantization + TTT evaluation script for the record submission.
records/track_10min_16mb/2026-03-24_Int5GPTQ_33M_LegalTTT/train_seed1337.log	Seed 1337 run log supporting the reported metrics/size.
records/track_10min_16mb/2026-03-24_Int5GPTQ_33M_LegalTTT/train_seed42.log	Seed 42 run log supporting the reported metrics/size.
records/track_10min_16mb/2026-03-24_Int5GPTQ_33M_LegalTTT/train_seed7.log	Seed 7 run log supporting the reported metrics/size.
records/track_10min_16mb/2026-03-24_Int5GPTQ_33M_LegalTTT/submission.json	Submission metadata capturing aggregate results, per-seed values, and configuration summary.
records/track_10min_16mb/2026-03-24_Int5GPTQ_33M_LegalTTT/README.md	Method/architecture write-up and reproduction instructions for the submission.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

In QAT paths you clamp weights with torch.clamp(..., -(cr+1), cr), which makes the quantization range asymmetric and increases the number of representable integer levels (e.g., int5 becomes [-16, 15] = 32 values, int6 becomes [-32, 31] = 64 values). This does not match the stated int5/int6 ranges used elsewhere in this script (e.g., GPTQ quantization clamps to [-cr, cr]) and can cause a train/inference mismatch. Consider clamping to [-cr, cr] (or otherwise aligning QAT and post-training quantization ranges and updating the documentation accordingly).

[Copilot]

mixed_quantize_int6_gptq uses plain print(...) for quantization stats. In distributed runs this will print from every rank, which matches the duplicated lines seen in the added logs and makes logs noisy/harder to parse. Prefer routing these messages through log0 / gating them on rank == 0 (or returning the stats to the caller and logging once).

Suggested change

	print(f"gptq_quantize: {gptq_count} GPTQ layers, {naive_count} naive layers", flush=True)
	print(f"mixed_precision: {int5_params} int5 params, {int6_params} int6 params", flush=True)
	msg1 = f"gptq_quantize: {gptq_count} GPTQ layers, {naive_count} naive layers"
	msg2 = f"mixed_precision: {int5_params} int5 params, {int6_params} int6 params"
	if not dist.is_available() or not dist.is_initialized() or dist.get_rank() == 0:
	print(msg1, flush=True)
	print(msg2, flush=True)

[Copilot]

LogisticContextMixer vocab size is computed as val_tokens.to(torch.int32).max().item() + 1, which forces a full dtype conversion of the entire validation tensor (tens of millions of tokens) just to get the max. Since vocab size is already known (args.vocab_size, or sp.vocab_size()), prefer using that directly (or int(val_tokens.max()) + 1 without casting) to avoid unnecessary CPU work and memory pressure during eval.

Suggested change

	vocab_size=val_tokens.to(torch.int32).max().item() + 1,
	vocab_size=args.vocab_size,

[Copilot]

This record script writes/reads final_model.int6.ptz and logs it as int6+..., but the submission/README describe int5 GPTQ (and most scripts in the repo use the final_model.int8.ptz artifact name). If there are any shared tooling expectations around artifact filenames or log labels, this inconsistency can break automation and is confusing for reproduction. Consider aligning the artifact filename/log labels with the actual format (e.g., final_model.int5.ptz or the repo-standard final_model.int8.ptz naming convention).

Suggested change

	with open("final_model.int6.ptz", "wb") as f:
	f.write(quant_blob)
	quant_file_bytes = len(quant_blob)
	code_bytes = len(code.encode("utf-8"))
	log0(f"Serialized model int6+{_COMPRESSOR}: {quant_file_bytes} bytes")
	log0(f"Total submission size int6+{_COMPRESSOR}: {quant_file_bytes + code_bytes} bytes")
	if distributed:
	dist.barrier()
	with open("final_model.int6.ptz", "rb") as f:
	with open("final_model.int5.ptz", "wb") as f:
	f.write(quant_blob)
	quant_file_bytes = len(quant_blob)
	code_bytes = len(code.encode("utf-8"))
	log0(f"Serialized model int5+{_COMPRESSOR}: {quant_file_bytes} bytes")
	log0(f"Total submission size int5+{_COMPRESSOR}: {quant_file_bytes + code_bytes} bytes")
	if distributed:
	dist.barrier()
	with open("final_model.int5.ptz", "rb") as f:

[Copilot]

The README describes the MLP as "relu²", but the implementation in train_gpt.py uses leaky_relu(..., negative_slope=0.5).square(). Updating the README to match the actual activation used will make the architecture description accurate and easier to reproduce/compare against other runs.

Suggested change

	- 11 layers, model_dim=512, 8 heads / 8 KV heads (MHA), MLP 3.5x relu²
	- 11 layers, model_dim=512, 8 heads / 8 KV heads (MHA), MLP 3.5x leaky_relu² (negative_slope=0.5)

[valerio-oai]

As per previous PRs, it's disallowed to use training data in any way during evaluation, which your GPTQ calibration is currently doing. This would be legal (or at least more likely to be legal) if you ended training early and calibrated as part of the training time. However, per the logs provided, it looks like you're training for the full 10 minutes, then doing 3.6s of calibration and then the rest of the eval, meaning this is included in the eval time and therefore currently illegal.