GolfStudent v2 14L: d=352, Value Residuals, GPTQ-lite, Schedule-Free, Muon+EMA

[whitestone1121-web]

GolfStudent v2 — 16MB Hybrid LM

Architecture: d=352, L=14 (10x GatedMLP + 4x Attention every 3rd layer), vocab=1024, weight-tied embedding/lm_head, SwiGLU FFN (3x expansion), RoPE on attention layers, orthogonal weight init

v2 improvements over v1:

• d=352 (from 288) — +65% model capacity, ~15MB INT8+zlib (94% of 16MB budget)
• Value Residuals — learned scalar skip gates (init=0, tanh-gated) every 3 blocks
• GPTQ-lite — 5 clip percentile candidates per row, min reconstruction MSE INT8
• Schedule-Free final 120s — constant LR floor (10%) + faster EMA (decay=0.97) instead of LR→0 warmdown

Training:

• Muon optimizer (momentum 0.85→0.99 warmup over 1500 steps, WD=0.04) for matrix params
• Adam (fused) for embeddings/scalars
• EMA decay=0.997, updated every step
• Wallclock-aware schedule: cosine + schedule-free final 120s
• Grad clip=0.3

Quantization: Per-row INT8 GPTQ-lite (5 clip percentiles, min MSE) + zlib level=9
Size: ~15.06MB / 16MB (94.1% budget)

[whitestone1121-web]

Updated for v2: Architecture is now d=352 (from 288), adding Value Residuals (learned tanh-gated skip connections every 3 blocks, init=0) and GPTQ-lite INT8 (5 clip percentile candidates per row, min reconstruction MSE). No distillation - pure CE on FineWeb binary shards. Quantization + zlib happens after the wallclock timer exits, matching standard contest format. Dry-run confirms ~15.06MB (94.1% of 16MB budget).