Columnar v5: grammar and derived columns, categorical bit packing, patched frames

[eliahilse]

One coordinated plan bump that implements the four remaining compression concepts, specified first and gated at every step.

What the wire gains (spec/plan-columnar-v5.md, profile v2)

• Grammar columns — a profile can declare the shape of machine-made ids (evt_{seq36}_{hex8}); matching values travel as one small integer per token lane, riding the existing delta/frame machinery. Four bits per hex digit instead of eight, and sequences for nearly nothing.
• Derived columns — a functional dependency (actorId → actorEmail) recorded once in the profile; conforming rows ship zero bytes.
• Categorical bit packing — enum indices at schema-derived width (six members = three bits), dictionary codes bit-packed at the width of the highest code used, escapes appended.
• Patched frame of reference (int mode 0x04) — outlier rows alone pay their high bits; durationMs-shaped columns stop paying outlier width on every row.

The old "declared count payable by remaining input" rule is incompatible with width-zero columns and is now scoped to row encoding; columnar counts are bounded by maxItems plus a new maxAmplification decode-policy limit, mirrored at encode so the encoder never emits what its decoder refuses. Row materialization is deferred until all columns decode.

Measured (500-message corpora, per message)

route	JSON	edge Brotli	v4 profiled	v5 profiled
audit events	12,687	2,512	795	574 (22.1×)
device telemetry	7,994	1,422	576	542
social feed	6,863	2,294	1,511	1,466
single order	782	408	187	180
OHLCV candles	3,225	842	362	362/384

On the profiled stream, stacking Brotli on top now loses bytes on four of five routes. Site and READMEs updated to the new numbers; the profiled rows drop Brotli.

Process

• Spec written first; codex gate on the spec text (1 round), then three gate rounds on the TS implementation — findings included two real design holes (FOR width minimality was never pinned; the affordability bound vs width-zero columns) now fixed normatively.
• Python and Rust ported independently against spec + vectors in parallel worktrees; both independently flagged the same TS deflate/maxByteLength asymmetry, fixed here.
• All golden vectors hand-computed from spec math before asserting the implementations reproduce them; every new decoder rule carries a discriminating vector (mutation-checked).
• TS 272 tests · Python 205 (3.10–3.13) · Rust 11 binaries/163 vector cases · live TS↔Python interop now 197 B (12.4×).

🤖 Generated with Claude Code