Phase 5 M2 desk-research spike: tokenizer integration architecture - #13

Open

perNyfelt wants to merge 5 commits into

mainfrom

phase5-m2-spike

Commits on Aug 18, 2026

Phase 5 M2 desk-research spike: tokenizer integration architecture

perNyfelt
and
claude
committed
531d86f
Phase 5 M2 spike: confirm Rust is unavoidable via tokenizers-cpp

perNyfelt
and
claude
committed
0631e23
Phase 5 M2 spike: mlx-c won't add tokenizer support, and DJL vs. HF's own Swift port un-settles the architecture choice

perNyfelt
and
claude
committed
f570d94
Phase 5 M2 spike: measure the pure-Java port size instead of guessing it

perNyfelt
and
claude
committed
3a2b880
Phase 5 M2 spike: no Java swift-jinja equivalent, but a smaller HF-maintained porting source and a third no-port option

perNyfelt
and
claude
committed
ee95f0c