Feature/463 bpe qwen#465

Merged

michalharakal merged 7 commits intodevelopfrom

feature/463-bpe-qwen

Apr 12, 2026

Commits on Apr 12, 2026

Add Tokenizer interface and retrofit TekkenTokenizer (#463 step 1)

michalharakal
and
claude
committed
Add GPT-2 byte-to-unicode table (#463 step 2)

michalharakal
and
claude
committed
Add QwenByteLevelBpeTokenizer core (#463 step 3)

michalharakal
and
claude
committed
Expose tokenizer fields in GgufModelMetadata (#463 step 4)

michalharakal
and
claude
committed
Add TokenizerFactory with per-architecture dispatch (#463 step 5)

michalharakal
and
claude
committed
Add Qwen2.5 end-to-end fixture test for byte-level BPE (#463 step 6)

michalharakal
and
claude
committed
Document #463 byte-level BPE fix in CHANGELOG (#463 step 7)

michalharakal
and
claude
committed