Add SentencePiece/LLaMA tokenizer path (#464)#466

Merged

michalharakal merged 4 commits intodevelopfrom

feature/464-tokenizer-path

Apr 13, 2026

Commits on Apr 12, 2026

Add SentencePieceTokenizer core (#464 step 1)

michalharakal
and
claude
committed
Wire SentencePieceTokenizer into TokenizerFactory (#464 step 2)

michalharakal
and
claude
committed

Commits on Apr 13, 2026

Add TinyLlama end-to-end fixture test for SentencePiece (#464 step 3)

michalharakal
and
claude
committed
Document #464 SentencePiece path in CHANGELOG (#464 step 4)

michalharakal
and
claude
committed