fix(train): drop unknown tokens during training and inference - #359

Merged

stephantul merged 3 commits into

MinishLab:mainfrom

serhiizghama:fix/train-drop-unk-tokens

Sep 5, 2026

Commits on Sep 3, 2026

fix(train): drop unknown tokens during training and inference
serhiizghama
committed
4976cc8
test: cover unk stripping against StaticModel tokenization
serhiizghama
committed
97beaae

Commits on Sep 4, 2026

test: cover the no-unk-token path in train tokenization
serhiizghama
committed
357b101