Decompose _tokenize_chat_view into a staged _ChatViewTokenizer - #997

Merged

bradhilton merged 12 commits into

mainfrom

refactor/tokenize-chat-view

Sep 27, 2026

Commits on Sep 27, 2026

Move _tokenize_chat_view into a _ChatViewTokenizer class
bradhilton
committed
89dbb65
Extract _ChatViewTokenizer._render_messages
bradhilton
committed
ecdce4e
Extract _ChatViewTokenizer._render_canonical_masks
bradhilton
committed
c4379bd
Extract _ChatViewTokenizer._substitute_exact_prefix
bradhilton
committed
2f813c7
Extract _ChatViewTokenizer._translate_masks
bradhilton
committed
f8b4d2b
Extract _ChatViewTokenizer._prepare_span_search
bradhilton
committed
5eb451f
Extract _ChatViewTokenizer._prove_marked_bounds
bradhilton
committed
f01e1a3
Extract _ChatViewTokenizer._prove_probed_bounds
bradhilton
committed
e22eca8
Extract _ChatViewTokenizer._tokenize_exact_length_stops
bradhilton
committed
cb838b8
Extract _ChatViewTokenizer._collect_replacements
bradhilton
committed
2779a64
Extract _ChatViewTokenizer._assemble
bradhilton
committed
3c3d1a6
Break the _ChatViewTokenizer render-cache reference cycle
bradhilton
committed
b9f9da8