Decompose _tokenize_chat_view into a staged _ChatViewTokenizer

[bradhilton]

Summary

Decomposes _tokenize_chat_view in src/art/trajectories/_tokenize.py — a single 1,713-line function with 14 nested closures, 215 local variables, 146 ifs and nesting depth 9 — into class _ChatViewTokenizer. _tokenize_chat_view keeps its exact signature and becomes a 19-line wrapper that builds a fresh instance and calls run().

• The 14 closures (raw_render, render_normalized_text, render_text, segmented_render, render, probe_render, part_ids, source_prompt_tokens, source_output_tokens, source_matches_context, locations, canonical_render_to_rendered, canonical_span_to_rendered, differing_span) became methods with verbatim bodies.
• The straight-line body became ten stage methods called in the original order from run(): __init__ (setup), _render_messages, _render_canonical_masks, _substitute_exact_prefix, _translate_masks, _prepare_span_search, _prove_marked_bounds, _prove_probed_bounds, _tokenize_exact_length_stops, _collect_replacements, _assemble. Each is a verbatim slice of the original with only local→attribute renames.
• The ~50 shared locals are flat self. attributes rather than grouped dataclasses: the state does not partition by lifecycle (rendered is written in three stages, direct_bounds in two, stop_mask is mutated in place late, segmented is read via late binding), so any grouping would split co-mutated names. Introducing dataclasses is a cheap follow-up if wanted.
• Kept in _tokenize.py rather than a new module because three tests monkeypatch _tokenize module globals that the body resolves at call time (_tokenize_exact_projected_chat_history, _history_matches_projection, _cached_tokenizer); a move would have made those silent no-ops.
• run() deletes self.prefix_render_cache in a finally so the render cache's bound-method reference does not form a self → cache → method → self cycle; per-call state is refcount-freed exactly as the closure version was (verified with gc disabled: 0 instances alive after return, 0 unreachable objects on collect).

_tokenize_chat_view: 1,713 → 19 lines. _tokenize.py: 6,981 → 7,125 lines (class is 1,837 lines after ruff format; largest method _collect_replacements at 566 lines — the natural seams for a second pass are a per-message context object with eight sub-methods, and five stages in _prove_marked_bounds).

Equivalence evidence

• A generator script sliced the original by line range and, at each of the 11 commits, checked with ast.dump that every method body equals the original slice after normalizing self.X → X, that every original statement is covered exactly once, and that closure signatures are ['self'] + original. symtable finds no unresolved names.
• Byte-identical output: with the companion golden harness (refactor/tokenize-golden-harness, Add a byte-exact golden regression harness for trajectory tokenization #996) merged on top — 776 traced tokenize_history calls across 279 existing tests plus 13 named cases covering every "preserve … boundaries" fix since August — ART_TOKENIZE_GOLDEN=check pytest tests/unit/trajectories: 756 passed, at the first commit and at HEAD.
• Independent review (thermo-nuclear rubric) independently re-derived the normalized-AST diff and added name-resolution and definite-assignment analyses (clean); confirmed the truthy-walrus → is not None change is inert (TokenizedHistory is a pydantic model with no __bool__/__len__, and the helper applies the original truthiness test internally); no closure had default arguments or nonlocal; no error path embeds frame names; nothing pickles the instance; perf unchanged within ±5% across five cases including the real Qwen3-0.6B tokenizer. Its one Medium (the reference cycle) is fixed in the last commit. Bugbot found no bugs.
• tests/unit/trajectories: 742 passed; ruff check, ruff format --check, ty check clean. Rebased onto 3ad9a5f8d.

One cosmetic change: the UserWarning from _warn_prefix_retokenization (stacklevel=3) now attributes to _ChatViewTokenizer.run instead of _tokenize_history; both frames are internal to _tokenize.py.

[bradhilton]

Independent consumer review by Hayek and his review delegate; posted by Jarvis at Brad's request. Validation below describes the reviewer's work.

Reviewed d0f718d against 3ad9a5f (all 12 commits). No blocking findings in this refactor. I did not author this change.

• The 14 extracted closure bodies/signatures match after local-to-instance renaming. The remaining 73 statements retain their order except three empty cache allocations moved into construction; the two tokenizer-variable substitutions reference the same resolved object. The other 138 top-level definitions are unchanged. This preserves the existing token replacement, mask translation, source ownership, and tool-history logic rather than changing their contracts.
• The final try/finally breaks the state → render cache → bound method → state cycle. Four independent checks with cyclic GC disabled confirm immediate release after ordinary success, exact early return, renderer failure, and assembly failure, while preserving input history and the primary exception.
• All 13 named golden cases from companion Add a byte-exact golden regression harness for trajectory tokenization #996 at a569226958d0078058767e28d321eb85c4cbf8fb, plus fixture completeness, passed against this head. This includes all four cached real Qwen tokenizer cases: zero skips, no download. Coverage includes tool calls/reasoning, literal thinking tags, multipart content, sampled multiturn histories, length stops, and NaN logprobs.

Validation limit: #996 is a separate PR, not part of this head or its canonical CI. I ran its named cases, not the separate 776-call trace corpus; that opt-in tracer excludes raising/private-worker calls and reports previously unknown tests without failing. The changed code stays in the existing module, so there is no added package/import manifest to ship. Hosted quality checks passed; package-install and GPU execution jobs were skipped. No GPU/performance or exhaustive semantic-equivalence claim. Warning/traceback frames move into the stage methods.

Head/base were rechecked unchanged after review. This disposition does not cover subsequent integration with the separate native-token correctness stack.