Reuse recorded tokens by default and preserve literal assistant content

[bradhilton]

Complete recorded responses could fail tokenization when a tool call’s structured projection was shorter than the sampled output. Literal <think> text could also be deleted, and copied responses could receive logprobs from the wrong prompt. This fixes those cases through the existing art.tokenize API, without a new opt-in.

Recorded prompt IDs, response IDs, logprobs and source ownership remain authoritative. Complete unchanged Chat histories end at the final recorded output, including length-limited and tool responses. ART proves nonterminal boundaries separately. Historical assistant roles require a full prompt match using the original request messages and tool order; edited histories and explicit template overrides retain rendering.

Plain content stays literal; structured reasoning retains its meaning. Bounded evidence reuse avoids repeated validation within callback-free phases and clears before user callbacks.

Public regressions, real Qwen tokenization and retained failure inputs validate the change. Removing unobserved terminal footers deliberately changes their OUTPUT/SFT inclusion. Captured proofs establish token ownership, not numerical-training or production-speed equivalence.

Validation, compatibility and review scope

• Current head: afabb3c9f. Canonical CI passed: lint, format, type and lock checks; 1,264 lightweight tests; and 2,294 unit tests (34 skipped, two deselected). The final test-only update separately checks native no-render behavior and explicit rendering; production remains 671a9dc1.
• The terminal correction changes three private tokenization definitions. Local coverage includes 366 lightweight cases plus five real-ART followups, 35 receiving controls (overlapping the focused tests), and two real-Qwen tool/length cases. Independent source reviews bind production 671a9dc1; a separate copied-context oracle has 30 positive and adversarial controls. Test-only fixture updates keep STOP-authority tests on histories that actually require role rendering.
• Retained Ef339 now preserves its one canonical history, all 11 original conditioning/ID/logprob/STOP proofs and 24,026 first-owned finite terms. E047, Ebc83 and E2ac also pass the unchanged generic objective witness. E3cab preserves two canonical histories: 14 complete sampled encounters and one authenticated context copy, with 20,593 finite terms and nine sampled STOP bits. Input/options and compact serialization remain unchanged.
• Ext09 retains the documented unknown-STOP behavior when no tokenizer was resolved for its model. A separate authority-supplied call verifies all six sources, 8,881 finite terms and four sampled STOP bits after effective renderer/kwargs equality. Batch208 still correctly refuses a nonnested, wrongly conditioned sampled span on both the previous and new source; this correction does not add history repartitioning. Retained input components were verified, but incomplete context snapshots were not used; reconstructed authority/cache limits remain explicit.
• Seven alternating public CPU rounds, using a controlled template with the real cached Qwen vocabulary, measured a complete single terminal-length case at 37.56→19.53 ms. Its two unrecorded terminal tokens are deliberately omitted; the retained prefix IDs/logprobs/flags are equal. Ordinary and multi-turn controls remained within noise. Earlier memo benchmarks remain component-specific; no private latency or GPU gain is claimed.
• Complete native calls still work offline without downloading metadata or guessing EOS. Same-call resolved authority is reused only for the same model. Explicit overrides, partial/edited projections, ambiguous historical roles and unsupported source evidence retain their checks. A copied suffix requires its complete original sampled occurrence in an earlier selected history.
• Parser normalization preserves literal Jinja data, structured reasoning and unrelated custom operations. Named-template collisions after normalization refuse before rendering.
• This supersedes the need for the extra public API/native opt-in in Add explicit sampled-output STOP certification #973/Add an explicit native representation for sampled tokenization #976 for this default behavior. Their strict proof and SAMPLED-only repartitioning utilities remain distinct and are not imported here. Neither PR is automatically closed or merged. Fable produced no final verdict before its account limit; no fresh two-model clearance is claimed. Frozen runs and native training dependencies remain unchanged.