Prove native conditioning before preserving literal reasoning markers

[bradhilton]

Some chat templates parse literal </think> text as reasoning even when the recorded generation disabled thinking. For example, HEAD</think>MIDDLE</think>TAIL can lose its first two segments and change the preceding turn’s length-stop boundary. This replaces #967’s exact-template hash exception with a behavioral proof over render copies and a final native-conditioning check.

For a generation whose own recorded request explicitly sets thinking off, the repair requires native output (apart from an independently evidenced sampled stop suffix) to decode exactly to the full content. The template must preserve the empty-message scaffold and render the complete literal content directly after the generation prompt. Only then are empty reasoning aliases added to a temporary render copy. The captured messages and inference template remain unchanged.

Every sampled source in an adapted history must then retain its complete native prompt/output IDs, logprobs, source ownership and sampled flags, including STOP. This catches a later source that was actually generated against an earlier lossy rendering. Structured reasoning and its aliases remain structured; thinking-on/unknown sources are not reinterpreted. Unsupported/partial protocol projections or unavailable optional probes decline the repair, while inconsistent final native conditioning raises.

Compatibility: unlike the old hash exception, this requires complete native evidence before accepting a repair. Missing evidence leaves normal rendering of the recorded messages in place. This does not invent finite logprobs, reconcile sampled STOP flags, or change source-free SFT behavior. The explicit sampled-STOP API in #973 is separate.

Validation:

• 536 focused public CPU tests across tokenization, histories, source identity and render caching; semantic template variants, literal/multiple/nested markers, real mixed reasoning ON/OFF histories, partial Responses projections, mutation refusals, native-prefix negatives and length/STOP ownership are covered.
• Six public Qwen BPE checks using revision 995ad96eacd98c81ed38be0c5b274b04031597b0 (tokenizer JSON SHA 5f9e4d49…) and the exact preserved template, a comment variant and a reasoning-alias variant; two-source IDs/logprobs/flags/prefixes and input nonmutation match. No private replay, model or GPU workload.
• Reuses Reuse exact chat-template prefixes during long-history tokenization #960’s admitted invocation-local cache. Added bounded known-full-render priming checks; cache keys still distinguish message context, settings and generation flag. Canonical CI passed lint, format, full types and lock checking, 1,247 Megatron lightweight tests, and 2,061 unit tests (34 skipped, 2 deselected). The GPU lane correctly skipped this change. Workstation-only full ty diagnostics were byte-identical to clean main.
• Independent source/API review passed on this exact head, including complete native-prefix/ownership proof, optional-probe compatibility, cache invalidation, bounds and measured overhead.

Cost is material and measured separately from #960’s cache improvements. A fixed single-CPU public Qwen BPE comparison against main (ABBA/BA order, three trials per variant and three iterations per trial) measured ordinary explicit-OFF overhead of 12–37.5% at 4/16/32 turns; the 32-turn case was 66.7 → 74.7 ms despite 68 → 66 render calls. The all-turn literal repair fixture was 77.3 → 194.6 ms (2.52×; 68 → 227 renders) at 32 turns. Unknown-mode and already-native tool fast paths retain their call counts. These small synthetic timings are not production throughput estimates. Before the final full-render reuse adjustment, the corresponding 32-turn results were +13.7% and 2.48×; both measurements are retained rather than claiming a timing win from the adjustment.