Preserve contained sampled-part bounds during history correction - #943
Merged
bradhilton merged 12 commits intoSep 24, 2026
Merged
Conversation
…story-part-bounds-followup-09-20260923
bradhilton
marked this pull request as ready for review
September 23, 2026 18:49
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Corrected complete-message bounds can leave valid sampled parts unrecognized or replay visible content around a length stop. Preserve contained part proofs and exact sampled IDs/logprobs, reject unproven complete boundaries, and retain an independent synthetic length STOP without replaying trailing visible text, including alternate captured EOS IDs. Restore source-proven STOP flags per corrected occurrence so repeated references to one sampled source retain both stops.
Validation: 311 tokenizer tests pass against the exact branch package composed with main
376c68b83, plus configured Ruff, format and whitespace checks. Native regressions confirm the prior opening-prefix, alternate-EOS, short-capture and repeated-source failures. Fresh independent source reviews approved, and exact-head CI passed for471b66e272db18cce2b439e0faccccd409b0cfc5. Fixtures are synthetic; the original failed R6/R7 trajectory was unavailable. No live training source was changed.