fix autocorrect and sentence_tokenize stopping after 32 matches#84
Open
CodingSelim wants to merge 1 commit into
Open
fix autocorrect and sentence_tokenize stopping after 32 matches#84CodingSelim wants to merge 1 commit into
CodingSelim wants to merge 1 commit into
Conversation
both were passing re.UNICODE as the 4th positional arg to re.sub, which is count not flags. re.UNICODE is 32 so every substitution silently stopped after 32 replacements. so autocorrect left long text under-corrected past the 32nd match and sentence_tokenize returned the tail as one unsplit blob. moved it to flags= so the count is unbounded again, added tests with >32 matches that fail before and pass after.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
noticed autocorrect and sentence_tokenize quietly give up partway through longer text. both call re.sub with re.UNICODE as the 4th positional arg, but that slot is count, not flags. re.UNICODE happens to be 32 so each sub just stops after 32 replacements.
so autocorrect leaves any real-length text under-corrected past the 32nd match, and sentence_tokenize only inserts 32 split markers and hands back the whole tail as one unsplit sentence.
quick repro:
fix is just moving it to flags=re.UNICODE so the count goes back to unbounded. added a case to each of the existing tests with more than 32 matches, they fail before and pass after. full test_araby suite is green.