Recognize Arabic and fullwidth CJK comma as name-part delimiters#281
Merged
Conversation
The "Lastname, Firstname" split only matched the ASCII comma, so Arabic (U+060C) and fullwidth (U+FF0C) commas failed to trigger the comma format and leaked into the parsed output. Adds a shared "commas" regex used for both the split and the trailing-comma strip. Fixes #265.
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #281 +/- ##
=======================================
Coverage 97.69% 97.69%
=======================================
Files 13 13
Lines 912 913 +1
=======================================
+ Hits 891 892 +1
Misses 21 21 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
RegexTupleManager falls back to EMPTY_REGEX for any key absent from a
custom regexes dict, and re.compile('').split() matches between every
character rather than not splitting -- so a Constants built with a
custom regexes set that omits "commas" shattered names into single
characters instead of just disabling the comma split. Also clarifies
the suffix_delimiter docstring to mention the comma variants.
Found in review of #281.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
parser.py) only matched the ASCII,, so the Arabic comma،(U+060C, standard in Arabic/Persian/Urdu text) never triggered "Last, First" detection and stuck to the last-name output.commasregex ([,،，], also covering the fullwidth CJK comma，U+FF0C) used both for splitting on commas and for stripping a trailing comma before parsing.Test plan
tests/test_comma_variants.pycovers Arabic comma splitting, fullwidth comma splitting, no pollution in output, and trailing-comma stripping (written first, watched fail, then fixed — TDD)pytest tests/— 1464 passed, 4 skipped, 22 xfailed, no regressionsmypy nameparser/ tests/— cleanFixes #265