From 3c7a14fe9b8f77e78986366315f24e21cdc6fc2e Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Wed, 26 Aug 2026 00:28:08 -0700 Subject: [PATCH 01/12] fix(extract,rules): a marker-led clause reads maiden, whatever pair encloses it MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit "Jane Smith née Jones" gives maiden 'Jones'; "Jane Smith (née Jones)" gave nickname 'née Jones', because the parens are a nickname pair by default and nothing looked inside them. The clause says "maiden" out loud, so the caller should not have to say it in Policy. extract already emits Role.MAIDEN regions and group already drops the marker inside one (#329), so this is a role swap at the existing emit site plus one predicate beside _suffix_shaped -- in the branch S1 has already declined, which keeps S1's precedence by construction. Extracting the region rather than masking the delimiters is what keeps "Jane (née Jones) Smith" reading family 'Smith': M2's take runs to the end of the name, and the closing paren says where the maiden name stops. A marker with no word after it stays a nickname, so M1's (Nee) boundary is not contradicted on the unconfigured path. M1's boundary example was this very input and moves to M1's own stated exception, the suffix-shaped clause S1 takes first. Two existing tests move with it. A quoted clause that is itself marker-led is now maiden too -- M3 is keyed on content, not on which pair matched -- so the #329 row that contrasted the two roles has no contrast left; it keeps its place recording that, and the role filter it pinned (_group.py:841) is pinned instead by a marker glued to punctuation, which M3's whitespace-split test declines and the tokenizer's token-level one still tags. And test_parser's corpus filter stripped only a trailing period, so a bracketed '(geb.' slipped past a filter whose comment says "no marker already present"; those names now carry a maiden clause of their own, and appending a second one is no longer a single variable. Closes #335 Co-Authored-By: Claude Opus 5 --- docs/design/rules.md | 39 +++++++- nameparser/_pipeline/_extract.py | 30 +++++- tests/v2/cases.py | 126 +++++++++++++++++++++++--- tests/v2/test_ledger_guards.py | 4 +- tests/v2/test_parser.py | 16 +++- tools/differential/corpus_rules.jsonl | 3 + 6 files changed, 194 insertions(+), 24 deletions(-) diff --git a/docs/design/rules.md b/docs/design/rules.md index 69f17dc9..542563a4 100644 --- a/docs/design/rules.md +++ b/docs/design/rules.md @@ -531,7 +531,7 @@ N3. Rationale: a person set down as a nickname plus one name word is ## Maiden names (M) -Background: a maiden name is written beside the current name, set off by a marker word or by enclosure. Markers are attested across French née/né, German geb./geborene, Dutch geboren, Czech/Slovak roz./rozená, Scandinavian født/fødd/född, Russian урожд. (both ё and е spellings), and Japanese 旧姓 — both grammatical genders where attested. Japanese more often writes the marker with a fullwidth colon (旧姓：佐藤), which is no separator, so marker and name arrive as a single word. Which enclosures mean "maiden" rather than "nickname" is a caller convention, so the maiden reading of a delimiter pair is opt-in. +Background: a maiden name is written beside the current name, set off by a marker word or by enclosure. Markers are attested across French née/né, German geb./geborene, Dutch geboren, Czech/Slovak roz./rozená, Scandinavian født/fødd/född, Russian урожд. (both ё and е spellings), and Japanese 旧姓 — both grammatical genders where attested. Japanese more often writes the marker with a fullwidth colon (旧姓：佐藤), which is no separator, so marker and name arrive as a single word. Which enclosures mean "maiden" rather than "nickname" is a caller convention, so the maiden reading of a delimiter pair is opt-in — except where the clause announces itself. A clause of two words or more led by a marker word has said which convention it means, and reads as the maiden name inside a nickname pair as well (M3); a lone marker word has not, and neither has one the colon spelling above glues to the name. M1. Rationale: an enclosure the caller has declared to mean maiden holds the former family name; a recognized marker word inside it @@ -543,12 +543,16 @@ M1. Rationale: an enclosure the caller has declared to mean maiden clause keeps its word, which may itself be a surname (Nee). Clauses are independent: two enclosures read as one maiden name, each dropping or keeping its own marker. A pair configured for - both maiden and nickname reads maiden. + both maiden and nickname reads maiden. Configuring the pair is + what this rule needs for a clause that does not announce itself + — markerless content, and a lone marker word alike; a clause of + two words or more led by a recognized marker reads as the maiden + name inside a nickname pair as well (M3). "Jane Smith (née Jones)" maiden-parens → maiden="Jones" "Jane Smith (Nee)" maiden-parens → maiden="Nee" · boundary "Jane Smith (Nee) (Jones)" maiden-parens → maiden="Nee Jones" - "Jane Smith (née Jones)" → nickname="née Jones" · boundary - history: decisions.md#M1 · interacts: S1, M2 · implemented: nameparser/_pipeline/_extract.py, nameparser/_pipeline/_group.py + "Andrew Perkins (MBA)" maiden-parens → suffix="MBA" · boundary + history: decisions.md#M1 · interacts: S1, M2, M3 · implemented: nameparser/_pipeline/_extract.py, nameparser/_pipeline/_group.py M2. Rationale: a maiden marker announces that what follows it is the former family name; the marker is an announcement, not a name. @@ -599,6 +603,33 @@ M2. Rationale: a maiden marker announces that what follows it is the "John née Jones Smith Ma" → maiden="Jones Smith Ma" history: decisions.md#M2 · interacts: P2, P3, P5, R2, M1, S2, H1 · implemented: nameparser/_pipeline/_group.py +M3. Rationale: an enclosure says nothing about whether it means + maiden, but a recognized marker word inside it does — the clause + announces itself, so the caller does not have to declare the + pair. + A bracketed clause whose content opens with a recognized marker + word and carries a word after it reads as the maiden name, + whichever bucket the enclosing pair sits in — unless the content + is suffix-shaped, which S1 takes first — the marker itself + dropped, as M1 drops it. A marker with no word after it is just + a word in brackets, and so is a marker no separator divides from + the name, the fullwidth-colon spelling M2 records. Where the + pair is already configured for maiden names M1 governs and this + adds nothing. Being keyed on the content rather than on the + pair, this reaches a nickname pair's clause too, and M1's + independence then governs what it produces: where a marker-led + clause stands beside another marker-led clause, both read as + maiden and join into one maiden name, leaving no nickname. + "Jane Smith (née Jones)" → maiden="Jones" + "Jane (née Jones) Smith" → family="Smith" + "Jane Smith (née)" → nickname="née" · boundary + Accepted: the word taken after the marker is not tested for + being a name word, so unlike M2's bare take this one does not + stop at a suffix word — the same two words read one way + bracketed and another way bare. + "Jane Smith (née V)" → maiden="V" + interacts: M1, M2, S1, N1 · implemented: nameparser/_pipeline/_extract.py + ## Commas & structure (C) Background: a comma in a name signals one of two conventions — the listing form "Family, Given" or trailing credentials "Name, PhD" — and which is meant can only be judged from what stands after the first comma. Recognizing a credential run is by nature a vocabulary judgment, so this is the one structural decision that consults the suffix word lists. Which characters COUNT as the comma is part of the rule: the Arabic comma (U+060C) and the fullwidth comma (U+FF0C) both signal the listing form, while the ideographic comma (U+3001) is not a name-structure comma at all (#265). diff --git a/nameparser/_pipeline/_extract.py b/nameparser/_pipeline/_extract.py index 3e33af62..e27c08ae 100644 --- a/nameparser/_pipeline/_extract.py +++ b/nameparser/_pipeline/_extract.py @@ -7,11 +7,16 @@ A Role.MAIDEN region is the WHOLE inner span, marker word included -- nothing here strips one. classify tags a marker inside it like any other token, and group drops it from a multi-token clause (#329). -Reads: Policy.nickname_delimiters, Policy.maiden_delimiters, and +A region reaches that role two ways: the pair that matched sits in +Policy.maiden_delimiters (M1), or the content itself opens with a +marker word (M3), which reassigns the role after the match and so is +the one thing here that a bucket alone does not decide. +Reads: Policy.nickname_delimiters, Policy.maiden_delimiters, +Lexicon.maiden_markers, and Lexicon.suffix_words / suffix_acronyms / suffix_acronyms_ambiguous through _suffix_shaped. -Implements rules N1, N2, S1 and M1 of docs/design/rules.md (the #273 +Implements rules N1, N2, S1, M1 and M3 of docs/design/rules.md (the #273 matching mechanism); each is cited at its code below. One scan mechanic worth stating up front: matching is one left-to-right pass, no nesting, and delimiter characters inside a matched region are @@ -52,6 +57,24 @@ def _suffix_shaped(content: str, lexicon: Lexicon) -> bool: or content.endswith(".")) +# rules.md#M3: "a bracketed clause whose content opens with a +# recognized marker word and carries a word after it reads as the +# maiden name, whichever bucket the enclosing pair sits in" +def _maiden_marked(content: str, lexicon: Lexicon) -> bool: + """The clause says 'maiden' out loud, so the caller does not have to + say it in Policy. Requires a word AFTER the marker: a lone marker in + brackets is a word in brackets, and M1 deliberately keeps a one-word + clause's word (it may be the surname Nee). The word after is not + tested for anything -- M3's Accepted line, and the reason a + bracketed '(née V)' reads maiden 'V' where the bare 'née V' gives + M2 a suffix. Whitespace-split, so a marker the writer glued to + punctuation is not one here ('née,'); the tokenizer splits that + comma off and still tags the token, which is what keeps _group's + Role.MAIDEN filter reachable.""" + words = content.split() + return len(words) > 1 and _normalize(words[0]) in lexicon.maiden_markers + + # rules.md#N2: "a quote whose open and close are the same character # opens only at a word start and closes only at a word end, so an # apostrophe inside or at the end of a word is literal" @@ -203,6 +226,9 @@ def extract_delimited(state: ParseState) -> ParseState: masked.append(Span(j, j + len(close))) else: if inner.start < inner.end: + if (role is Role.NICKNAME and _maiden_marked( + text[inner.start:inner.end], state.lexicon)): + role = Role.MAIDEN extracted.append((role, inner)) masked.append(Span(i, j + len(close))) # position-driven scanning makes overlapping matches diff --git a/tests/v2/cases.py b/tests/v2/cases.py index 82f13360..67bbfd21 100644 --- a/tests/v2/cases.py +++ b/tests/v2/cases.py @@ -1209,7 +1209,15 @@ def __post_init__(self) -> None: "nickname_delimiters.pop('parenthesis'), it gave first " "Jane / last Smith / maiden 'née Jones' -- same name " "fields, marker still inside the value, which is the " - "single field this change moves"), + "single field this change moves. Since #335 the same " + "input reads identically with NO policy at all " + "(maiden_marked_clause_reads_maiden_by_default below), " + "which does not make this row redundant: the pair " + "sitting in the maiden bucket settles the role before " + "M3 is consulted, so this row exercises M1's path and " + "that one exercises M3's. Rewriting it to drop the " + "policy would delete the configured path's coverage " + "rather than move it"), Case("maiden_marker_delimited_unaccented", "Jane Smith (nee Jones)", {"given": "Jane", "family": "Smith", "maiden": "Jones"}, policy=Policy(maiden_delimiters=frozenset({("(", ")")})), @@ -1228,6 +1236,57 @@ def __post_init__(self) -> None: "maiden 'nee Jones' (2026-08-03) -- the same diff the " "accented row records, which is the point: the two " "spellings behave alike on both sides"), + Case("maiden_marked_clause_reads_maiden_by_default", + "Jane Smith (née Jones)", + {"given": "Jane", "family": "Smith", "maiden": "Jones"}, + classification="fix(#335)", + notes="rules.md#M3 -- the clause says 'maiden' out loud, so the " + "pair enclosing it does not have to be configured. " + "maiden_marker_delimited above is the same input under " + "Policy(maiden_delimiters=...) and reads identically -- " + "what M3 adds is the DEFAULT reading, where 1.4.0 and " + "2.1 alike gave nickname 'née Jones'"), + Case("maiden_marked_clause_interior_keeps_the_family", + "Jane (née Jones) Smith", + {"given": "Jane", "family": "Smith", "maiden": "Jones"}, + classification="fix(#335)", + notes="the row that decides the MECHANISM. Extracting the " + "clause as a Role.MAIDEN region keeps the closing " + "delimiter as the maiden name's right boundary; masking " + "the delimiters and letting M2's bare-marker rule " + "consume the content instead would read maiden 'Jones " + "Smith' with an empty family, because M2's take runs to " + "the end of the name. The parens say where it stops"), + Case("maiden_marked_clause_one_word_stays_a_nickname", + "Jane Smith (née)", + {"given": "Jane", "family": "Smith", "nickname": "née"}, + notes="M3's boundary: a marker with no word after it is not a " + "maiden clause. Without this condition the default " + "reading of a lone parenthesized marker would flip to " + "maiden 'née', and M1's own (Nee) boundary -- a " + "one-word clause keeps its word, which may be the " + "surname Nee -- would be contradicted on the " + "unconfigured path. Parity: 1.4.0 read nickname 'née'"), + Case("markerless_parenthesized_clause_stays_a_nickname", + "Cherice J. (Johnson) Williams", + {"given": "Cherice", "middle": "J.", "family": "Williams", + "nickname": "Johnson"}, + notes="M3's other boundary, and the reason the maiden " + "delimiters remain worth configuring: the parenthesized " + "birth surname without a marker is a real US convention " + "and a corpus name (corpus_issues.jsonl), but nothing in " + "the clause says 'maiden', so it stays a nickname by " + "default. Only a caller who knows their data can say " + "otherwise, which is what Policy(maiden_delimiters=...) " + "is for. Parity"), + Case("maiden_marked_clause_beside_a_nickname", + 'Jane "Janey" Smith (née Jones)', + {"given": "Jane", "family": "Smith", "nickname": "Janey", + "maiden": "Jones"}, + classification="fix(#335)", + notes="two clauses, two roles. Through 2.1 both were " + "nicknames and the facade joined them into one value, " + "'Janey née Jones' -- the merged-nickname half of #335"), Case("maiden_marker_delimited_unmarked_content", "Jane Smith (Mary Jones)", {"given": "Jane", "family": "Smith", "maiden": "Mary Jones"}, @@ -1286,19 +1345,25 @@ def __post_init__(self) -> None: "Smith / maiden 'Jones née'"), Case("maiden_marker_delimited_beside_a_nickname_clause", 'Jane "née Janie" Smith {née Jones}', - {"given": "Jane", "family": "Smith", "nickname": "née Janie", - "maiden": "Jones"}, + {"given": "Jane", "family": "Smith", "maiden": "Janie Jones"}, policy=Policy(maiden_delimiters=frozenset({("{", "}")})), - classification="fix(#329)", - notes="the pass is scoped to MAIDEN clauses, and this is the " - "row that says so: two extracted clauses, both opening " - "with a marker word, and only the maiden one loses it. " - "Braces route to maiden here precisely so the default " - "nickname set survives untouched -- the parenthesis " - "rows above cannot show this, since Policy's " - "maiden-wins canonicalization would take ( ) away from " - "nickname. Without the role filter the nickname reads " - "'Janie'. 1.4.0 cannot express a brace delimiter at " + classification="fix(#335)", + notes="#335 took this row's job away, and the row is kept to " + "record that. It was the pin for the #329 drop pass " + "being scoped to MAIDEN clauses -- two extracted " + "clauses, both opening with a marker word, only the " + "maiden one losing it, nickname 'née Janie' and maiden " + "'Jones'. M3 now reads the QUOTED clause as maiden too, " + "since it is marker-led like the braced one and M3 is " + "keyed on content rather than on which pair matched, so " + "there is no nickname left to contrast: both clauses " + "are maiden and M1's independence rule joins them into " + "one value. The role filter it used to discriminate " + "(_group.py:841) is still reachable and still pinned -- " + "that job moved to " + "marker_glued_to_punctuation_keeps_the_clause_a_nickname " + "below, which reaches a marker-led clause M3 declines. " + "1.4.0 cannot express a brace delimiter at " "all (its buckets hold the NAMES of compiled regexes " "and there is no brace one; measured 2026-08-03, " "maiden_delimiters['brace'] = ('{', '}') is accepted " @@ -1308,8 +1373,39 @@ def __post_init__(self) -> None: "first Jane / middle 'Smith {née' / last 'Jones}' / " "nickname 'née Janie' -- braces as name text, the same " "convention maiden_marker_kyusei_delimited uses for a " - "knob with no v1 spelling. The nickname agreed even " - "there"), + "knob with no v1 spelling (re-measured 2026-08-26, " + "unchanged)"), + Case("marker_glued_to_punctuation_keeps_the_clause_a_nickname", + 'Jane "née, Janie" Smith (née Jones)', + {"given": "Jane", "family": "Smith", "nickname": "née Janie", + "maiden": "Jones"}, + classification="fix(#335)", + notes="M3 and the #329 drop pass ask the marker question of " + "different things, and this row is where the two " + "answers differ. M3 splits the clause on WHITESPACE and " + "normalizes the first word: 'née,' normalizes to " + "'née,' -- _normalize strips a trailing period but not " + "a comma -- so M3 declines and the quoted clause stays " + "a nickname. tokenize splits the comma off as a " + "separator, so the clause's first TOKEN is 'née' and " + "carries vocab:maiden-marker, which is exactly what " + "_group.py:841's 'role is not Role.MAIDEN' branch " + "exists to refuse. Measured 2026-08-26: with that " + "branch removed this reads nickname 'Janie', the " + "marker dropped out of a nickname. BOTH clauses are " + "load-bearing -- the drop pass is gated on the name " + "holding a MAIDEN region at all, so the same quoted " + "clause alone ('Jane \"née, Janie\" Smith') leaves the " + "branch unexercised, removing it measurably changes " + "nothing there. The paren clause is what opens the " + "block, and M3 is what makes it maiden. The comma is " + "absent from the nickname VALUE because tokenize " + "treats COMMA_CHARS as a separator inside every region " + "including an extracted one, which predates #335 and " + "is not part of it. 1.4.0 gave first Jane / last Smith " + "/ nickname 'née, Janie née Jones' (2026-08-26) -- " + "comma kept, both clauses merged into the one field, " + "which is the merged-nickname half of #335"), Case("maiden_marker_delimited_two_clauses", "Jane Smith (Nee) (Jones)", {"given": "Jane", "family": "Smith", "maiden": "Nee Jones"}, diff --git a/tests/v2/test_ledger_guards.py b/tests/v2/test_ledger_guards.py index 819c5c57..a0893636 100644 --- a/tests/v2/test_ledger_guards.py +++ b/tests/v2/test_ledger_guards.py @@ -1276,7 +1276,7 @@ def _claim(rule: dict) -> _Claim: "fix(comma-precomma-family) pre-comma run reads as family, not given": _Claim(279, ('family', 'given'), "28a62b622a48"), "fix(suffix-routing) two-token name with unambiguous trailing suffix stays suffix": - _Claim(1075, ('family', 'given', 'suffix'), "97934f29bdc8"), + _Claim(1078, ('family', 'given', 'suffix'), "235a385447cb"), "fix(suffix-delimiter-rendering) no-space delimiter core token kept whole": _Claim(0, ('suffix',), "e3b0c44298fc"), "ambiguous-surname-acronym data change: parenthesized (MA)/(DO) now stays nickname": @@ -1801,7 +1801,7 @@ class _Excluded(NamedTuple): ("fix(comma-family)", "fix(comma-precomma-family)", "fix(suffix-routing)")), '(^|[\\w.]\\s+)[("\'][^)"\']+[)"\'](\\s+\\w|\\s*$)': - _Excluded(46, "71eb2fa94553", ()), + _Excluded(49, "aa0609aef13c", ()), } diff --git a/tests/v2/test_parser.py b/tests/v2/test_parser.py index 5181f818..e2b8a9d3 100644 --- a/tests/v2/test_parser.py +++ b/tests/v2/test_parser.py @@ -1310,12 +1310,26 @@ def test_stacked_activation_warns_only_for_uncovered_scripts() -> None: def _clause_free_latin_corpus_names() -> list[str]: + from nameparser import DEFAULT_NICKNAME_DELIMITERS from nameparser.config.maiden_markers import MAIDEN_MARKERS from ._differential_fixtures import _CORPUS_NAMES + # A marker glued to a delimiter character is still a marker, and + # the membership test is per WORD, so '(geb.' must lose the + # bracket as well as the abbreviating period before it is asked. + # Stripping only the period admitted the three corpus names that + # bracket their marker, and once rules.md#M3 read such a clause as + # the maiden name they had one of their own -- two clauses, and + # the appended one no longer the only variable. Textual and so + # deliberately conservative: it also turns away the one-word + # '(Nee)', which M3 declines and which would have been safe to + # keep. Delimiter characters come from the shipped set rather than + # a literal, so a pair added there cannot quietly reopen this. + strip = "".join({ch for pair in DEFAULT_NICKNAME_DELIMITERS + for ch in pair}) + "." return [name for name in _CORPUS_NAMES if _LATIN.match(name) and "," not in name - and not any(word.lower().rstrip(".") in MAIDEN_MARKERS + and not any(word.lower().strip(strip) in MAIDEN_MARKERS for word in name.split())] diff --git a/tools/differential/corpus_rules.jsonl b/tools/differential/corpus_rules.jsonl index b365291f..22197876 100644 --- a/tools/differential/corpus_rules.jsonl +++ b/tools/differential/corpus_rules.jsonl @@ -43,9 +43,12 @@ "J.R. Smith" "Jack Ma." "Jack Wei Ma" +"Jane (née Jones) Smith" "Jane Smith (Nee)" "Jane Smith (Nee) (Jones)" "Jane Smith (née Jones)" +"Jane Smith (née V)" +"Jane Smith (née)" "Jane Smith née Jones" "Jane Smith née Jones PhD" "Jane Smith née V" From bbaad0a4110a92ea70c1a1c7d65137d568c3bd61 Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Wed, 26 Aug 2026 00:37:30 -0700 Subject: [PATCH 02/12] test(differential): classify the names the marker-led clause moves MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Six corpus names change reading under rules.md#M3 against 2.1.0 and 2.0.0, five against 1.4.0. One rule per ledger, keyed on the names by literal: the shape is "a bracketed clause whose first word is a marker", which is a question about the parse and not about the string, so a fields-only rule standing in for it would pre-excuse every future nickname/maiden confusion in the corpus. `fields` is exactly what each baseline's diff moves. Against 2.1.0 all six move {nickname, maiden}. Against 2.0.0 the five Latin names move that pair while the CJK name also flips given/family, so that ledger gets two rules rather than one declaring the union -- a union would have let a given/family regression on any Latin name pass unreported, which is what PR #444 found behind fix(#424) and had to narrow. Against 1.4.0 the CJK name is absent, and that is the interesting half. Its diff there is now {given, middle, family, maiden}: 1.4 read the fullwidth brackets as name text, so no nickname changes hands, and fix(cjk-maiden-marker) already describes exactly that diff and takes the name. fix(cjk-fullwidth-paren-nickname), whose only name it was, is declared dormant rather than deleted -- the reasoning it carries about its `fields` boundary is recorded nowhere else, and the declaration is load bearing in reverse. _carries() could not see the markers in question: it strips a token's abbreviating period but not its bracket, so 'Anna Müller (geb. Schmidt)' read as carrying no maiden vocabulary and the presence guard refused a rule that was right. Stripping delimiter characters only ever finds markers that are really there. Co-Authored-By: Claude Opus 5 --- tests/v2/test_ledger_guards.py | 25 ++++++++++- tools/differential/expected_since_1.4.0.toml | 44 ++++++++++++++++++++ tools/differential/expected_since_2.0.0.toml | 38 +++++++++++++++++ tools/differential/expected_since_2.1.0.toml | 30 +++++++++++++ 4 files changed, 135 insertions(+), 2 deletions(-) diff --git a/tests/v2/test_ledger_guards.py b/tests/v2/test_ledger_guards.py index a0893636..48d2162d 100644 --- a/tests/v2/test_ledger_guards.py +++ b/tests/v2/test_ledger_guards.py @@ -35,7 +35,7 @@ class declares, which members an alternation offers. Those are exact import pytest -from nameparser import _policy +from nameparser import DEFAULT_NICKNAME_DELIMITERS, _policy from nameparser._policy import Script # The parser's own fold, imported rather than reimplemented: a # hand-written one here stripped commas, parens, brackets and quotes, @@ -1116,8 +1116,21 @@ def _carries(name: str, vocabulary: frozenset[str]) -> bool: but exactly one corpus name reaches that branch today, and it is the 旧姓 one. Tighten this before admitting a vocabulary whose short non-ASCII entries occur inside ordinary names. + + Delimiter characters come off the token before the membership + test, because a marker glued to a bracket is still a marker to the + parser: rules.md#M3 reads '(geb. Schmidt)' as a maiden clause on + the strength of that very word, and _normalize strips the + abbreviating period but not the paren, so 'Anna Müller (geb. + Schmidt)' read as carrying no maiden vocabulary at all. This + direction is the safe one -- an unstripped token cannot be a + vocabulary entry, so the strip only ever finds markers that are + really there, and the names it rescues are exactly the ones a + maiden rule may legitimately claim. """ - tokens = {_normalize(token) for token in name.split()} + delimiters = "".join({ch for pair in DEFAULT_NICKNAME_DELIMITERS + for ch in pair}) + tokens = {_normalize(token.strip(delimiters)) for token in name.split()} return bool(tokens & vocabulary) or any( entry in name for entry in vocabulary if not entry.isascii()) @@ -1239,6 +1252,8 @@ def _claim(rule: dict) -> _Claim: #: both is growth into names the rule genuinely describes. _CORPUS_CLAIMS: dict[str, dict[str, _Claim]] = { "expected_since_1.4.0.toml": { + "fix(#335) a marker-led bracketed clause reads as the maiden name whatever pair encloses it": + _Claim(5, ('maiden', 'nickname'), "a419f74143e3"), "fix(#410) a title and one name word name the family, whatever annotation stands beside it": _Claim(3, ('family', 'given'), "24d6223e472f"), "fix(#410) the maiden flavor, where 1.4.0 read the marker as a middle name": @@ -1349,6 +1364,10 @@ def _claim(rule: dict) -> _Claim: _Claim(1, ('family', 'given'), "e62caedec864"), }, "expected_since_2.0.0.toml": { + "fix(#335) a marker-led bracketed clause reads as the maiden name whatever pair encloses it": + _Claim(5, ('maiden', 'nickname'), "a419f74143e3"), + "fix(#335) a marker-led bracketed clause reads as the maiden name, compounding with the CJK order flip": + _Claim(1, ('family', 'given', 'maiden', 'nickname'), "cf370e856ae7"), "fix(#410) a title and one name word name the family, whatever annotation stands beside it": _Claim(4, ('family', 'given'), "da1dd1473145"), "fix(#430) a credential run does not end at the roman numeral describing it": @@ -1453,6 +1472,8 @@ def _claim(rule: dict) -> _Claim: _Claim(1, ('family', 'maiden'), "2150936a8c55"), }, "expected_since_2.1.0.toml": { + "fix(#335) a marker-led bracketed clause reads as the maiden name whatever pair encloses it": + _Claim(6, ('maiden', 'nickname'), "d0e857deddb2"), "fix(#410) a title and one name word name the family, whatever annotation stands beside it": _Claim(4, ('family', 'given'), "da1dd1473145"), "fix(#430) a credential run does not end at the roman numeral describing it": diff --git a/tools/differential/expected_since_1.4.0.toml b/tools/differential/expected_since_1.4.0.toml index 194b8a11..474e0ced 100644 --- a/tools/differential/expected_since_1.4.0.toml +++ b/tools/differential/expected_since_1.4.0.toml @@ -647,6 +647,25 @@ issue = "fix(cjk-fullwidth-paren-nickname) fullwidth-parenthesis recognition com # above carry, pinned by the same auto-discovering test. The slug is # free-form; the #271/#272 substrings it avoids were a constraint of a # selector retired in #333. +# +# DORMANT since 2.2, and the reason is a migration rather than a +# revert. Its one name moved to fix(cjk-maiden-marker) above: under +# rules.md#M3 the fullwidth clause is read on the strength of the 旧姓 +# inside it, so the clause lands in `maiden` and `nickname` no longer +# diffs at all -- 1.4 read the brackets as name text and had no +# nickname here either. The diff is now {given, middle, family, +# maiden}, which is a subset of that rule's fields and not of this +# one's, so classify() hands the name over. That is the right home: +# what the name demonstrates today IS a marker consumption, and the +# fullwidth pair is only what makes the marker a separate token. +# +# Kept rather than deleted because the reasoning below is about the +# `fields` boundary between this rule and fix(cjk-delimited-nickname), +# which nothing else records, and because the declaration is load +# bearing in reverse: if a fullwidth-bracketed CJK NICKNAME diff ever +# appears again, this rule wakes and the harness says the dormant +# reason has gone false. +dormant = "its one name, '山田 花子（旧姓 佐藤）', diffs in `maiden` rather than `nickname` since rules.md#M3 and is claimed by fix(cjk-maiden-marker); no other corpus name pairs a fullwidth bracket with a CJK script" name_regex = "(?s)(?=.*[（）])(?=.*[\\u3005-\\u3006\\u3040-\\u309F\\u30A0-\\u30FF\\u3400-\\u4DBF\\u4E00-\\u9FFF\\uF900-\\uFAFF\\uAC00-\\uD7A3\\uFF65-\\uFF65])" fields = ["given", "middle", "family", "nickname"] @@ -1373,3 +1392,28 @@ issue = "fix(#410) the maiden flavor, where 1.4.0 read the marker as a middle na # marker extraction and the H1 fix land together. name_regex = "(?i)^Dr\\. Smith n[ée]e Jones$" fields = ["given", "middle", "family", "maiden"] + +[[change]] +issue = "fix(#335) a marker-led bracketed clause reads as the maiden name whatever pair encloses it" +# 'Jane Smith (née Jones)', 'Jane (née Jones) Smith', 'Jane Smith +# (née V)', 'Anna Müller (geb. Schmidt)', 'Ursula von der Leyen +# (geb. Albrecht)': rules.md#M3 reads a bracketed clause whose content +# opens with a marker word and carries a word after it as the maiden +# name. 1.4 extracted the same five clauses as NICKNAMES -- v1 knew +# the parenthesis pair and had no maiden reading to give an +# undeclared one -- so the diff is the role and only the role. +# +# `fields` is exactly the pair that moves. Five names, two fields, +# measured; the union with what these names move against 2.0.0 would +# be wider, and declaring a field a diff does not move is what let a +# real given/family movement hide behind fix(#424) until PR #444. +# +# The CJK name is NOT here, unlike in the two 2.x ledgers. Against +# 1.4 it diffs in {given, middle, family, maiden}: v1 read the +# fullwidth brackets as name text, so there is no nickname on either +# side to change hands, and fix(cjk-maiden-marker) above already +# describes exactly that diff. See the dormancy note on +# fix(cjk-fullwidth-paren-nickname), which is the rule that used to +# hold it. +name_regex = "(?i)^(?:Jane Smith \\(n[ée]e Jones\\)|Jane \\(n[ée]e Jones\\) Smith|Jane Smith \\(n[ée]e V\\)|Anna Müller \\(geb\\. Schmidt\\)|Ursula von der Leyen \\(geb\\. Albrecht\\))$" +fields = ["nickname", "maiden"] diff --git a/tools/differential/expected_since_2.0.0.toml b/tools/differential/expected_since_2.0.0.toml index 3931348a..be5ba35e 100644 --- a/tools/differential/expected_since_2.0.0.toml +++ b/tools/differential/expected_since_2.0.0.toml @@ -949,3 +949,41 @@ issue = "fix(#410) a title and one name word name the family, whatever annotatio # own baseline sees. name_regex = "(?i)^(?:Senator \"Rick\" Edmonds|Xyz\\. \\(Bud\\) Smith|Dr\\. Smith n[ée]e Jones|'Smitty' Dr\\. Jones)$" fields = ["given", "family"] + +[[change]] +issue = "fix(#335) a marker-led bracketed clause reads as the maiden name whatever pair encloses it" +# 'Jane Smith (née Jones)', 'Jane (née Jones) Smith', 'Jane Smith +# (née V)', 'Anna Müller (geb. Schmidt)', 'Ursula von der Leyen +# (geb. Albrecht)': rules.md#M3 reads a bracketed clause whose content +# opens with a marker word and carries a word after it as the maiden +# name, whichever bucket the enclosing pair sits in. 2.0 read the +# parenthesis pair as a nickname delimiter and looked no further, so +# the clause came back a nickname. +# +# `fields` is exactly the pair that moves. The five names here are +# Latin and the clause changes ROLE and nothing else in them, so a +# `given`, `middle` or `family` movement on any of them is a +# regression and stays loud. +name_regex = "(?i)^(?:Jane Smith \\(n[ée]e Jones\\)|Jane \\(n[ée]e Jones\\) Smith|Jane Smith \\(n[ée]e V\\)|Anna Müller \\(geb\\. Schmidt\\)|Ursula von der Leyen \\(geb\\. Albrecht\\))$" +fields = ["nickname", "maiden"] + +[[change]] +issue = "fix(#335) a marker-led bracketed clause reads as the maiden name, compounding with the CJK order flip" +# '山田 花子（旧姓 佐藤）', and its own rule rather than a sixth +# alternative in the one above, for the reason the CJK rules in this +# file keep restating: it moves FOUR fields where the five Latin names +# move two. 2.0 read the fullwidth clause as a nickname and the +# wholly-Han remainder given-first, so consuming the clause flips +# `given` and `family` (#271) on top of the role change. +# +# One rule holding all six would have to declare the union, and the +# union is a superset for five of them -- a given/family regression on +# any Latin name here would then pass unreported. That is the shape PR +# #444 found behind fix(#424) and had to narrow, and it is why this is +# split rather than widened. +# +# The literal is the whole regex: no shape test can separate "the +# fullwidth clause is a maiden clause now" from "the fullwidth clause +# is a nickname", which is the distinction the fields above turn on. +name_regex = "^山田 花子（旧姓 佐藤）$" +fields = ["given", "family", "nickname", "maiden"] diff --git a/tools/differential/expected_since_2.1.0.toml b/tools/differential/expected_since_2.1.0.toml index af8dbb11..4987c5ea 100644 --- a/tools/differential/expected_since_2.1.0.toml +++ b/tools/differential/expected_since_2.1.0.toml @@ -879,3 +879,33 @@ issue = "fix(#410) a title and one name word name the family, whatever annotatio # tests/v2/cases.py::suffix_stays_suffix_title instead. name_regex = "(?i)^(?:Senator \"Rick\" Edmonds|Xyz\\. \\(Bud\\) Smith|Dr\\. Smith n[ée]e Jones|'Smitty' Dr\\. Jones)$" fields = ["given", "family"] + +[[change]] +issue = "fix(#335) a marker-led bracketed clause reads as the maiden name whatever pair encloses it" +# 'Jane Smith (née Jones)', 'Jane (née Jones) Smith', 'Jane Smith +# (née V)', 'Anna Müller (geb. Schmidt)', 'Ursula von der Leyen +# (geb. Albrecht)', '山田 花子（旧姓 佐藤）': rules.md#M3 reads a bracketed +# clause whose content opens with a marker word and carries a word +# after it as the maiden name, whichever bucket the enclosing pair +# sits in. Through 2.1 the parenthesis pairs -- ASCII and fullwidth +# alike -- were NICKNAME delimiters by default and nothing looked +# inside them, so all six clauses came back as nicknames. +# +# `fields` is exactly the pair that moves and not one field more. The +# clause changes ROLE and nothing else here: the name words around it +# keep their readings, so a `given`, `middle` or `family` movement on +# any of these six is a regression and stays loud. Worth stating +# because the convenient thing is to declare the union of what the +# same six names move against OTHER baselines -- against 2.0.0 the +# CJK name also flips given/family -- and PR #444 is what that costs: +# fix(#424) there declared a field its diff had stopped moving, and a +# real movement was absorbed silently behind it until #444 narrowed +# the rule. +# +# Keyed on the six names by literal, like the #410 rule above. The +# shape is "a bracketed clause whose first word is a marker", which is +# a question about the parse rather than about the string, and a +# fields-only rule standing in for it would pre-excuse every future +# nickname/maiden confusion in the corpus. +name_regex = "(?i)^(?:Jane Smith \\(n[ée]e Jones\\)|Jane \\(n[ée]e Jones\\) Smith|Jane Smith \\(n[ée]e V\\)|Anna Müller \\(geb\\. Schmidt\\)|Ursula von der Leyen \\(geb\\. Albrecht\\)|山田 花子（旧姓 佐藤）)$" +fields = ["nickname", "maiden"] From dd61f3a4b7aac13dab76a543669da6cdc55f05ef Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Wed, 26 Aug 2026 01:03:11 -0700 Subject: [PATCH 03/12] docs(decisions,README,maiden_markers,release_log): record M3 decisions.md gains an M3 section: why the clause is EXTRACTED as a region rather than having its delimiters masked for M2 to consume (the two agree on a trailing clause and diverge on an interior one, because M2's take runs to the end of the name), why the condition is a word after the marker, the real division of labour -- one word versus a marker plus a word, not markerless versus marker-led -- the suffix-word divergence as an accepted consequence, and why implemented: names _extract.py alone. The durable finding is recorded there too: _maiden_marked asks a whitespace-split question and tokenize asks a token-level one, and that divergence is the only thing keeping _group.py:841's role filter reachable. Anyone unifying the two tests would orphan it silently. M1's entry records that its opt-in narrowed, and corrects its own 2026-08-05 premise: #329 was never gate-visible. Under the default policy no corpus name reached the drop pass, so the CJK name's 2.0.0-to-2.1.0 movement was the order flip. It is not gate-visible now either -- a ledger rule narrows by which fields move, never by what they hold, so reverting the drop leaves the same six names moving the same pair and all three gates green (measured by mutation). The README's conclusion under its maiden table says the same beside the table it belongs to, in place of a claim that the gate had gained the coverage. maiden_markers.py said the bracketed Japanese form needs Policy(maiden_delimiters=...). It does not, since M3. Also recorded, not resolved: M3 puts the markers to work in a position they never acted on by default -- the first word of any nickname clause -- and the vocabulary-collision criterion was never applied to it. 'Jane "Roz Smith" Jones' read nickname 'Roz Smith' through 2.1 and now reads maiden 'Smith' with no nickname. Co-Authored-By: Claude Opus 5 --- docs/design/decisions.md | 14 ++++++++- docs/design/rules.md | 2 +- docs/release_log.rst | 2 ++ nameparser/config/maiden_markers.py | 36 ++++++++++++++-------- tools/differential/README.md | 46 ++++++++++++++++++++++------- 5 files changed, 75 insertions(+), 25 deletions(-) diff --git a/docs/design/decisions.md b/docs/design/decisions.md index 9171771b..8edfc3a2 100644 --- a/docs/design/decisions.md +++ b/docs/design/decisions.md @@ -491,7 +491,19 @@ Declined: - 2026-08-05 #329/#335 — marker auto-detection inside a nickname-delimited clause was deferred to #335 on a corpus measurement: 山田 花子（旧姓 佐藤） is in the CJK differential corpus so the #329 change was gate-visible, while "Jane Smith (née Jones)" is in no corpus — shipping auto-detection in 2.1 would have let a real Latin-affecting change ride under a "0 Latin-only" gate report. -Open: [#335](https://github.com/derek73/python-nameparser/issues/335) should a marker inside a NICKNAME-delimited clause flip it to maiden without configuration. +- 2026-08-26 #335 — correction to the 2026-08-05 entry above, which reasoned from "the #329 change was gate-visible". It was not. Under the default policy the corpora run, no name reached the drop pass at all, so 山田 花子（旧姓 佐藤） moving between 2.0.0 and 2.1.0 was the East Asian order flip and not #329. Nor is #329 gate-visible now that M3 routes marker-led clauses to maiden without configuration: a ledger rule narrows by which FIELDS move and never by what they hold, so reverting the drop leaves the same six names moving the same {nickname, maiden} pair and all three gates green (measured by mutation 2026-08-26). The deferral #329 argued for was still the right call — it was about a Latin-affecting change riding under a gate report, and that half held — but the premise is corrected here rather than left to be re-used. tools/differential/README.md carries the same correction beside the table it belongs to. +- 2026-08-26 #335 (M3) — the opt-in this section opens with narrowed, and the boundary is not where the 2026-07-03 entry drew it. A clause whose content announces itself no longer needs its pair declared, so what configuring maiden_delimiters buys is now markerless content and one-word clauses. See #M3 for the whole reasoning; M1 still governs every clause whose pair is configured, and M3 defers to it for the marker drop itself. + +### M3 — the marker-led clause + +- 2026-08-26 #335 (M3, PR on fix/335-maiden-marked-clause) — a bracketed clause whose content opens with a recognized marker word and carries a word after it reads as the maiden name, whichever bucket the enclosing pair sits in. The parse already had the information: "Jane Smith née Jones" gave maiden "Jones" while "Jane Smith (née Jones)" gave nickname "née Jones", because ( ) is a nickname pair by default and nothing looked inside it. The clause says "maiden" out loud, so the caller should not have to say it in Policy. +- 2026-08-26 #335 — the MECHANISM, and the issue proposed the other one. Masking the delimiters and letting M2's bare-marker rule consume the content would avoid a second implementation, and it was rejected on measurement: the two readings agree on a trailing clause and diverge on an interior one. "Jane (née Jones) Smith" reads given "Jane", family "Smith", maiden "Jones" under extraction; masking turns it into the bare "Jane née Jones Smith", which reads maiden "Jones Smith" and family "" (measured 2026-08-26), because M2's take runs to the end of the name. The closing delimiter is a boundary the writer typed and masking throws it away. Extraction was also the SMALLER change — extract already emits Role.MAIDEN regions and group already drops the marker inside one (#329), so this is a role swap at the existing emit site plus one predicate, in the branch S1 has already declined, which keeps S1's precedence by construction. +- 2026-08-26 #335 — the condition is a word AFTER the marker, not the marker alone. The reason is M1's: a one-word clause keeps its word because Nee is an attested surname (Irish Ní/Nee, and a Chinese romanization), and a marker with nothing after it marks nothing. Be precise about what pins it, because the obvious answer is wrong: M1's own boundary line carries the maiden-parens annotation, so it asserts the CONFIGURED reading and passes either way — built and measured 2026-08-26, a len(words) > 0 predicate still gives it maiden "Nee". What the loosened predicate actually breaks is M3's own boundary example ("Jane Smith (née)" → nickname "née") and tests/v2/cases.py::maiden_marked_clause_one_word_stays_a_nickname, three failures and no more. Anyone re-litigating the second-word test should check those, not M1's line. So the real division of labour is NOT markerless versus marker-led, which is how it was first written into rules.md and had to be corrected at review: it is one word versus a marker plus a word. "Jane Smith (Nee)" is marker-led and still reads nickname "Nee" by default, maiden "Nee" only with the pair configured (both measured 2026-08-26). +- 2026-08-26 #335 — accepted consequence: the word taken after the marker is not tested for being a name word, so M3 does not stop at a suffix word where M2's bare take does. "Jane Smith (née V)" reads maiden "V" while the bare "Jane Smith née V" reads suffix "V" and family "née" (both measured). The reason is structural rather than an oversight: M3 hands the WHOLE clause to Role.MAIDEN and the bracket is its right boundary, so the trailing-suffix rule never sees the word as trailing the name at all. Pinned as M3's Accepted line in rules.md, which puts "Jane Smith (née V)" in the differential corpus and classified in all three ledgers. +- 2026-08-26 #335 — rules.md#M3 lists implemented: nameparser/_pipeline/_extract.py alone, and that is deliberate rather than an omission. M3 decides a REGION'S ROLE; the marker drop it describes ("the marker itself dropped, as M1 drops it") is M1's mechanism in _group.py, which cites M1 and is governed by it. Adding _group.py to M3's list would mean inventing an M3 citation there for behavior M1 already owns — and test_doc_citations requires implemented: to equal the set of modules that cite the rule, so the two would have to move together. Do not re-open this. +- 2026-08-26 #335 — the durable finding, and the one worth reading before touching either side: _maiden_marked asks a WHITESPACE-SPLIT, _normalize'd question of the clause's first word, while tokenize and classify ask a token-level one. _normalize strips a trailing period but not a comma, so 'née,' is not a marker to M3 while the tokenizer splits the comma off and still tags the token vocab:maiden-marker. That divergence is the only thing keeping _group.py:841's "role is not Role.MAIDEN" branch reachable at all: without it, a marker-led NICKNAME clause could no longer exist, since M3 would have converted every one to MAIDEN before group ran. Anyone "unifying" the two tests would orphan that filter silently. The pin is tests/v2/cases.py::marker_glued_to_punctuation_keeps_the_clause_a_nickname, and it needs BOTH of its clauses — the drop pass is gated on the name holding a maiden region at all, so a marker-glued nickname clause standing alone leaves the branch unexercised (measured by mutation, 2026-08-26). The row that used to pin the filter, maiden_marker_delimited_beside_a_nickname_clause, lost the job to M3: both its clauses are marker-led, so both are maiden now and there is no contrast left in it. +- 2026-08-26 #335 — RECORDED, NOT RESOLVED: M3 puts maiden_markers to work in a position they never acted on under the default policy — the first word of any nickname-delimited clause — and decisions.md#vocabulary-collisions C-i (a word belongs in the ambiguous subset iff it is borne as an ordinary name IN THE POSITION THE VOCABULARY CLAIM ACTS ON) was never applied to that new position. maiden_markers has no ambiguous subset at all. The shipped entry that most obviously bears the test is roz: Roz is an English diminutive, and a diminutive is exactly what a nickname clause holds. Measured 2026-08-26: 'Jane "Roz Smith" Jones' read nickname "Roz Smith" through 2.1 and now reads maiden "Smith" with no nickname, and 'Rosalind (Roz Smith) Jones' likewise. The nickname is lost and a maiden name invented, by default. M3 itself has no switch, so the only lever is the vocabulary: Parser(lexicon=dataclasses.replace(lex, maiden_markers=...)) without roz restores nickname "Roz Smith" (measured), which is a heavy instrument for one word and not something a caller would think to reach for. Surfaced by review after the mechanism was approved; no judgement recorded here yet, and no issue filed. +- 2026-08-26 #335 — what configuring maiden_delimiters still buys, since M3 removes the commonest reason to reach for it: markerless clauses, and one-word clauses. "Cherice J. (Johnson) Williams" is a real US convention and a corpus name, and nothing in that clause says maiden, so it stays a nickname by default — only a caller who knows their data can say otherwise. The knob is a declaration about the DATA; M3 is a reading of what a clause declares about itself. ### O1 — East Slavic rotation diff --git a/docs/design/rules.md b/docs/design/rules.md index 542563a4..3f5d29de 100644 --- a/docs/design/rules.md +++ b/docs/design/rules.md @@ -628,7 +628,7 @@ M3. Rationale: an enclosure says nothing about whether it means stop at a suffix word — the same two words read one way bracketed and another way bare. "Jane Smith (née V)" → maiden="V" - interacts: M1, M2, S1, N1 · implemented: nameparser/_pipeline/_extract.py + history: decisions.md#M3 · interacts: M1, M2, S1, N1 · implemented: nameparser/_pipeline/_extract.py ## Commas & structure (C) diff --git a/docs/release_log.rst b/docs/release_log.rst index 30d701ee..ae836087 100644 --- a/docs/release_log.rst +++ b/docs/release_log.rst @@ -45,6 +45,8 @@ Release Log - Fix a bound given name joining past a credential that the suffix rule then takes, leaving no family: ``"abdul Smith Jr Ma"`` read given ``abdul Smith`` with no family, where ``"John Smith Jr Ma"`` reads family ``Smith`` with suffix ``Jr, Ma``. The join's reserve had re-derived the suffix rule's trailing walk by hand and left out one of its two forks -- a bare acronym such as ``Ma`` is a credential only with words to spare -- so it counted the acronym as a name word, joined, and the suffix rule then peeled it. The two stages now share one walk: the reserve tries the join, runs the suffix rule's own peel over the pieces as the join would leave them, and joins only when two name words became one and nothing else changed. ``"abdul Smith Jr Ma"`` reads family ``Smith``, suffix ``Jr, Ma``; ``"abdul Smith Ma"`` reads family ``Smith``, suffix ``Ma`` rather than joining the pair and keeping the credential as the family -- both as 1.4.0 read them and as ``John`` reads in their place; ``"abdul Smith Berg Ma"`` keeps its join, and a title word standing after the bound word joins again as it did in 1.4.0 (``"Berg, abdul Sir"`` reads given ``abdul Sir``) without the pair ever being read as a title (``"abdul Sheikh and Ahmad Bakar"`` reads given ``abdul Sheikh and Ahmad``). No name of the differential corpora has the shapes; the ``Ma`` rules examples carry the fix, byte-identical to 1.4.0 and moving against 2.0.0 and 2.1.0, and the title-word shapes are pinned in tests (closes #425) + - Fix a bracketed maiden clause reading as a nickname because its brackets were not declared. ``"Jane Smith nee Jones"`` gave maiden ``Jones``, while ``"Jane Smith (née Jones)"`` gave nickname ``née Jones`` -- the parenthesis pair is a nickname delimiter by default and nothing looked inside it, so the marker word the writer put there was read as part of an alias. A bracketed clause whose content opens with a recognized marker word and carries a word after it now reads as the maiden name whichever bucket its pair sits in: ``"Jane Smith (née Jones)"`` reads family ``Smith``, maiden ``Jones``, and so does the Japanese ``"山田 花子（旧姓 佐藤）"`` (family 山田, given 花子, maiden 佐藤), which needed ``Policy(maiden_delimiters=...)`` through 2.1. The clause is lifted out as a region rather than having its brackets erased, which is what keeps an INTERIOR clause from eating the name behind it: ``"Jane (née Jones) Smith"`` reads given ``Jane``, family ``Smith``, maiden ``Jones``, where erasing the brackets would leave the bare ``"Jane née Jones Smith"``, whose marker takes every word to the end of the name -- maiden ``Jones Smith`` and no family at all. The closing bracket is a boundary the writer typed, and it is now where the maiden name stops. Two clauses beside each other no longer merge: ``"Jane "Janey" Smith (née Jones)"`` read nickname ``Janey née Jones`` through 2.1 and now reads nickname ``Janey``, maiden ``Jones``, each clause keeping its own role, and this reaches ``HumanName``. What does NOT move is a clause that says nothing about maiden names. A lone marker in brackets is just a word in brackets -- ``"Jane Smith (née)"`` stays nickname ``née``, as 1.4.0 read it -- because a marker with no name after it marks nothing, and because a one-word clause may be the surname it looks like (``Nee`` is Irish Ní/Nee and a Chinese romanization). A markerless clause stays a nickname too: ``"Cherice J. (Johnson) Williams"`` keeps nickname ``Johnson``, which is what ``Policy(maiden_delimiters=...)`` remains for -- the parenthesized birth surname is a real convention, and only a caller who knows the data can say that is what these brackets mean. One accepted consequence: the word after the marker is not tested for being a name word, so a bracketed clause does not stop at a credential where the bare form does -- ``"Jane Smith (née V)"`` reads maiden ``V`` while ``"Jane Smith née V"`` reads suffix ``V``. The clause goes to ``maiden`` whole and the bracket is its boundary, so the trailing-suffix rule never sees the word as trailing the name. Six of the 1,078 differential corpus names move against the 2.1.0 and 2.0.0 baselines and five against 1.4.0: the Japanese name is the sixth, and against 1.4.0 it is not a new diff at all, since 1.4 read the fullwidth brackets as name text and had no nickname there to change hands. Those six are also every name in the corpora that HAS the shape (measured), so the count says how few bracketed marker clauses the corpora hold rather than how narrow the change is; four of the six are hand-written rule and issue examples, and the coverage that bounds the behavior is ``tests/v2/cases.py`` (closes #335) + - Fix a particle chain and a maiden name taking a trailing generational numeral as a name word: ``"John van der Berg V"`` read family ``van der Berg V`` and ``"John née Jones Smith V"`` read maiden ``Jones Smith V``, where ``"John Smith V"`` reads suffix ``V``. Both stops asked "is this a suffix?" with the test that vetoes a bare ``V`` as an initial -- the same question the bound given-name join asked until #401 -- and now ask the suffix rule's own walk, over the name as they would leave it: family ``van der Berg``, maiden ``Jones Smith``, suffix ``V``, for ``I`` and ``X`` alike; ``"John van der J. V"`` and ``"J. née Jones Smith V"`` keep their readings, the word before the numeral being an initial as written or as left. The chain stops before a bare credential with words to spare too, as ``"John Smith Ma"`` does, where the suffix rule will still find them once the chain has run: ``"John van der Berg Ma"`` reads suffix ``Ma``, which is how 1.4.0 read it, while ``"Freiherr von Berg MA"`` keeps family ``von Berg MA``; a maiden name keeps such a credential, since the words the marker takes are the very words that made it one. A particle that is also suffix vocabulary now ends the chain where it opens the trailing run: ``"John Smith Mc V"`` reads suffix ``Mc, V``. The chain also no longer swallows the given name behind an unlisted abbreviation: ``"Xyz. van Johnson"`` and ``"Esq. van Gogh"`` read given ``van``, as ``"Dr. van Johnson"`` has since #367, the chain and the bound given-name join now asking the same test that reads the abbreviation as a title (closes #424) - Fix a name losing its given/family split when a comma is followed only by an honorific: ``"John Smith, Mr."`` now gives given ``John``, family ``Smith``, title ``Mr.`` where it returned the whole of ``"John Smith"`` as the family name. The family-comma rule that the part before the comma is wholly the surname rests on the writer having marked where the surname ends, and a comma followed by nothing but titles marked no such thing -- ``"John Smith, Mr."`` is ``"Mr. John Smith"`` with the honorific moved to the end. A comma followed by an actual name still fixes the family (``"John Smith, Jones"`` keeps family ``John Smith``), a title followed by a postnominal is each read where it stands (``"Smith, Dr. Jr."`` is unchanged), and a single pre-comma piece has no split to keep, so ``"Smith, Dr."`` is unchanged. Because the pre-comma name is now read positionally, it also picks up the two things the merged read suppressed: the script-order table, the declared name order (``"de Mesnil Juan, Dr."`` keeps family ``de Mesnil`` under a family-first order) and the particle-or-given ambiguity report (``"Van Johnson, Mr."``). The same holds when the comma is followed by titles and postnominals with no name word between them: ``"John Smith, Mr. Jr."`` keeps its split, while ``"Smith Jr., Mr."`` -- one name word -- keeps its family diff --git a/nameparser/config/maiden_markers.py b/nameparser/config/maiden_markers.py index d064333d..2ee3f3e5 100644 --- a/nameparser/config/maiden_markers.py +++ b/nameparser/config/maiden_markers.py @@ -41,19 +41,31 @@ Matching being whole-token, the marker has to BE a token -- which for Japanese means something has to divide it from the name it marks. A -space does, and so does a configured delimiter: extract masks the -whole bracketed region, delimiter characters included, before tokenize -runs, so a bracket bounds a token exactly as a space does and -"山田（旧姓 佐藤）" needs no space in front of 旧姓 at all. The bare -"山田花子 旧姓 佐藤" and -- since #329 -- the bracketed -"山田 花子（旧姓 佐藤）" under Policy(maiden_delimiters=...) alike give -maiden 佐藤. What divides nothing is the fullwidth colon that the form +space does, and so does a delimiter: extract masks the whole bracketed +region, delimiter characters included, before tokenize runs, so a +bracket bounds a token exactly as a space does and "山田（旧姓 佐藤）" +needs no space in front of 旧姓 at all. The bare "山田花子 旧姓 佐藤" +and the bracketed "山田 花子（旧姓 佐藤）" alike give maiden 佐藤, and +since #335 the bracketed form needs no configuration to do it: the +fullwidth pair is a NICKNAME delimiter by default, and rules.md#M3 +reads a clause that opens with a marker word AND carries a word after +it as the maiden name, whichever bucket its pair sits in. The second +word is part of the condition, not a detail of it -- a lone "（旧姓）" +is a word in brackets and stays a nickname. #329, which drops the marker from inside the +clause, is what makes the value 佐藤 rather than "旧姓 佐藤". Declaring +the pair in Policy(maiden_delimiters=...) reaches the same reading by +M1's path. What divides nothing is the fullwidth colon that the form Japanese more often writes puts after the marker: "山田（旧姓：佐藤）" -still yields maiden "旧姓：佐藤" with the marker and its colon attached. -Not because delimited content escapes classification -- classify tags -a marker wherever it is a token -- but because ： is no separator -tokenize knows, so marker and name arrive as ONE token and there is -nothing to drop. The wholly unspaced "山田花子（旧姓佐藤）" reads as one +under Policy(maiden_delimiters=...) still yields maiden "旧姓：佐藤" +with the marker and its colon attached. Not because delimited content +escapes classification -- classify tags a marker wherever it is a +token -- but because ： is no separator tokenize knows, so marker and +name arrive as ONE token and there is nothing to drop. M3 does not +reach that form either, and by its own test rather than by tokenize's: +it splits the clause on WHITESPACE, and "旧姓：佐藤" is one whitespace +word, so there is no marker word for the clause to open with and by +default it stays a nickname. The two tests agree here and are +deliberately not the same test -- see decisions.md#M3. The wholly unspaced "山田花子（旧姓佐藤）" reads as one token for the same reason. Peeling a marker off the head of a token is #317's job. diff --git a/tools/differential/README.md b/tools/differential/README.md index e2b5b749..a2086db3 100644 --- a/tools/differential/README.md +++ b/tools/differential/README.md @@ -412,13 +412,16 @@ The corpora run under the **default policy**, so any behavior gated behind a non-default `Policy` field is invisible here. Default *vocabulary* is a different matter: it is fully in EFFECT, never gated off the way a `Policy` field is, so a change to it can show up -here. That is not the same as coverage -- only 3 of the 17 shipped +here. That is not the same as coverage -- only 4 of the 17 shipped `maiden_markers` and 8 of the 15 `honorific_tails` appear anywhere in -the corpora (measured 2026-08-05), so an entry no corpus name -exercises is as invisible as an opt-in policy. +the corpora as whole tokens (re-measured 2026-08-26; the marker count +was 3 until #414's rules corpus brought in a parenthesized `Nee`), so +an entry no corpus name exercises is as invisible as an opt-in +policy. -Two independent mechanisms put a birth surname in `maiden`, and only -one is opt-in (measured 2026-08-05): +Two independent mechanisms put a birth surname in `maiden`, and what +is opt-in about them is narrower than it looks (rows measured +2026-08-05, re-measured 2026-08-26): | input | default policy | `maiden_delimiters={("(", ")")}` | |---|---|---| @@ -431,9 +434,30 @@ policy says they do. Row 2 carries no brackets, so it isolates the marker: `Lexicon.maiden_markers` ships 17 entries by default, `nee` among them, and the bare form needs no configuration at all. -So what is opt-in is not the marker words — it is only the delimited -path. #329 changed what happens when both are in play (the marker -inside a delimited clause is now dropped from the value), and this -gate cannot see it, because the corpora never configure the delimiter. -Opt-in behavior is covered by `tests/v2/cases.py`, whose rows carry -their own `policy=`. +So what is opt-in is neither the marker words nor the delimited path +as a whole: it is the delimited path for content that does not +announce itself. Since #335 a clause of two words or more led by a +marker reads as the maiden name whichever bucket its pair sits in +(`rules.md#M3`), so `Jane Smith (née Jones)` needs no configuration +either. Row 1 is exactly the shape that still does — a markerless +clause — along with a one-word clause like `Jane Smith (Nee)`, where +nothing in the content says maiden and only a caller who knows the +data can. + +It does NOT put #329 within reach of this gate, and the reason +generalizes past this one change. #329 governs what a delimited maiden +clause CONTAINS -- the marker word is dropped from the value -- while +a ledger rule narrows by which FIELDS move, never by what they hold. +The six names classified under `fix(#335)` move `{nickname, maiden}` +whether the marker is dropped or not, so that rule absorbs a #329 +regression in silence. Measured 2026-08-26 by reverting the drop pass +in `_group.py`: `Jane Smith (née Jones)` reads maiden `née Jones`, and +all three gates still report 0 unexplained. #329 was out of reach +before #335 too, for a different reason -- under the default policy no +corpus name reached the drop at all, so the movement on +`山田 花子（旧姓 佐藤）` between 2.0.0 and 2.1.0 was the East Asian order +flip rather than #329 (`decisions.md#M1`'s 2026-08-05 entry calls that +change gate-visible; the 2026-08-26 entry beside it records the +correction). Value-level coverage for both is `tests/v2/cases.py`, +whose rows assert values, and whose opt-in rows carry their own +`policy=`. From d1f79ac5cadc565a0711aaa1e6bffad5a42b10a3 Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Wed, 26 Aug 2026 01:23:32 -0700 Subject: [PATCH 04/12] fix(maiden_markers): roz leaves the default vocabulary MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit "Rosalind Roz Smith" read maiden 'Smith' and no family name at all, and "Rosalind Roz Jones Smith" read maiden 'Jones Smith'. Roz is the ordinary English diminutive of Rosalind; marker matching is whole-token, case-folded and period-insensitive, so it was the same string as the Czech abbreviation roz., and a maiden marker takes every word after it. The defect is as old as the marker vocabulary and lives on M2's bare path -- it is not #335's, and predates the 2.2 work entirely. It was found reviewing #335, which is a different thing. Grounded in decisions.md#vocabulary-collisions C-i: a word belongs in its set's ambiguous subset iff it is borne as an ordinary name in the position the vocabulary claim acts on, and Roz is borne exactly there, the word after a given name. MAIDEN_MARKERS has no ambiguous subset and no code path consults one, so ship and do-not-ship are the only two expressions of C-i this set has -- which is why the remedy is removal rather than a marking. The full participle rozená stays and is unambiguous. What stops working is the abbreviation, for Czech and Slovak callers: "Anna Nováková roz. Svobodová" now reads middle 'Nováková roz.', family 'Svobodová', which is how 1.4.0 read it. Both moving names return to their 1.4.0 readings. A caller who needs the abbreviation adds it to their own Lexicon. The fix(#274) ledger rule hand-copies this vocabulary, so its roz alternative goes too, and the guard that requires every alternative to match a shipped entry is what caught it. No corpus name contains the string in any case or position: the alternative claimed nothing, the rule's recorded corpus claim is unchanged, and all three gates report the same counts as before the removal. Four hard-recorded counts moved with the entry and were recounted rather than decremented: the set ships 16, fix(#274)'s members reach 3 of them, the corpora hold 4 markers (nee arrived with #414) and the rule covers 3. Co-Authored-By: Claude Opus 5 --- docs/design/decisions.md | 5 ++- docs/design/rules.md | 2 +- docs/release_log.rst | 2 + nameparser/config/maiden_markers.py | 27 +++++++++---- tests/v2/cases.py | 40 ++++++++++++++++++++ tests/v2/test_ledger_guards.py | 10 +++-- tools/differential/README.md | 8 ++-- tools/differential/expected_since_1.4.0.toml | 16 +++++--- 8 files changed, 86 insertions(+), 24 deletions(-) diff --git a/docs/design/decisions.md b/docs/design/decisions.md index 8edfc3a2..d9adffb8 100644 --- a/docs/design/decisions.md +++ b/docs/design/decisions.md @@ -431,7 +431,8 @@ Excluded (MAIDEN_MARKERS, per nameparser/config/maiden_markers.py): #274 flagging "is nee safe as a default (it's also a rare surname)" as open; the question resolved silently with the shipped set. Recorded here because the included risky member deserves its analysis as much as the excluded ones; M1's (Nee) boundary covers only the enclosure path, not this marker path. - "born" — never shipped: a release-log drafting invention, caught by the 2.0 milestone audit and corrected (5ccf9f3). Recorded so nobody "restores" it; if ever proposed for real, Max Born is the counterexample to analyze. -- Scandinavian "f." — collides with the initial F.; only the full participles (født/fødd/född) are safe. Czech masculine "rozený" awaits the same vetting. +- Scandinavian "f." — collides with the initial F.; only the full participles (født/fødd/född) are safe. Czech masculine "rozený" awaits the same vetting, on its own merits rather than following the feminine form in. +- Czech/Slovak "roz." — SHIPPED through 2.1 and removed 2026-08-26 (Derek's call). Roz is an ordinary English diminutive of Rosalind, and matching is whole-token, case-folded and period-insensitive, so "Roz", "roz" and "roz." are one string to this set. The collision is not theoretical and it is not new: on the bare path, which predates the 2.2 work entirely, "Rosalind Roz Smith" read maiden "Smith" and NO family name at all, and "Rosalind Roz Jones Smith" read maiden "Jones Smith", because M2 hands the marker every word after it. Grounded in #vocabulary-collisions C-i — a word belongs in its set's ambiguous subset iff it is borne as an ordinary name IN THE POSITION THE VOCABULARY CLAIM ACTS ON, and under uncertainty default to ambiguous. Roz is borne exactly there: the word after a given name, which is the only position M2's claim acts on. What C-i cannot do for this set is the marking: MAIDEN_MARKERS has no ambiguous subset, and no code path consults one, so the only two expressions of C-i available here are ship and do not ship. That is why the remedy is removal rather than an ambiguous marking, and it is worth knowing before someone proposes marking the next collision instead. The full participle rozená stays and is unambiguous; a Czech caller who needs the abbreviation adds it to their own Lexicon, which is the same answer this section gives for every locale-specific vocabulary. The cost is accepted and stated: "Anna Nováková roz. Svobodová" now reads middle "Nováková roz.", family "Svobodová" — which is how 1.4.0 read it — and "Rosalind Roz Smith" is back to 1.4.0's given/middle/family. Pinned by tests/v2/cases.py::diminutive_that_was_a_marker_keeps_the_family and ::full_participle_marker_still_consumes. ### C1 — the suffix-comma decision @@ -502,7 +503,7 @@ Declined: - 2026-08-26 #335 — accepted consequence: the word taken after the marker is not tested for being a name word, so M3 does not stop at a suffix word where M2's bare take does. "Jane Smith (née V)" reads maiden "V" while the bare "Jane Smith née V" reads suffix "V" and family "née" (both measured). The reason is structural rather than an oversight: M3 hands the WHOLE clause to Role.MAIDEN and the bracket is its right boundary, so the trailing-suffix rule never sees the word as trailing the name at all. Pinned as M3's Accepted line in rules.md, which puts "Jane Smith (née V)" in the differential corpus and classified in all three ledgers. - 2026-08-26 #335 — rules.md#M3 lists implemented: nameparser/_pipeline/_extract.py alone, and that is deliberate rather than an omission. M3 decides a REGION'S ROLE; the marker drop it describes ("the marker itself dropped, as M1 drops it") is M1's mechanism in _group.py, which cites M1 and is governed by it. Adding _group.py to M3's list would mean inventing an M3 citation there for behavior M1 already owns — and test_doc_citations requires implemented: to equal the set of modules that cite the rule, so the two would have to move together. Do not re-open this. - 2026-08-26 #335 — the durable finding, and the one worth reading before touching either side: _maiden_marked asks a WHITESPACE-SPLIT, _normalize'd question of the clause's first word, while tokenize and classify ask a token-level one. _normalize strips a trailing period but not a comma, so 'née,' is not a marker to M3 while the tokenizer splits the comma off and still tags the token vocab:maiden-marker. That divergence is the only thing keeping _group.py:841's "role is not Role.MAIDEN" branch reachable at all: without it, a marker-led NICKNAME clause could no longer exist, since M3 would have converted every one to MAIDEN before group ran. Anyone "unifying" the two tests would orphan that filter silently. The pin is tests/v2/cases.py::marker_glued_to_punctuation_keeps_the_clause_a_nickname, and it needs BOTH of its clauses — the drop pass is gated on the name holding a maiden region at all, so a marker-glued nickname clause standing alone leaves the branch unexercised (measured by mutation, 2026-08-26). The row that used to pin the filter, maiden_marker_delimited_beside_a_nickname_clause, lost the job to M3: both its clauses are marker-led, so both are maiden now and there is no contrast left in it. -- 2026-08-26 #335 — RECORDED, NOT RESOLVED: M3 puts maiden_markers to work in a position they never acted on under the default policy — the first word of any nickname-delimited clause — and decisions.md#vocabulary-collisions C-i (a word belongs in the ambiguous subset iff it is borne as an ordinary name IN THE POSITION THE VOCABULARY CLAIM ACTS ON) was never applied to that new position. maiden_markers has no ambiguous subset at all. The shipped entry that most obviously bears the test is roz: Roz is an English diminutive, and a diminutive is exactly what a nickname clause holds. Measured 2026-08-26: 'Jane "Roz Smith" Jones' read nickname "Roz Smith" through 2.1 and now reads maiden "Smith" with no nickname, and 'Rosalind (Roz Smith) Jones' likewise. The nickname is lost and a maiden name invented, by default. M3 itself has no switch, so the only lever is the vocabulary: Parser(lexicon=dataclasses.replace(lex, maiden_markers=...)) without roz restores nickname "Roz Smith" (measured), which is a heavy instrument for one word and not something a caller would think to reach for. Surfaced by review after the mechanism was approved; no judgement recorded here yet, and no issue filed. +- 2026-08-26 #335 — M3 makes the marker vocabulary act in a THIRD position. M2's claim acts on the word after a name word; M1's on the content of a configured pair; M3's now on the first word of any bracketed clause, under the default policy. #vocabulary-collisions C-i is keyed on the position a claim acts on, so extending where a rule acts re-asks C-i of the whole set it reads, and nothing in the process prompts that — the rule was designed, reviewed and approved without the set being re-examined. Review afterwards found one entry that fails C-i, the Czech abbreviation roz, and its disposition is the Excluded block in #M1: it was removed. Two things worth separating, since the finding surfaced during this work and is easy to file under it — the defect was M2's and predates #335 (bare "Rosalind Roz Smith" lost its family name on master), and M3's own boundary already excluded the common one-word "(Roz)" spelling. What belongs to M3 is only the widening, and the lesson is the general one: re-ask C-i of a vocabulary set whenever a rule extends where its claim acts. - 2026-08-26 #335 — what configuring maiden_delimiters still buys, since M3 removes the commonest reason to reach for it: markerless clauses, and one-word clauses. "Cherice J. (Johnson) Williams" is a real US convention and a corpus name, and nothing in that clause says maiden, so it stays a nickname by default — only a caller who knows their data can say otherwise. The knob is a declaration about the DATA; M3 is a reading of what a clause declares about itself. ### O1 — East Slavic rotation diff --git a/docs/design/rules.md b/docs/design/rules.md index 3f5d29de..5a2d0854 100644 --- a/docs/design/rules.md +++ b/docs/design/rules.md @@ -531,7 +531,7 @@ N3. Rationale: a person set down as a nickname plus one name word is ## Maiden names (M) -Background: a maiden name is written beside the current name, set off by a marker word or by enclosure. Markers are attested across French née/né, German geb./geborene, Dutch geboren, Czech/Slovak roz./rozená, Scandinavian født/fødd/född, Russian урожд. (both ё and е spellings), and Japanese 旧姓 — both grammatical genders where attested. Japanese more often writes the marker with a fullwidth colon (旧姓：佐藤), which is no separator, so marker and name arrive as a single word. Which enclosures mean "maiden" rather than "nickname" is a caller convention, so the maiden reading of a delimiter pair is opt-in — except where the clause announces itself. A clause of two words or more led by a marker word has said which convention it means, and reads as the maiden name inside a nickname pair as well (M3); a lone marker word has not, and neither has one the colon spelling above glues to the name. +Background: a maiden name is written beside the current name, set off by a marker word or by enclosure. Markers are attested across French née/né, German geb./geborene, Dutch geboren, Czech/Slovak rozená (the abbreviation roz. is deliberately not shipped -- it collides with the English diminutive Roz), Scandinavian født/fødd/född, Russian урожд. (both ё and е spellings), and Japanese 旧姓 — both grammatical genders where attested. Japanese more often writes the marker with a fullwidth colon (旧姓：佐藤), which is no separator, so marker and name arrive as a single word. Which enclosures mean "maiden" rather than "nickname" is a caller convention, so the maiden reading of a delimiter pair is opt-in — except where the clause announces itself. A clause of two words or more led by a marker word has said which convention it means, and reads as the maiden name inside a nickname pair as well (M3); a lone marker word has not, and neither has one the colon spelling above glues to the name. M1. Rationale: an enclosure the caller has declared to mean maiden holds the former family name; a recognized marker word inside it diff --git a/docs/release_log.rst b/docs/release_log.rst index ae836087..626d8f5c 100644 --- a/docs/release_log.rst +++ b/docs/release_log.rst @@ -45,6 +45,8 @@ Release Log - Fix a bound given name joining past a credential that the suffix rule then takes, leaving no family: ``"abdul Smith Jr Ma"`` read given ``abdul Smith`` with no family, where ``"John Smith Jr Ma"`` reads family ``Smith`` with suffix ``Jr, Ma``. The join's reserve had re-derived the suffix rule's trailing walk by hand and left out one of its two forks -- a bare acronym such as ``Ma`` is a credential only with words to spare -- so it counted the acronym as a name word, joined, and the suffix rule then peeled it. The two stages now share one walk: the reserve tries the join, runs the suffix rule's own peel over the pieces as the join would leave them, and joins only when two name words became one and nothing else changed. ``"abdul Smith Jr Ma"`` reads family ``Smith``, suffix ``Jr, Ma``; ``"abdul Smith Ma"`` reads family ``Smith``, suffix ``Ma`` rather than joining the pair and keeping the credential as the family -- both as 1.4.0 read them and as ``John`` reads in their place; ``"abdul Smith Berg Ma"`` keeps its join, and a title word standing after the bound word joins again as it did in 1.4.0 (``"Berg, abdul Sir"`` reads given ``abdul Sir``) without the pair ever being read as a title (``"abdul Sheikh and Ahmad Bakar"`` reads given ``abdul Sheikh and Ahmad``). No name of the differential corpora has the shapes; the ``Ma`` rules examples carry the fix, byte-identical to 1.4.0 and moving against 2.0.0 and 2.1.0, and the title-word shapes are pinned in tests (closes #425) + - Remove the Czech/Slovak abbreviation ``roz.`` from the default maiden markers, where it had been eating English given names. Marker matching is whole-token, case-folded and period-insensitive, so ``Roz`` -- the ordinary diminutive of Rosalind -- was the same string as the marker, and a maiden marker takes every word after it: ``"Rosalind Roz Smith"`` read maiden ``Smith`` with **no family name at all**, and ``"Rosalind Roz Jones Smith"`` read maiden ``Jones Smith``. Both now read the way 1.4.0 read them, given ``Rosalind`` with ``Roz`` a middle name and ``Smith`` the family. The defect is as old as the marker vocabulary and has nothing to do with the bracketed-clause change above; it was found while reviewing that change. What stops working is the abbreviation itself, for Czech and Slovak callers: ``"Anna Nováková roz. Svobodová"`` now reads middle ``Nováková roz.``, family ``Svobodová`` rather than family ``Nováková``, maiden ``Svobodová`` -- which is also how 1.4.0 read it, so this is a return to the v1 reading rather than a new one. The full participle is untouched and needs no configuration: ``"Anna Nováková rozená Svobodová"`` still reads family ``Nováková``, maiden ``Svobodová``, and it is only the abbreviation that was ambiguous. A caller who wants the abbreviation back adds it to their own lexicon -- ``Parser(lexicon=dataclasses.replace(lex, maiden_markers=lex.maiden_markers | {"roz"}))`` -- which restores the old reading exactly. The default set is now 16 entries. No differential corpus name contains the string in any case or position, so nothing moves against any baseline (closes the collision found in #335's review) + - Fix a bracketed maiden clause reading as a nickname because its brackets were not declared. ``"Jane Smith nee Jones"`` gave maiden ``Jones``, while ``"Jane Smith (née Jones)"`` gave nickname ``née Jones`` -- the parenthesis pair is a nickname delimiter by default and nothing looked inside it, so the marker word the writer put there was read as part of an alias. A bracketed clause whose content opens with a recognized marker word and carries a word after it now reads as the maiden name whichever bucket its pair sits in: ``"Jane Smith (née Jones)"`` reads family ``Smith``, maiden ``Jones``, and so does the Japanese ``"山田 花子（旧姓 佐藤）"`` (family 山田, given 花子, maiden 佐藤), which needed ``Policy(maiden_delimiters=...)`` through 2.1. The clause is lifted out as a region rather than having its brackets erased, which is what keeps an INTERIOR clause from eating the name behind it: ``"Jane (née Jones) Smith"`` reads given ``Jane``, family ``Smith``, maiden ``Jones``, where erasing the brackets would leave the bare ``"Jane née Jones Smith"``, whose marker takes every word to the end of the name -- maiden ``Jones Smith`` and no family at all. The closing bracket is a boundary the writer typed, and it is now where the maiden name stops. Two clauses beside each other no longer merge: ``"Jane "Janey" Smith (née Jones)"`` read nickname ``Janey née Jones`` through 2.1 and now reads nickname ``Janey``, maiden ``Jones``, each clause keeping its own role, and this reaches ``HumanName``. What does NOT move is a clause that says nothing about maiden names. A lone marker in brackets is just a word in brackets -- ``"Jane Smith (née)"`` stays nickname ``née``, as 1.4.0 read it -- because a marker with no name after it marks nothing, and because a one-word clause may be the surname it looks like (``Nee`` is Irish Ní/Nee and a Chinese romanization). A markerless clause stays a nickname too: ``"Cherice J. (Johnson) Williams"`` keeps nickname ``Johnson``, which is what ``Policy(maiden_delimiters=...)`` remains for -- the parenthesized birth surname is a real convention, and only a caller who knows the data can say that is what these brackets mean. One accepted consequence: the word after the marker is not tested for being a name word, so a bracketed clause does not stop at a credential where the bare form does -- ``"Jane Smith (née V)"`` reads maiden ``V`` while ``"Jane Smith née V"`` reads suffix ``V``. The clause goes to ``maiden`` whole and the bracket is its boundary, so the trailing-suffix rule never sees the word as trailing the name. Six of the 1,078 differential corpus names move against the 2.1.0 and 2.0.0 baselines and five against 1.4.0: the Japanese name is the sixth, and against 1.4.0 it is not a new diff at all, since 1.4 read the fullwidth brackets as name text and had no nickname there to change hands. Those six are also every name in the corpora that HAS the shape (measured), so the count says how few bracketed marker clauses the corpora hold rather than how narrow the change is; four of the six are hand-written rule and issue examples, and the coverage that bounds the behavior is ``tests/v2/cases.py`` (closes #335) - Fix a particle chain and a maiden name taking a trailing generational numeral as a name word: ``"John van der Berg V"`` read family ``van der Berg V`` and ``"John née Jones Smith V"`` read maiden ``Jones Smith V``, where ``"John Smith V"`` reads suffix ``V``. Both stops asked "is this a suffix?" with the test that vetoes a bare ``V`` as an initial -- the same question the bound given-name join asked until #401 -- and now ask the suffix rule's own walk, over the name as they would leave it: family ``van der Berg``, maiden ``Jones Smith``, suffix ``V``, for ``I`` and ``X`` alike; ``"John van der J. V"`` and ``"J. née Jones Smith V"`` keep their readings, the word before the numeral being an initial as written or as left. The chain stops before a bare credential with words to spare too, as ``"John Smith Ma"`` does, where the suffix rule will still find them once the chain has run: ``"John van der Berg Ma"`` reads suffix ``Ma``, which is how 1.4.0 read it, while ``"Freiherr von Berg MA"`` keeps family ``von Berg MA``; a maiden name keeps such a credential, since the words the marker takes are the very words that made it one. A particle that is also suffix vocabulary now ends the chain where it opens the trailing run: ``"John Smith Mc V"`` reads suffix ``Mc, V``. The chain also no longer swallows the given name behind an unlisted abbreviation: ``"Xyz. van Johnson"`` and ``"Esq. van Gogh"`` read given ``van``, as ``"Dr. van Johnson"`` has since #367, the chain and the bound given-name join now asking the same test that reads the abbreviation as a title (closes #424) diff --git a/nameparser/config/maiden_markers.py b/nameparser/config/maiden_markers.py index 2ee3f3e5..5e897992 100644 --- a/nameparser/config/maiden_markers.py +++ b/nameparser/config/maiden_markers.py @@ -7,7 +7,6 @@ 'geb', 'geborene', 'geboren', - 'roz', 'rozená', 'født', 'fødd', @@ -22,13 +21,15 @@ """ Marker words that introduce a birth surname, e.g. "Jane Smith née Jones" (#274). French née/né/nee, German geb./geborene, Dutch geboren, -Czech/Slovak roz./rozená, Danish/Norwegian født (Nynorsk fødd), Swedish +Czech/Slovak rozená, Danish/Norwegian født (Nynorsk fødd), Swedish född, Russian урожд./урождённая/урождённый (both ё and е spellings — case normalization does not fold them, and running text routinely writes е). Both grammatical genders are listed where #274 or review attested them (née/né, урождённая/урождённый); Czech masculine rozený -awaits the same vetting. Entries are stored normalized: lowercase, no -periods. +awaits the same vetting, and would be vetted on its own merits rather +than following the feminine form in — the abbreviation roz. shared by +both genders is deliberately absent, below. Entries are stored +normalized: lowercase, no periods. Japanese 旧姓 is here rather than in locales.JA, on the rule that admitted the Cyrillic entries: a native-script marker cannot collide @@ -65,17 +66,27 @@ it splits the clause on WHITESPACE, and "旧姓：佐藤" is one whitespace word, so there is no marker word for the clause to open with and by default it stays a nickname. The two tests agree here and are -deliberately not the same test -- see decisions.md#M3. The wholly unspaced "山田花子（旧姓佐藤）" reads as one -token for the same reason. Peeling a marker off the head of a token is +deliberately not the same test -- see decisions.md#M3. The wholly +unspaced "山田花子（旧姓佐藤）" reads as one token for the same reason. Peeling a marker off the head of a token is #317's job. Consumed by the 2.0 parser's default lexicon. The 1.x parser does not read this module. Deliberately absent: Polish "z domu" (a two-token marker; pending the -2.0 pipeline's multi-token matching decision) and the Scandinavian +2.0 pipeline's multi-token matching decision), the Scandinavian abbreviation "f." (collides with the initial "F." — only the full -participles are safe). +participles are safe), and the Czech/Slovak abbreviation "roz." +(shipped through 2.1 and removed in 2.2). Roz is an ordinary English +diminutive of Rosalind, and matching is whole-token, case-folded and +period-insensitive, so "Roz", "roz" and "roz." are one string to this +set: with it shipped, "Rosalind Roz Smith" read maiden "Smith" and NO +family name at all. The collision is in the position the claim acts +on, which is the test decisions.md#vocabulary-collisions states, and +this set has no ambiguous subset to express a partial answer in — see +decisions.md#M1's Excluded block. The full participle rozená stays, +being unambiguous; a caller who needs the abbreviation adds it to +their own Lexicon. """ diff --git a/tests/v2/cases.py b/tests/v2/cases.py index 67bbfd21..9adc8d82 100644 --- a/tests/v2/cases.py +++ b/tests/v2/cases.py @@ -784,6 +784,46 @@ def __post_init__(self) -> None: {"given": "Jane", "family": "Smith", "maiden": "Jones"}, classification="fix(#274)", notes="v1 mangles to middle='Smith née'"), + Case("diminutive_that_was_a_marker_keeps_the_family", + "Rosalind Roz Smith", + {"given": "Rosalind", "middle": "Roz", "family": "Smith"}, + notes="'roz', the Czech/Slovak abbreviation, shipped in " + "MAIDEN_MARKERS through 2.1 and collided with the " + "English diminutive of Rosalind -- matching is " + "whole-token, case-folded and period-insensitive, so " + "Roz, roz and roz. are one string. This name read " + "maiden 'Smith' with NO family name at all (measured on " + "the pre-removal tree), because M2 hands the marker " + "every word after it. The entry is gone in 2.2, which " + "is what this row pins. Nothing to do with the " + "delimited path: the defect is M2's, it predates #335, " + "and the one-word '(Roz)' spelling was never affected " + "since M3 declines a lone marker. Parity, and it is " + "RESTORED parity rather than untouched -- 1.4.0 has no " + "maiden support and read first Rosalind / middle Roz / " + "last Smith (2026-08-26), which is where the removal " + "puts this name back"), + Case("full_participle_marker_still_consumes", + "Anna Nováková rozená Svobodová", + {"given": "Anna", "family": "Nováková", "maiden": "Svobodová"}, + classification="fix(#274)", + notes="the other half of the roz removal, and the reason it " + "was a removal and not a retreat from Czech: the full " + "participle rozená stays, being a word no one is " + "called. Pinned because deleting a vocabulary entry " + "invites deleting its neighbours, and because nothing " + "else in the suite reaches this entry: removing " + "rozená from MAIDEN_MARKERS fails exactly this row's " + "two tests, one per runner, and nothing else (measured " + "2026-08-26) -- the position " + "maiden_marker_delimited_unaccented holds for 'nee'. " + "The cost the removal accepts is the " + "abbreviation: 'Anna Nováková roz. Svobodová' now reads " + "middle 'Nováková roz.', family 'Svobodová' (measured), " + "which is exactly how 1.4.0 read it. 1.4.0 read this " + "row middle 'Nováková rozená' / last Svobodová " + "(2026-08-26) -- the marker inside the name, the " + "ordinary v1 reading of every marker"), Case("maiden_marker_after_particle_chain", "Ursula von der Leyen geb. Albrecht", {"given": "Ursula", "family": "von der Leyen", diff --git a/tests/v2/test_ledger_guards.py b/tests/v2/test_ledger_guards.py index 48d2162d..457162bd 100644 --- a/tests/v2/test_ledger_guards.py +++ b/tests/v2/test_ledger_guards.py @@ -845,13 +845,15 @@ class _LatinCopy(NamedTuple): #: removal: drop an entry a member covers and the snapshot shrinks. #: #: Three nearby counts differ and are easy to conflate, all for -#: fix(#274) specifically: MAIDEN_MARKERS ships 17 entries; that rule's -#: members reach 4 of them; the corpora contain 3 markers in total -#: (geb, née, 旧姓), only 2 of which it covers. +#: fix(#274) specifically, and all four numbers moved in 2.2 -- recount +#: rather than adjust them: MAIDEN_MARKERS ships 16 entries (roz left +#: the vocabulary); that rule's members reach 3 of them; the corpora +#: contain 4 markers in total (geb, nee, née, 旧姓 -- nee arrived with +#: #414's rules corpus), 3 of which it covers. _LATIN_ALTERNATION_SOURCES: dict[str, _LatinCopy] = { "fix(#274)": _LatinCopy( vocabulary=MAIDEN_MARKERS, - covers=frozenset({"geb", "nee", "née", "roz"})), + covers=frozenset({"geb", "nee", "née"})), "ambiguous-surname-acronym": _LatinCopy( vocabulary=SUFFIX_ACRONYMS_AMBIGUOUS, covers=frozenset({"do", "ma"})), diff --git a/tools/differential/README.md b/tools/differential/README.md index a2086db3..113af81b 100644 --- a/tools/differential/README.md +++ b/tools/differential/README.md @@ -412,10 +412,12 @@ The corpora run under the **default policy**, so any behavior gated behind a non-default `Policy` field is invisible here. Default *vocabulary* is a different matter: it is fully in EFFECT, never gated off the way a `Policy` field is, so a change to it can show up -here. That is not the same as coverage -- only 4 of the 17 shipped +here. That is not the same as coverage -- only 4 of the 16 shipped `maiden_markers` and 8 of the 15 `honorific_tails` appear anywhere in the corpora as whole tokens (re-measured 2026-08-26; the marker count -was 3 until #414's rules corpus brought in a parenthesized `Nee`), so +was 3 until #414's rules corpus brought in a parenthesized `Nee`, and +the denominator was 17 until `roz` left the vocabulary in 2.2 -- it +appeared in no corpus name, so only the denominator moved), so an entry no corpus name exercises is as invisible as an opt-in policy. @@ -431,7 +433,7 @@ is opt-in about them is narrower than it looks (rows measured Row 1 carries no marker word, so it isolates the delimiter: the brackets alone route their content to `maiden`, and only once the policy says they do. Row 2 carries no brackets, so it isolates the -marker: `Lexicon.maiden_markers` ships 17 entries by default, `nee` +marker: `Lexicon.maiden_markers` ships 16 entries by default, `nee` among them, and the bare form needs no configuration at all. So what is opt-in is neither the marker words nor the delimited path diff --git a/tools/differential/expected_since_1.4.0.toml b/tools/differential/expected_since_1.4.0.toml index 474e0ced..6677c762 100644 --- a/tools/differential/expected_since_1.4.0.toml +++ b/tools/differential/expected_since_1.4.0.toml @@ -67,11 +67,15 @@ issue = "fix(#274) maiden markers consumed" # tests/v2/test_ledger_guards.py: every alternative must match at least # one entry the config ships, and none may match ordinary name text. # -# Its members reach 4 of the 17 entries -- geb, nee, née, roz -- which -# is not the same set as the markers the corpora contain (geb, née and -# 旧姓; only the first two are covered here). Growing it toward the -# other 12 buys nothing: none appears as a token in any corpus, so -# there is no diff for them to classify. +# Its members reach 3 of the 16 entries -- geb, nee, née -- which is +# not the same set as the markers the corpora contain (geb, nee, née +# and 旧姓; the first three are covered here). Growing it toward the +# other 13 buys nothing: none appears as a token in any corpus, so +# there is no diff for them to classify. The roz alternative was +# dropped in 2.2 with the vocabulary entry it copied (see +# maiden_markers.py's Deliberately absent note): no corpus name +# contains the string in any case or position, so it classified +# nothing and its removal moved no name. # # 旧姓 is a different case, and it has been got wrong three times, so # state it precisely. Widening this rule to cover it changes no corpus @@ -126,7 +130,7 @@ issue = "fix(#274) maiden markers consumed" # would need the regex to ask a vocabulary question -- suffix or # connective membership -- a second alternation this rule cannot # spend. -name_regex = "(?i)(? Date: Wed, 26 Aug 2026 01:38:08 -0700 Subject: [PATCH 05/12] docs(rules,decisions,extract,README): pin S1 over M3, and say what the role test does MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Four things the mutation pass turned up, none of them behavior. S1's precedence over M3 was stated in rules.md and executed by nothing. It is now an M3 example line, "Jane Smith (née Jr.)", which reads family 'née' and suffix 'Jr.' identically on 1.4.0, 2.1.0 and here, so the corpus row it adds diffs against nothing. Be exact about which mutation that pins, because the obvious one is inert: hoisting the role swap above the _suffix_shaped call changes no reading at all, since the S1 branch never reads `role` -- it masks the two delimiter spans and hands the content to the token stream, and the role it would have carried is never consulted. What inverts the precedence is giving M3 the clause first, an early append of a Role.MAIDEN region ahead of the S1 test: that reads the example as family 'Smith', maiden 'Jr.', and with the line in place it fails test_example[M3-2]. The Accepted block gained a sentence, because the two suffix-word examples now sitting in one rule look like a contradiction until you know S1 asks about the WHOLE clause: "née Jr." is suffix-shaped by its trailing period, "née V" is not, and only the second reaches M3. The `role is Role.NICKNAME` test cannot change the outcome -- `order` holds exactly two roles, so a role that is not NICKNAME is already MAIDEN and the assignment is a no-op either way -- and no test can catch its removal. Kept, for the day `order` gains a third bucket, and now commented rather than left to look load-bearing. decisions.md#M3 said loosening the second-word test causes "three failures and no more". True of pytest and the weaker half: it also turns the 2.1.0 gate red with three unexplained names, and one of them splits its two bracket pairs across the maiden and nickname fields, contradicting M1 outright. That is the signal a future reader should check first. Three bare _group.py:841 citations become _group.group plus the quoted predicate, per the dotted-symbol convention the pipeline comments already use: nothing recomputes a line number, and one edit above it invalidated all three at once. The README's marker count gains the recompute command decisions.md ships beside its own counts, and says which counting convention it uses -- whole tokens, where the neighbouring _carries guard deliberately asks a wider question and gets 5 rather than 4. Co-Authored-By: Claude Opus 5 --- docs/design/decisions.md | 4 +-- docs/design/rules.md | 6 ++++- docs/release_log.rst | 2 +- nameparser/_pipeline/_extract.py | 10 ++++++++ nameparser/config/maiden_markers.py | 4 +-- tests/v2/cases.py | 8 +++--- tests/v2/test_ledger_guards.py | 4 +-- tools/differential/README.md | 35 ++++++++++++++++++++++----- tools/differential/corpus_rules.jsonl | 1 + 9 files changed, 57 insertions(+), 17 deletions(-) diff --git a/docs/design/decisions.md b/docs/design/decisions.md index d9adffb8..2ad4f29b 100644 --- a/docs/design/decisions.md +++ b/docs/design/decisions.md @@ -499,10 +499,10 @@ Declined: - 2026-08-26 #335 (M3, PR on fix/335-maiden-marked-clause) — a bracketed clause whose content opens with a recognized marker word and carries a word after it reads as the maiden name, whichever bucket the enclosing pair sits in. The parse already had the information: "Jane Smith née Jones" gave maiden "Jones" while "Jane Smith (née Jones)" gave nickname "née Jones", because ( ) is a nickname pair by default and nothing looked inside it. The clause says "maiden" out loud, so the caller should not have to say it in Policy. - 2026-08-26 #335 — the MECHANISM, and the issue proposed the other one. Masking the delimiters and letting M2's bare-marker rule consume the content would avoid a second implementation, and it was rejected on measurement: the two readings agree on a trailing clause and diverge on an interior one. "Jane (née Jones) Smith" reads given "Jane", family "Smith", maiden "Jones" under extraction; masking turns it into the bare "Jane née Jones Smith", which reads maiden "Jones Smith" and family "" (measured 2026-08-26), because M2's take runs to the end of the name. The closing delimiter is a boundary the writer typed and masking throws it away. Extraction was also the SMALLER change — extract already emits Role.MAIDEN regions and group already drops the marker inside one (#329), so this is a role swap at the existing emit site plus one predicate, in the branch S1 has already declined, which keeps S1's precedence by construction. -- 2026-08-26 #335 — the condition is a word AFTER the marker, not the marker alone. The reason is M1's: a one-word clause keeps its word because Nee is an attested surname (Irish Ní/Nee, and a Chinese romanization), and a marker with nothing after it marks nothing. Be precise about what pins it, because the obvious answer is wrong: M1's own boundary line carries the maiden-parens annotation, so it asserts the CONFIGURED reading and passes either way — built and measured 2026-08-26, a len(words) > 0 predicate still gives it maiden "Nee". What the loosened predicate actually breaks is M3's own boundary example ("Jane Smith (née)" → nickname "née") and tests/v2/cases.py::maiden_marked_clause_one_word_stays_a_nickname, three failures and no more. Anyone re-litigating the second-word test should check those, not M1's line. So the real division of labour is NOT markerless versus marker-led, which is how it was first written into rules.md and had to be corrected at review: it is one word versus a marker plus a word. "Jane Smith (Nee)" is marker-led and still reads nickname "Nee" by default, maiden "Nee" only with the pair configured (both measured 2026-08-26). +- 2026-08-26 #335 — the condition is a word AFTER the marker, not the marker alone. The reason is M1's: a one-word clause keeps its word because Nee is an attested surname (Irish Ní/Nee, and a Chinese romanization), and a marker with nothing after it marks nothing. Be precise about what pins it, because the obvious answer is wrong: M1's own boundary line carries the maiden-parens annotation, so it asserts the CONFIGURED reading and passes either way — built and measured 2026-08-26, a len(words) > 0 predicate still gives it maiden "Nee". What the loosened predicate actually breaks, measured by mutation 2026-08-26, is M3's own boundary example ("Jane Smith (née)" → nickname "née") and tests/v2/cases.py::maiden_marked_clause_one_word_stays_a_nickname in both runners — three pytest failures — AND the 2.1.0 differential gate, which goes red with three unexplained names: "Jane Smith (Nee)", "Jane Smith (Nee) (Jones)" and "Jane Smith (née)". The gate is the stronger signal and the one to look at first. The middle name is why: loosened, it reads maiden "Nee" and nickname "Jones", splitting one bracket pair into the maiden field and the next into the nickname field, which is the direct contradiction of M1's "two enclosures read as one maiden name". Anyone re-litigating the second-word test should check those, not M1's line. So the real division of labour is NOT markerless versus marker-led, which is how it was first written into rules.md and had to be corrected at review: it is one word versus a marker plus a word. "Jane Smith (Nee)" is marker-led and still reads nickname "Nee" by default, maiden "Nee" only with the pair configured (both measured 2026-08-26). - 2026-08-26 #335 — accepted consequence: the word taken after the marker is not tested for being a name word, so M3 does not stop at a suffix word where M2's bare take does. "Jane Smith (née V)" reads maiden "V" while the bare "Jane Smith née V" reads suffix "V" and family "née" (both measured). The reason is structural rather than an oversight: M3 hands the WHOLE clause to Role.MAIDEN and the bracket is its right boundary, so the trailing-suffix rule never sees the word as trailing the name at all. Pinned as M3's Accepted line in rules.md, which puts "Jane Smith (née V)" in the differential corpus and classified in all three ledgers. - 2026-08-26 #335 — rules.md#M3 lists implemented: nameparser/_pipeline/_extract.py alone, and that is deliberate rather than an omission. M3 decides a REGION'S ROLE; the marker drop it describes ("the marker itself dropped, as M1 drops it") is M1's mechanism in _group.py, which cites M1 and is governed by it. Adding _group.py to M3's list would mean inventing an M3 citation there for behavior M1 already owns — and test_doc_citations requires implemented: to equal the set of modules that cite the rule, so the two would have to move together. Do not re-open this. -- 2026-08-26 #335 — the durable finding, and the one worth reading before touching either side: _maiden_marked asks a WHITESPACE-SPLIT, _normalize'd question of the clause's first word, while tokenize and classify ask a token-level one. _normalize strips a trailing period but not a comma, so 'née,' is not a marker to M3 while the tokenizer splits the comma off and still tags the token vocab:maiden-marker. That divergence is the only thing keeping _group.py:841's "role is not Role.MAIDEN" branch reachable at all: without it, a marker-led NICKNAME clause could no longer exist, since M3 would have converted every one to MAIDEN before group ran. Anyone "unifying" the two tests would orphan that filter silently. The pin is tests/v2/cases.py::marker_glued_to_punctuation_keeps_the_clause_a_nickname, and it needs BOTH of its clauses — the drop pass is gated on the name holding a maiden region at all, so a marker-glued nickname clause standing alone leaves the branch unexercised (measured by mutation, 2026-08-26). The row that used to pin the filter, maiden_marker_delimited_beside_a_nickname_clause, lost the job to M3: both its clauses are marker-led, so both are maiden now and there is no contrast left in it. +- 2026-08-26 #335 — the durable finding, and the one worth reading before touching either side: _maiden_marked asks a WHITESPACE-SPLIT, _normalize'd question of the clause's first word, while tokenize and classify ask a token-level one. _normalize strips a trailing period but not a comma, so 'née,' is not a marker to M3 while the tokenizer splits the comma off and still tags the token vocab:maiden-marker. That divergence is the only thing keeping the "role is not Role.MAIDEN" branch of _group.group's clause-scoped drop pass reachable at all: without it, a marker-led NICKNAME clause could no longer exist, since M3 would have converted every one to MAIDEN before group ran. Anyone "unifying" the two tests would orphan that filter silently. The pin is tests/v2/cases.py::marker_glued_to_punctuation_keeps_the_clause_a_nickname, and it needs BOTH of its clauses — the drop pass is gated on the name holding a maiden region at all, so a marker-glued nickname clause standing alone leaves the branch unexercised (measured by mutation, 2026-08-26). The row that used to pin the filter, maiden_marker_delimited_beside_a_nickname_clause, lost the job to M3: both its clauses are marker-led, so both are maiden now and there is no contrast left in it. - 2026-08-26 #335 — M3 makes the marker vocabulary act in a THIRD position. M2's claim acts on the word after a name word; M1's on the content of a configured pair; M3's now on the first word of any bracketed clause, under the default policy. #vocabulary-collisions C-i is keyed on the position a claim acts on, so extending where a rule acts re-asks C-i of the whole set it reads, and nothing in the process prompts that — the rule was designed, reviewed and approved without the set being re-examined. Review afterwards found one entry that fails C-i, the Czech abbreviation roz, and its disposition is the Excluded block in #M1: it was removed. Two things worth separating, since the finding surfaced during this work and is easy to file under it — the defect was M2's and predates #335 (bare "Rosalind Roz Smith" lost its family name on master), and M3's own boundary already excluded the common one-word "(Roz)" spelling. What belongs to M3 is only the widening, and the lesson is the general one: re-ask C-i of a vocabulary set whenever a rule extends where its claim acts. - 2026-08-26 #335 — what configuring maiden_delimiters still buys, since M3 removes the commonest reason to reach for it: markerless clauses, and one-word clauses. "Cherice J. (Johnson) Williams" is a real US convention and a corpus name, and nothing in that clause says maiden, so it stays a nickname by default — only a caller who knows their data can say otherwise. The knob is a declaration about the DATA; M3 is a reading of what a clause declares about itself. diff --git a/docs/design/rules.md b/docs/design/rules.md index 5a2d0854..718e81bc 100644 --- a/docs/design/rules.md +++ b/docs/design/rules.md @@ -622,11 +622,15 @@ M3. Rationale: an enclosure says nothing about whether it means maiden and join into one maiden name, leaving no nickname. "Jane Smith (née Jones)" → maiden="Jones" "Jane (née Jones) Smith" → family="Smith" + "Jane Smith (née Jr.)" → suffix="Jr." "Jane Smith (née)" → nickname="née" · boundary Accepted: the word taken after the marker is not tested for being a name word, so unlike M2's bare take this one does not stop at a suffix word — the same two words read one way - bracketed and another way bare. + bracketed and another way bare. This does not contradict the + S1 example above: S1 asks whether the WHOLE clause is + suffix-shaped, which the trailing period makes true of the one + and false of the other, so only the V clause reaches this rule. "Jane Smith (née V)" → maiden="V" history: decisions.md#M3 · interacts: M1, M2, S1, N1 · implemented: nameparser/_pipeline/_extract.py diff --git a/docs/release_log.rst b/docs/release_log.rst index 626d8f5c..b4ce27a7 100644 --- a/docs/release_log.rst +++ b/docs/release_log.rst @@ -47,7 +47,7 @@ Release Log - Remove the Czech/Slovak abbreviation ``roz.`` from the default maiden markers, where it had been eating English given names. Marker matching is whole-token, case-folded and period-insensitive, so ``Roz`` -- the ordinary diminutive of Rosalind -- was the same string as the marker, and a maiden marker takes every word after it: ``"Rosalind Roz Smith"`` read maiden ``Smith`` with **no family name at all**, and ``"Rosalind Roz Jones Smith"`` read maiden ``Jones Smith``. Both now read the way 1.4.0 read them, given ``Rosalind`` with ``Roz`` a middle name and ``Smith`` the family. The defect is as old as the marker vocabulary and has nothing to do with the bracketed-clause change above; it was found while reviewing that change. What stops working is the abbreviation itself, for Czech and Slovak callers: ``"Anna Nováková roz. Svobodová"`` now reads middle ``Nováková roz.``, family ``Svobodová`` rather than family ``Nováková``, maiden ``Svobodová`` -- which is also how 1.4.0 read it, so this is a return to the v1 reading rather than a new one. The full participle is untouched and needs no configuration: ``"Anna Nováková rozená Svobodová"`` still reads family ``Nováková``, maiden ``Svobodová``, and it is only the abbreviation that was ambiguous. A caller who wants the abbreviation back adds it to their own lexicon -- ``Parser(lexicon=dataclasses.replace(lex, maiden_markers=lex.maiden_markers | {"roz"}))`` -- which restores the old reading exactly. The default set is now 16 entries. No differential corpus name contains the string in any case or position, so nothing moves against any baseline (closes the collision found in #335's review) - - Fix a bracketed maiden clause reading as a nickname because its brackets were not declared. ``"Jane Smith nee Jones"`` gave maiden ``Jones``, while ``"Jane Smith (née Jones)"`` gave nickname ``née Jones`` -- the parenthesis pair is a nickname delimiter by default and nothing looked inside it, so the marker word the writer put there was read as part of an alias. A bracketed clause whose content opens with a recognized marker word and carries a word after it now reads as the maiden name whichever bucket its pair sits in: ``"Jane Smith (née Jones)"`` reads family ``Smith``, maiden ``Jones``, and so does the Japanese ``"山田 花子（旧姓 佐藤）"`` (family 山田, given 花子, maiden 佐藤), which needed ``Policy(maiden_delimiters=...)`` through 2.1. The clause is lifted out as a region rather than having its brackets erased, which is what keeps an INTERIOR clause from eating the name behind it: ``"Jane (née Jones) Smith"`` reads given ``Jane``, family ``Smith``, maiden ``Jones``, where erasing the brackets would leave the bare ``"Jane née Jones Smith"``, whose marker takes every word to the end of the name -- maiden ``Jones Smith`` and no family at all. The closing bracket is a boundary the writer typed, and it is now where the maiden name stops. Two clauses beside each other no longer merge: ``"Jane "Janey" Smith (née Jones)"`` read nickname ``Janey née Jones`` through 2.1 and now reads nickname ``Janey``, maiden ``Jones``, each clause keeping its own role, and this reaches ``HumanName``. What does NOT move is a clause that says nothing about maiden names. A lone marker in brackets is just a word in brackets -- ``"Jane Smith (née)"`` stays nickname ``née``, as 1.4.0 read it -- because a marker with no name after it marks nothing, and because a one-word clause may be the surname it looks like (``Nee`` is Irish Ní/Nee and a Chinese romanization). A markerless clause stays a nickname too: ``"Cherice J. (Johnson) Williams"`` keeps nickname ``Johnson``, which is what ``Policy(maiden_delimiters=...)`` remains for -- the parenthesized birth surname is a real convention, and only a caller who knows the data can say that is what these brackets mean. One accepted consequence: the word after the marker is not tested for being a name word, so a bracketed clause does not stop at a credential where the bare form does -- ``"Jane Smith (née V)"`` reads maiden ``V`` while ``"Jane Smith née V"`` reads suffix ``V``. The clause goes to ``maiden`` whole and the bracket is its boundary, so the trailing-suffix rule never sees the word as trailing the name. Six of the 1,078 differential corpus names move against the 2.1.0 and 2.0.0 baselines and five against 1.4.0: the Japanese name is the sixth, and against 1.4.0 it is not a new diff at all, since 1.4 read the fullwidth brackets as name text and had no nickname there to change hands. Those six are also every name in the corpora that HAS the shape (measured), so the count says how few bracketed marker clauses the corpora hold rather than how narrow the change is; four of the six are hand-written rule and issue examples, and the coverage that bounds the behavior is ``tests/v2/cases.py`` (closes #335) + - Fix a bracketed maiden clause reading as a nickname because its brackets were not declared. ``"Jane Smith nee Jones"`` gave maiden ``Jones``, while ``"Jane Smith (née Jones)"`` gave nickname ``née Jones`` -- the parenthesis pair is a nickname delimiter by default and nothing looked inside it, so the marker word the writer put there was read as part of an alias. A bracketed clause whose content opens with a recognized marker word and carries a word after it now reads as the maiden name whichever bucket its pair sits in: ``"Jane Smith (née Jones)"`` reads family ``Smith``, maiden ``Jones``, and so does the Japanese ``"山田 花子（旧姓 佐藤）"`` (family 山田, given 花子, maiden 佐藤), which needed ``Policy(maiden_delimiters=...)`` through 2.1. The clause is lifted out as a region rather than having its brackets erased, which is what keeps an INTERIOR clause from eating the name behind it: ``"Jane (née Jones) Smith"`` reads given ``Jane``, family ``Smith``, maiden ``Jones``, where erasing the brackets would leave the bare ``"Jane née Jones Smith"``, whose marker takes every word to the end of the name -- maiden ``Jones Smith`` and no family at all. The closing bracket is a boundary the writer typed, and it is now where the maiden name stops. Two clauses beside each other no longer merge: ``"Jane "Janey" Smith (née Jones)"`` read nickname ``Janey née Jones`` through 2.1 and now reads nickname ``Janey``, maiden ``Jones``, each clause keeping its own role, and this reaches ``HumanName``. What does NOT move is a clause that says nothing about maiden names. A lone marker in brackets is just a word in brackets -- ``"Jane Smith (née)"`` stays nickname ``née``, as 1.4.0 read it -- because a marker with no name after it marks nothing, and because a one-word clause may be the surname it looks like (``Nee`` is Irish Ní/Nee and a Chinese romanization). A markerless clause stays a nickname too: ``"Cherice J. (Johnson) Williams"`` keeps nickname ``Johnson``, which is what ``Policy(maiden_delimiters=...)`` remains for -- the parenthesized birth surname is a real convention, and only a caller who knows the data can say that is what these brackets mean. One accepted consequence: the word after the marker is not tested for being a name word, so a bracketed clause does not stop at a credential where the bare form does -- ``"Jane Smith (née V)"`` reads maiden ``V`` while ``"Jane Smith née V"`` reads suffix ``V``. The clause goes to ``maiden`` whole and the bracket is its boundary, so the trailing-suffix rule never sees the word as trailing the name. Six of the 1,079 differential corpus names move against the 2.1.0 and 2.0.0 baselines and five against 1.4.0: the Japanese name is the sixth, and against 1.4.0 it is not a new diff at all, since 1.4 read the fullwidth brackets as name text and had no nickname there to change hands. Those six are also every name in the corpora that HAS the shape (measured), so the count says how few bracketed marker clauses the corpora hold rather than how narrow the change is; four of the six are hand-written rule and issue examples, and the coverage that bounds the behavior is ``tests/v2/cases.py`` (closes #335) - Fix a particle chain and a maiden name taking a trailing generational numeral as a name word: ``"John van der Berg V"`` read family ``van der Berg V`` and ``"John née Jones Smith V"`` read maiden ``Jones Smith V``, where ``"John Smith V"`` reads suffix ``V``. Both stops asked "is this a suffix?" with the test that vetoes a bare ``V`` as an initial -- the same question the bound given-name join asked until #401 -- and now ask the suffix rule's own walk, over the name as they would leave it: family ``van der Berg``, maiden ``Jones Smith``, suffix ``V``, for ``I`` and ``X`` alike; ``"John van der J. V"`` and ``"J. née Jones Smith V"`` keep their readings, the word before the numeral being an initial as written or as left. The chain stops before a bare credential with words to spare too, as ``"John Smith Ma"`` does, where the suffix rule will still find them once the chain has run: ``"John van der Berg Ma"`` reads suffix ``Ma``, which is how 1.4.0 read it, while ``"Freiherr von Berg MA"`` keeps family ``von Berg MA``; a maiden name keeps such a credential, since the words the marker takes are the very words that made it one. A particle that is also suffix vocabulary now ends the chain where it opens the trailing run: ``"John Smith Mc V"`` reads suffix ``Mc, V``. The chain also no longer swallows the given name behind an unlisted abbreviation: ``"Xyz. van Johnson"`` and ``"Esq. van Gogh"`` read given ``van``, as ``"Dr. van Johnson"`` has since #367, the chain and the bound given-name join now asking the same test that reads the abbreviation as a title (closes #424) diff --git a/nameparser/_pipeline/_extract.py b/nameparser/_pipeline/_extract.py index e27c08ae..c0b7182a 100644 --- a/nameparser/_pipeline/_extract.py +++ b/nameparser/_pipeline/_extract.py @@ -226,6 +226,16 @@ def extract_delimited(state: ParseState) -> ParseState: masked.append(Span(j, j + len(close))) else: if inner.start < inner.end: + # M3 upgrades a nickname clause; a configured maiden + # pair is M1's and is left alone. The role test cannot + # be false today and no test can catch its removal: + # `order` above holds exactly two roles, so a role that + # is not NICKNAME is already MAIDEN and the assignment + # would be a no-op either way. It is kept for the day + # `order` gains a third bucket, when it becomes the + # difference between M3 claiming that bucket's clauses + # and leaving them. Measured 2026-08-26: dropping it + # leaves the suite and all three gates green. if (role is Role.NICKNAME and _maiden_marked( text[inner.start:inner.end], state.lexicon)): role = Role.MAIDEN diff --git a/nameparser/config/maiden_markers.py b/nameparser/config/maiden_markers.py index 5e897992..0410c9de 100644 --- a/nameparser/config/maiden_markers.py +++ b/nameparser/config/maiden_markers.py @@ -67,8 +67,8 @@ word, so there is no marker word for the clause to open with and by default it stays a nickname. The two tests agree here and are deliberately not the same test -- see decisions.md#M3. The wholly -unspaced "山田花子（旧姓佐藤）" reads as one token for the same reason. Peeling a marker off the head of a token is -#317's job. +unspaced "山田花子（旧姓佐藤）" reads as one token for the same reason. +Peeling a marker off the head of a token is #317's job. Consumed by the 2.0 parser's default lexicon. The 1.x parser does not read this module. diff --git a/tests/v2/cases.py b/tests/v2/cases.py index 9adc8d82..6d2e42d5 100644 --- a/tests/v2/cases.py +++ b/tests/v2/cases.py @@ -1399,7 +1399,9 @@ def __post_init__(self) -> None: "there is no nickname left to contrast: both clauses " "are maiden and M1's independence rule joins them into " "one value. The role filter it used to discriminate " - "(_group.py:841) is still reachable and still pinned -- " + "(the 'role is not Role.MAIDEN' branch of " + "_group.group's drop pass) is still reachable and " + "still pinned -- " "that job moved to " "marker_glued_to_punctuation_keeps_the_clause_a_nickname " "below, which reaches a marker-led clause M3 declines. " @@ -1429,8 +1431,8 @@ def __post_init__(self) -> None: "a nickname. tokenize splits the comma off as a " "separator, so the clause's first TOKEN is 'née' and " "carries vocab:maiden-marker, which is exactly what " - "_group.py:841's 'role is not Role.MAIDEN' branch " - "exists to refuse. Measured 2026-08-26: with that " + "the 'role is not Role.MAIDEN' branch of " + "_group.group's drop pass exists to refuse. Measured 2026-08-26: with that " "branch removed this reads nickname 'Janie', the " "marker dropped out of a nickname. BOTH clauses are " "load-bearing -- the drop pass is gated on the name " diff --git a/tests/v2/test_ledger_guards.py b/tests/v2/test_ledger_guards.py index 457162bd..60dbf58a 100644 --- a/tests/v2/test_ledger_guards.py +++ b/tests/v2/test_ledger_guards.py @@ -1293,7 +1293,7 @@ def _claim(rule: dict) -> _Claim: "fix(comma-precomma-family) pre-comma run reads as family, not given": _Claim(279, ('family', 'given'), "28a62b622a48"), "fix(suffix-routing) two-token name with unambiguous trailing suffix stays suffix": - _Claim(1078, ('family', 'given', 'suffix'), "235a385447cb"), + _Claim(1079, ('family', 'given', 'suffix'), "3cc8bbe00b2d"), "fix(suffix-delimiter-rendering) no-space delimiter core token kept whole": _Claim(0, ('suffix',), "e3b0c44298fc"), "ambiguous-surname-acronym data change: parenthesized (MA)/(DO) now stays nickname": @@ -1824,7 +1824,7 @@ class _Excluded(NamedTuple): ("fix(comma-family)", "fix(comma-precomma-family)", "fix(suffix-routing)")), '(^|[\\w.]\\s+)[("\'][^)"\']+[)"\'](\\s+\\w|\\s*$)': - _Excluded(49, "aa0609aef13c", ()), + _Excluded(50, "241623062660", ()), } diff --git a/tools/differential/README.md b/tools/differential/README.md index 113af81b..a231fc5b 100644 --- a/tools/differential/README.md +++ b/tools/differential/README.md @@ -414,12 +414,35 @@ behind a non-default `Policy` field is invisible here. Default gated off the way a `Policy` field is, so a change to it can show up here. That is not the same as coverage -- only 4 of the 16 shipped `maiden_markers` and 8 of the 15 `honorific_tails` appear anywhere in -the corpora as whole tokens (re-measured 2026-08-26; the marker count -was 3 until #414's rules corpus brought in a parenthesized `Nee`, and -the denominator was 17 until `roz` left the vocabulary in 2.2 -- it -appeared in no corpus name, so only the denominator moved), so -an entry no corpus name exercises is as invisible as an opt-in -policy. +the corpora (re-measured 2026-08-26; the marker count was 3 until +#414's rules corpus brought in a parenthesized `Nee`, and the +denominator was 17 until `roz` left the vocabulary in 2.2 -- it +appeared in no corpus name, so only the denominator moved), so an +entry no corpus name exercises is as invisible as an opt-in policy. + +Those two numbers count WHOLE TOKENS, delimiters stripped: `née` +counts because `(née` is a token of `Anna Müller (née Jones)` once its +bracket comes off, and `né` does not, being only a substring of that +same token. The convention matters because the neighbouring guard +`tests/v2/test_ledger_guards.py::_carries` deliberately asks a wider +question -- it also matches a non-ASCII entry anywhere inside a name, +since 旧姓 is written flush against the name it marks -- and under that +reading the marker count is 5, not 4. Both are right about different +questions. Recompute: + +``` +uv run python -c " +import glob, json +from nameparser import Parser +from nameparser._lexicon import _normalize +L = Parser().lexicon +names = [json.loads(l) for f in glob.glob('tools/differential/corpus*.jsonl') for l in open(f, encoding='utf-8') if l.strip()] +toks = {_normalize(t.strip('()\'"«»“”„「」『』（）')) for n in names for t in n.split()} +for s in ('maiden_markers', 'honorific_tails'): + v = getattr(L, s) + print(s, len(toks & v), 'of', len(v), sorted(toks & v))" +``` + Two independent mechanisms put a birth surname in `maiden`, and what is opt-in about them is narrower than it looks (rows measured diff --git a/tools/differential/corpus_rules.jsonl b/tools/differential/corpus_rules.jsonl index 22197876..8cade004 100644 --- a/tools/differential/corpus_rules.jsonl +++ b/tools/differential/corpus_rules.jsonl @@ -47,6 +47,7 @@ "Jane Smith (Nee)" "Jane Smith (Nee) (Jones)" "Jane Smith (née Jones)" +"Jane Smith (née Jr.)" "Jane Smith (née V)" "Jane Smith (née)" "Jane Smith née Jones" From 7a611a00e79aaedf581c827da81ad67f4e498b22 Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Wed, 26 Aug 2026 10:34:39 -0700 Subject: [PATCH 06/12] docs(differential,decisions,rules,release_log): the five-reviewer round MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Thirteen factual corrections, no behavior. Each was re-measured here before landing; two reviewer figures conflicted and both are settled below by measurement. The ledger's marker arithmetic lost a carve-out when I recounted it: thirteen entries go unreached, but 旧姓 is one of them and DOES appear as a corpus token, which is why it has a rule of its own. Twelve is the number that means "no diff to classify", as it was before. Counts that were stale or never right: the isascii note said 12 of 17 (the set ships 16) and claimed one corpus name depends on the substring branch -- it is two, both 旧姓, and it was seven before this branch's delimiter strip moved the parenthesized née names onto the token branch. The alternation roster still described a roz fragment this branch deleted. Lexicon.maiden_markers' public docstring gave "roz." as an example of the defaults. test_parser's filter comment said three names; it turns away nine, five carrying a maiden clause under M3 and four -- three one-word clauses and the S1-diverted "(née Jr.)" -- that M3 declines and that would have been safe. The release note said six corpus names are every name with the shape. Seven have it; the seventh is "Jane Smith (née Jr.)", which S1 takes. Five of the six movers are rule or issue examples, not four. Its roz bullet pointed at "the bracketed-clause change above", which is below it. The Excluded block is keyed by vocabulary set, not by rule, and two cross-references called it "#M1's". rules.md called roz "deliberately not shipped" when it shipped through 2.1 and was removed here -- the distinction that block draws on purpose, and a Czech caller reading rules.md would have concluded nothing changed for them. Two claims were too strong. The roz defect predates M3, but M3 would have widened it: with the entry restored, "Jane Smith (Roz Jones)" reads maiden 'Jones' where 2.1.0 read nickname 'Roz Jones'. And the {nickname, maiden} field pair is the 2.1.0 ledger's alone -- at 2.0.0 the CJK name declares four fields, at 1.4.0 it is not a fix(#335) name -- which is the premise for keeping those ledgers split. The role-test comment said the test "cannot be false today". It is false whenever a maiden pair matched; what it cannot do is change the outcome. Co-Authored-By: Claude Opus 5 --- docs/design/decisions.md | 6 +++--- docs/design/rules.md | 2 +- docs/release_log.rst | 4 ++-- nameparser/_lexicon.py | 2 +- nameparser/_pipeline/_extract.py | 12 +++++++----- nameparser/config/maiden_markers.py | 3 ++- tests/v2/cases.py | 10 +++++++--- tests/v2/test_ledger_guards.py | 18 ++++++++++++------ tests/v2/test_parser.py | 16 +++++++++------- tools/differential/README.md | 19 ++++++++++++------- tools/differential/expected_since_1.4.0.toml | 10 +++++++--- 11 files changed, 63 insertions(+), 39 deletions(-) diff --git a/docs/design/decisions.md b/docs/design/decisions.md index 2ad4f29b..b2817497 100644 --- a/docs/design/decisions.md +++ b/docs/design/decisions.md @@ -492,18 +492,18 @@ Declined: - 2026-08-05 #329/#335 — marker auto-detection inside a nickname-delimited clause was deferred to #335 on a corpus measurement: 山田 花子（旧姓 佐藤） is in the CJK differential corpus so the #329 change was gate-visible, while "Jane Smith (née Jones)" is in no corpus — shipping auto-detection in 2.1 would have let a real Latin-affecting change ride under a "0 Latin-only" gate report. -- 2026-08-26 #335 — correction to the 2026-08-05 entry above, which reasoned from "the #329 change was gate-visible". It was not. Under the default policy the corpora run, no name reached the drop pass at all, so 山田 花子（旧姓 佐藤） moving between 2.0.0 and 2.1.0 was the East Asian order flip and not #329. Nor is #329 gate-visible now that M3 routes marker-led clauses to maiden without configuration: a ledger rule narrows by which FIELDS move and never by what they hold, so reverting the drop leaves the same six names moving the same {nickname, maiden} pair and all three gates green (measured by mutation 2026-08-26). The deferral #329 argued for was still the right call — it was about a Latin-affecting change riding under a gate report, and that half held — but the premise is corrected here rather than left to be re-used. tools/differential/README.md carries the same correction beside the table it belongs to. +- 2026-08-26 #335 — correction to the 2026-08-05 entry above, which reasoned from "the #329 change was gate-visible". It was not. Under the default policy the corpora run, no name reached the drop pass at all, so 山田 花子（旧姓 佐藤） moving between 2.0.0 and 2.1.0 was the East Asian order flip and not #329. Nor is #329 gate-visible now that M3 routes marker-led clauses to maiden without configuration: a ledger rule narrows by which FIELDS move and never by what they hold, so reverting the drop leaves the same six names moving the same fields and all three gates green (measured by mutation 2026-08-26). The {nickname, maiden} pair is the 2.1.0 ledger's; the other two baselines classify the CJK name differently, which is why those ledgers are split and must stay so. The deferral #329 argued for was still the right call — it was about a Latin-affecting change riding under a gate report, and that half held — but the premise is corrected here rather than left to be re-used. tools/differential/README.md carries the same correction beside the table it belongs to. - 2026-08-26 #335 (M3) — the opt-in this section opens with narrowed, and the boundary is not where the 2026-07-03 entry drew it. A clause whose content announces itself no longer needs its pair declared, so what configuring maiden_delimiters buys is now markerless content and one-word clauses. See #M3 for the whole reasoning; M1 still governs every clause whose pair is configured, and M3 defers to it for the marker drop itself. ### M3 — the marker-led clause - 2026-08-26 #335 (M3, PR on fix/335-maiden-marked-clause) — a bracketed clause whose content opens with a recognized marker word and carries a word after it reads as the maiden name, whichever bucket the enclosing pair sits in. The parse already had the information: "Jane Smith née Jones" gave maiden "Jones" while "Jane Smith (née Jones)" gave nickname "née Jones", because ( ) is a nickname pair by default and nothing looked inside it. The clause says "maiden" out loud, so the caller should not have to say it in Policy. -- 2026-08-26 #335 — the MECHANISM, and the issue proposed the other one. Masking the delimiters and letting M2's bare-marker rule consume the content would avoid a second implementation, and it was rejected on measurement: the two readings agree on a trailing clause and diverge on an interior one. "Jane (née Jones) Smith" reads given "Jane", family "Smith", maiden "Jones" under extraction; masking turns it into the bare "Jane née Jones Smith", which reads maiden "Jones Smith" and family "" (measured 2026-08-26), because M2's take runs to the end of the name. The closing delimiter is a boundary the writer typed and masking throws it away. Extraction was also the SMALLER change — extract already emits Role.MAIDEN regions and group already drops the marker inside one (#329), so this is a role swap at the existing emit site plus one predicate, in the branch S1 has already declined, which keeps S1's precedence by construction. +- 2026-08-26 #335 — the MECHANISM, and the issue proposed the other one. Masking the delimiters and letting M2's bare-marker rule consume the content would avoid a second implementation, and it was rejected on measurement: the two readings agree on a trailing clause and diverge on an interior one. "Jane (née Jones) Smith" reads given "Jane", family "Smith", maiden "Jones" under extraction; masking turns it into the bare "Jane née Jones Smith", which reads maiden "Jones Smith" and family "" (measured 2026-08-26), because M2's take runs to the end of the name. The closing delimiter is a boundary the writer typed and masking throws it away. Extraction was also the SMALLER change — extract already emits Role.MAIDEN regions and group already drops the marker inside one (#329), so this is a role swap at the existing emit site plus one predicate, in the branch S1 has already declined. That placement is what keeps S1 first, but be exact about the mechanism, because the obvious mutation does not test it: hoisting the swap above the _suffix_shaped call changes nothing at all (measured 2026-08-26), since the S1 branch never reads `role` -- it masks the two delimiter spans and hands the content to the token stream, and the role it would have carried is never consulted. What inverts the precedence is giving M3 the clause first, an early `extracted.append((Role.MAIDEN, inner))` ahead of the S1 test: that reads "Jane Smith (née Jr.)" as family "Smith", maiden "Jr." instead of family "née", suffix "Jr.", and M3's own example line for it fails. So the precedence is pinned, and by that example rather than by the branch shape. - 2026-08-26 #335 — the condition is a word AFTER the marker, not the marker alone. The reason is M1's: a one-word clause keeps its word because Nee is an attested surname (Irish Ní/Nee, and a Chinese romanization), and a marker with nothing after it marks nothing. Be precise about what pins it, because the obvious answer is wrong: M1's own boundary line carries the maiden-parens annotation, so it asserts the CONFIGURED reading and passes either way — built and measured 2026-08-26, a len(words) > 0 predicate still gives it maiden "Nee". What the loosened predicate actually breaks, measured by mutation 2026-08-26, is M3's own boundary example ("Jane Smith (née)" → nickname "née") and tests/v2/cases.py::maiden_marked_clause_one_word_stays_a_nickname in both runners — three pytest failures — AND the 2.1.0 differential gate, which goes red with three unexplained names: "Jane Smith (Nee)", "Jane Smith (Nee) (Jones)" and "Jane Smith (née)". The gate is the stronger signal and the one to look at first. The middle name is why: loosened, it reads maiden "Nee" and nickname "Jones", splitting one bracket pair into the maiden field and the next into the nickname field, which is the direct contradiction of M1's "two enclosures read as one maiden name". Anyone re-litigating the second-word test should check those, not M1's line. So the real division of labour is NOT markerless versus marker-led, which is how it was first written into rules.md and had to be corrected at review: it is one word versus a marker plus a word. "Jane Smith (Nee)" is marker-led and still reads nickname "Nee" by default, maiden "Nee" only with the pair configured (both measured 2026-08-26). - 2026-08-26 #335 — accepted consequence: the word taken after the marker is not tested for being a name word, so M3 does not stop at a suffix word where M2's bare take does. "Jane Smith (née V)" reads maiden "V" while the bare "Jane Smith née V" reads suffix "V" and family "née" (both measured). The reason is structural rather than an oversight: M3 hands the WHOLE clause to Role.MAIDEN and the bracket is its right boundary, so the trailing-suffix rule never sees the word as trailing the name at all. Pinned as M3's Accepted line in rules.md, which puts "Jane Smith (née V)" in the differential corpus and classified in all three ledgers. - 2026-08-26 #335 — rules.md#M3 lists implemented: nameparser/_pipeline/_extract.py alone, and that is deliberate rather than an omission. M3 decides a REGION'S ROLE; the marker drop it describes ("the marker itself dropped, as M1 drops it") is M1's mechanism in _group.py, which cites M1 and is governed by it. Adding _group.py to M3's list would mean inventing an M3 citation there for behavior M1 already owns — and test_doc_citations requires implemented: to equal the set of modules that cite the rule, so the two would have to move together. Do not re-open this. - 2026-08-26 #335 — the durable finding, and the one worth reading before touching either side: _maiden_marked asks a WHITESPACE-SPLIT, _normalize'd question of the clause's first word, while tokenize and classify ask a token-level one. _normalize strips a trailing period but not a comma, so 'née,' is not a marker to M3 while the tokenizer splits the comma off and still tags the token vocab:maiden-marker. That divergence is the only thing keeping the "role is not Role.MAIDEN" branch of _group.group's clause-scoped drop pass reachable at all: without it, a marker-led NICKNAME clause could no longer exist, since M3 would have converted every one to MAIDEN before group ran. Anyone "unifying" the two tests would orphan that filter silently. The pin is tests/v2/cases.py::marker_glued_to_punctuation_keeps_the_clause_a_nickname, and it needs BOTH of its clauses — the drop pass is gated on the name holding a maiden region at all, so a marker-glued nickname clause standing alone leaves the branch unexercised (measured by mutation, 2026-08-26). The row that used to pin the filter, maiden_marker_delimited_beside_a_nickname_clause, lost the job to M3: both its clauses are marker-led, so both are maiden now and there is no contrast left in it. -- 2026-08-26 #335 — M3 makes the marker vocabulary act in a THIRD position. M2's claim acts on the word after a name word; M1's on the content of a configured pair; M3's now on the first word of any bracketed clause, under the default policy. #vocabulary-collisions C-i is keyed on the position a claim acts on, so extending where a rule acts re-asks C-i of the whole set it reads, and nothing in the process prompts that — the rule was designed, reviewed and approved without the set being re-examined. Review afterwards found one entry that fails C-i, the Czech abbreviation roz, and its disposition is the Excluded block in #M1: it was removed. Two things worth separating, since the finding surfaced during this work and is easy to file under it — the defect was M2's and predates #335 (bare "Rosalind Roz Smith" lost its family name on master), and M3's own boundary already excluded the common one-word "(Roz)" spelling. What belongs to M3 is only the widening, and the lesson is the general one: re-ask C-i of a vocabulary set whenever a rule extends where its claim acts. +- 2026-08-26 #335 — M3 makes the marker vocabulary act in a THIRD position. M2's claim acts on the word after a name word; M1's on the content of a configured pair; M3's now on the first word of any bracketed clause, under the default policy. #vocabulary-collisions C-i is keyed on the position a claim acts on, so extending where a rule acts re-asks C-i of the whole set it reads, and nothing in the process prompts that — the rule was designed, reviewed and approved without the set being re-examined. Review afterwards found one entry that fails C-i, the Czech abbreviation roz, and its disposition is this file's Excluded (MAIDEN_MARKERS) block, which is keyed to the vocabulary set rather than to any rule: it was removed. Two things worth separating, since the finding surfaced during this work and is easy to file under it — the defect was M2's and predates #335 (bare "Rosalind Roz Smith" lost its family name on master), and M3's own boundary already excluded the common one-word "(Roz)" spelling. What belongs to M3 is only the widening, and the lesson is the general one: re-ask C-i of a vocabulary set whenever a rule extends where its claim acts. - 2026-08-26 #335 — what configuring maiden_delimiters still buys, since M3 removes the commonest reason to reach for it: markerless clauses, and one-word clauses. "Cherice J. (Johnson) Williams" is a real US convention and a corpus name, and nothing in that clause says maiden, so it stays a nickname by default — only a caller who knows their data can say otherwise. The knob is a declaration about the DATA; M3 is a reading of what a clause declares about itself. ### O1 — East Slavic rotation diff --git a/docs/design/rules.md b/docs/design/rules.md index 718e81bc..d2791f4b 100644 --- a/docs/design/rules.md +++ b/docs/design/rules.md @@ -531,7 +531,7 @@ N3. Rationale: a person set down as a nickname plus one name word is ## Maiden names (M) -Background: a maiden name is written beside the current name, set off by a marker word or by enclosure. Markers are attested across French née/né, German geb./geborene, Dutch geboren, Czech/Slovak rozená (the abbreviation roz. is deliberately not shipped -- it collides with the English diminutive Roz), Scandinavian født/fødd/född, Russian урожд. (both ё and е spellings), and Japanese 旧姓 — both grammatical genders where attested. Japanese more often writes the marker with a fullwidth colon (旧姓：佐藤), which is no separator, so marker and name arrive as a single word. Which enclosures mean "maiden" rather than "nickname" is a caller convention, so the maiden reading of a delimiter pair is opt-in — except where the clause announces itself. A clause of two words or more led by a marker word has said which convention it means, and reads as the maiden name inside a nickname pair as well (M3); a lone marker word has not, and neither has one the colon spelling above glues to the name. +Background: a maiden name is written beside the current name, set off by a marker word or by enclosure. Markers are attested across French née/né, German geb./geborene, Dutch geboren, Czech/Slovak rozená (the abbreviation roz. shipped through 2.1 and was removed in 2.2 -- it collides with the English diminutive Roz, and a caller who needs it adds it to their own Lexicon), Scandinavian født/fødd/född, Russian урожд. (both ё and е spellings), and Japanese 旧姓 — both grammatical genders where attested. Japanese more often writes the marker with a fullwidth colon (旧姓：佐藤), which is no separator, so marker and name arrive as a single word. Which enclosures mean "maiden" rather than "nickname" is a caller convention, so the maiden reading of a delimiter pair is opt-in — except where the clause announces itself. A clause of two words or more led by a marker word has said which convention it means, and reads as the maiden name inside a nickname pair as well (M3); a lone marker word has not, and neither has one the colon spelling above glues to the name. M1. Rationale: an enclosure the caller has declared to mean maiden holds the former family name; a recognized marker word inside it diff --git a/docs/release_log.rst b/docs/release_log.rst index b4ce27a7..2c781641 100644 --- a/docs/release_log.rst +++ b/docs/release_log.rst @@ -45,9 +45,9 @@ Release Log - Fix a bound given name joining past a credential that the suffix rule then takes, leaving no family: ``"abdul Smith Jr Ma"`` read given ``abdul Smith`` with no family, where ``"John Smith Jr Ma"`` reads family ``Smith`` with suffix ``Jr, Ma``. The join's reserve had re-derived the suffix rule's trailing walk by hand and left out one of its two forks -- a bare acronym such as ``Ma`` is a credential only with words to spare -- so it counted the acronym as a name word, joined, and the suffix rule then peeled it. The two stages now share one walk: the reserve tries the join, runs the suffix rule's own peel over the pieces as the join would leave them, and joins only when two name words became one and nothing else changed. ``"abdul Smith Jr Ma"`` reads family ``Smith``, suffix ``Jr, Ma``; ``"abdul Smith Ma"`` reads family ``Smith``, suffix ``Ma`` rather than joining the pair and keeping the credential as the family -- both as 1.4.0 read them and as ``John`` reads in their place; ``"abdul Smith Berg Ma"`` keeps its join, and a title word standing after the bound word joins again as it did in 1.4.0 (``"Berg, abdul Sir"`` reads given ``abdul Sir``) without the pair ever being read as a title (``"abdul Sheikh and Ahmad Bakar"`` reads given ``abdul Sheikh and Ahmad``). No name of the differential corpora has the shapes; the ``Ma`` rules examples carry the fix, byte-identical to 1.4.0 and moving against 2.0.0 and 2.1.0, and the title-word shapes are pinned in tests (closes #425) - - Remove the Czech/Slovak abbreviation ``roz.`` from the default maiden markers, where it had been eating English given names. Marker matching is whole-token, case-folded and period-insensitive, so ``Roz`` -- the ordinary diminutive of Rosalind -- was the same string as the marker, and a maiden marker takes every word after it: ``"Rosalind Roz Smith"`` read maiden ``Smith`` with **no family name at all**, and ``"Rosalind Roz Jones Smith"`` read maiden ``Jones Smith``. Both now read the way 1.4.0 read them, given ``Rosalind`` with ``Roz`` a middle name and ``Smith`` the family. The defect is as old as the marker vocabulary and has nothing to do with the bracketed-clause change above; it was found while reviewing that change. What stops working is the abbreviation itself, for Czech and Slovak callers: ``"Anna Nováková roz. Svobodová"`` now reads middle ``Nováková roz.``, family ``Svobodová`` rather than family ``Nováková``, maiden ``Svobodová`` -- which is also how 1.4.0 read it, so this is a return to the v1 reading rather than a new one. The full participle is untouched and needs no configuration: ``"Anna Nováková rozená Svobodová"`` still reads family ``Nováková``, maiden ``Svobodová``, and it is only the abbreviation that was ambiguous. A caller who wants the abbreviation back adds it to their own lexicon -- ``Parser(lexicon=dataclasses.replace(lex, maiden_markers=lex.maiden_markers | {"roz"}))`` -- which restores the old reading exactly. The default set is now 16 entries. No differential corpus name contains the string in any case or position, so nothing moves against any baseline (closes the collision found in #335's review) + - Remove the Czech/Slovak abbreviation ``roz.`` from the default maiden markers, where it had been eating English given names. Marker matching is whole-token, case-folded and period-insensitive, so ``Roz`` -- the ordinary diminutive of Rosalind -- was the same string as the marker, and a maiden marker takes every word after it: ``"Rosalind Roz Smith"`` read maiden ``Smith`` with **no family name at all**, and ``"Rosalind Roz Jones Smith"`` read maiden ``Jones Smith``. Both now read the way 1.4.0 read them, given ``Rosalind`` with ``Roz`` a middle name and ``Smith`` the family. The defect is as old as the marker vocabulary and belongs to the bare marker rule, not to the bracketed-clause change in the bullet below; it was found while reviewing that change, which would have widened it -- with ``roz`` still shipped, ``"Jane Smith (Roz Jones)"`` reads maiden ``Jones`` where 2.1.0 read nickname ``Roz Jones``. What stops working is the abbreviation itself, for Czech and Slovak callers: ``"Anna Nováková roz. Svobodová"`` now reads middle ``Nováková roz.``, family ``Svobodová`` rather than family ``Nováková``, maiden ``Svobodová`` -- which is also how 1.4.0 read it, so this is a return to the v1 reading rather than a new one. The full participle is untouched and needs no configuration: ``"Anna Nováková rozená Svobodová"`` still reads family ``Nováková``, maiden ``Svobodová``, and it is only the abbreviation that was ambiguous. A caller who wants the abbreviation back adds it to their own lexicon -- ``Parser(lexicon=dataclasses.replace(lex, maiden_markers=lex.maiden_markers | {"roz"}))`` -- which restores the old reading exactly. The default set is now 16 entries. No differential corpus name contains the string in any case or position, so nothing moves against any baseline (closes the collision found in #335's review) - - Fix a bracketed maiden clause reading as a nickname because its brackets were not declared. ``"Jane Smith nee Jones"`` gave maiden ``Jones``, while ``"Jane Smith (née Jones)"`` gave nickname ``née Jones`` -- the parenthesis pair is a nickname delimiter by default and nothing looked inside it, so the marker word the writer put there was read as part of an alias. A bracketed clause whose content opens with a recognized marker word and carries a word after it now reads as the maiden name whichever bucket its pair sits in: ``"Jane Smith (née Jones)"`` reads family ``Smith``, maiden ``Jones``, and so does the Japanese ``"山田 花子（旧姓 佐藤）"`` (family 山田, given 花子, maiden 佐藤), which needed ``Policy(maiden_delimiters=...)`` through 2.1. The clause is lifted out as a region rather than having its brackets erased, which is what keeps an INTERIOR clause from eating the name behind it: ``"Jane (née Jones) Smith"`` reads given ``Jane``, family ``Smith``, maiden ``Jones``, where erasing the brackets would leave the bare ``"Jane née Jones Smith"``, whose marker takes every word to the end of the name -- maiden ``Jones Smith`` and no family at all. The closing bracket is a boundary the writer typed, and it is now where the maiden name stops. Two clauses beside each other no longer merge: ``"Jane "Janey" Smith (née Jones)"`` read nickname ``Janey née Jones`` through 2.1 and now reads nickname ``Janey``, maiden ``Jones``, each clause keeping its own role, and this reaches ``HumanName``. What does NOT move is a clause that says nothing about maiden names. A lone marker in brackets is just a word in brackets -- ``"Jane Smith (née)"`` stays nickname ``née``, as 1.4.0 read it -- because a marker with no name after it marks nothing, and because a one-word clause may be the surname it looks like (``Nee`` is Irish Ní/Nee and a Chinese romanization). A markerless clause stays a nickname too: ``"Cherice J. (Johnson) Williams"`` keeps nickname ``Johnson``, which is what ``Policy(maiden_delimiters=...)`` remains for -- the parenthesized birth surname is a real convention, and only a caller who knows the data can say that is what these brackets mean. One accepted consequence: the word after the marker is not tested for being a name word, so a bracketed clause does not stop at a credential where the bare form does -- ``"Jane Smith (née V)"`` reads maiden ``V`` while ``"Jane Smith née V"`` reads suffix ``V``. The clause goes to ``maiden`` whole and the bracket is its boundary, so the trailing-suffix rule never sees the word as trailing the name. Six of the 1,079 differential corpus names move against the 2.1.0 and 2.0.0 baselines and five against 1.4.0: the Japanese name is the sixth, and against 1.4.0 it is not a new diff at all, since 1.4 read the fullwidth brackets as name text and had no nickname there to change hands. Those six are also every name in the corpora that HAS the shape (measured), so the count says how few bracketed marker clauses the corpora hold rather than how narrow the change is; four of the six are hand-written rule and issue examples, and the coverage that bounds the behavior is ``tests/v2/cases.py`` (closes #335) + - Fix a bracketed maiden clause reading as a nickname because its brackets were not declared. ``"Jane Smith nee Jones"`` gave maiden ``Jones``, while ``"Jane Smith (née Jones)"`` gave nickname ``née Jones`` -- the parenthesis pair is a nickname delimiter by default and nothing looked inside it, so the marker word the writer put there was read as part of an alias. A bracketed clause whose content opens with a recognized marker word and carries a word after it now reads as the maiden name whichever bucket its pair sits in: ``"Jane Smith (née Jones)"`` reads family ``Smith``, maiden ``Jones``, and so does the Japanese ``"山田 花子（旧姓 佐藤）"`` (family 山田, given 花子, maiden 佐藤), which needed ``Policy(maiden_delimiters=...)`` through 2.1. The clause is lifted out as a region rather than having its brackets erased, which is what keeps an INTERIOR clause from eating the name behind it: ``"Jane (née Jones) Smith"`` reads given ``Jane``, family ``Smith``, maiden ``Jones``, where erasing the brackets would leave the bare ``"Jane née Jones Smith"``, whose marker takes every word to the end of the name -- maiden ``Jones Smith`` and no family at all. The closing bracket is a boundary the writer typed, and it is now where the maiden name stops. Two clauses beside each other no longer merge: ``"Jane "Janey" Smith (née Jones)"`` read nickname ``Janey née Jones`` through 2.1 and now reads nickname ``Janey``, maiden ``Jones``, each clause keeping its own role, and this reaches ``HumanName``. What does NOT move is a clause that says nothing about maiden names. A lone marker in brackets is just a word in brackets -- ``"Jane Smith (née)"`` stays nickname ``née``, as 1.4.0 read it -- because a marker with no name after it marks nothing, and because a one-word clause may be the surname it looks like (``Nee`` is Irish Ní/Nee and a Chinese romanization). A markerless clause stays a nickname too: ``"Cherice J. (Johnson) Williams"`` keeps nickname ``Johnson``, which is what ``Policy(maiden_delimiters=...)`` remains for -- the parenthesized birth surname is a real convention, and only a caller who knows the data can say that is what these brackets mean. One accepted consequence: the word after the marker is not tested for being a name word, so a bracketed clause does not stop at a credential where the bare form does -- ``"Jane Smith (née V)"`` reads maiden ``V`` while ``"Jane Smith née V"`` reads suffix ``V``. The clause goes to ``maiden`` whole and the bracket is its boundary, so the trailing-suffix rule never sees the word as trailing the name. Six of the 1,079 differential corpus names move against the 2.1.0 and 2.0.0 baselines and five against 1.4.0: the Japanese name is the sixth, and against 1.4.0 it is not a new diff at all, since 1.4 read the fullwidth brackets as name text and had no nickname there to change hands. Seven corpus names have the shape and six move (measured); the seventh is ``"Jane Smith (née Jr.)"``, which the suffix rule takes first. So the count says how few bracketed marker clauses the corpora hold rather than how narrow the change is -- five of the six are hand-written rule or issue examples, only the Japanese name arriving from elsewhere, and the coverage that bounds the behavior is ``tests/v2/cases.py`` (closes #335) - Fix a particle chain and a maiden name taking a trailing generational numeral as a name word: ``"John van der Berg V"`` read family ``van der Berg V`` and ``"John née Jones Smith V"`` read maiden ``Jones Smith V``, where ``"John Smith V"`` reads suffix ``V``. Both stops asked "is this a suffix?" with the test that vetoes a bare ``V`` as an initial -- the same question the bound given-name join asked until #401 -- and now ask the suffix rule's own walk, over the name as they would leave it: family ``van der Berg``, maiden ``Jones Smith``, suffix ``V``, for ``I`` and ``X`` alike; ``"John van der J. V"`` and ``"J. née Jones Smith V"`` keep their readings, the word before the numeral being an initial as written or as left. The chain stops before a bare credential with words to spare too, as ``"John Smith Ma"`` does, where the suffix rule will still find them once the chain has run: ``"John van der Berg Ma"`` reads suffix ``Ma``, which is how 1.4.0 read it, while ``"Freiherr von Berg MA"`` keeps family ``von Berg MA``; a maiden name keeps such a credential, since the words the marker takes are the very words that made it one. A particle that is also suffix vocabulary now ends the chain where it opens the trailing run: ``"John Smith Mc V"`` reads suffix ``Mc, V``. The chain also no longer swallows the given name behind an unlisted abbreviation: ``"Xyz. van Johnson"`` and ``"Esq. van Gogh"`` read given ``van``, as ``"Dr. van Johnson"`` has since #367, the chain and the bound given-name join now asking the same test that reads the abbreviation as a title (closes #424) diff --git a/nameparser/_lexicon.py b/nameparser/_lexicon.py index fc6d2abe..5c0a2cf6 100644 --- a/nameparser/_lexicon.py +++ b/nameparser/_lexicon.py @@ -366,7 +366,7 @@ class Lexicon: #: :data:`~nameparser.config.bound_given_names.BOUND_GIVEN_NAMES`. bound_given_names: frozenset[str] = frozenset() #: Marker words introducing a birth surname, routed to the maiden - #: field ("née", "geb.", "roz.", ...). Full default list: + #: field ("née", "geb.", "rozená", ...). Full default list: #: :data:`~nameparser.config.maiden_markers.MAIDEN_MARKERS`. maiden_markers: frozenset[str] = frozenset() #: Family names for the unspaced-name segmentation stage (#271), diff --git a/nameparser/_pipeline/_extract.py b/nameparser/_pipeline/_extract.py index c0b7182a..f4e1b3c0 100644 --- a/nameparser/_pipeline/_extract.py +++ b/nameparser/_pipeline/_extract.py @@ -227,11 +227,13 @@ def extract_delimited(state: ParseState) -> ParseState: else: if inner.start < inner.end: # M3 upgrades a nickname clause; a configured maiden - # pair is M1's and is left alone. The role test cannot - # be false today and no test can catch its removal: - # `order` above holds exactly two roles, so a role that - # is not NICKNAME is already MAIDEN and the assignment - # would be a no-op either way. It is kept for the day + # pair is M1's and is left alone. The role test is + # False whenever a maiden pair matched, but it cannot + # change the OUTCOME, and no test can catch its + # removal: `order` above holds exactly two roles, so a + # role that is not NICKNAME is already MAIDEN and the + # assignment would be a no-op either way. It is kept + # for the day # `order` gains a third bucket, when it becomes the # difference between M3 claiming that bucket's clauses # and leaving them. Measured 2026-08-26: dropping it diff --git a/nameparser/config/maiden_markers.py b/nameparser/config/maiden_markers.py index 0410c9de..26ee225b 100644 --- a/nameparser/config/maiden_markers.py +++ b/nameparser/config/maiden_markers.py @@ -84,7 +84,8 @@ family name at all. The collision is in the position the claim acts on, which is the test decisions.md#vocabulary-collisions states, and this set has no ambiguous subset to express a partial answer in — see -decisions.md#M1's Excluded block. The full participle rozená stays, +decisions.md's Excluded (MAIDEN_MARKERS) block. The full participle +rozená stays, being unambiguous; a caller who needs the abbreviation adds it to their own Lexicon. """ diff --git a/tests/v2/cases.py b/tests/v2/cases.py index 6d2e42d5..7ef30b8e 100644 --- a/tests/v2/cases.py +++ b/tests/v2/cases.py @@ -796,9 +796,13 @@ def __post_init__(self) -> None: "the pre-removal tree), because M2 hands the marker " "every word after it. The entry is gone in 2.2, which " "is what this row pins. Nothing to do with the " - "delimited path: the defect is M2's, it predates #335, " - "and the one-word '(Roz)' spelling was never affected " - "since M3 declines a lone marker. Parity, and it is " + "delimited path, though M3 would have widened it: " + "the defect is M2's and predates #335, and the one-word " + "'(Roz)' spelling was never affected since M3 declines " + "a lone marker -- but 'Jane Smith (Roz Jones)' reads " + "maiden 'Jones' with the entry restored, where 2.1.0 " + "read nickname 'Roz Jones' (measured 2026-08-26). " + "Parity, and it is " "RESTORED parity rather than untouched -- 1.4.0 has no " "maiden support and read first Rosalind / middle Roz / " "last Smith (2026-08-26), which is where the removal " diff --git a/tests/v2/test_ledger_guards.py b/tests/v2/test_ledger_guards.py index 60dbf58a..ddb7034e 100644 --- a/tests/v2/test_ledger_guards.py +++ b/tests/v2/test_ledger_guards.py @@ -835,8 +835,8 @@ class _LatinCopy(NamedTuple): #: by a substring of the rule's `issue`. Kept apart from #: _HONORIFIC_SOURCES because the relationship is not set equality: #: these members are regex FRAGMENTS, not entries -- "n[ée]e" covers two -#: markers at once, "geb\.?" and "roz\.?" one each -- so there is no set -#: to compare against. +#: markers at once and "geb\.?" one -- so there is no set to compare +#: against. #: #: `covers` is recorded rather than equated to the whole vocabulary. #: Equality would force a rule to grow alternatives for markers it has @@ -1110,14 +1110,20 @@ def _carries(name: str, vocabulary: frozenset[str]) -> bool: marker like 旧姓 is written against the name it marks rather than spaced off it. - Note what the isascii() split actually covers: 12 of the 17 + Note what the isascii() split actually covers: 12 of the 16 entries, not only the CJK one. `né` is two characters, so the substring branch reads `René` as carrying a marker. Every over-match here SHRINKS the set of unexplained names and so weakens the guard -- the direction this module exists to close -- - but exactly one corpus name reaches that branch today, and it is - the 旧姓 one. Tighten this before admitting a vocabulary whose - short non-ASCII entries occur inside ordinary names. + but only two corpus names DEPEND on that branch today, meaning the + token test below says no and the substring test says yes, and both + are 旧姓 ones: the fullwidth-bracketed clause and the + fullwidth-colon spelling. Seven depended on it before the + delimiter strip below arrived (2026-08-26); that strip moved the + parenthesized née names onto the token branch, where the answer + does not rest on a substring. Tighten this before admitting a + vocabulary whose short non-ASCII entries occur inside ordinary + names. Delimiter characters come off the token before the membership test, because a marker glued to a bracket is still a marker to the diff --git a/tests/v2/test_parser.py b/tests/v2/test_parser.py index e2b8a9d3..db2235b0 100644 --- a/tests/v2/test_parser.py +++ b/tests/v2/test_parser.py @@ -1317,13 +1317,15 @@ def _clause_free_latin_corpus_names() -> list[str]: # A marker glued to a delimiter character is still a marker, and # the membership test is per WORD, so '(geb.' must lose the # bracket as well as the abbreviating period before it is asked. - # Stripping only the period admitted the three corpus names that - # bracket their marker, and once rules.md#M3 read such a clause as - # the maiden name they had one of their own -- two clauses, and - # the appended one no longer the only variable. Textual and so - # deliberately conservative: it also turns away the one-word - # '(Nee)', which M3 declines and which would have been safe to - # keep. Delimiter characters come from the shipped set rather than + # Stripping only the period admitted every corpus name that + # brackets its marker, and once rules.md#M3 read such a clause as + # the maiden name, five of them had a maiden clause of their own -- + # two clauses, and the appended one no longer the only variable. + # The strip turns away NINE names in all (measured 2026-08-26): + # those five, plus four M3 declines and which would have been safe + # to keep -- the one-word '(Nee)', '(Nee) (Jones)' and '(née)', and + # '(née Jr.)', which S1 takes before M3 sees it. Textual, and so + # deliberately conservative in exactly that direction. Delimiter characters come from the shipped set rather than # a literal, so a pair added there cannot quietly reopen this. strip = "".join({ch for pair in DEFAULT_NICKNAME_DELIMITERS for ch in pair}) + "." diff --git a/tools/differential/README.md b/tools/differential/README.md index a231fc5b..52fbd099 100644 --- a/tools/differential/README.md +++ b/tools/differential/README.md @@ -420,10 +420,12 @@ denominator was 17 until `roz` left the vocabulary in 2.2 -- it appeared in no corpus name, so only the denominator moved), so an entry no corpus name exercises is as invisible as an opt-in policy. -Those two numbers count WHOLE TOKENS, delimiters stripped: `née` -counts because `(née` is a token of `Anna Müller (née Jones)` once its -bracket comes off, and `né` does not, being only a substring of that -same token. The convention matters because the neighbouring guard +Those two numbers count WHOLE TOKENS, delimiters stripped. The strip +is what earns exactly one of the four: `nee`, whose only corpus +appearance is the bracketed `Jane Smith (Nee)`, where the token is +`(Nee)` until the brackets come off. `née` needs no strip, appearing +bare in thirty-odd names, and `né` is not counted at all -- it occurs +only as a substring of `née`, never as a token. The convention matters because the neighbouring guard `tests/v2/test_ledger_guards.py::_carries` deliberately asks a wider question -- it also matches a non-ASCII entry anywhere inside a name, since 旧姓 is written flush against the name it marks -- and under that @@ -473,9 +475,12 @@ It does NOT put #329 within reach of this gate, and the reason generalizes past this one change. #329 governs what a delimited maiden clause CONTAINS -- the marker word is dropped from the value -- while a ledger rule narrows by which FIELDS move, never by what they hold. -The six names classified under `fix(#335)` move `{nickname, maiden}` -whether the marker is dropped or not, so that rule absorbs a #329 -regression in silence. Measured 2026-08-26 by reverting the drop pass +The six names the 2.1.0 ledger classifies under `fix(#335)` move +`{nickname, maiden}` whether the marker is dropped or not, so that rule +absorbs a #329 regression in silence. The field sets are per baseline +and only that ledger's are uniform: at 2.0.0 the CJK name declares four +fields and has a rule to itself, and at 1.4.0 it is not a `fix(#335)` +name at all. Measured 2026-08-26 by reverting the drop pass in `_group.py`: `Jane Smith (née Jones)` reads maiden `née Jones`, and all three gates still report 0 unexplained. #329 was out of reach before #335 too, for a different reason -- under the default policy no diff --git a/tools/differential/expected_since_1.4.0.toml b/tools/differential/expected_since_1.4.0.toml index 6677c762..fc875a88 100644 --- a/tools/differential/expected_since_1.4.0.toml +++ b/tools/differential/expected_since_1.4.0.toml @@ -69,9 +69,13 @@ issue = "fix(#274) maiden markers consumed" # # Its members reach 3 of the 16 entries -- geb, nee, née -- which is # not the same set as the markers the corpora contain (geb, nee, née -# and 旧姓; the first three are covered here). Growing it toward the -# other 13 buys nothing: none appears as a token in any corpus, so -# there is no diff for them to classify. The roz alternative was +# and 旧姓; the first three are covered here). Thirteen entries go +# unreached, and the carve-out in that arithmetic is load-bearing: +# 旧姓 is one of the thirteen and DOES appear as a corpus token, which +# is why it has fix(cjk-maiden-marker) of its own below rather than an +# alternative here. Growing this rule toward the other TWELVE buys +# nothing: none of them appears as a token in any corpus, so there is +# no diff for them to classify. The roz alternative was # dropped in 2.2 with the vocabulary entry it copied (see # maiden_markers.py's Deliberately absent note): no corpus name # contains the string in any case or position, so it classified From 7355de63b5a03457394268ec8dd794c9313c12eb Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Wed, 26 Aug 2026 10:43:07 -0700 Subject: [PATCH 07/12] test(cases): pin what M3 reaches MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Six coverage gaps, each found by a mutation that left the suite and the gates green. The OPENS-WITH half of the predicate was pinned by nothing: widening it from the first word to any word passed everything. "Jane Smith (Jones née)" closes it, and is parity -- 1.4.0 and 2.1.0 read the nickname too. M3 reaching pairs other than the parenthesis was pinned by nothing either. Gating the swap to '(' and the double quote left the whole suite green AND the 2.1.0 gate green, its fix(#335) rule quietly falling from six names to five, with only 1.4.0 going red. The fullwidth pair is the one the docstring and the release note both advertise as newly working without configuration, so it gets a default-policy row of its own, beside a quote-pair row for the commonest way a nickname is actually written. markerless_parenthesized_clause_stays_a_nickname executed without pinning anything: its clause is one word, so the length condition refused it before the vocabulary condition was consulted, duplicating the boundary above it. Two words now, so the vocabulary test is the one under load; the corpus spelling stays named in the note. "Smith (née Jones)" is N3's shape meeting M3, and N3 was not amended when M3 landed. A marker-led clause is not a nickname clause, so N3 does not reach it and the one name word keeps the given reading its bare spelling has had since 1.4.0. Recorded as an N3 Accepted line, pinned as a row, and classified in all three ledgers -- its own rule rather than a sixth alternative in the fix(#335) one, since it moves four fields where those move two. S1-over-M3 rested on a doc example asserting one field. The full reading is now a row, because the surprising part is not the suffix: it is that the marker word becomes the family name. _clause_free_latin_corpus_names had no non-emptiness assertion, and this branch widened its filter. An empty parametrization passes as a skip. Co-Authored-By: Claude Opus 5 --- docs/design/rules.md | 7 +- docs/release_log.rst | 2 +- tests/v2/cases.py | 132 +++++++++++++++++-- tests/v2/test_ledger_guards.py | 16 ++- tests/v2/test_parser.py | 11 ++ tools/differential/corpus_rules.jsonl | 1 + tools/differential/expected_since_1.4.0.toml | 18 +++ tools/differential/expected_since_2.0.0.toml | 18 +++ tools/differential/expected_since_2.1.0.toml | 18 +++ 9 files changed, 205 insertions(+), 18 deletions(-) diff --git a/docs/design/rules.md b/docs/design/rules.md index d2791f4b..48bb0590 100644 --- a/docs/design/rules.md +++ b/docs/design/rules.md @@ -527,7 +527,12 @@ N3. Rationale: a person set down as a nickname plus one name word is "'Smitty' Jones Jr." → family="" "'Smitty' Dr. Jones" → family="Jones" "'Smitty' Sir John" → given="John" - history: decisions.md#N3 · interacts: H1 · implemented: nameparser/_pipeline/_assign.py + Accepted: a marker-led clause is a maiden clause and not a + nickname one (M3), so this rule does not reach a name written + that way, and the one name word keeps the reading the bare + spelling gives it rather than becoming the family name. + "Smith (née Jones)" → given="Smith" + history: decisions.md#N3 · interacts: H1, M3 · implemented: nameparser/_pipeline/_assign.py ## Maiden names (M) diff --git a/docs/release_log.rst b/docs/release_log.rst index 2c781641..cad35d7c 100644 --- a/docs/release_log.rst +++ b/docs/release_log.rst @@ -47,7 +47,7 @@ Release Log - Remove the Czech/Slovak abbreviation ``roz.`` from the default maiden markers, where it had been eating English given names. Marker matching is whole-token, case-folded and period-insensitive, so ``Roz`` -- the ordinary diminutive of Rosalind -- was the same string as the marker, and a maiden marker takes every word after it: ``"Rosalind Roz Smith"`` read maiden ``Smith`` with **no family name at all**, and ``"Rosalind Roz Jones Smith"`` read maiden ``Jones Smith``. Both now read the way 1.4.0 read them, given ``Rosalind`` with ``Roz`` a middle name and ``Smith`` the family. The defect is as old as the marker vocabulary and belongs to the bare marker rule, not to the bracketed-clause change in the bullet below; it was found while reviewing that change, which would have widened it -- with ``roz`` still shipped, ``"Jane Smith (Roz Jones)"`` reads maiden ``Jones`` where 2.1.0 read nickname ``Roz Jones``. What stops working is the abbreviation itself, for Czech and Slovak callers: ``"Anna Nováková roz. Svobodová"`` now reads middle ``Nováková roz.``, family ``Svobodová`` rather than family ``Nováková``, maiden ``Svobodová`` -- which is also how 1.4.0 read it, so this is a return to the v1 reading rather than a new one. The full participle is untouched and needs no configuration: ``"Anna Nováková rozená Svobodová"`` still reads family ``Nováková``, maiden ``Svobodová``, and it is only the abbreviation that was ambiguous. A caller who wants the abbreviation back adds it to their own lexicon -- ``Parser(lexicon=dataclasses.replace(lex, maiden_markers=lex.maiden_markers | {"roz"}))`` -- which restores the old reading exactly. The default set is now 16 entries. No differential corpus name contains the string in any case or position, so nothing moves against any baseline (closes the collision found in #335's review) - - Fix a bracketed maiden clause reading as a nickname because its brackets were not declared. ``"Jane Smith nee Jones"`` gave maiden ``Jones``, while ``"Jane Smith (née Jones)"`` gave nickname ``née Jones`` -- the parenthesis pair is a nickname delimiter by default and nothing looked inside it, so the marker word the writer put there was read as part of an alias. A bracketed clause whose content opens with a recognized marker word and carries a word after it now reads as the maiden name whichever bucket its pair sits in: ``"Jane Smith (née Jones)"`` reads family ``Smith``, maiden ``Jones``, and so does the Japanese ``"山田 花子（旧姓 佐藤）"`` (family 山田, given 花子, maiden 佐藤), which needed ``Policy(maiden_delimiters=...)`` through 2.1. The clause is lifted out as a region rather than having its brackets erased, which is what keeps an INTERIOR clause from eating the name behind it: ``"Jane (née Jones) Smith"`` reads given ``Jane``, family ``Smith``, maiden ``Jones``, where erasing the brackets would leave the bare ``"Jane née Jones Smith"``, whose marker takes every word to the end of the name -- maiden ``Jones Smith`` and no family at all. The closing bracket is a boundary the writer typed, and it is now where the maiden name stops. Two clauses beside each other no longer merge: ``"Jane "Janey" Smith (née Jones)"`` read nickname ``Janey née Jones`` through 2.1 and now reads nickname ``Janey``, maiden ``Jones``, each clause keeping its own role, and this reaches ``HumanName``. What does NOT move is a clause that says nothing about maiden names. A lone marker in brackets is just a word in brackets -- ``"Jane Smith (née)"`` stays nickname ``née``, as 1.4.0 read it -- because a marker with no name after it marks nothing, and because a one-word clause may be the surname it looks like (``Nee`` is Irish Ní/Nee and a Chinese romanization). A markerless clause stays a nickname too: ``"Cherice J. (Johnson) Williams"`` keeps nickname ``Johnson``, which is what ``Policy(maiden_delimiters=...)`` remains for -- the parenthesized birth surname is a real convention, and only a caller who knows the data can say that is what these brackets mean. One accepted consequence: the word after the marker is not tested for being a name word, so a bracketed clause does not stop at a credential where the bare form does -- ``"Jane Smith (née V)"`` reads maiden ``V`` while ``"Jane Smith née V"`` reads suffix ``V``. The clause goes to ``maiden`` whole and the bracket is its boundary, so the trailing-suffix rule never sees the word as trailing the name. Six of the 1,079 differential corpus names move against the 2.1.0 and 2.0.0 baselines and five against 1.4.0: the Japanese name is the sixth, and against 1.4.0 it is not a new diff at all, since 1.4 read the fullwidth brackets as name text and had no nickname there to change hands. Seven corpus names have the shape and six move (measured); the seventh is ``"Jane Smith (née Jr.)"``, which the suffix rule takes first. So the count says how few bracketed marker clauses the corpora hold rather than how narrow the change is -- five of the six are hand-written rule or issue examples, only the Japanese name arriving from elsewhere, and the coverage that bounds the behavior is ``tests/v2/cases.py`` (closes #335) + - Fix a bracketed maiden clause reading as a nickname because its brackets were not declared. ``"Jane Smith nee Jones"`` gave maiden ``Jones``, while ``"Jane Smith (née Jones)"`` gave nickname ``née Jones`` -- the parenthesis pair is a nickname delimiter by default and nothing looked inside it, so the marker word the writer put there was read as part of an alias. A bracketed clause whose content opens with a recognized marker word and carries a word after it now reads as the maiden name whichever bucket its pair sits in: ``"Jane Smith (née Jones)"`` reads family ``Smith``, maiden ``Jones``, and so does the Japanese ``"山田 花子（旧姓 佐藤）"`` (family 山田, given 花子, maiden 佐藤), which needed ``Policy(maiden_delimiters=...)`` through 2.1. The clause is lifted out as a region rather than having its brackets erased, which is what keeps an INTERIOR clause from eating the name behind it: ``"Jane (née Jones) Smith"`` reads given ``Jane``, family ``Smith``, maiden ``Jones``, where erasing the brackets would leave the bare ``"Jane née Jones Smith"``, whose marker takes every word to the end of the name -- maiden ``Jones Smith`` and no family at all. The closing bracket is a boundary the writer typed, and it is now where the maiden name stops. Two clauses beside each other no longer merge: ``"Jane "Janey" Smith (née Jones)"`` read nickname ``Janey née Jones`` through 2.1 and now reads nickname ``Janey``, maiden ``Jones``, each clause keeping its own role, and this reaches ``HumanName``. What does NOT move is a clause that says nothing about maiden names. A lone marker in brackets is just a word in brackets -- ``"Jane Smith (née)"`` stays nickname ``née``, as 1.4.0 read it -- because a marker with no name after it marks nothing, and because a one-word clause may be the surname it looks like (``Nee`` is Irish Ní/Nee and a Chinese romanization). A markerless clause stays a nickname too: ``"Cherice J. (Johnson) Williams"`` keeps nickname ``Johnson``, which is what ``Policy(maiden_delimiters=...)`` remains for -- the parenthesized birth surname is a real convention, and only a caller who knows the data can say that is what these brackets mean. One accepted consequence: the word after the marker is not tested for being a name word, so a bracketed clause does not stop at a credential where the bare form does -- ``"Jane Smith (née V)"`` reads maiden ``V`` while ``"Jane Smith née V"`` reads suffix ``V``. The clause goes to ``maiden`` whole and the bracket is its boundary, so the trailing-suffix rule never sees the word as trailing the name. One more shape moves with it, and it is the one to know about if you parse names that are a surname and a clause: a name that is ONLY a marker clause and one name word now reads that word as the given name rather than the family. ``"Smith (née Jones)"`` reads given ``Smith``, maiden ``Jones`` where every earlier version read family ``Smith``, nickname ``née Jones``. That is the bracketed form agreeing with the bare one -- ``"Smith née Jones"`` has read given ``Smith`` since 1.4.0 -- because a marker-led clause is not a nickname clause and so does not reach the rule that would have made the word a family name. The empty ``family`` it leaves behind is the bare form's behavior and is not new here. Seven of the 1,080 differential corpus names move against the 2.1.0 and 2.0.0 baselines and six against 1.4.0: the Japanese name is the odd one, and against 1.4.0 it is not a new diff at all, since 1.4 read the fullwidth brackets as name text and had no nickname there to change hands. Eight corpus names have the shape and seven move (measured); the eighth is ``"Jane Smith (née Jr.)"``, which the suffix rule takes first. So the count says how few bracketed marker clauses the corpora hold rather than how narrow the change is -- six of the seven are hand-written rule or issue examples, only the Japanese name arriving from elsewhere, and the coverage that bounds the behavior is ``tests/v2/cases.py`` (closes #335) - Fix a particle chain and a maiden name taking a trailing generational numeral as a name word: ``"John van der Berg V"`` read family ``van der Berg V`` and ``"John née Jones Smith V"`` read maiden ``Jones Smith V``, where ``"John Smith V"`` reads suffix ``V``. Both stops asked "is this a suffix?" with the test that vetoes a bare ``V`` as an initial -- the same question the bound given-name join asked until #401 -- and now ask the suffix rule's own walk, over the name as they would leave it: family ``van der Berg``, maiden ``Jones Smith``, suffix ``V``, for ``I`` and ``X`` alike; ``"John van der J. V"`` and ``"J. née Jones Smith V"`` keep their readings, the word before the numeral being an initial as written or as left. The chain stops before a bare credential with words to spare too, as ``"John Smith Ma"`` does, where the suffix rule will still find them once the chain has run: ``"John van der Berg Ma"`` reads suffix ``Ma``, which is how 1.4.0 read it, while ``"Freiherr von Berg MA"`` keeps family ``von Berg MA``; a maiden name keeps such a credential, since the words the marker takes are the very words that made it one. A particle that is also suffix vocabulary now ends the chain where it opens the trailing run: ``"John Smith Mc V"`` reads suffix ``Mc, V``. The chain also no longer swallows the given name behind an unlisted abbreviation: ``"Xyz. van Johnson"`` and ``"Esq. van Gogh"`` read given ``van``, as ``"Dr. van Johnson"`` has since #367, the chain and the bound given-name join now asking the same test that reads the abbreviation as a title (closes #424) diff --git a/tests/v2/cases.py b/tests/v2/cases.py index 7ef30b8e..820b4a84 100644 --- a/tests/v2/cases.py +++ b/tests/v2/cases.py @@ -1312,17 +1312,93 @@ def __post_init__(self) -> None: "surname Nee -- would be contradicted on the " "unconfigured path. Parity: 1.4.0 read nickname 'née'"), Case("markerless_parenthesized_clause_stays_a_nickname", - "Cherice J. (Johnson) Williams", + "Cherice J. (Mary Johnson) Williams", {"given": "Cherice", "middle": "J.", "family": "Williams", - "nickname": "Johnson"}, + "nickname": "Mary Johnson"}, notes="M3's other boundary, and the reason the maiden " "delimiters remain worth configuring: the parenthesized " "birth surname without a marker is a real US convention " - "and a corpus name (corpus_issues.jsonl), but nothing in " - "the clause says 'maiden', so it stays a nickname by " - "default. Only a caller who knows their data can say " - "otherwise, which is what Policy(maiden_delimiters=...) " - "is for. Parity"), + "-- 'Cherice J. (Johnson) Williams' is the corpus name " + "(corpus_issues.jsonl) -- but nothing in the clause " + "says 'maiden', so it stays a nickname by default. Only " + "a caller who knows their data can say otherwise, which " + "is what Policy(maiden_delimiters=...) is for. The " + "clause is TWO words here, and that is the whole point " + "of the row: M3 tests the clause length before it tests " + "the vocabulary, so the corpus spelling's one-word " + "clause is refused by the length condition and never " + "reaches the vocabulary one -- it would duplicate " + "maiden_marked_clause_one_word_stays_a_nickname rather " + "than fence the other condition. Measured 2026-08-26: " + "with the vocabulary test dropped this reads maiden " + "'Johnson', and with it the one-word spelling reads " + "nickname either way. Parity: 1.4.0 and 2.1.0 both read " + "nickname 'Mary Johnson'"), + Case("maiden_marker_not_first_stays_a_nickname", + "Jane Smith (Jones née)", + {"given": "Jane", "family": "Smith", "nickname": "Jones née"}, + notes="the OPENS-WITH half of M3, which nothing else reaches: " + "a marker inside the clause but not first leaves the " + "clause a nickname. Measured 2026-08-26, widening the " + "predicate from the first word to any word left the " + "whole suite green and all three gates at 0 unexplained " + "-- this row is what closes that. The bracketed twin of " + "maiden_marker_delimited_trailing_marker, which pins the " + "same asymmetry one layer down, inside a clause already " + "routed to maiden by policy: no marker the shipped " + "vocabulary carries is written after the name it marks. " + "Parity: 1.4.0 and 2.1.0 both read nickname 'Jones née'"), + Case("marker_led_clause_with_one_name_word", + "Smith (née Jones)", + {"given": "Smith", "maiden": "Jones"}, + classification="fix(#335)", + notes="N3's shape meeting M3, and the row exists because the " + "two rules disagree about what a clause is. N3 reads a " + "name that is only a nickname plus one name word as " + "'that word is the family name' -- but a marker-led " + "clause is not a nickname clause, so N3 never sees this " + "one and 'Smith' keeps the given-name reading the bare " + "spelling gives it. Recorded as an N3 Accepted line. " + "The empty family is the bare path's, not this " + "change's: 'Smith née Jones' reads given 'Smith', " + "family '' on 2.1.0 too, and 1.4.0 read first 'Smith' / " + "last 'Jones' -- the given side has agreed since 1.4.0, " + "the emptying is 2.x's and is tracked separately as the " + "#410 analogue. What #335 moves is only the bracketed " + "spelling: 1.4.0, 2.0.0 and 2.1.0 all read it family " + "'Smith', nickname 'née Jones' (measured 2026-08-26)"), + Case("marker_led_clause_in_a_quote_pair", + 'Jane Smith "née Jones"', + {"given": "Jane", "family": "Smith", "maiden": "Jones"}, + classification="fix(#335)", + notes="M3 is keyed on the CONTENT, not on which pair matched, " + "and this is the row that says so in the commonest " + "spelling: a quote pair is how nicknames are usually " + "written, and the same clause inside one reads maiden " + "exactly as it does inside parentheses. Nine of the " + "eleven shipped nickname pairs still have no row of " + "their own -- gating the swap to '(' and '\"' leaves " + "the whole suite green (measured 2026-08-26) -- so this " + "row and the fullwidth one below fence the two ends of " + "the set rather than the whole of it. 1.4.0 and 2.1.0 " + "both read nickname 'née Jones'"), + Case("maiden_marked_clause_takes_the_suffix_reading_from_s1", + "Jane Smith (née Jr.)", + {"given": "Jane", "middle": "Smith", "family": "née", + "suffix": "Jr."}, + notes="S1 takes a suffix-shaped clause before M3 is " + "consulted, and the whole reading is here because the " + "surprising part is not the suffix: it is that the " + "MARKER becomes the family name. S1 drops the brackets " + "and lets the content read as if written bare, and " + "bare 'Jane Smith née Jr.' has no name word after the " + "marker for M2 to take, so 'née' stays an ordinary " + "word and lands in the family. rules.md#M3 carries the " + "same input as an example line, but the runner checks " + "one field per line; this row is the other four. " + "Parity, and unchanged by #335 -- 1.4.0 and 2.1.0 read " + "it the same way, which is why the corpus row it added " + "diffs against no baseline"), Case("maiden_marked_clause_beside_a_nickname", 'Jane "Janey" Smith (née Jones)', {"given": "Jane", "family": "Smith", "nickname": "Janey", @@ -1554,10 +1630,44 @@ def __post_init__(self) -> None: "ko_honorific_period_under_strict_comma_suffixes uses " "for a knob with no v1 spelling. That reading is also " "what the differential harness sees, since it runs the " - "corpus under the DEFAULT policy where （） is a #273 " - "NICKNAME delimiter and nothing in #329 is reachable; " - "the diff is classified there under " - "fix(cjk-fullwidth-paren-nickname)"), + "corpus under the DEFAULT policy. What the harness does " + "with it changed in 2.2: through 2.1 the （） pair was a " + "#273 NICKNAME delimiter and nothing in #329 was " + "reachable, so the diff classified under " + "fix(cjk-fullwidth-paren-nickname). Since #335 the marker " + "inside the clause is enough on its own, so this name " + "reads maiden under the default policy too -- see " + "maiden_marked_fullwidth_clause_by_default below -- and " + "the diff classifies under fix(#335) at 2.1.0 and 2.0.0 " + "while at 1.4.0 it moved to fix(cjk-maiden-marker), " + "leaving the fullwidth-paren rule dormant in that ledger. " + "This row keeps its policy because M1 still governs a " + "configured pair and settles the role before M3 is " + "consulted"), + Case("maiden_marked_fullwidth_clause_by_default", + "山田 花子（旧姓 佐藤）", + {"given": "花子", "family": "山田", "maiden": "佐藤"}, + classification="feat(#273) + fix(#271) + fix(#335)", + notes="the row above without its policy, and the one " + "that fences M3 across the delimiter SET rather " + "than at the parenthesis: gating the swap to '(' " + "and the double quote leaves the entire suite " + "green and even the 2.1.0 gate green, its " + "fix(#335) rule quietly falling from six names to " + "five, with only the 1.4.0 gate going red " + "(measured 2026-08-26). The fullwidth pair is the " + "one the maiden_markers docstring and the 2.2 " + "release note both advertise as newly working " + "without configuration, so it is the one that " + "most needs a row. Three changes compound in the " + "classification: #273 taught the parser the " + "fullwidth pair, #271 gives the wholly-Han " + "remainder its family-first reading, and #335 " + "makes the marker inside the clause enough on its " + "own. 1.4.0 read first 山田 / middle '花子（旧姓' / " + "last '佐藤）' with the brackets as name text; " + "2.1.0 read given 花子 / family 山田 / nickname " + "'旧姓 佐藤' (both measured 2026-08-26)"), Case("east_slavic", "Сидоров Иван Петрович", {"given": "Иван", "middle": "Петрович", "family": "Сидоров"}, policy=_ES), diff --git a/tests/v2/test_ledger_guards.py b/tests/v2/test_ledger_guards.py index ddb7034e..5347dc15 100644 --- a/tests/v2/test_ledger_guards.py +++ b/tests/v2/test_ledger_guards.py @@ -1260,7 +1260,9 @@ def _claim(rule: dict) -> _Claim: #: both is growth into names the rule genuinely describes. _CORPUS_CLAIMS: dict[str, dict[str, _Claim]] = { "expected_since_1.4.0.toml": { - "fix(#335) a marker-led bracketed clause reads as the maiden name whatever pair encloses it": + "fix(#335) a marker-led clause leaves the one name word its bare reading": + _Claim(1, ('family', 'given', 'maiden', 'nickname'), "c09cc7dba88b"), + "fix(#335) a marker-led bracketed clause reads as the maiden name whatever pair encloses it": _Claim(5, ('maiden', 'nickname'), "a419f74143e3"), "fix(#410) a title and one name word name the family, whatever annotation stands beside it": _Claim(3, ('family', 'given'), "24d6223e472f"), @@ -1299,7 +1301,7 @@ def _claim(rule: dict) -> _Claim: "fix(comma-precomma-family) pre-comma run reads as family, not given": _Claim(279, ('family', 'given'), "28a62b622a48"), "fix(suffix-routing) two-token name with unambiguous trailing suffix stays suffix": - _Claim(1079, ('family', 'given', 'suffix'), "3cc8bbe00b2d"), + _Claim(1080, ('family', 'given', 'suffix'), "0cb2cda1ed6b"), "fix(suffix-delimiter-rendering) no-space delimiter core token kept whole": _Claim(0, ('suffix',), "e3b0c44298fc"), "ambiguous-surname-acronym data change: parenthesized (MA)/(DO) now stays nickname": @@ -1372,7 +1374,9 @@ def _claim(rule: dict) -> _Claim: _Claim(1, ('family', 'given'), "e62caedec864"), }, "expected_since_2.0.0.toml": { - "fix(#335) a marker-led bracketed clause reads as the maiden name whatever pair encloses it": + "fix(#335) a marker-led clause leaves the one name word its bare reading": + _Claim(1, ('family', 'given', 'maiden', 'nickname'), "c09cc7dba88b"), + "fix(#335) a marker-led bracketed clause reads as the maiden name whatever pair encloses it": _Claim(5, ('maiden', 'nickname'), "a419f74143e3"), "fix(#335) a marker-led bracketed clause reads as the maiden name, compounding with the CJK order flip": _Claim(1, ('family', 'given', 'maiden', 'nickname'), "cf370e856ae7"), @@ -1480,7 +1484,9 @@ def _claim(rule: dict) -> _Claim: _Claim(1, ('family', 'maiden'), "2150936a8c55"), }, "expected_since_2.1.0.toml": { - "fix(#335) a marker-led bracketed clause reads as the maiden name whatever pair encloses it": + "fix(#335) a marker-led clause leaves the one name word its bare reading": + _Claim(1, ('family', 'given', 'maiden', 'nickname'), "c09cc7dba88b"), + "fix(#335) a marker-led bracketed clause reads as the maiden name whatever pair encloses it": _Claim(6, ('maiden', 'nickname'), "d0e857deddb2"), "fix(#410) a title and one name word name the family, whatever annotation stands beside it": _Claim(4, ('family', 'given'), "da1dd1473145"), @@ -1830,7 +1836,7 @@ class _Excluded(NamedTuple): ("fix(comma-family)", "fix(comma-precomma-family)", "fix(suffix-routing)")), '(^|[\\w.]\\s+)[("\'][^)"\']+[)"\'](\\s+\\w|\\s*$)': - _Excluded(50, "241623062660", ()), + _Excluded(51, "770738271273", ()), } diff --git a/tests/v2/test_parser.py b/tests/v2/test_parser.py index db2235b0..38645d84 100644 --- a/tests/v2/test_parser.py +++ b/tests/v2/test_parser.py @@ -1335,6 +1335,17 @@ def _clause_free_latin_corpus_names() -> list[str]: for word in name.split())] +def test_the_clause_free_corpus_is_not_empty() -> None: + """The invariant below is parametrized over a FILTERED corpus, and + an empty parametrization passes as a skip rather than failing -- + the shape #329 left behind. The filter has been widened once + already (the delimiter strip, 2026-08-26, which took it from 637 + names to 628), so the floor is what says a future widening emptied + it. Deliberately far below today's count: this asks whether the + filter still selects a corpus, not what the corpus holds.""" + assert len(_clause_free_latin_corpus_names()) > 100 + + @pytest.mark.parametrize("name", _clause_free_latin_corpus_names()) def test_a_maiden_clause_changes_nothing_else(name: str) -> None: """The grouping rules count and join only the words that remain diff --git a/tools/differential/corpus_rules.jsonl b/tools/differential/corpus_rules.jsonl index 8cade004..136ff033 100644 --- a/tools/differential/corpus_rules.jsonl +++ b/tools/differential/corpus_rules.jsonl @@ -134,6 +134,7 @@ "Sir John" "Sir abdul van der Berg" "Sir de Mesnil" +"Smith (née Jones)" "Smith Jr." "Smith Jr., Mr." "Smith, Abd" diff --git a/tools/differential/expected_since_1.4.0.toml b/tools/differential/expected_since_1.4.0.toml index fc875a88..2ae0598c 100644 --- a/tools/differential/expected_since_1.4.0.toml +++ b/tools/differential/expected_since_1.4.0.toml @@ -1425,3 +1425,21 @@ issue = "fix(#335) a marker-led bracketed clause reads as the maiden name whatev # hold it. name_regex = "(?i)^(?:Jane Smith \\(n[ée]e Jones\\)|Jane \\(n[ée]e Jones\\) Smith|Jane Smith \\(n[ée]e V\\)|Anna Müller \\(geb\\. Schmidt\\)|Ursula von der Leyen \\(geb\\. Albrecht\\))$" fields = ["nickname", "maiden"] + +[[change]] +issue = "fix(#335) a marker-led clause leaves the one name word its bare reading" +# 'Smith (née Jones)', which rules.md#N3 carries as an Accepted line. +# N3 reads a name that is only a nickname plus one name word as "that +# word is the family name", and a marker-led clause is not a nickname +# clause, so N3 no longer reaches this shape: the word keeps the +# reading the bare "Smith née Jones" gives it, which is `given`. +# +# Its own rule rather than a sixth alternative in the fix(#335) rule +# above, for the reason that rule states: this diff moves FOUR fields +# where those move two, and one rule holding both would have to +# declare the union, pre-excusing a given/family regression on the +# five. The empty `family` is not this change's -- the bare spelling +# reads that way on 2.0.0 and 2.1.0 too -- but it is inside this +# diff, so the rule declares it. +name_regex = "^Smith \\(n[\u00e9e]e Jones\\)$" +fields = ["given", "family", "nickname", "maiden"] diff --git a/tools/differential/expected_since_2.0.0.toml b/tools/differential/expected_since_2.0.0.toml index be5ba35e..bf85b76a 100644 --- a/tools/differential/expected_since_2.0.0.toml +++ b/tools/differential/expected_since_2.0.0.toml @@ -987,3 +987,21 @@ issue = "fix(#335) a marker-led bracketed clause reads as the maiden name, compo # is a nickname", which is the distinction the fields above turn on. name_regex = "^山田 花子（旧姓 佐藤）$" fields = ["given", "family", "nickname", "maiden"] + +[[change]] +issue = "fix(#335) a marker-led clause leaves the one name word its bare reading" +# 'Smith (née Jones)', which rules.md#N3 carries as an Accepted line. +# N3 reads a name that is only a nickname plus one name word as "that +# word is the family name", and a marker-led clause is not a nickname +# clause, so N3 no longer reaches this shape: the word keeps the +# reading the bare "Smith née Jones" gives it, which is `given`. +# +# Its own rule rather than a sixth alternative in the fix(#335) rule +# above, for the reason that rule states: this diff moves FOUR fields +# where those move two, and one rule holding both would have to +# declare the union, pre-excusing a given/family regression on the +# five. The empty `family` is not this change's -- the bare spelling +# reads that way on 2.0.0 and 2.1.0 too -- but it is inside this +# diff, so the rule declares it. +name_regex = "^Smith \\(n[\u00e9e]e Jones\\)$" +fields = ["given", "family", "nickname", "maiden"] diff --git a/tools/differential/expected_since_2.1.0.toml b/tools/differential/expected_since_2.1.0.toml index 4987c5ea..b23e412c 100644 --- a/tools/differential/expected_since_2.1.0.toml +++ b/tools/differential/expected_since_2.1.0.toml @@ -909,3 +909,21 @@ issue = "fix(#335) a marker-led bracketed clause reads as the maiden name whatev # nickname/maiden confusion in the corpus. name_regex = "(?i)^(?:Jane Smith \\(n[ée]e Jones\\)|Jane \\(n[ée]e Jones\\) Smith|Jane Smith \\(n[ée]e V\\)|Anna Müller \\(geb\\. Schmidt\\)|Ursula von der Leyen \\(geb\\. Albrecht\\)|山田 花子（旧姓 佐藤）)$" fields = ["nickname", "maiden"] + +[[change]] +issue = "fix(#335) a marker-led clause leaves the one name word its bare reading" +# 'Smith (née Jones)', which rules.md#N3 carries as an Accepted line. +# N3 reads a name that is only a nickname plus one name word as "that +# word is the family name", and a marker-led clause is not a nickname +# clause, so N3 no longer reaches this shape: the word keeps the +# reading the bare "Smith née Jones" gives it, which is `given`. +# +# Its own rule rather than a sixth alternative in the fix(#335) rule +# above, for the reason that rule states: this diff moves FOUR fields +# where those move two, and one rule holding both would have to +# declare the union, pre-excusing a given/family regression on the +# five. The empty `family` is not this change's -- the bare spelling +# reads that way on 2.0.0 and 2.1.0 too -- but it is inside this +# diff, so the rule declares it. +name_regex = "^Smith \\(n[\u00e9e]e Jones\\)$" +fields = ["given", "family", "nickname", "maiden"] From f4b9e4e0084c0e74049b966c3a4b72cf5b200144 Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Wed, 26 Aug 2026 10:45:33 -0700 Subject: [PATCH 08/12] docs(customize,usage,release_log): M3 for callers MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The contributor docs recorded M3 the day it landed; the guide a caller actually reads did not, and CI doctests the examples rather than the prose around them, so nothing noticed. customize.rst said "a delimiter pair routes to exactly one field, and maiden_delimiters states the more specific intent". Since M3 the pair is asked second: the content decides first, and only a clause that says nothing about itself is left for the pair to route. Its maiden_delimiters table row used "Jane Smith (née Jones)" as the worked example, which is now the canonical case for NOT needing the knob; it uses the markerless parenthesized surname instead, which is what only a caller can declare. usage.rst's "Delimited content is not always a nickname" section knew about the suffix reading and not the maiden one. It now opens with M3 and its two boundaries, with doctests, and says the suffix reading is taken first. The release note said "bracketed" nine times over parenthesis examples alone. M3 reaches all eleven shipped pairs, and quotes are how a nickname is most often written, so a caller had no way to anticipate that a quoted clause moves too. rules.md's M Background and M1 amendment stated the two-word rule without S1's carve-out, which both now carry. Co-Authored-By: Claude Opus 5 --- docs/customize.rst | 34 +++++++++++++++++++++------------- docs/design/rules.md | 5 +++-- docs/release_log.rst | 2 +- docs/usage.rst | 31 +++++++++++++++++++++++++++++-- 4 files changed, 54 insertions(+), 18 deletions(-) diff --git a/docs/customize.rst b/docs/customize.rst index 6b4e08d7..4a5f6cdd 100644 --- a/docs/customize.rst +++ b/docs/customize.rst @@ -288,15 +288,19 @@ listed below. * - ``maiden_delimiters`` - ``frozenset[tuple[str, str]]`` - Routes content enclosed by these delimiter pairs to ``maiden`` - instead, and drops them from the effective nickname set. A - marker word opening the enclosed content is dropped from the - value, so ``"Jane Smith (née Jones)"`` gives maiden ``Jones`` - — but only where that content holds more than one *token*, - since a lone ``"(Nee)"`` is a maiden name rather than a - marker. Tokens, not words: a marker written against the name - it marks is one token with them, so ``"山田花子（旧姓佐藤）"`` - keeps its ``旧姓``. Defaults to empty — see the routing - example below. + instead, and drops them from the effective nickname set. Set + this for clauses that carry no marker word: since 2.2 a clause + opening with one reads as a maiden name whatever pair encloses + it, so ``"Jane Smith (née Jones)"`` needs no configuration at + all, while ``"Cherice J. (Johnson) Williams"`` — the + parenthesized birth surname written bare — is what only you + can declare. A marker word opening the enclosed content is + dropped from the value either way, but only where that content + holds more than one *token*, since a lone ``"(Nee)"`` is a + maiden name rather than a marker. Tokens, not words: a marker + written against the name it marks is one token with them, so + ``"山田花子（旧姓佐藤）"`` keeps its ``旧姓``. Defaults to empty — + see the routing example below. * - ``extra_suffix_delimiters`` - ``frozenset[str]`` - Adds separators that split suffix groups, e.g. ``" - "`` for @@ -496,10 +500,14 @@ off. Nicknames, maiden names, and brackets ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -A delimiter pair routes to exactly one field, and ``maiden_delimiters`` -states the more specific intent — so listing a pair there drops it from -the effective ``nickname_delimiters`` set automatically, and the -one-liner is the whole recipe: +A delimiter pair carries no meaning of its own, so what a clause reads +as is settled in two steps. The content is asked first: a clause that +opens with a recognized maiden marker and carries a name word after it +is a maiden name whatever encloses it, and needs nothing configured. +Only for the rest does the PAIR decide, and that is what this knob is +for — listing a pair here drops it from the effective +``nickname_delimiters`` set automatically, and the one-liner is the +whole recipe: .. doctest:: diff --git a/docs/design/rules.md b/docs/design/rules.md index 48bb0590..31e12570 100644 --- a/docs/design/rules.md +++ b/docs/design/rules.md @@ -536,7 +536,7 @@ N3. Rationale: a person set down as a nickname plus one name word is ## Maiden names (M) -Background: a maiden name is written beside the current name, set off by a marker word or by enclosure. Markers are attested across French née/né, German geb./geborene, Dutch geboren, Czech/Slovak rozená (the abbreviation roz. shipped through 2.1 and was removed in 2.2 -- it collides with the English diminutive Roz, and a caller who needs it adds it to their own Lexicon), Scandinavian født/fødd/född, Russian урожд. (both ё and е spellings), and Japanese 旧姓 — both grammatical genders where attested. Japanese more often writes the marker with a fullwidth colon (旧姓：佐藤), which is no separator, so marker and name arrive as a single word. Which enclosures mean "maiden" rather than "nickname" is a caller convention, so the maiden reading of a delimiter pair is opt-in — except where the clause announces itself. A clause of two words or more led by a marker word has said which convention it means, and reads as the maiden name inside a nickname pair as well (M3); a lone marker word has not, and neither has one the colon spelling above glues to the name. +Background: a maiden name is written beside the current name, set off by a marker word or by enclosure. Markers are attested across French née/né, German geb./geborene, Dutch geboren, Czech/Slovak rozená (the abbreviation roz. shipped through 2.1 and was removed in 2.2 -- it collides with the English diminutive Roz, and a caller who needs it adds it to their own Lexicon), Scandinavian født/fødd/född, Russian урожд. (both ё and е spellings), and Japanese 旧姓 — both grammatical genders where attested. Japanese more often writes the marker with a fullwidth colon (旧姓：佐藤), which is no separator, so marker and name arrive as a single word. Which enclosures mean "maiden" rather than "nickname" is a caller convention, so the maiden reading of a delimiter pair is opt-in — except where the clause announces itself. A clause of two words or more led by a marker word has said which convention it means, and reads as the maiden name inside a nickname pair as well (M3) — unless its content is suffix-shaped, which S1 takes ahead of both. A lone marker word has said nothing, and neither has one the colon spelling above glues to the name. M1. Rationale: an enclosure the caller has declared to mean maiden holds the former family name; a recognized marker word inside it @@ -552,7 +552,8 @@ M1. Rationale: an enclosure the caller has declared to mean maiden what this rule needs for a clause that does not announce itself — markerless content, and a lone marker word alike; a clause of two words or more led by a recognized marker reads as the maiden - name inside a nickname pair as well (M3). + name inside a nickname pair as well (M3), the suffix-shaped + content S1 takes excepted there as it is here. "Jane Smith (née Jones)" maiden-parens → maiden="Jones" "Jane Smith (Nee)" maiden-parens → maiden="Nee" · boundary "Jane Smith (Nee) (Jones)" maiden-parens → maiden="Nee Jones" diff --git a/docs/release_log.rst b/docs/release_log.rst index cad35d7c..440475bc 100644 --- a/docs/release_log.rst +++ b/docs/release_log.rst @@ -47,7 +47,7 @@ Release Log - Remove the Czech/Slovak abbreviation ``roz.`` from the default maiden markers, where it had been eating English given names. Marker matching is whole-token, case-folded and period-insensitive, so ``Roz`` -- the ordinary diminutive of Rosalind -- was the same string as the marker, and a maiden marker takes every word after it: ``"Rosalind Roz Smith"`` read maiden ``Smith`` with **no family name at all**, and ``"Rosalind Roz Jones Smith"`` read maiden ``Jones Smith``. Both now read the way 1.4.0 read them, given ``Rosalind`` with ``Roz`` a middle name and ``Smith`` the family. The defect is as old as the marker vocabulary and belongs to the bare marker rule, not to the bracketed-clause change in the bullet below; it was found while reviewing that change, which would have widened it -- with ``roz`` still shipped, ``"Jane Smith (Roz Jones)"`` reads maiden ``Jones`` where 2.1.0 read nickname ``Roz Jones``. What stops working is the abbreviation itself, for Czech and Slovak callers: ``"Anna Nováková roz. Svobodová"`` now reads middle ``Nováková roz.``, family ``Svobodová`` rather than family ``Nováková``, maiden ``Svobodová`` -- which is also how 1.4.0 read it, so this is a return to the v1 reading rather than a new one. The full participle is untouched and needs no configuration: ``"Anna Nováková rozená Svobodová"`` still reads family ``Nováková``, maiden ``Svobodová``, and it is only the abbreviation that was ambiguous. A caller who wants the abbreviation back adds it to their own lexicon -- ``Parser(lexicon=dataclasses.replace(lex, maiden_markers=lex.maiden_markers | {"roz"}))`` -- which restores the old reading exactly. The default set is now 16 entries. No differential corpus name contains the string in any case or position, so nothing moves against any baseline (closes the collision found in #335's review) - - Fix a bracketed maiden clause reading as a nickname because its brackets were not declared. ``"Jane Smith nee Jones"`` gave maiden ``Jones``, while ``"Jane Smith (née Jones)"`` gave nickname ``née Jones`` -- the parenthesis pair is a nickname delimiter by default and nothing looked inside it, so the marker word the writer put there was read as part of an alias. A bracketed clause whose content opens with a recognized marker word and carries a word after it now reads as the maiden name whichever bucket its pair sits in: ``"Jane Smith (née Jones)"`` reads family ``Smith``, maiden ``Jones``, and so does the Japanese ``"山田 花子（旧姓 佐藤）"`` (family 山田, given 花子, maiden 佐藤), which needed ``Policy(maiden_delimiters=...)`` through 2.1. The clause is lifted out as a region rather than having its brackets erased, which is what keeps an INTERIOR clause from eating the name behind it: ``"Jane (née Jones) Smith"`` reads given ``Jane``, family ``Smith``, maiden ``Jones``, where erasing the brackets would leave the bare ``"Jane née Jones Smith"``, whose marker takes every word to the end of the name -- maiden ``Jones Smith`` and no family at all. The closing bracket is a boundary the writer typed, and it is now where the maiden name stops. Two clauses beside each other no longer merge: ``"Jane "Janey" Smith (née Jones)"`` read nickname ``Janey née Jones`` through 2.1 and now reads nickname ``Janey``, maiden ``Jones``, each clause keeping its own role, and this reaches ``HumanName``. What does NOT move is a clause that says nothing about maiden names. A lone marker in brackets is just a word in brackets -- ``"Jane Smith (née)"`` stays nickname ``née``, as 1.4.0 read it -- because a marker with no name after it marks nothing, and because a one-word clause may be the surname it looks like (``Nee`` is Irish Ní/Nee and a Chinese romanization). A markerless clause stays a nickname too: ``"Cherice J. (Johnson) Williams"`` keeps nickname ``Johnson``, which is what ``Policy(maiden_delimiters=...)`` remains for -- the parenthesized birth surname is a real convention, and only a caller who knows the data can say that is what these brackets mean. One accepted consequence: the word after the marker is not tested for being a name word, so a bracketed clause does not stop at a credential where the bare form does -- ``"Jane Smith (née V)"`` reads maiden ``V`` while ``"Jane Smith née V"`` reads suffix ``V``. The clause goes to ``maiden`` whole and the bracket is its boundary, so the trailing-suffix rule never sees the word as trailing the name. One more shape moves with it, and it is the one to know about if you parse names that are a surname and a clause: a name that is ONLY a marker clause and one name word now reads that word as the given name rather than the family. ``"Smith (née Jones)"`` reads given ``Smith``, maiden ``Jones`` where every earlier version read family ``Smith``, nickname ``née Jones``. That is the bracketed form agreeing with the bare one -- ``"Smith née Jones"`` has read given ``Smith`` since 1.4.0 -- because a marker-led clause is not a nickname clause and so does not reach the rule that would have made the word a family name. The empty ``family`` it leaves behind is the bare form's behavior and is not new here. Seven of the 1,080 differential corpus names move against the 2.1.0 and 2.0.0 baselines and six against 1.4.0: the Japanese name is the odd one, and against 1.4.0 it is not a new diff at all, since 1.4 read the fullwidth brackets as name text and had no nickname there to change hands. Eight corpus names have the shape and seven move (measured); the eighth is ``"Jane Smith (née Jr.)"``, which the suffix rule takes first. So the count says how few bracketed marker clauses the corpora hold rather than how narrow the change is -- six of the seven are hand-written rule or issue examples, only the Japanese name arriving from elsewhere, and the coverage that bounds the behavior is ``tests/v2/cases.py`` (closes #335) + - Fix a bracketed maiden clause reading as a nickname because its brackets were not declared. ``"Jane Smith nee Jones"`` gave maiden ``Jones``, while ``"Jane Smith (née Jones)"`` gave nickname ``née Jones`` -- the parenthesis pair is a nickname delimiter by default and nothing looked inside it, so the marker word the writer put there was read as part of an alias. This is about every delimiter pair the parser ships, not only the parentheses these examples use: quotes are how a nickname is most often written, and ``"Jane Smith "née Jones""`` and ``"Jane Smith 'née Jones'"`` move from nickname to maiden exactly as the parenthesized spelling does. A bracketed clause whose content opens with a recognized marker word and carries a word after it now reads as the maiden name whichever bucket its pair sits in: ``"Jane Smith (née Jones)"`` reads family ``Smith``, maiden ``Jones``, and so does the Japanese ``"山田 花子（旧姓 佐藤）"`` (family 山田, given 花子, maiden 佐藤), which needed ``Policy(maiden_delimiters=...)`` through 2.1. The clause is lifted out as a region rather than having its brackets erased, which is what keeps an INTERIOR clause from eating the name behind it: ``"Jane (née Jones) Smith"`` reads given ``Jane``, family ``Smith``, maiden ``Jones``, where erasing the brackets would leave the bare ``"Jane née Jones Smith"``, whose marker takes every word to the end of the name -- maiden ``Jones Smith`` and no family at all. The closing bracket is a boundary the writer typed, and it is now where the maiden name stops. Two clauses beside each other no longer merge: ``"Jane "Janey" Smith (née Jones)"`` read nickname ``Janey née Jones`` through 2.1 and now reads nickname ``Janey``, maiden ``Jones``, each clause keeping its own role, and this reaches ``HumanName``. What does NOT move is a clause that says nothing about maiden names. A lone marker in brackets is just a word in brackets -- ``"Jane Smith (née)"`` stays nickname ``née``, as 1.4.0 read it -- because a marker with no name after it marks nothing, and because a one-word clause may be the surname it looks like (``Nee`` is Irish Ní/Nee and a Chinese romanization). A markerless clause stays a nickname too: ``"Cherice J. (Johnson) Williams"`` keeps nickname ``Johnson``, which is what ``Policy(maiden_delimiters=...)`` remains for -- the parenthesized birth surname is a real convention, and only a caller who knows the data can say that is what these brackets mean. One accepted consequence: the word after the marker is not tested for being a name word, so a bracketed clause does not stop at a credential where the bare form does -- ``"Jane Smith (née V)"`` reads maiden ``V`` while ``"Jane Smith née V"`` reads suffix ``V``. The clause goes to ``maiden`` whole and the bracket is its boundary, so the trailing-suffix rule never sees the word as trailing the name. One more shape moves with it, and it is the one to know about if you parse names that are a surname and a clause: a name that is ONLY a marker clause and one name word now reads that word as the given name rather than the family. ``"Smith (née Jones)"`` reads given ``Smith``, maiden ``Jones`` where every earlier version read family ``Smith``, nickname ``née Jones``. That is the bracketed form agreeing with the bare one -- ``"Smith née Jones"`` has read given ``Smith`` since 1.4.0 -- because a marker-led clause is not a nickname clause and so does not reach the rule that would have made the word a family name. The empty ``family`` it leaves behind is the bare form's behavior and is not new here. Seven of the 1,080 differential corpus names move against the 2.1.0 and 2.0.0 baselines and six against 1.4.0: the Japanese name is the odd one, and against 1.4.0 it is not a new diff at all, since 1.4 read the fullwidth brackets as name text and had no nickname there to change hands. Eight corpus names have the shape and seven move (measured); the eighth is ``"Jane Smith (née Jr.)"``, which the suffix rule takes first. So the count says how few bracketed marker clauses the corpora hold rather than how narrow the change is -- six of the seven are hand-written rule or issue examples, only the Japanese name arriving from elsewhere, and the coverage that bounds the behavior is ``tests/v2/cases.py`` (closes #335) - Fix a particle chain and a maiden name taking a trailing generational numeral as a name word: ``"John van der Berg V"`` read family ``van der Berg V`` and ``"John née Jones Smith V"`` read maiden ``Jones Smith V``, where ``"John Smith V"`` reads suffix ``V``. Both stops asked "is this a suffix?" with the test that vetoes a bare ``V`` as an initial -- the same question the bound given-name join asked until #401 -- and now ask the suffix rule's own walk, over the name as they would leave it: family ``van der Berg``, maiden ``Jones Smith``, suffix ``V``, for ``I`` and ``X`` alike; ``"John van der J. V"`` and ``"J. née Jones Smith V"`` keep their readings, the word before the numeral being an initial as written or as left. The chain stops before a bare credential with words to spare too, as ``"John Smith Ma"`` does, where the suffix rule will still find them once the chain has run: ``"John van der Berg Ma"`` reads suffix ``Ma``, which is how 1.4.0 read it, while ``"Freiherr von Berg MA"`` keeps family ``von Berg MA``; a maiden name keeps such a credential, since the words the marker takes are the very words that made it one. A particle that is also suffix vocabulary now ends the chain where it opens the trailing run: ``"John Smith Mc V"`` reads suffix ``Mc, V``. The chain also no longer swallows the given name behind an unlisted abbreviation: ``"Xyz. van Johnson"`` and ``"Esq. van Gogh"`` read given ``van``, as ``"Dr. van Johnson"`` has since #367, the chain and the bound given-name join now asking the same test that reads the abbreviation as a title (closes #424) diff --git a/docs/usage.rst b/docs/usage.rst index 5aedd40e..8a8f68ab 100644 --- a/docs/usage.rst +++ b/docs/usage.rst @@ -655,9 +655,36 @@ instead of relying on the default: >>> parse(text).maiden 'Jones' -Delimited content is not always a nickname. If what's inside is a known +Delimited content is not always a nickname. If it opens with a marker +word and has a name word after it, the clause is a maiden name, whatever +pair encloses it and with nothing configured — the clause has said which +convention it means, so you do not have to: + +.. doctest:: + + >>> parse("Jane Smith (née Jones)").maiden + 'Jones' + >>> parse('Jane Smith "née Jones"').maiden + 'Jones' + >>> parse("Jane (née Jones) Smith").family + 'Smith' + +A marker with no name after it is just a word in brackets, and a clause +with no marker at all stays a nickname — the parenthesized birth surname +is a real convention, but nothing in the clause says so, and only you +can declare that with ``maiden_delimiters`` (see :doc:`customize`): + +.. doctest:: + + >>> parse("Jane Smith (née)").nickname + 'née' + >>> parse("Cherice J. (Johnson) Williams").nickname + 'Johnson' + +If what's inside is a known suffix, or simply ends in a period, it is read as a suffix instead — -parenthesized credentials and retired ranks are far more common than +that reading is taken before the maiden one, and parenthesized +credentials and retired ranks are far more common than parenthesized nicknames that happen to be credentials: .. doctest:: From ea0bbe8f3ff9309eb2cea42c66dab69b97abd207 Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Wed, 26 Aug 2026 10:55:30 -0700 Subject: [PATCH 09/12] docs(rules,release_log): point the one-word maiden shape at #445 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit N3's Accepted block records that a marker-led clause is not a nickname clause, so the rule does not reach `Smith (née Jones)` and the family comes back empty. Whether that empty family is right is a separate question: 1.4.0 kept it, the emptying arrived with #274's marker consumption in 2.0, and the same shape has been repaired twice since -- at H1 (#410) and at P5's reserve (#411). #445 carries the third. Co-Authored-By: Claude Opus 5 --- docs/design/rules.md | 6 +++++- docs/release_log.rst | 2 +- 2 files changed, 6 insertions(+), 2 deletions(-) diff --git a/docs/design/rules.md b/docs/design/rules.md index 31e12570..93b569a7 100644 --- a/docs/design/rules.md +++ b/docs/design/rules.md @@ -530,7 +530,11 @@ N3. Rationale: a person set down as a nickname plus one name word is Accepted: a marker-led clause is a maiden clause and not a nickname one (M3), so this rule does not reach a name written that way, and the one name word keeps the reading the bare - spelling gives it rather than becoming the family name. + spelling gives it rather than becoming the family name. The + empty family that leaves is the bare spelling's, and #445 tracks + whether it is right — 1.4.0 kept the family there, and the same + empty family has been repaired twice already, at H1 (#410) and + at P5's reserve (#411). "Smith (née Jones)" → given="Smith" history: decisions.md#N3 · interacts: H1, M3 · implemented: nameparser/_pipeline/_assign.py diff --git a/docs/release_log.rst b/docs/release_log.rst index 440475bc..91a1bfbc 100644 --- a/docs/release_log.rst +++ b/docs/release_log.rst @@ -47,7 +47,7 @@ Release Log - Remove the Czech/Slovak abbreviation ``roz.`` from the default maiden markers, where it had been eating English given names. Marker matching is whole-token, case-folded and period-insensitive, so ``Roz`` -- the ordinary diminutive of Rosalind -- was the same string as the marker, and a maiden marker takes every word after it: ``"Rosalind Roz Smith"`` read maiden ``Smith`` with **no family name at all**, and ``"Rosalind Roz Jones Smith"`` read maiden ``Jones Smith``. Both now read the way 1.4.0 read them, given ``Rosalind`` with ``Roz`` a middle name and ``Smith`` the family. The defect is as old as the marker vocabulary and belongs to the bare marker rule, not to the bracketed-clause change in the bullet below; it was found while reviewing that change, which would have widened it -- with ``roz`` still shipped, ``"Jane Smith (Roz Jones)"`` reads maiden ``Jones`` where 2.1.0 read nickname ``Roz Jones``. What stops working is the abbreviation itself, for Czech and Slovak callers: ``"Anna Nováková roz. Svobodová"`` now reads middle ``Nováková roz.``, family ``Svobodová`` rather than family ``Nováková``, maiden ``Svobodová`` -- which is also how 1.4.0 read it, so this is a return to the v1 reading rather than a new one. The full participle is untouched and needs no configuration: ``"Anna Nováková rozená Svobodová"`` still reads family ``Nováková``, maiden ``Svobodová``, and it is only the abbreviation that was ambiguous. A caller who wants the abbreviation back adds it to their own lexicon -- ``Parser(lexicon=dataclasses.replace(lex, maiden_markers=lex.maiden_markers | {"roz"}))`` -- which restores the old reading exactly. The default set is now 16 entries. No differential corpus name contains the string in any case or position, so nothing moves against any baseline (closes the collision found in #335's review) - - Fix a bracketed maiden clause reading as a nickname because its brackets were not declared. ``"Jane Smith nee Jones"`` gave maiden ``Jones``, while ``"Jane Smith (née Jones)"`` gave nickname ``née Jones`` -- the parenthesis pair is a nickname delimiter by default and nothing looked inside it, so the marker word the writer put there was read as part of an alias. This is about every delimiter pair the parser ships, not only the parentheses these examples use: quotes are how a nickname is most often written, and ``"Jane Smith "née Jones""`` and ``"Jane Smith 'née Jones'"`` move from nickname to maiden exactly as the parenthesized spelling does. A bracketed clause whose content opens with a recognized marker word and carries a word after it now reads as the maiden name whichever bucket its pair sits in: ``"Jane Smith (née Jones)"`` reads family ``Smith``, maiden ``Jones``, and so does the Japanese ``"山田 花子（旧姓 佐藤）"`` (family 山田, given 花子, maiden 佐藤), which needed ``Policy(maiden_delimiters=...)`` through 2.1. The clause is lifted out as a region rather than having its brackets erased, which is what keeps an INTERIOR clause from eating the name behind it: ``"Jane (née Jones) Smith"`` reads given ``Jane``, family ``Smith``, maiden ``Jones``, where erasing the brackets would leave the bare ``"Jane née Jones Smith"``, whose marker takes every word to the end of the name -- maiden ``Jones Smith`` and no family at all. The closing bracket is a boundary the writer typed, and it is now where the maiden name stops. Two clauses beside each other no longer merge: ``"Jane "Janey" Smith (née Jones)"`` read nickname ``Janey née Jones`` through 2.1 and now reads nickname ``Janey``, maiden ``Jones``, each clause keeping its own role, and this reaches ``HumanName``. What does NOT move is a clause that says nothing about maiden names. A lone marker in brackets is just a word in brackets -- ``"Jane Smith (née)"`` stays nickname ``née``, as 1.4.0 read it -- because a marker with no name after it marks nothing, and because a one-word clause may be the surname it looks like (``Nee`` is Irish Ní/Nee and a Chinese romanization). A markerless clause stays a nickname too: ``"Cherice J. (Johnson) Williams"`` keeps nickname ``Johnson``, which is what ``Policy(maiden_delimiters=...)`` remains for -- the parenthesized birth surname is a real convention, and only a caller who knows the data can say that is what these brackets mean. One accepted consequence: the word after the marker is not tested for being a name word, so a bracketed clause does not stop at a credential where the bare form does -- ``"Jane Smith (née V)"`` reads maiden ``V`` while ``"Jane Smith née V"`` reads suffix ``V``. The clause goes to ``maiden`` whole and the bracket is its boundary, so the trailing-suffix rule never sees the word as trailing the name. One more shape moves with it, and it is the one to know about if you parse names that are a surname and a clause: a name that is ONLY a marker clause and one name word now reads that word as the given name rather than the family. ``"Smith (née Jones)"`` reads given ``Smith``, maiden ``Jones`` where every earlier version read family ``Smith``, nickname ``née Jones``. That is the bracketed form agreeing with the bare one -- ``"Smith née Jones"`` has read given ``Smith`` since 1.4.0 -- because a marker-led clause is not a nickname clause and so does not reach the rule that would have made the word a family name. The empty ``family`` it leaves behind is the bare form's behavior and is not new here. Seven of the 1,080 differential corpus names move against the 2.1.0 and 2.0.0 baselines and six against 1.4.0: the Japanese name is the odd one, and against 1.4.0 it is not a new diff at all, since 1.4 read the fullwidth brackets as name text and had no nickname there to change hands. Eight corpus names have the shape and seven move (measured); the eighth is ``"Jane Smith (née Jr.)"``, which the suffix rule takes first. So the count says how few bracketed marker clauses the corpora hold rather than how narrow the change is -- six of the seven are hand-written rule or issue examples, only the Japanese name arriving from elsewhere, and the coverage that bounds the behavior is ``tests/v2/cases.py`` (closes #335) + - Fix a bracketed maiden clause reading as a nickname because its brackets were not declared. ``"Jane Smith nee Jones"`` gave maiden ``Jones``, while ``"Jane Smith (née Jones)"`` gave nickname ``née Jones`` -- the parenthesis pair is a nickname delimiter by default and nothing looked inside it, so the marker word the writer put there was read as part of an alias. This is about every delimiter pair the parser ships, not only the parentheses these examples use: quotes are how a nickname is most often written, and ``"Jane Smith "née Jones""`` and ``"Jane Smith 'née Jones'"`` move from nickname to maiden exactly as the parenthesized spelling does. A bracketed clause whose content opens with a recognized marker word and carries a word after it now reads as the maiden name whichever bucket its pair sits in: ``"Jane Smith (née Jones)"`` reads family ``Smith``, maiden ``Jones``, and so does the Japanese ``"山田 花子（旧姓 佐藤）"`` (family 山田, given 花子, maiden 佐藤), which needed ``Policy(maiden_delimiters=...)`` through 2.1. The clause is lifted out as a region rather than having its brackets erased, which is what keeps an INTERIOR clause from eating the name behind it: ``"Jane (née Jones) Smith"`` reads given ``Jane``, family ``Smith``, maiden ``Jones``, where erasing the brackets would leave the bare ``"Jane née Jones Smith"``, whose marker takes every word to the end of the name -- maiden ``Jones Smith`` and no family at all. The closing bracket is a boundary the writer typed, and it is now where the maiden name stops. Two clauses beside each other no longer merge: ``"Jane "Janey" Smith (née Jones)"`` read nickname ``Janey née Jones`` through 2.1 and now reads nickname ``Janey``, maiden ``Jones``, each clause keeping its own role, and this reaches ``HumanName``. What does NOT move is a clause that says nothing about maiden names. A lone marker in brackets is just a word in brackets -- ``"Jane Smith (née)"`` stays nickname ``née``, as 1.4.0 read it -- because a marker with no name after it marks nothing, and because a one-word clause may be the surname it looks like (``Nee`` is Irish Ní/Nee and a Chinese romanization). A markerless clause stays a nickname too: ``"Cherice J. (Johnson) Williams"`` keeps nickname ``Johnson``, which is what ``Policy(maiden_delimiters=...)`` remains for -- the parenthesized birth surname is a real convention, and only a caller who knows the data can say that is what these brackets mean. One accepted consequence: the word after the marker is not tested for being a name word, so a bracketed clause does not stop at a credential where the bare form does -- ``"Jane Smith (née V)"`` reads maiden ``V`` while ``"Jane Smith née V"`` reads suffix ``V``. The clause goes to ``maiden`` whole and the bracket is its boundary, so the trailing-suffix rule never sees the word as trailing the name. One more shape moves with it, and it is the one to know about if you parse names that are a surname and a clause: a name that is ONLY a marker clause and one name word now reads that word as the given name rather than the family. ``"Smith (née Jones)"`` reads given ``Smith``, maiden ``Jones`` where every earlier version read family ``Smith``, nickname ``née Jones``. That is the bracketed form agreeing with the bare one -- ``"Smith née Jones"`` has read given ``Smith`` since 1.4.0 -- because a marker-led clause is not a nickname clause and so does not reach the rule that would have made the word a family name. The empty ``family`` it leaves behind is the bare form's behavior and is not new here, though it is not 1.4.0's either -- v1 read ``"Smith née Jones"`` as first ``Smith``, middle ``née``, last ``Jones``, and #445 tracks whether the family should be kept. Seven of the 1,080 differential corpus names move against the 2.1.0 and 2.0.0 baselines and six against 1.4.0: the Japanese name is the odd one, and against 1.4.0 it is not a new diff at all, since 1.4 read the fullwidth brackets as name text and had no nickname there to change hands. Eight corpus names have the shape and seven move (measured); the eighth is ``"Jane Smith (née Jr.)"``, which the suffix rule takes first. So the count says how few bracketed marker clauses the corpora hold rather than how narrow the change is -- six of the seven are hand-written rule or issue examples, only the Japanese name arriving from elsewhere, and the coverage that bounds the behavior is ``tests/v2/cases.py`` (closes #335) - Fix a particle chain and a maiden name taking a trailing generational numeral as a name word: ``"John van der Berg V"`` read family ``van der Berg V`` and ``"John née Jones Smith V"`` read maiden ``Jones Smith V``, where ``"John Smith V"`` reads suffix ``V``. Both stops asked "is this a suffix?" with the test that vetoes a bare ``V`` as an initial -- the same question the bound given-name join asked until #401 -- and now ask the suffix rule's own walk, over the name as they would leave it: family ``van der Berg``, maiden ``Jones Smith``, suffix ``V``, for ``I`` and ``X`` alike; ``"John van der J. V"`` and ``"J. née Jones Smith V"`` keep their readings, the word before the numeral being an initial as written or as left. The chain stops before a bare credential with words to spare too, as ``"John Smith Ma"`` does, where the suffix rule will still find them once the chain has run: ``"John van der Berg Ma"`` reads suffix ``Ma``, which is how 1.4.0 read it, while ``"Freiherr von Berg MA"`` keeps family ``von Berg MA``; a maiden name keeps such a credential, since the words the marker takes are the very words that made it one. A particle that is also suffix vocabulary now ends the chain where it opens the trailing run: ``"John Smith Mc V"`` reads suffix ``Mc, V``. The chain also no longer swallows the given name behind an unlisted abbreviation: ``"Xyz. van Johnson"`` and ``"Esq. van Gogh"`` read given ``van``, as ``"Dr. van Johnson"`` has since #367, the chain and the bound given-name join now asking the same test that reads the abbreviation as a title (closes #424) From e801ce7da9e280fe66611d3d6b872ae5a732f0b4 Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Wed, 26 Aug 2026 13:13:27 -0700 Subject: [PATCH 10/12] docs(cases,differential,customize): the toolkit round on the corrections MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit One cause under most of this: 7a611a0 corrected a batch of corpus counts and 7355de6 then added a corpus row, falsifying six of them two commits later. A corpus row is a global variable and only the machine-checked constants fail loudly, so every count here is recounted and the three that quantify over the corpus now carry the one-liner that recomputes them, the way tools/differential/README.md does. The filter turns away TEN names, not nine, six of them carrying a maiden clause; it went from 638 names to 628, not 637; eight names depended on the substring branch before the delimiter strip, not seven. Three claims about mutations did not hold. The quote-pair row said gating the swap to '(' and the double quote leaves the suite green -- its own sibling 270 lines below, added in the same commit, fails. It said nine of eleven pairs have no row; three pairs are exercised and eight are not. The fullwidth row said only 1.4.0 reddens; 2.0.0 reddens too, on the four-field rule it has for that name. And the markerless row said dropping the vocabulary test reads maiden 'Johnson' -- it reads 'Mary Johnson', because #329's drop is gated on the first token carrying vocab:maiden-marker and Mary does not. Smaller: the S1 row is the other three fields, not four; 1.4.0 read the bare one-word shape with middle 'née', which is why the sibling rule declares middle; the 2.1.0 ledger's rule carries six alternatives where its siblings carry five; nee appears in two corpus names; and an exclusion comment naming test_ledger_guards as its authority said 46 where that test pins 51 -- self-falsifying rather than merely stale. customize.rst's maiden_delimiters row made the exact error decisions.md records having made once already: markerless versus marker-led, when the line is one word versus a marker plus a word. A caller with '(Nee)' data would have concluded they need no configuration; measured, the default gives nickname there. Both customize.rst paragraphs now state the second-word condition and S1's carve-out, and neither says "name word" -- M3 does not test for one. The three new ledger blocks gain a revisit marker: they declare `family`, and `family` is what #445 would change, so that fix would land inside a rule written to explain something else. The corpus spelling 'Cherice J. (Johnson) Williams' gets its row back beside the two-word one that fences the vocabulary condition. Co-Authored-By: Claude Opus 5 --- docs/customize.rst | 37 ++++++++------ docs/design/rules.md | 7 +-- docs/usage.rst | 2 +- nameparser/_pipeline/_extract.py | 4 +- nameparser/config/maiden_markers.py | 5 +- tests/v2/cases.py | 52 ++++++++++++++------ tests/v2/test_ledger_guards.py | 29 ++++++++--- tests/v2/test_parser.py | 38 ++++++++++---- tools/differential/README.md | 7 +-- tools/differential/expected_since_1.4.0.toml | 16 ++++-- tools/differential/expected_since_2.0.0.toml | 8 ++- tools/differential/expected_since_2.1.0.toml | 16 ++++-- 12 files changed, 155 insertions(+), 66 deletions(-) diff --git a/docs/customize.rst b/docs/customize.rst index 4a5f6cdd..0636d823 100644 --- a/docs/customize.rst +++ b/docs/customize.rst @@ -289,16 +289,21 @@ listed below. - ``frozenset[tuple[str, str]]`` - Routes content enclosed by these delimiter pairs to ``maiden`` instead, and drops them from the effective nickname set. Set - this for clauses that carry no marker word: since 2.2 a clause - opening with one reads as a maiden name whatever pair encloses - it, so ``"Jane Smith (née Jones)"`` needs no configuration at - all, while ``"Cherice J. (Johnson) Williams"`` — the - parenthesized birth surname written bare — is what only you - can declare. A marker word opening the enclosed content is - dropped from the value either way, but only where that content - holds more than one *token*, since a lone ``"(Nee)"`` is a - maiden name rather than a marker. Tokens, not words: a marker - written against the name it marks is one token with them, so + this for a clause that says nothing about itself, which is two + kinds of clause and not one: content with no marker word in it + (``"Cherice J. (Johnson) Williams"``, the parenthesized birth + surname written bare) AND a lone marker word + (``"Jane Smith (Nee)"``, which reads nickname ``Nee`` by default + and maiden ``Nee`` only with the pair listed here). What needs + no configuration since 2.2 is a clause that opens with a marker + word AND has a word after it: ``"Jane Smith (née Jones)"`` reads + maiden ``Jones`` whatever pair encloses it, unless the content is + suffix-shaped, which is read as a suffix ahead of both. A marker + word opening the enclosed content is dropped from the value + either way, but only where that content holds more than one + *token* — the same reason a lone ``"(Nee)"`` is a maiden name + rather than a marker. Tokens, not words: a marker written + against the name it marks is one token with them, so ``"山田花子（旧姓佐藤）"`` keeps its ``旧姓``. Defaults to empty — see the routing example below. * - ``extra_suffix_delimiters`` @@ -501,11 +506,13 @@ Nicknames, maiden names, and brackets ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ A delimiter pair carries no meaning of its own, so what a clause reads -as is settled in two steps. The content is asked first: a clause that -opens with a recognized maiden marker and carries a name word after it -is a maiden name whatever encloses it, and needs nothing configured. -Only for the rest does the PAIR decide, and that is what this knob is -for — listing a pair here drops it from the effective +as is settled in steps. Suffix-shaped content is taken first and reads +as a suffix. Then the content is asked whether it announces itself: a +clause opening with a recognized maiden marker and carrying a word +after it is a maiden name whatever encloses it, and needs nothing +configured. Only for what is left — markerless content, and a lone +marker word — does the PAIR decide, and that is what this knob is for. +Listing a pair here drops it from the effective ``nickname_delimiters`` set automatically, and the one-liner is the whole recipe: diff --git a/docs/design/rules.md b/docs/design/rules.md index 93b569a7..4648dc80 100644 --- a/docs/design/rules.md +++ b/docs/design/rules.md @@ -532,9 +532,10 @@ N3. Rationale: a person set down as a nickname plus one name word is that way, and the one name word keeps the reading the bare spelling gives it rather than becoming the family name. The empty family that leaves is the bare spelling's, and #445 tracks - whether it is right — 1.4.0 kept the family there, and the same - empty family has been repaired twice already, at H1 (#410) and - at P5's reserve (#411). + whether it is right — 1.4.0 read the bare spelling as first + Smith, middle née, last Jones, emptying nothing, and the same + empty family has been repaired twice already, at H1 (#410) and at + P5's reserve (#411). "Smith (née Jones)" → given="Smith" history: decisions.md#N3 · interacts: H1, M3 · implemented: nameparser/_pipeline/_assign.py diff --git a/docs/usage.rst b/docs/usage.rst index 8a8f68ab..13017133 100644 --- a/docs/usage.rst +++ b/docs/usage.rst @@ -656,7 +656,7 @@ instead of relying on the default: 'Jones' Delimited content is not always a nickname. If it opens with a marker -word and has a name word after it, the clause is a maiden name, whatever +word and has a word after it, the clause is a maiden name, whatever pair encloses it and with nothing configured — the clause has said which convention it means, so you do not have to: diff --git a/nameparser/_pipeline/_extract.py b/nameparser/_pipeline/_extract.py index f4e1b3c0..3fa3227c 100644 --- a/nameparser/_pipeline/_extract.py +++ b/nameparser/_pipeline/_extract.py @@ -233,8 +233,8 @@ def extract_delimited(state: ParseState) -> ParseState: # removal: `order` above holds exactly two roles, so a # role that is not NICKNAME is already MAIDEN and the # assignment would be a no-op either way. It is kept - # for the day - # `order` gains a third bucket, when it becomes the + # for the day `order` gains a third bucket, when it + # becomes the # difference between M3 claiming that bucket's clauses # and leaving them. Measured 2026-08-26: dropping it # leaves the suite and all three gates green. diff --git a/nameparser/config/maiden_markers.py b/nameparser/config/maiden_markers.py index 26ee225b..e9d26265 100644 --- a/nameparser/config/maiden_markers.py +++ b/nameparser/config/maiden_markers.py @@ -85,9 +85,8 @@ on, which is the test decisions.md#vocabulary-collisions states, and this set has no ambiguous subset to express a partial answer in — see decisions.md's Excluded (MAIDEN_MARKERS) block. The full participle -rozená stays, -being unambiguous; a caller who needs the abbreviation adds it to -their own Lexicon. +rozená stays, being unambiguous; a caller who needs the abbreviation +adds it to their own Lexicon. """ diff --git a/tests/v2/cases.py b/tests/v2/cases.py index 820b4a84..46f28fd5 100644 --- a/tests/v2/cases.py +++ b/tests/v2/cases.py @@ -1331,9 +1331,23 @@ def __post_init__(self) -> None: "maiden_marked_clause_one_word_stays_a_nickname rather " "than fence the other condition. Measured 2026-08-26: " "with the vocabulary test dropped this reads maiden " - "'Johnson', and with it the one-word spelling reads " - "nickname either way. Parity: 1.4.0 and 2.1.0 both read " - "nickname 'Mary Johnson'"), + "'Mary Johnson' -- the WHOLE clause, because #329's " + "drop is gated on the first token carrying " + "vocab:maiden-marker and 'Mary' does not, so nothing is " + "dropped -- and with the vocabulary test in place the " + "one-word spelling reads nickname either way. Parity: " + "1.4.0 and 2.1.0 both read nickname 'Mary Johnson'"), + Case("markerless_one_word_clause_stays_a_nickname", + "Cherice J. (Johnson) Williams", + {"given": "Cherice", "middle": "J.", "family": "Williams", + "nickname": "Johnson"}, + notes="the corpus spelling (corpus_issues.jsonl) of the row " + "above, kept beside it rather than replaced by it. It " + "reaches M3's length condition and stops there, so it " + "cannot fence the vocabulary one -- which is why the " + "row above widens the clause to two words -- but it is " + "the name real US data actually carries, and a row for " + "the corpus name is worth its two lines. Parity"), Case("maiden_marker_not_first_stays_a_nickname", "Jane Smith (Jones née)", {"given": "Jane", "family": "Smith", "nickname": "Jones née"}, @@ -1362,7 +1376,10 @@ def __post_init__(self) -> None: "The empty family is the bare path's, not this " "change's: 'Smith née Jones' reads given 'Smith', " "family '' on 2.1.0 too, and 1.4.0 read first 'Smith' / " - "last 'Jones' -- the given side has agreed since 1.4.0, " + "middle 'née' / last 'Jones' -- the marker as a middle " + "name, which is why the sibling fix(#410) rule at 1.4.0 " + "declares `middle`. The given side has agreed since " + "1.4.0, " "the emptying is 2.x's and is tracked separately as the " "#410 analogue. What #335 moves is only the bracketed " "spelling: 1.4.0, 2.0.0 and 2.1.0 all read it family " @@ -1375,13 +1392,17 @@ def __post_init__(self) -> None: "and this is the row that says so in the commonest " "spelling: a quote pair is how nicknames are usually " "written, and the same clause inside one reads maiden " - "exactly as it does inside parentheses. Nine of the " - "eleven shipped nickname pairs still have no row of " - "their own -- gating the swap to '(' and '\"' leaves " - "the whole suite green (measured 2026-08-26) -- so this " - "row and the fullwidth one below fence the two ends of " - "the set rather than the whole of it. 1.4.0 and 2.1.0 " - "both read nickname 'née Jones'"), + "exactly as it does inside parentheses. Three of the " + "eleven shipped nickname pairs are exercised by a " + "marker-led clause anywhere in the suite -- this one, " + "the parenthesis, and the fullwidth pair below -- and " + "the other EIGHT have no row. Measured 2026-08-26 by " + "disabling the swap one pair at a time: those three " + "redden and the eight do not. So this row and the " + "fullwidth one fence two ends of the set rather than " + "the whole of it; a set-level roster is what would " + "cover the rest. 1.4.0 and 2.1.0 both read nickname " + "'née Jones'"), Case("maiden_marked_clause_takes_the_suffix_reading_from_s1", "Jane Smith (née Jr.)", {"given": "Jane", "middle": "Smith", "family": "née", @@ -1395,7 +1416,7 @@ def __post_init__(self) -> None: "marker for M2 to take, so 'née' stays an ordinary " "word and lands in the family. rules.md#M3 carries the " "same input as an example line, but the runner checks " - "one field per line; this row is the other four. " + "one field per line; this row is the other three. " "Parity, and unchanged by #335 -- 1.4.0 and 2.1.0 read " "it the same way, which is why the corpus row it added " "diffs against no baseline"), @@ -1654,8 +1675,11 @@ def __post_init__(self) -> None: "and the double quote leaves the entire suite " "green and even the 2.1.0 gate green, its " "fix(#335) rule quietly falling from six names to " - "five, with only the 1.4.0 gate going red " - "(measured 2026-08-26). The fullwidth pair is the " + "five, and the 1.4.0 and 2.0.0 gates going red while " + "2.1.0 stays green (measured 2026-08-26; the 2.0.0 " + "ledger gives this name a four-field rule of its own, " + "which is what catches it there). The suite catches it " + "too, at this row. The fullwidth pair is the " "one the maiden_markers docstring and the 2.2 " "release note both advertise as newly working " "without configuration, so it is the one that " diff --git a/tests/v2/test_ledger_guards.py b/tests/v2/test_ledger_guards.py index 5347dc15..d4284d8d 100644 --- a/tests/v2/test_ledger_guards.py +++ b/tests/v2/test_ledger_guards.py @@ -1118,12 +1118,27 @@ def _carries(name: str, vocabulary: frozenset[str]) -> bool: but only two corpus names DEPEND on that branch today, meaning the token test below says no and the substring test says yes, and both are 旧姓 ones: the fullwidth-bracketed clause and the - fullwidth-colon spelling. Seven depended on it before the + fullwidth-colon spelling. Eight depended on it before the delimiter strip below arrived (2026-08-26); that strip moved the parenthesized née names onto the token branch, where the answer - does not rest on a substring. Tighten this before admitting a - vocabulary whose short non-ASCII entries occur inside ordinary - names. + does not rest on a substring. Both figures quantify over the + corpus and go stale on any row added to it, so recount rather than + adjust: + uv run python -c " + import glob, json + from nameparser import DEFAULT_NICKNAME_DELIMITERS as D + from nameparser.config.maiden_markers import MAIDEN_MARKERS as V + from nameparser._lexicon import _normalize + names = {json.loads(l) for f in glob.glob('tools/differential/corpus*.jsonl') + for l in open(f, encoding='utf-8') if l.strip()} + strip = ''.join({c for p in D for c in p}) + sub = lambda n: any(e in n for e in V if not e.isascii()) + print(sum(not {_normalize(t.strip(strip)) for t in n.split()} & V + and sub(n) for n in names), + sum(not {_normalize(t) for t in n.split()} & V + and sub(n) for n in names))" + Tighten this before admitting a vocabulary whose short non-ASCII + entries occur inside ordinary names. Delimiter characters come off the token before the membership test, because a marker glued to a bracket is still a marker to the @@ -1262,7 +1277,7 @@ def _claim(rule: dict) -> _Claim: "expected_since_1.4.0.toml": { "fix(#335) a marker-led clause leaves the one name word its bare reading": _Claim(1, ('family', 'given', 'maiden', 'nickname'), "c09cc7dba88b"), - "fix(#335) a marker-led bracketed clause reads as the maiden name whatever pair encloses it": + "fix(#335) a marker-led bracketed clause reads as the maiden name whatever pair encloses it": _Claim(5, ('maiden', 'nickname'), "a419f74143e3"), "fix(#410) a title and one name word name the family, whatever annotation stands beside it": _Claim(3, ('family', 'given'), "24d6223e472f"), @@ -1376,7 +1391,7 @@ def _claim(rule: dict) -> _Claim: "expected_since_2.0.0.toml": { "fix(#335) a marker-led clause leaves the one name word its bare reading": _Claim(1, ('family', 'given', 'maiden', 'nickname'), "c09cc7dba88b"), - "fix(#335) a marker-led bracketed clause reads as the maiden name whatever pair encloses it": + "fix(#335) a marker-led bracketed clause reads as the maiden name whatever pair encloses it": _Claim(5, ('maiden', 'nickname'), "a419f74143e3"), "fix(#335) a marker-led bracketed clause reads as the maiden name, compounding with the CJK order flip": _Claim(1, ('family', 'given', 'maiden', 'nickname'), "cf370e856ae7"), @@ -1486,7 +1501,7 @@ def _claim(rule: dict) -> _Claim: "expected_since_2.1.0.toml": { "fix(#335) a marker-led clause leaves the one name word its bare reading": _Claim(1, ('family', 'given', 'maiden', 'nickname'), "c09cc7dba88b"), - "fix(#335) a marker-led bracketed clause reads as the maiden name whatever pair encloses it": + "fix(#335) a marker-led bracketed clause reads as the maiden name whatever pair encloses it": _Claim(6, ('maiden', 'nickname'), "d0e857deddb2"), "fix(#410) a title and one name word name the family, whatever annotation stands beside it": _Claim(4, ('family', 'given'), "da1dd1473145"), diff --git a/tests/v2/test_parser.py b/tests/v2/test_parser.py index 38645d84..b725c63f 100644 --- a/tests/v2/test_parser.py +++ b/tests/v2/test_parser.py @@ -1319,14 +1319,33 @@ def _clause_free_latin_corpus_names() -> list[str]: # bracket as well as the abbreviating period before it is asked. # Stripping only the period admitted every corpus name that # brackets its marker, and once rules.md#M3 read such a clause as - # the maiden name, five of them had a maiden clause of their own -- + # the maiden name, six of them had a maiden clause of their own -- # two clauses, and the appended one no longer the only variable. - # The strip turns away NINE names in all (measured 2026-08-26): - # those five, plus four M3 declines and which would have been safe - # to keep -- the one-word '(Nee)', '(Nee) (Jones)' and '(née)', and - # '(née Jr.)', which S1 takes before M3 sees it. Textual, and so - # deliberately conservative in exactly that direction. Delimiter characters come from the shipped set rather than - # a literal, so a pair added there cannot quietly reopen this. + # The strip turns away TEN names in all: those six, plus four that + # M3 declines and that would have been safe to keep -- the + # one-word '(Nee)', '(Nee) (Jones)' and '(née)', and '(née Jr.)', + # which S1 takes before M3 sees it. Textual, and so deliberately + # conservative in exactly that direction. + # + # Every count in this comment quantifies over the corpus, so one + # added corpus row falsifies it silently. Recount rather than + # adjust: + # uv run python -c " + # import re, sys; sys.path.insert(0, 'tests') + # from nameparser import DEFAULT_NICKNAME_DELIMITERS as D + # from nameparser.config.maiden_markers import MAIDEN_MARKERS as M + # from v2._differential_fixtures import _CORPUS_NAMES + # base = [n for n in _CORPUS_NAMES + # if re.match(r'^[\x00-\u024f]*$', n) and ',' not in n] + # strip = ''.join({c for p in D for c in p}) + '.' + # keep = lambda f: [n for n in base + # if not any(f(w) in M for w in n.split())] + # old = keep(lambda w: w.lower().rstrip('.')) + # new = keep(lambda w: w.lower().strip(strip)) + # print(len(old), len(new), sorted(set(old) - set(new)))" + # + # Delimiter characters come from the shipped set rather than a + # literal, so a pair added there cannot quietly reopen this. strip = "".join({ch for pair in DEFAULT_NICKNAME_DELIMITERS for ch in pair}) + "." return [name for name in _CORPUS_NAMES @@ -1339,8 +1358,9 @@ def test_the_clause_free_corpus_is_not_empty() -> None: """The invariant below is parametrized over a FILTERED corpus, and an empty parametrization passes as a skip rather than failing -- the shape #329 left behind. The filter has been widened once - already (the delimiter strip, 2026-08-26, which took it from 637 - names to 628), so the floor is what says a future widening emptied + already (the delimiter strip, 2026-08-26, which took it from 638 + names to 628 -- the one-liner in that filter's comment recounts + both), so the floor is what says a future widening emptied it. Deliberately far below today's count: this asks whether the filter still selects a corpus, not what the corpus holds.""" assert len(_clause_free_latin_corpus_names()) > 100 diff --git a/tools/differential/README.md b/tools/differential/README.md index 52fbd099..d5eac394 100644 --- a/tools/differential/README.md +++ b/tools/differential/README.md @@ -421,9 +421,10 @@ appeared in no corpus name, so only the denominator moved), so an entry no corpus name exercises is as invisible as an opt-in policy. Those two numbers count WHOLE TOKENS, delimiters stripped. The strip -is what earns exactly one of the four: `nee`, whose only corpus -appearance is the bracketed `Jane Smith (Nee)`, where the token is -`(Nee)` until the brackets come off. `née` needs no strip, appearing +is what earns exactly one of the four: `nee`, which appears in the +corpora only inside brackets -- `Jane Smith (Nee)` and +`Jane Smith (Nee) (Jones)` -- where the token carries them until they +come off. `née` needs no strip, appearing bare in thirty-odd names, and `né` is not counted at all -- it occurs only as a substring of `née`, never as a token. The convention matters because the neighbouring guard `tests/v2/test_ledger_guards.py::_carries` deliberately asks a wider diff --git a/tools/differential/expected_since_1.4.0.toml b/tools/differential/expected_since_1.4.0.toml index 2ae0598c..2bf866d8 100644 --- a/tools/differential/expected_since_1.4.0.toml +++ b/tools/differential/expected_since_1.4.0.toml @@ -1259,8 +1259,12 @@ why = "feat(#273) recognizes TYPOGRAPHIC nickname delimiters; the ASCII pairs we # ('"Rick" Edmonds'), medial ('John (Jack) Kennedy') or trailing # ('Franklin, Benjamin (Ben)'), with a word character or a period as # the inner flank so a middle initial counts ('Cherice J. (Johnson) -# Williams'). 46 corpus names (recounted 2026-08-25; the figure is -# the one test_ledger_guards.py pins as this entry's `captures`). +# Williams'). 51 corpus names (recounted 2026-08-26). The figure is +# the one test_ledger_guards.py pins as this entry's `captures`, so it +# is not merely stale when it drifts, it contradicts a machine-checked +# constant: read it there rather than trusting this line, and grep +# _EXCLUSION_EFFECT for the pattern. It was 46 for three rounds and +# moved twice in one delta, once per rules-corpus row added. # # It was medial-only for three rounds, on the theory that the trailing # position is where credentials live and that widening would silence @@ -1441,5 +1445,11 @@ issue = "fix(#335) a marker-led clause leaves the one name word its bare reading # five. The empty `family` is not this change's -- the bare spelling # reads that way on 2.0.0 and 2.1.0 too -- but it is inside this # diff, so the rule declares it. -name_regex = "^Smith \\(n[\u00e9e]e Jones\\)$" +# +# revisit when #445 lands: this rule declares `family`, and `family` +# is exactly the field #445 would change. Fixing it there moves this +# diff, and this rule would absorb the change in silence at all three +# baselines -- the same failure the four-field split exists to avoid, +# displaced onto a different field. Re-measure this rule then. +name_regex = "(?i)^Smith \\(n[ée]e Jones\\)$" fields = ["given", "family", "nickname", "maiden"] diff --git a/tools/differential/expected_since_2.0.0.toml b/tools/differential/expected_since_2.0.0.toml index bf85b76a..5f49b2aa 100644 --- a/tools/differential/expected_since_2.0.0.toml +++ b/tools/differential/expected_since_2.0.0.toml @@ -1003,5 +1003,11 @@ issue = "fix(#335) a marker-led clause leaves the one name word its bare reading # five. The empty `family` is not this change's -- the bare spelling # reads that way on 2.0.0 and 2.1.0 too -- but it is inside this # diff, so the rule declares it. -name_regex = "^Smith \\(n[\u00e9e]e Jones\\)$" +# +# revisit when #445 lands: this rule declares `family`, and `family` +# is exactly the field #445 would change. Fixing it there moves this +# diff, and this rule would absorb the change in silence at all three +# baselines -- the same failure the four-field split exists to avoid, +# displaced onto a different field. Re-measure this rule then. +name_regex = "(?i)^Smith \\(n[ée]e Jones\\)$" fields = ["given", "family", "nickname", "maiden"] diff --git a/tools/differential/expected_since_2.1.0.toml b/tools/differential/expected_since_2.1.0.toml index b23e412c..43c6bf6e 100644 --- a/tools/differential/expected_since_2.1.0.toml +++ b/tools/differential/expected_since_2.1.0.toml @@ -918,12 +918,18 @@ issue = "fix(#335) a marker-led clause leaves the one name word its bare reading # clause, so N3 no longer reaches this shape: the word keeps the # reading the bare "Smith née Jones" gives it, which is `given`. # -# Its own rule rather than a sixth alternative in the fix(#335) rule -# above, for the reason that rule states: this diff moves FOUR fields -# where those move two, and one rule holding both would have to +# Its own rule rather than a seventh alternative in the fix(#335) +# rule above, for the reason that rule states: this diff moves FOUR +# fields where those move two, and one rule holding both would have to # declare the union, pre-excusing a given/family regression on the -# five. The empty `family` is not this change's -- the bare spelling +# six. The empty `family` is not this change's -- the bare spelling # reads that way on 2.0.0 and 2.1.0 too -- but it is inside this # diff, so the rule declares it. -name_regex = "^Smith \\(n[\u00e9e]e Jones\\)$" +# +# revisit when #445 lands: this rule declares `family`, and `family` +# is exactly the field #445 would change. Fixing it there moves this +# diff, and this rule would absorb the change in silence at all three +# baselines -- the same failure the four-field split exists to avoid, +# displaced onto a different field. Re-measure this rule then. +name_regex = "(?i)^Smith \\(n[ée]e Jones\\)$" fields = ["given", "family", "nickname", "maiden"] From 46f58ee4ab3e0d2a983e199b3f62395c10c5b297 Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Wed, 26 Aug 2026 14:12:07 -0700 Subject: [PATCH 11/12] docs(rules,mechanisms,cases): a case row pins a fork, not a wordlist entry MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three places a reader looks, none of which said this. rules.md's Not in scope gains the goal: no shipped wordlist parses every name and none is meant to -- the promise is a mechanism for every TYPE of vocabulary, with callers configuring the members their data needs. mechanisms.md gains the reviewable criterion under Verification shapes. cases.py's docstring said what a row must carry and never what earns one. Written because a review round reported "11 of 16 maiden markers have no behavioral pin" as a coverage gap and proposed a roster to close it. It is not a gap: født, fødd and född are one class, and every fork the code actually takes was already pinned. A count over a wordlist reads as a finding because it carries a number, and it inflates with every entry added -- which is the tell that it measures the data rather than the behavior. Co-Authored-By: Claude Opus 5 --- docs/design/mechanisms.md | 4 ++++ docs/design/rules.md | 1 + tests/v2/cases.py | 7 +++++++ 3 files changed, 12 insertions(+) diff --git a/docs/design/mechanisms.md b/docs/design/mechanisms.md index f010cde2..1374cf40 100644 --- a/docs/design/mechanisms.md +++ b/docs/design/mechanisms.md @@ -139,6 +139,10 @@ Contract statement. A worker that must run under a pinned dependency is rendered Contract statement. A user-supplied pattern is rejected as over-matching by probing it against a small set of inputs sharing no script, vocabulary or punctuation; matching all of them means it targets no behavior family. Measured: `.`, `.+`, `\b` and `[\s\S]` all decline the empty string — the naive probe — and still match every corpus name. Lives in tools/differential/compare.py (_SENTINELS). +### VOCABULARY-EXERCISES-FORKS — pin the branch, not the wordlist + +Contract statement. A vocabulary set is data; the code that reads it is what has behavior, so a case row earns its place by exercising a FORK — a branch the parser takes for one member and not another — never by covering a member. Sixteen maiden markers are not sixteen behaviors: født, fødd and född are one class, and a row for each pins nothing the first does not. It follows that "N of M entries have no test" is not a finding; it reads as one because it carries a number. The finding is "this fork has no test", and the forks are findable by reading the code that consults the set and enumerating the branches it takes — for maiden markers, the substring-versus-token split (ASCII against not), the edge-period normalization that makes geb. and geb one word, the native-script whole-token case that cannot collide with Latin text, and a member borne as an ordinary name (decisions.md#vocabulary-collisions). Four classes, four rows; the other twelve entries are data. The same reduction applies to the delimiter pairs, where the forks are same-character against distinct-character open and close (N2) and ASCII against fullwidth, not eleven pairs. Coverage measured against a wordlist grows with every entry added and narrows nothing, which is the tell: a criterion that inflates with the data is measuring the data. + ### FORCE-A-DECISION-TABLE — no silent defaults on growth Contract statement. Where adding an enum member or a file must not silently inherit a default, a local table's key set is asserted equal to the population, so growth fails the suite until someone decides — against a local table, not the constant under test. Exemplar: tests/v2/pipeline/test_vocab.py's per-script initials check; reused for _CORPUS_FLOORS in tools/differential/compare.py, and for the pack registry (test_registry_is_the_pack_contract: every registered pack must ship DEVIATES and its rotator list or the suite fails structurally — decisions.md#deviates-registry). Known gap it exposes: DEFAULT_SCRIPT_ORDERS has no such guard. diff --git a/docs/design/rules.md b/docs/design/rules.md index 4648dc80..78afa97c 100644 --- a/docs/design/rules.md +++ b/docs/design/rules.md @@ -20,6 +20,7 @@ An `annotation` names a policy, locale (`[ru]`), or extras gate (`[ja+segmenter] readings). Language-specific behavior is opt-in configuration. Script-conditional behavior exists only where the script itself settles the convention (see the W section). - **Grammatical inflection.** Names inflect in many languages (vocative, genitive); this library neither produces nor consumes inflected forms. CLDR personNames draws the same line. - **Validation.** Deciding whether a string IS a person's name is not parsing; `parse()` is total over strings and never rejects input. +- **Vocabulary completeness.** No shipped wordlist parses every name, and none is meant to. The promise is that every TYPE of vocabulary has a mechanism — markers, particles, titles, suffixes, bound given names — and that callers configure the members their data needs. A missing entry is a configuration answer, not a defect, and what the tests owe is one exercise per behavioral fork rather than one per entry (mechanisms.md#VOCABULARY-EXERCISES-FORKS). - **Comparison.** matches()/comparison_key() are a value-API surface, not parsing; their design record is decisions.md#comparison-surface. ## Titles & honorifics (H) diff --git a/tests/v2/cases.py b/tests/v2/cases.py index 46f28fd5..649c3a27 100644 --- a/tests/v2/cases.py +++ b/tests/v2/cases.py @@ -18,6 +18,13 @@ The v1 suite's full corpus is extracted into this table by the migration plan (facade runner consumes the same rows); this file seeds it with the pinned battery. + +What earns a row is a FORK: a branch taken for one input and not +another -- a rule's boundary, a precedence contest between two rules, +a policy that changes the answer. A row demonstrating one more member +of a vocabulary set pins nothing its other members do not, and one row +per entry grows this table without narrowing what can break +(mechanisms.md#VOCABULARY-EXERCISES-FORKS). """ from __future__ import annotations From 15bf4e3f7f29aad5013d8b0d351bb1fe2a99ebc7 Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Wed, 26 Aug 2026 15:24:30 -0700 Subject: [PATCH 12/12] docs(mechanisms,cases,customize,differential): the fork criterion, measured MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The VOCABULARY-EXERCISES-FORKS entry named four forks for maiden_markers. Measured, the set has none: it is read at exactly two sites, both plain whole-token membership after _normalize, and all sixteen members collapse to one structural class across bare, bracketed and lone-bracketed shapes with the marker text normalised out. Two of the four named forks were not parser behavior at all -- the substring split is _carries, a test guard that deliberately asks wider, and edge-period normalization is _normalize, which discriminates spellings across every vocabulary rather than members of this one. That makes the entry's argument stronger, and it now says so: rows on this set earn their place on clause SHAPE, never on which member appeared. The unit is the INPUT, matching cases.py's wording rather than the entry's old "one member and not another"; a row plainly earns its place with no member fork behind it. Three repairs beyond that. The delimiter enumeration claimed exhaustiveness and missed WORD_INTERNAL_DELIMITERS, which splits the same-character class in two -- 'Jane Smith"' reports UNBALANCED_DELIMITER and "Jane Smith'" reports nothing -- while listing ASCII against fullwidth, which is not a fork: swapping the pair across all 45 corpus names holding a paren changes no field and no ambiguity kind. The entry blurred the line Derek drew, so it states it: a delimiter pair the library SHIPS is caller-visible behavior and deleting one reddens three tests, while vocabulary a caller configures is data. And a row may earn its place a second way, with no fork behind it, by pinning a name a release note advertises by name. Its counts are gone rather than corrected. They were present-tense prose over a drifting wordlist, which root AGENTS.md forbids, this PR moved 17 to 16 by removing roz, and the descriptive figure was wrong anyway -- five members carry rows, so eleven have none. Two merge blockers. A cases.py note carried "leaves the entire suite green" beside the correcting sentence that says the suite catches it; the previous commit appended the truth and left the falsehood standing. And neither recompute one-liner ran as pasted: both bodies were indented inside the string handed to python -c, so both raised IndentationError. I had run them flush-left in my own shell, which is the inert-measurement shape they exist to prevent. Both are dedented, carry a note saying why, and were re-run by reconstructing them from the source and pasting. The #445 marker went on the bracketed rules only. The bare spelling is claimed at 1.4.0 by fix(#274), which also declares `family`, so a "keep the family" fix would land inside it with the gate green -- the same absorption on the same field at the same baseline. Marked there too, with the names to re-measure listed. Also: three ledger regexes gained (?i) in the previous commit without being mentioned, a real widening (the class now matches NEE and NÉE) bounded by _Claim's digest; customize.rst said suffix-shaped content "is read as a suffix" when S1 drops the brackets and parses the content as if written bare; _policy.py's autodoc'd docstring never got that clarification; and a 63/9 count is replaced by its recompute command, since 9 does not reproduce under its own definition. Co-Authored-By: Claude Opus 5 --- docs/customize.rst | 20 +++++--- docs/design/mechanisms.md | 16 ++++++- nameparser/_pipeline/_extract.py | 8 ++-- nameparser/_policy.py | 14 ++++-- tests/v2/cases.py | 50 ++++++++++++-------- tests/v2/test_ledger_guards.py | 26 +++++----- tests/v2/test_parser.py | 26 +++++----- tools/differential/README.md | 6 +-- tools/differential/expected_since_1.4.0.toml | 31 ++++++++++-- 9 files changed, 131 insertions(+), 66 deletions(-) diff --git a/docs/customize.rst b/docs/customize.rst index 0636d823..38595d46 100644 --- a/docs/customize.rst +++ b/docs/customize.rst @@ -298,11 +298,14 @@ listed below. no configuration since 2.2 is a clause that opens with a marker word AND has a word after it: ``"Jane Smith (née Jones)"`` reads maiden ``Jones`` whatever pair encloses it, unless the content is - suffix-shaped, which is read as a suffix ahead of both. A marker - word opening the enclosed content is dropped from the value - either way, but only where that content holds more than one - *token* — the same reason a lone ``"(Nee)"`` is a maiden name - rather than a marker. Tokens, not words: a marker written + suffix-shaped, which is taken ahead of both: the brackets are + dropped and the content parses as if written bare, so + ``"Jane Smith (née Jr.)"`` gives family ``née``, suffix ``Jr.`` + rather than a suffix of the whole clause. A marker word opening + the enclosed content is dropped from the value either way, but + only where that content holds more than one *token* — the same + reason a lone ``"(Nee)"`` listed here keeps ``Nee`` as the + maiden value rather than reading it as a marker. Tokens, not words: a marker written against the name it marks is one token with them, so ``"山田花子（旧姓佐藤）"`` keeps its ``旧姓``. Defaults to empty — see the routing example below. @@ -506,8 +509,11 @@ Nicknames, maiden names, and brackets ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ A delimiter pair carries no meaning of its own, so what a clause reads -as is settled in steps. Suffix-shaped content is taken first and reads -as a suffix. Then the content is asked whether it announces itself: a +as is settled in steps. Suffix-shaped content is taken first: the +brackets are dropped and what was inside parses as if it had been +written bare, which is not the same as the clause becoming the suffix +(``"Jane Smith (née Jr.)"`` gives family ``née``, suffix ``Jr.``). +Then the content is asked whether it announces itself: a clause opening with a recognized maiden marker and carrying a word after it is a maiden name whatever encloses it, and needs nothing configured. Only for what is left — markerless content, and a lone diff --git a/docs/design/mechanisms.md b/docs/design/mechanisms.md index 1374cf40..8bc04b10 100644 --- a/docs/design/mechanisms.md +++ b/docs/design/mechanisms.md @@ -141,7 +141,21 @@ Contract statement. A user-supplied pattern is rejected as over-matching by prob ### VOCABULARY-EXERCISES-FORKS — pin the branch, not the wordlist -Contract statement. A vocabulary set is data; the code that reads it is what has behavior, so a case row earns its place by exercising a FORK — a branch the parser takes for one member and not another — never by covering a member. Sixteen maiden markers are not sixteen behaviors: født, fødd and född are one class, and a row for each pins nothing the first does not. It follows that "N of M entries have no test" is not a finding; it reads as one because it carries a number. The finding is "this fork has no test", and the forks are findable by reading the code that consults the set and enumerating the branches it takes — for maiden markers, the substring-versus-token split (ASCII against not), the edge-period normalization that makes geb. and geb one word, the native-script whole-token case that cannot collide with Latin text, and a member borne as an ordinary name (decisions.md#vocabulary-collisions). Four classes, four rows; the other twelve entries are data. The same reduction applies to the delimiter pairs, where the forks are same-character against distinct-character open and close (N2) and ASCII against fullwidth, not eleven pairs. Coverage measured against a wordlist grows with every entry added and narrows nothing, which is the tell: a criterion that inflates with the data is measuring the data. +Contract statement. A vocabulary set a caller configures is data; the code that reads it is what has behavior, so a case row earns its place by exercising a FORK — a branch the parser takes for one INPUT and not another — never by covering one more member. The unit is the input, not the member, because most forks are keyed on the shape of the surrounding name rather than on which word appeared. Find them by reading the sites that consult the set and enumerating the branches they take, and expect that enumeration to come back empty: `maiden_markers` is read at exactly two sites (`_classify.py`'s tagging and `_extract.py`'s `_maiden_marked`), both plain whole-token membership after `_normalize`, and every shipped member parses to one structural class in bare, bracketed and lone-bracketed shapes with the marker text normalised out — zero member forks. What earns the maiden rows is clause SHAPE: marker first against marker later, a word after the marker against none, a marker a separator divides from the name against one glued to it. Two adjacent things are NOT forks of this set and do not want rows: the substring-versus-token split is `_carries` in test_ledger_guards.py, a test guard deliberately asking a wider question than the parser (tools/differential/README.md says so), and edge-period normalization lives in `_lexicon._normalize`, applying to every token of every vocabulary, so it discriminates spellings rather than members. Only "a member borne as an ordinary name" is member-keyed (decisions.md#vocabulary-collisions), and it has its own rows. The reduction does NOT carry across to delimiter pairs, and the difference is the line to hold: a pair the library SHIPS is caller-visible behavior, and vocabulary a caller configures is data — deleting `("«", "»")` from the defaults reddens three tests, so cases.py's one-row-per-shipped-pair convention is load-bearing and this criterion is not permission to delete those rows. Within the pairs the code does fork, on whether open and close are the same character (N2) and again inside that class, where `WORD_INTERNAL_DELIMITERS` carves the apostrophe out: `Jane Smith"` reports UNBALANCED_DELIMITER and `Jane Smith'` reports nothing. ASCII against fullwidth is not one of them — swapping `( )` for `（ ）` across every corpus name holding a paren changes no field and no ambiguity kind. A row may also earn its place a second way, with no fork behind it: pinning a name a release note advertises BY NAME, which is a promise to a reader rather than a branch in the parser. It follows that "N of M entries have no test" is not a finding; it reads as one because it carries a number, and it inflates with every entry added, which is the tell that it measures the data. The finding is "this fork has no test". State the argument without present-tense counts over a drifting wordlist — see docs/design/AGENTS.md — and where a count is genuinely wanted, ship the one-liner instead: + +``` +uv run python -c " +from nameparser import parse +from nameparser.config.maiden_markers import MAIDEN_MARKERS +shapes = ['Jane Smith {m} Jones', 'Jane Smith ({m} Jones)', 'Jane Smith ({m})'] +cls = {} +for m in sorted(MAIDEN_MARKERS): + sig = tuple(tuple((f, (getattr(parse(s.format(m=m)), f) or '').replace(m, '')) + for f in ('given', 'middle', 'family', 'suffix', 'nickname', 'maiden')) + for s in shapes) + cls.setdefault(sig, []).append(m) +print(len(MAIDEN_MARKERS), 'members ->', len(cls), 'structural classes')" +``` ### FORCE-A-DECISION-TABLE — no silent defaults on growth diff --git a/nameparser/_pipeline/_extract.py b/nameparser/_pipeline/_extract.py index 3fa3227c..da5f876c 100644 --- a/nameparser/_pipeline/_extract.py +++ b/nameparser/_pipeline/_extract.py @@ -234,10 +234,10 @@ def extract_delimited(state: ParseState) -> ParseState: # role that is not NICKNAME is already MAIDEN and the # assignment would be a no-op either way. It is kept # for the day `order` gains a third bucket, when it - # becomes the - # difference between M3 claiming that bucket's clauses - # and leaving them. Measured 2026-08-26: dropping it - # leaves the suite and all three gates green. + # becomes the difference between M3 claiming that + # bucket's clauses and leaving them. Measured + # 2026-08-26: dropping it leaves the suite and all + # three gates green. if (role is Role.NICKNAME and _maiden_marked( text[inner.start:inner.end], state.lexicon)): role = Role.MAIDEN diff --git a/nameparser/_policy.py b/nameparser/_policy.py index c8978274..a3000f04 100644 --- a/nameparser/_policy.py +++ b/nameparser/_policy.py @@ -623,9 +623,17 @@ class Policy: #: field instead; a pair listed here is dropped from the effective #: nickname set (maiden wins, see __post_init__), so #: maiden_delimiters=frozenset({("(", ")")}) is the whole recipe (#274). - #: A maiden_markers word opening the enclosed content is dropped - #: from the value, but only where that content holds more than one - #: token: a lone "(Nee)" is a maiden NAME, not a marker (#329). + #: Set this for a clause that says nothing about itself, which is + #: two kinds and not one: content with no marker word in it, and a + #: LONE marker word. Since #335 a clause that opens with a marker + #: word AND has a word after it reads as the maiden name whatever + #: pair encloses it, so "Jane Smith (née Jones)" needs no + #: configuration -- unless the content is suffix-shaped, which is + #: taken first, the brackets dropped and the content read as if + #: written bare. A maiden_markers word opening the enclosed + #: content is dropped from the value, but only where that content + #: holds more than one token: a lone "(Nee)" is a maiden NAME, + #: not a marker (#329). maiden_delimiters: frozenset[tuple[str, str]] = frozenset() #: Additional separators that split suffix groups (e.g. " - " for #: "Jane Smith, RN - CRNA"). Additive only: the comma always diff --git a/tests/v2/cases.py b/tests/v2/cases.py index 649c3a27..b78b278f 100644 --- a/tests/v2/cases.py +++ b/tests/v2/cases.py @@ -24,7 +24,11 @@ a policy that changes the answer. A row demonstrating one more member of a vocabulary set pins nothing its other members do not, and one row per entry grows this table without narrowing what can break -(mechanisms.md#VOCABULARY-EXERCISES-FORKS). +(mechanisms.md#VOCABULARY-EXERCISES-FORKS -- a restatement, and +nothing checks it against the entry: test_doc_citations verifies an +excerpt only where the reference is followed by a colon and a quoted +span, which running prose like this one is not. Read the entry, not +this paragraph, if the two ever disagree). """ from __future__ import annotations @@ -1383,10 +1387,14 @@ def __post_init__(self) -> None: "The empty family is the bare path's, not this " "change's: 'Smith née Jones' reads given 'Smith', " "family '' on 2.1.0 too, and 1.4.0 read first 'Smith' / " - "middle 'née' / last 'Jones' -- the marker as a middle " - "name, which is why the sibling fix(#410) rule at 1.4.0 " - "declares `middle`. The given side has agreed since " - "1.4.0, " + "middle 'née' / last 'Jones' -- the marker as a " + "middle name. The parallel evidence at 1.4.0 is not " + "the fix(#410) maiden-flavor rule, which is keyed on " + "the TITLED 'Dr. Smith née Jones', but fix(#274), " + "which claims the bare one-word names 'Janey née " + "Jones' and 'abd née Jones' and declares `family` for " + "exactly the reason this row records. The given side " + "has agreed since 1.4.0, " "the emptying is 2.x's and is tracked separately as the " "#410 analogue. What #335 moves is only the bracketed " "spelling: 1.4.0, 2.0.0 and 2.1.0 all read it family " @@ -1405,11 +1413,16 @@ def __post_init__(self) -> None: "the parenthesis, and the fullwidth pair below -- and " "the other EIGHT have no row. Measured 2026-08-26 by " "disabling the swap one pair at a time: those three " - "redden and the eight do not. So this row and the " - "fullwidth one fence two ends of the set rather than " - "the whole of it; a set-level roster is what would " - "cover the rest. 1.4.0 and 2.1.0 both read nickname " - "'née Jones'"), + "redden and the eight do not. That is a count over a " + "wordlist, so read it the way " + "mechanisms.md#VOCABULARY-EXERCISES-FORKS says to: the " + "eight are not eight gaps, since the pairs fork on " + "whether open and close are the same character and on " + "the apostrophe carve-out inside that, not on which " + "pair. This row is here because a quote pair is the " + "same-character branch, and because the release note " + "advertises the spelling. 1.4.0 and 2.1.0 both read " + "nickname 'née Jones'"), Case("maiden_marked_clause_takes_the_suffix_reading_from_s1", "Jane Smith (née Jr.)", {"given": "Jane", "middle": "Smith", "family": "née", @@ -1678,15 +1691,14 @@ def __post_init__(self) -> None: classification="feat(#273) + fix(#271) + fix(#335)", notes="the row above without its policy, and the one " "that fences M3 across the delimiter SET rather " - "than at the parenthesis: gating the swap to '(' " - "and the double quote leaves the entire suite " - "green and even the 2.1.0 gate green, its " - "fix(#335) rule quietly falling from six names to " - "five, and the 1.4.0 and 2.0.0 gates going red while " - "2.1.0 stays green (measured 2026-08-26; the 2.0.0 " - "ledger gives this name a four-field rule of its own, " - "which is what catches it there). The suite catches it " - "too, at this row. The fullwidth pair is the " + "than at the parenthesis. Measured 2026-08-26 by " + "gating the swap to '(' and the double quote: this row " + "and its facade twin are the only two failures in the " + "suite, and of the gates only 1.4.0 and 2.0.0 redden " + "-- 2.1.0 stays green with its fix(#335) rule quietly " + "falling from six names to five, while the 2.0.0 " + "ledger catches it on the four-field rule it gives " + "this name. The fullwidth pair is the " "one the maiden_markers docstring and the 2.2 " "release note both advertise as newly working " "without configuration, so it is the one that " diff --git a/tests/v2/test_ledger_guards.py b/tests/v2/test_ledger_guards.py index d4284d8d..b5391d03 100644 --- a/tests/v2/test_ledger_guards.py +++ b/tests/v2/test_ledger_guards.py @@ -1124,19 +1124,19 @@ def _carries(name: str, vocabulary: frozenset[str]) -> bool: does not rest on a substring. Both figures quantify over the corpus and go stale on any row added to it, so recount rather than adjust: - uv run python -c " - import glob, json - from nameparser import DEFAULT_NICKNAME_DELIMITERS as D - from nameparser.config.maiden_markers import MAIDEN_MARKERS as V - from nameparser._lexicon import _normalize - names = {json.loads(l) for f in glob.glob('tools/differential/corpus*.jsonl') - for l in open(f, encoding='utf-8') if l.strip()} - strip = ''.join({c for p in D for c in p}) - sub = lambda n: any(e in n for e in V if not e.isascii()) - print(sum(not {_normalize(t.strip(strip)) for t in n.split()} & V - and sub(n) for n in names), - sum(not {_normalize(t) for t in n.split()} & V - and sub(n) for n in names))" + The body must sit flush left: `python -c` compiles it as a module, + so an indented first line raises IndentationError on paste. + +uv run python -c " +import glob, json +from nameparser import DEFAULT_NICKNAME_DELIMITERS as D +from nameparser.config.maiden_markers import MAIDEN_MARKERS as V +from nameparser._lexicon import _normalize +names = {json.loads(l) for f in glob.glob('tools/differential/corpus*.jsonl') for l in open(f, encoding='utf-8') if l.strip()} +strip = ''.join({c for p in D for c in p}) +sub = lambda n: any(e in n for e in V if not e.isascii()) +print(sum(not {_normalize(t.strip(strip)) for t in n.split()} & V and sub(n) for n in names), + sum(not {_normalize(t) for t in n.split()} & V and sub(n) for n in names))" Tighten this before admitting a vocabulary whose short non-ASCII entries occur inside ordinary names. diff --git a/tests/v2/test_parser.py b/tests/v2/test_parser.py index b725c63f..55e75da6 100644 --- a/tests/v2/test_parser.py +++ b/tests/v2/test_parser.py @@ -1330,19 +1330,19 @@ def _clause_free_latin_corpus_names() -> list[str]: # Every count in this comment quantifies over the corpus, so one # added corpus row falsifies it silently. Recount rather than # adjust: - # uv run python -c " - # import re, sys; sys.path.insert(0, 'tests') - # from nameparser import DEFAULT_NICKNAME_DELIMITERS as D - # from nameparser.config.maiden_markers import MAIDEN_MARKERS as M - # from v2._differential_fixtures import _CORPUS_NAMES - # base = [n for n in _CORPUS_NAMES - # if re.match(r'^[\x00-\u024f]*$', n) and ',' not in n] - # strip = ''.join({c for p in D for c in p}) + '.' - # keep = lambda f: [n for n in base - # if not any(f(w) in M for w in n.split())] - # old = keep(lambda w: w.lower().rstrip('.')) - # new = keep(lambda w: w.lower().strip(strip)) - # print(len(old), len(new), sorted(set(old) - set(new)))" + # The body must sit flush left: `python -c` compiles it as a module, + # so an indented first line raises IndentationError on paste. + # uv run python -c " + # import re, sys; sys.path.insert(0, 'tests') + # from nameparser import DEFAULT_NICKNAME_DELIMITERS as D + # from nameparser.config.maiden_markers import MAIDEN_MARKERS as M + # from v2._differential_fixtures import _CORPUS_NAMES + # base = [n for n in _CORPUS_NAMES if re.match(r'^[\x00-\u024f]*$', n) and ',' not in n] + # strip = ''.join({c for p in D for c in p}) + '.' + # keep = lambda f: [n for n in base if not any(f(w) in M for w in n.split())] + # old = keep(lambda w: w.lower().rstrip('.')) + # new = keep(lambda w: w.lower().strip(strip)) + # print(len(old), len(new), sorted(set(old) - set(new)))" # # Delimiter characters come from the shipped set rather than a # literal, so a pair added there cannot quietly reopen this. diff --git a/tools/differential/README.md b/tools/differential/README.md index d5eac394..6c2f45cf 100644 --- a/tools/differential/README.md +++ b/tools/differential/README.md @@ -424,9 +424,9 @@ Those two numbers count WHOLE TOKENS, delimiters stripped. The strip is what earns exactly one of the four: `nee`, which appears in the corpora only inside brackets -- `Jane Smith (Nee)` and `Jane Smith (Nee) (Jones)` -- where the token carries them until they -come off. `née` needs no strip, appearing -bare in thirty-odd names, and `né` is not counted at all -- it occurs -only as a substring of `née`, never as a token. The convention matters because the neighbouring guard +come off. `née` needs no strip, appearing bare in many names, and `né` +is not counted at all: it occurs only as a substring of `née`, never +as a token. The convention matters because the neighbouring guard `tests/v2/test_ledger_guards.py::_carries` deliberately asks a wider question -- it also matches a non-ASCII entry anywhere inside a name, since 旧姓 is written flush against the name it marks -- and under that diff --git a/tools/differential/expected_since_1.4.0.toml b/tools/differential/expected_since_1.4.0.toml index 2bf866d8..3b76a3f8 100644 --- a/tools/differential/expected_since_1.4.0.toml +++ b/tools/differential/expected_since_1.4.0.toml @@ -134,6 +134,21 @@ issue = "fix(#274) maiden markers consumed" # would need the regex to ask a vocabulary question -- suffix or # connective membership -- a second alternation this rule cannot # spend. +# +# revisit when #445 lands, and this is the half of that shape the +# bracketed fix(#335) rules do NOT cover. This rule declares `family`, +# and eight names it claims read family '' because the marker takes +# every word after it -- the bare spelling of exactly what #445 asks +# about. If #445 lands as "keep the family", their 1.4.0 diff stays a +# subset of {maiden, middle, family} and this rule keeps explaining +# them with the gate green, which is the silent absorption the +# four-field split exists to prevent, on the same field and the same +# baseline. The checklist to re-measure, all one name word before the +# marker: 'Janey née Jones' and 'abd née Jones' (the exact N3-parallel +# shape, one word after the marker too), plus 'Jane née Jones Smith', +# 'Jane née and Jones Smith', 'John née Jones Smith V', +# 'John née Jones Smith Ma', 'J. née Jones Smith V' and +# 'Jane née Jones J. V'. name_regex = "(?i)(?