diff --git a/docs/customize.rst b/docs/customize.rst index 6b4e08d7..38595d46 100644 --- a/docs/customize.rst +++ b/docs/customize.rst @@ -288,15 +288,27 @@ listed below. * - ``maiden_delimiters`` - ``frozenset[tuple[str, str]]`` - Routes content enclosed by these delimiter pairs to ``maiden`` - instead, and drops them from the effective nickname set. A - marker word opening the enclosed content is dropped from the - value, so ``"Jane Smith (née Jones)"`` gives maiden ``Jones`` - — but only where that content holds more than one *token*, - since a lone ``"(Nee)"`` is a maiden name rather than a - marker. Tokens, not words: a marker written against the name - it marks is one token with them, so ``"山田花子（旧姓佐藤）"`` - keeps its ``旧姓``. Defaults to empty — see the routing - example below. + instead, and drops them from the effective nickname set. Set + this for a clause that says nothing about itself, which is two + kinds of clause and not one: content with no marker word in it + (``"Cherice J. (Johnson) Williams"``, the parenthesized birth + surname written bare) AND a lone marker word + (``"Jane Smith (Nee)"``, which reads nickname ``Nee`` by default + and maiden ``Nee`` only with the pair listed here). What needs + no configuration since 2.2 is a clause that opens with a marker + word AND has a word after it: ``"Jane Smith (née Jones)"`` reads + maiden ``Jones`` whatever pair encloses it, unless the content is + suffix-shaped, which is taken ahead of both: the brackets are + dropped and the content parses as if written bare, so + ``"Jane Smith (née Jr.)"`` gives family ``née``, suffix ``Jr.`` + rather than a suffix of the whole clause. A marker word opening + the enclosed content is dropped from the value either way, but + only where that content holds more than one *token* — the same + reason a lone ``"(Nee)"`` listed here keeps ``Nee`` as the + maiden value rather than reading it as a marker. Tokens, not words: a marker written + against the name it marks is one token with them, so + ``"山田花子（旧姓佐藤）"`` keeps its ``旧姓``. Defaults to empty — + see the routing example below. * - ``extra_suffix_delimiters`` - ``frozenset[str]`` - Adds separators that split suffix groups, e.g. ``" - "`` for @@ -496,10 +508,19 @@ off. Nicknames, maiden names, and brackets ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -A delimiter pair routes to exactly one field, and ``maiden_delimiters`` -states the more specific intent — so listing a pair there drops it from -the effective ``nickname_delimiters`` set automatically, and the -one-liner is the whole recipe: +A delimiter pair carries no meaning of its own, so what a clause reads +as is settled in steps. Suffix-shaped content is taken first: the +brackets are dropped and what was inside parses as if it had been +written bare, which is not the same as the clause becoming the suffix +(``"Jane Smith (née Jr.)"`` gives family ``née``, suffix ``Jr.``). +Then the content is asked whether it announces itself: a +clause opening with a recognized maiden marker and carrying a word +after it is a maiden name whatever encloses it, and needs nothing +configured. Only for what is left — markerless content, and a lone +marker word — does the PAIR decide, and that is what this knob is for. +Listing a pair here drops it from the effective +``nickname_delimiters`` set automatically, and the one-liner is the +whole recipe: .. doctest:: diff --git a/docs/design/decisions.md b/docs/design/decisions.md index 9171771b..b2817497 100644 --- a/docs/design/decisions.md +++ b/docs/design/decisions.md @@ -431,7 +431,8 @@ Excluded (MAIDEN_MARKERS, per nameparser/config/maiden_markers.py): #274 flagging "is nee safe as a default (it's also a rare surname)" as open; the question resolved silently with the shipped set. Recorded here because the included risky member deserves its analysis as much as the excluded ones; M1's (Nee) boundary covers only the enclosure path, not this marker path. - "born" — never shipped: a release-log drafting invention, caught by the 2.0 milestone audit and corrected (5ccf9f3). Recorded so nobody "restores" it; if ever proposed for real, Max Born is the counterexample to analyze. -- Scandinavian "f." — collides with the initial F.; only the full participles (født/fødd/född) are safe. Czech masculine "rozený" awaits the same vetting. +- Scandinavian "f." — collides with the initial F.; only the full participles (født/fødd/född) are safe. Czech masculine "rozený" awaits the same vetting, on its own merits rather than following the feminine form in. +- Czech/Slovak "roz." — SHIPPED through 2.1 and removed 2026-08-26 (Derek's call). Roz is an ordinary English diminutive of Rosalind, and matching is whole-token, case-folded and period-insensitive, so "Roz", "roz" and "roz." are one string to this set. The collision is not theoretical and it is not new: on the bare path, which predates the 2.2 work entirely, "Rosalind Roz Smith" read maiden "Smith" and NO family name at all, and "Rosalind Roz Jones Smith" read maiden "Jones Smith", because M2 hands the marker every word after it. Grounded in #vocabulary-collisions C-i — a word belongs in its set's ambiguous subset iff it is borne as an ordinary name IN THE POSITION THE VOCABULARY CLAIM ACTS ON, and under uncertainty default to ambiguous. Roz is borne exactly there: the word after a given name, which is the only position M2's claim acts on. What C-i cannot do for this set is the marking: MAIDEN_MARKERS has no ambiguous subset, and no code path consults one, so the only two expressions of C-i available here are ship and do not ship. That is why the remedy is removal rather than an ambiguous marking, and it is worth knowing before someone proposes marking the next collision instead. The full participle rozená stays and is unambiguous; a Czech caller who needs the abbreviation adds it to their own Lexicon, which is the same answer this section gives for every locale-specific vocabulary. The cost is accepted and stated: "Anna Nováková roz. Svobodová" now reads middle "Nováková roz.", family "Svobodová" — which is how 1.4.0 read it — and "Rosalind Roz Smith" is back to 1.4.0's given/middle/family. Pinned by tests/v2/cases.py::diminutive_that_was_a_marker_keeps_the_family and ::full_participle_marker_still_consumes. ### C1 — the suffix-comma decision @@ -491,7 +492,19 @@ Declined: - 2026-08-05 #329/#335 — marker auto-detection inside a nickname-delimited clause was deferred to #335 on a corpus measurement: 山田 花子（旧姓 佐藤） is in the CJK differential corpus so the #329 change was gate-visible, while "Jane Smith (née Jones)" is in no corpus — shipping auto-detection in 2.1 would have let a real Latin-affecting change ride under a "0 Latin-only" gate report. -Open: [#335](https://github.com/derek73/python-nameparser/issues/335) should a marker inside a NICKNAME-delimited clause flip it to maiden without configuration. +- 2026-08-26 #335 — correction to the 2026-08-05 entry above, which reasoned from "the #329 change was gate-visible". It was not. Under the default policy the corpora run, no name reached the drop pass at all, so 山田 花子（旧姓 佐藤） moving between 2.0.0 and 2.1.0 was the East Asian order flip and not #329. Nor is #329 gate-visible now that M3 routes marker-led clauses to maiden without configuration: a ledger rule narrows by which FIELDS move and never by what they hold, so reverting the drop leaves the same six names moving the same fields and all three gates green (measured by mutation 2026-08-26). The {nickname, maiden} pair is the 2.1.0 ledger's; the other two baselines classify the CJK name differently, which is why those ledgers are split and must stay so. The deferral #329 argued for was still the right call — it was about a Latin-affecting change riding under a gate report, and that half held — but the premise is corrected here rather than left to be re-used. tools/differential/README.md carries the same correction beside the table it belongs to. +- 2026-08-26 #335 (M3) — the opt-in this section opens with narrowed, and the boundary is not where the 2026-07-03 entry drew it. A clause whose content announces itself no longer needs its pair declared, so what configuring maiden_delimiters buys is now markerless content and one-word clauses. See #M3 for the whole reasoning; M1 still governs every clause whose pair is configured, and M3 defers to it for the marker drop itself. + +### M3 — the marker-led clause + +- 2026-08-26 #335 (M3, PR on fix/335-maiden-marked-clause) — a bracketed clause whose content opens with a recognized marker word and carries a word after it reads as the maiden name, whichever bucket the enclosing pair sits in. The parse already had the information: "Jane Smith née Jones" gave maiden "Jones" while "Jane Smith (née Jones)" gave nickname "née Jones", because ( ) is a nickname pair by default and nothing looked inside it. The clause says "maiden" out loud, so the caller should not have to say it in Policy. +- 2026-08-26 #335 — the MECHANISM, and the issue proposed the other one. Masking the delimiters and letting M2's bare-marker rule consume the content would avoid a second implementation, and it was rejected on measurement: the two readings agree on a trailing clause and diverge on an interior one. "Jane (née Jones) Smith" reads given "Jane", family "Smith", maiden "Jones" under extraction; masking turns it into the bare "Jane née Jones Smith", which reads maiden "Jones Smith" and family "" (measured 2026-08-26), because M2's take runs to the end of the name. The closing delimiter is a boundary the writer typed and masking throws it away. Extraction was also the SMALLER change — extract already emits Role.MAIDEN regions and group already drops the marker inside one (#329), so this is a role swap at the existing emit site plus one predicate, in the branch S1 has already declined. That placement is what keeps S1 first, but be exact about the mechanism, because the obvious mutation does not test it: hoisting the swap above the _suffix_shaped call changes nothing at all (measured 2026-08-26), since the S1 branch never reads `role` -- it masks the two delimiter spans and hands the content to the token stream, and the role it would have carried is never consulted. What inverts the precedence is giving M3 the clause first, an early `extracted.append((Role.MAIDEN, inner))` ahead of the S1 test: that reads "Jane Smith (née Jr.)" as family "Smith", maiden "Jr." instead of family "née", suffix "Jr.", and M3's own example line for it fails. So the precedence is pinned, and by that example rather than by the branch shape. +- 2026-08-26 #335 — the condition is a word AFTER the marker, not the marker alone. The reason is M1's: a one-word clause keeps its word because Nee is an attested surname (Irish Ní/Nee, and a Chinese romanization), and a marker with nothing after it marks nothing. Be precise about what pins it, because the obvious answer is wrong: M1's own boundary line carries the maiden-parens annotation, so it asserts the CONFIGURED reading and passes either way — built and measured 2026-08-26, a len(words) > 0 predicate still gives it maiden "Nee". What the loosened predicate actually breaks, measured by mutation 2026-08-26, is M3's own boundary example ("Jane Smith (née)" → nickname "née") and tests/v2/cases.py::maiden_marked_clause_one_word_stays_a_nickname in both runners — three pytest failures — AND the 2.1.0 differential gate, which goes red with three unexplained names: "Jane Smith (Nee)", "Jane Smith (Nee) (Jones)" and "Jane Smith (née)". The gate is the stronger signal and the one to look at first. The middle name is why: loosened, it reads maiden "Nee" and nickname "Jones", splitting one bracket pair into the maiden field and the next into the nickname field, which is the direct contradiction of M1's "two enclosures read as one maiden name". Anyone re-litigating the second-word test should check those, not M1's line. So the real division of labour is NOT markerless versus marker-led, which is how it was first written into rules.md and had to be corrected at review: it is one word versus a marker plus a word. "Jane Smith (Nee)" is marker-led and still reads nickname "Nee" by default, maiden "Nee" only with the pair configured (both measured 2026-08-26). +- 2026-08-26 #335 — accepted consequence: the word taken after the marker is not tested for being a name word, so M3 does not stop at a suffix word where M2's bare take does. "Jane Smith (née V)" reads maiden "V" while the bare "Jane Smith née V" reads suffix "V" and family "née" (both measured). The reason is structural rather than an oversight: M3 hands the WHOLE clause to Role.MAIDEN and the bracket is its right boundary, so the trailing-suffix rule never sees the word as trailing the name at all. Pinned as M3's Accepted line in rules.md, which puts "Jane Smith (née V)" in the differential corpus and classified in all three ledgers. +- 2026-08-26 #335 — rules.md#M3 lists implemented: nameparser/_pipeline/_extract.py alone, and that is deliberate rather than an omission. M3 decides a REGION'S ROLE; the marker drop it describes ("the marker itself dropped, as M1 drops it") is M1's mechanism in _group.py, which cites M1 and is governed by it. Adding _group.py to M3's list would mean inventing an M3 citation there for behavior M1 already owns — and test_doc_citations requires implemented: to equal the set of modules that cite the rule, so the two would have to move together. Do not re-open this. +- 2026-08-26 #335 — the durable finding, and the one worth reading before touching either side: _maiden_marked asks a WHITESPACE-SPLIT, _normalize'd question of the clause's first word, while tokenize and classify ask a token-level one. _normalize strips a trailing period but not a comma, so 'née,' is not a marker to M3 while the tokenizer splits the comma off and still tags the token vocab:maiden-marker. That divergence is the only thing keeping the "role is not Role.MAIDEN" branch of _group.group's clause-scoped drop pass reachable at all: without it, a marker-led NICKNAME clause could no longer exist, since M3 would have converted every one to MAIDEN before group ran. Anyone "unifying" the two tests would orphan that filter silently. The pin is tests/v2/cases.py::marker_glued_to_punctuation_keeps_the_clause_a_nickname, and it needs BOTH of its clauses — the drop pass is gated on the name holding a maiden region at all, so a marker-glued nickname clause standing alone leaves the branch unexercised (measured by mutation, 2026-08-26). The row that used to pin the filter, maiden_marker_delimited_beside_a_nickname_clause, lost the job to M3: both its clauses are marker-led, so both are maiden now and there is no contrast left in it. +- 2026-08-26 #335 — M3 makes the marker vocabulary act in a THIRD position. M2's claim acts on the word after a name word; M1's on the content of a configured pair; M3's now on the first word of any bracketed clause, under the default policy. #vocabulary-collisions C-i is keyed on the position a claim acts on, so extending where a rule acts re-asks C-i of the whole set it reads, and nothing in the process prompts that — the rule was designed, reviewed and approved without the set being re-examined. Review afterwards found one entry that fails C-i, the Czech abbreviation roz, and its disposition is this file's Excluded (MAIDEN_MARKERS) block, which is keyed to the vocabulary set rather than to any rule: it was removed. Two things worth separating, since the finding surfaced during this work and is easy to file under it — the defect was M2's and predates #335 (bare "Rosalind Roz Smith" lost its family name on master), and M3's own boundary already excluded the common one-word "(Roz)" spelling. What belongs to M3 is only the widening, and the lesson is the general one: re-ask C-i of a vocabulary set whenever a rule extends where its claim acts. +- 2026-08-26 #335 — what configuring maiden_delimiters still buys, since M3 removes the commonest reason to reach for it: markerless clauses, and one-word clauses. "Cherice J. (Johnson) Williams" is a real US convention and a corpus name, and nothing in that clause says maiden, so it stays a nickname by default — only a caller who knows their data can say otherwise. The knob is a declaration about the DATA; M3 is a reading of what a clause declares about itself. ### O1 — East Slavic rotation diff --git a/docs/design/mechanisms.md b/docs/design/mechanisms.md index f010cde2..8bc04b10 100644 --- a/docs/design/mechanisms.md +++ b/docs/design/mechanisms.md @@ -139,6 +139,24 @@ Contract statement. A worker that must run under a pinned dependency is rendered Contract statement. A user-supplied pattern is rejected as over-matching by probing it against a small set of inputs sharing no script, vocabulary or punctuation; matching all of them means it targets no behavior family. Measured: `.`, `.+`, `\b` and `[\s\S]` all decline the empty string — the naive probe — and still match every corpus name. Lives in tools/differential/compare.py (_SENTINELS). +### VOCABULARY-EXERCISES-FORKS — pin the branch, not the wordlist + +Contract statement. A vocabulary set a caller configures is data; the code that reads it is what has behavior, so a case row earns its place by exercising a FORK — a branch the parser takes for one INPUT and not another — never by covering one more member. The unit is the input, not the member, because most forks are keyed on the shape of the surrounding name rather than on which word appeared. Find them by reading the sites that consult the set and enumerating the branches they take, and expect that enumeration to come back empty: `maiden_markers` is read at exactly two sites (`_classify.py`'s tagging and `_extract.py`'s `_maiden_marked`), both plain whole-token membership after `_normalize`, and every shipped member parses to one structural class in bare, bracketed and lone-bracketed shapes with the marker text normalised out — zero member forks. What earns the maiden rows is clause SHAPE: marker first against marker later, a word after the marker against none, a marker a separator divides from the name against one glued to it. Two adjacent things are NOT forks of this set and do not want rows: the substring-versus-token split is `_carries` in test_ledger_guards.py, a test guard deliberately asking a wider question than the parser (tools/differential/README.md says so), and edge-period normalization lives in `_lexicon._normalize`, applying to every token of every vocabulary, so it discriminates spellings rather than members. Only "a member borne as an ordinary name" is member-keyed (decisions.md#vocabulary-collisions), and it has its own rows. The reduction does NOT carry across to delimiter pairs, and the difference is the line to hold: a pair the library SHIPS is caller-visible behavior, and vocabulary a caller configures is data — deleting `("«", "»")` from the defaults reddens three tests, so cases.py's one-row-per-shipped-pair convention is load-bearing and this criterion is not permission to delete those rows. Within the pairs the code does fork, on whether open and close are the same character (N2) and again inside that class, where `WORD_INTERNAL_DELIMITERS` carves the apostrophe out: `Jane Smith"` reports UNBALANCED_DELIMITER and `Jane Smith'` reports nothing. ASCII against fullwidth is not one of them — swapping `( )` for `（ ）` across every corpus name holding a paren changes no field and no ambiguity kind. A row may also earn its place a second way, with no fork behind it: pinning a name a release note advertises BY NAME, which is a promise to a reader rather than a branch in the parser. It follows that "N of M entries have no test" is not a finding; it reads as one because it carries a number, and it inflates with every entry added, which is the tell that it measures the data. The finding is "this fork has no test". State the argument without present-tense counts over a drifting wordlist — see docs/design/AGENTS.md — and where a count is genuinely wanted, ship the one-liner instead: + +``` +uv run python -c " +from nameparser import parse +from nameparser.config.maiden_markers import MAIDEN_MARKERS +shapes = ['Jane Smith {m} Jones', 'Jane Smith ({m} Jones)', 'Jane Smith ({m})'] +cls = {} +for m in sorted(MAIDEN_MARKERS): + sig = tuple(tuple((f, (getattr(parse(s.format(m=m)), f) or '').replace(m, '')) + for f in ('given', 'middle', 'family', 'suffix', 'nickname', 'maiden')) + for s in shapes) + cls.setdefault(sig, []).append(m) +print(len(MAIDEN_MARKERS), 'members ->', len(cls), 'structural classes')" +``` + ### FORCE-A-DECISION-TABLE — no silent defaults on growth Contract statement. Where adding an enum member or a file must not silently inherit a default, a local table's key set is asserted equal to the population, so growth fails the suite until someone decides — against a local table, not the constant under test. Exemplar: tests/v2/pipeline/test_vocab.py's per-script initials check; reused for _CORPUS_FLOORS in tools/differential/compare.py, and for the pack registry (test_registry_is_the_pack_contract: every registered pack must ship DEVIATES and its rotator list or the suite fails structurally — decisions.md#deviates-registry). Known gap it exposes: DEFAULT_SCRIPT_ORDERS has no such guard. diff --git a/docs/design/rules.md b/docs/design/rules.md index 69f17dc9..78afa97c 100644 --- a/docs/design/rules.md +++ b/docs/design/rules.md @@ -20,6 +20,7 @@ An `annotation` names a policy, locale (`[ru]`), or extras gate (`[ja+segmenter] readings). Language-specific behavior is opt-in configuration. Script-conditional behavior exists only where the script itself settles the convention (see the W section). - **Grammatical inflection.** Names inflect in many languages (vocative, genitive); this library neither produces nor consumes inflected forms. CLDR personNames draws the same line. - **Validation.** Deciding whether a string IS a person's name is not parsing; `parse()` is total over strings and never rejects input. +- **Vocabulary completeness.** No shipped wordlist parses every name, and none is meant to. The promise is that every TYPE of vocabulary has a mechanism — markers, particles, titles, suffixes, bound given names — and that callers configure the members their data needs. A missing entry is a configuration answer, not a defect, and what the tests owe is one exercise per behavioral fork rather than one per entry (mechanisms.md#VOCABULARY-EXERCISES-FORKS). - **Comparison.** matches()/comparison_key() are a value-API surface, not parsing; their design record is decisions.md#comparison-surface. ## Titles & honorifics (H) @@ -527,11 +528,21 @@ N3. Rationale: a person set down as a nickname plus one name word is "'Smitty' Jones Jr." → family="" "'Smitty' Dr. Jones" → family="Jones" "'Smitty' Sir John" → given="John" - history: decisions.md#N3 · interacts: H1 · implemented: nameparser/_pipeline/_assign.py + Accepted: a marker-led clause is a maiden clause and not a + nickname one (M3), so this rule does not reach a name written + that way, and the one name word keeps the reading the bare + spelling gives it rather than becoming the family name. The + empty family that leaves is the bare spelling's, and #445 tracks + whether it is right — 1.4.0 read the bare spelling as first + Smith, middle née, last Jones, emptying nothing, and the same + empty family has been repaired twice already, at H1 (#410) and at + P5's reserve (#411). + "Smith (née Jones)" → given="Smith" + history: decisions.md#N3 · interacts: H1, M3 · implemented: nameparser/_pipeline/_assign.py ## Maiden names (M) -Background: a maiden name is written beside the current name, set off by a marker word or by enclosure. Markers are attested across French née/né, German geb./geborene, Dutch geboren, Czech/Slovak roz./rozená, Scandinavian født/fødd/född, Russian урожд. (both ё and е spellings), and Japanese 旧姓 — both grammatical genders where attested. Japanese more often writes the marker with a fullwidth colon (旧姓：佐藤), which is no separator, so marker and name arrive as a single word. Which enclosures mean "maiden" rather than "nickname" is a caller convention, so the maiden reading of a delimiter pair is opt-in. +Background: a maiden name is written beside the current name, set off by a marker word or by enclosure. Markers are attested across French née/né, German geb./geborene, Dutch geboren, Czech/Slovak rozená (the abbreviation roz. shipped through 2.1 and was removed in 2.2 -- it collides with the English diminutive Roz, and a caller who needs it adds it to their own Lexicon), Scandinavian født/fødd/född, Russian урожд. (both ё and е spellings), and Japanese 旧姓 — both grammatical genders where attested. Japanese more often writes the marker with a fullwidth colon (旧姓：佐藤), which is no separator, so marker and name arrive as a single word. Which enclosures mean "maiden" rather than "nickname" is a caller convention, so the maiden reading of a delimiter pair is opt-in — except where the clause announces itself. A clause of two words or more led by a marker word has said which convention it means, and reads as the maiden name inside a nickname pair as well (M3) — unless its content is suffix-shaped, which S1 takes ahead of both. A lone marker word has said nothing, and neither has one the colon spelling above glues to the name. M1. Rationale: an enclosure the caller has declared to mean maiden holds the former family name; a recognized marker word inside it @@ -543,12 +554,17 @@ M1. Rationale: an enclosure the caller has declared to mean maiden clause keeps its word, which may itself be a surname (Nee). Clauses are independent: two enclosures read as one maiden name, each dropping or keeping its own marker. A pair configured for - both maiden and nickname reads maiden. + both maiden and nickname reads maiden. Configuring the pair is + what this rule needs for a clause that does not announce itself + — markerless content, and a lone marker word alike; a clause of + two words or more led by a recognized marker reads as the maiden + name inside a nickname pair as well (M3), the suffix-shaped + content S1 takes excepted there as it is here. "Jane Smith (née Jones)" maiden-parens → maiden="Jones" "Jane Smith (Nee)" maiden-parens → maiden="Nee" · boundary "Jane Smith (Nee) (Jones)" maiden-parens → maiden="Nee Jones" - "Jane Smith (née Jones)" → nickname="née Jones" · boundary - history: decisions.md#M1 · interacts: S1, M2 · implemented: nameparser/_pipeline/_extract.py, nameparser/_pipeline/_group.py + "Andrew Perkins (MBA)" maiden-parens → suffix="MBA" · boundary + history: decisions.md#M1 · interacts: S1, M2, M3 · implemented: nameparser/_pipeline/_extract.py, nameparser/_pipeline/_group.py M2. Rationale: a maiden marker announces that what follows it is the former family name; the marker is an announcement, not a name. @@ -599,6 +615,37 @@ M2. Rationale: a maiden marker announces that what follows it is the "John née Jones Smith Ma" → maiden="Jones Smith Ma" history: decisions.md#M2 · interacts: P2, P3, P5, R2, M1, S2, H1 · implemented: nameparser/_pipeline/_group.py +M3. Rationale: an enclosure says nothing about whether it means + maiden, but a recognized marker word inside it does — the clause + announces itself, so the caller does not have to declare the + pair. + A bracketed clause whose content opens with a recognized marker + word and carries a word after it reads as the maiden name, + whichever bucket the enclosing pair sits in — unless the content + is suffix-shaped, which S1 takes first — the marker itself + dropped, as M1 drops it. A marker with no word after it is just + a word in brackets, and so is a marker no separator divides from + the name, the fullwidth-colon spelling M2 records. Where the + pair is already configured for maiden names M1 governs and this + adds nothing. Being keyed on the content rather than on the + pair, this reaches a nickname pair's clause too, and M1's + independence then governs what it produces: where a marker-led + clause stands beside another marker-led clause, both read as + maiden and join into one maiden name, leaving no nickname. + "Jane Smith (née Jones)" → maiden="Jones" + "Jane (née Jones) Smith" → family="Smith" + "Jane Smith (née Jr.)" → suffix="Jr." + "Jane Smith (née)" → nickname="née" · boundary + Accepted: the word taken after the marker is not tested for + being a name word, so unlike M2's bare take this one does not + stop at a suffix word — the same two words read one way + bracketed and another way bare. This does not contradict the + S1 example above: S1 asks whether the WHOLE clause is + suffix-shaped, which the trailing period makes true of the one + and false of the other, so only the V clause reaches this rule. + "Jane Smith (née V)" → maiden="V" + history: decisions.md#M3 · interacts: M1, M2, S1, N1 · implemented: nameparser/_pipeline/_extract.py + ## Commas & structure (C) Background: a comma in a name signals one of two conventions — the listing form "Family, Given" or trailing credentials "Name, PhD" — and which is meant can only be judged from what stands after the first comma. Recognizing a credential run is by nature a vocabulary judgment, so this is the one structural decision that consults the suffix word lists. Which characters COUNT as the comma is part of the rule: the Arabic comma (U+060C) and the fullwidth comma (U+FF0C) both signal the listing form, while the ideographic comma (U+3001) is not a name-structure comma at all (#265). diff --git a/docs/release_log.rst b/docs/release_log.rst index 30d701ee..91a1bfbc 100644 --- a/docs/release_log.rst +++ b/docs/release_log.rst @@ -45,6 +45,10 @@ Release Log - Fix a bound given name joining past a credential that the suffix rule then takes, leaving no family: ``"abdul Smith Jr Ma"`` read given ``abdul Smith`` with no family, where ``"John Smith Jr Ma"`` reads family ``Smith`` with suffix ``Jr, Ma``. The join's reserve had re-derived the suffix rule's trailing walk by hand and left out one of its two forks -- a bare acronym such as ``Ma`` is a credential only with words to spare -- so it counted the acronym as a name word, joined, and the suffix rule then peeled it. The two stages now share one walk: the reserve tries the join, runs the suffix rule's own peel over the pieces as the join would leave them, and joins only when two name words became one and nothing else changed. ``"abdul Smith Jr Ma"`` reads family ``Smith``, suffix ``Jr, Ma``; ``"abdul Smith Ma"`` reads family ``Smith``, suffix ``Ma`` rather than joining the pair and keeping the credential as the family -- both as 1.4.0 read them and as ``John`` reads in their place; ``"abdul Smith Berg Ma"`` keeps its join, and a title word standing after the bound word joins again as it did in 1.4.0 (``"Berg, abdul Sir"`` reads given ``abdul Sir``) without the pair ever being read as a title (``"abdul Sheikh and Ahmad Bakar"`` reads given ``abdul Sheikh and Ahmad``). No name of the differential corpora has the shapes; the ``Ma`` rules examples carry the fix, byte-identical to 1.4.0 and moving against 2.0.0 and 2.1.0, and the title-word shapes are pinned in tests (closes #425) + - Remove the Czech/Slovak abbreviation ``roz.`` from the default maiden markers, where it had been eating English given names. Marker matching is whole-token, case-folded and period-insensitive, so ``Roz`` -- the ordinary diminutive of Rosalind -- was the same string as the marker, and a maiden marker takes every word after it: ``"Rosalind Roz Smith"`` read maiden ``Smith`` with **no family name at all**, and ``"Rosalind Roz Jones Smith"`` read maiden ``Jones Smith``. Both now read the way 1.4.0 read them, given ``Rosalind`` with ``Roz`` a middle name and ``Smith`` the family. The defect is as old as the marker vocabulary and belongs to the bare marker rule, not to the bracketed-clause change in the bullet below; it was found while reviewing that change, which would have widened it -- with ``roz`` still shipped, ``"Jane Smith (Roz Jones)"`` reads maiden ``Jones`` where 2.1.0 read nickname ``Roz Jones``. What stops working is the abbreviation itself, for Czech and Slovak callers: ``"Anna Nováková roz. Svobodová"`` now reads middle ``Nováková roz.``, family ``Svobodová`` rather than family ``Nováková``, maiden ``Svobodová`` -- which is also how 1.4.0 read it, so this is a return to the v1 reading rather than a new one. The full participle is untouched and needs no configuration: ``"Anna Nováková rozená Svobodová"`` still reads family ``Nováková``, maiden ``Svobodová``, and it is only the abbreviation that was ambiguous. A caller who wants the abbreviation back adds it to their own lexicon -- ``Parser(lexicon=dataclasses.replace(lex, maiden_markers=lex.maiden_markers | {"roz"}))`` -- which restores the old reading exactly. The default set is now 16 entries. No differential corpus name contains the string in any case or position, so nothing moves against any baseline (closes the collision found in #335's review) + + - Fix a bracketed maiden clause reading as a nickname because its brackets were not declared. ``"Jane Smith nee Jones"`` gave maiden ``Jones``, while ``"Jane Smith (née Jones)"`` gave nickname ``née Jones`` -- the parenthesis pair is a nickname delimiter by default and nothing looked inside it, so the marker word the writer put there was read as part of an alias. This is about every delimiter pair the parser ships, not only the parentheses these examples use: quotes are how a nickname is most often written, and ``"Jane Smith "née Jones""`` and ``"Jane Smith 'née Jones'"`` move from nickname to maiden exactly as the parenthesized spelling does. A bracketed clause whose content opens with a recognized marker word and carries a word after it now reads as the maiden name whichever bucket its pair sits in: ``"Jane Smith (née Jones)"`` reads family ``Smith``, maiden ``Jones``, and so does the Japanese ``"山田 花子（旧姓 佐藤）"`` (family 山田, given 花子, maiden 佐藤), which needed ``Policy(maiden_delimiters=...)`` through 2.1. The clause is lifted out as a region rather than having its brackets erased, which is what keeps an INTERIOR clause from eating the name behind it: ``"Jane (née Jones) Smith"`` reads given ``Jane``, family ``Smith``, maiden ``Jones``, where erasing the brackets would leave the bare ``"Jane née Jones Smith"``, whose marker takes every word to the end of the name -- maiden ``Jones Smith`` and no family at all. The closing bracket is a boundary the writer typed, and it is now where the maiden name stops. Two clauses beside each other no longer merge: ``"Jane "Janey" Smith (née Jones)"`` read nickname ``Janey née Jones`` through 2.1 and now reads nickname ``Janey``, maiden ``Jones``, each clause keeping its own role, and this reaches ``HumanName``. What does NOT move is a clause that says nothing about maiden names. A lone marker in brackets is just a word in brackets -- ``"Jane Smith (née)"`` stays nickname ``née``, as 1.4.0 read it -- because a marker with no name after it marks nothing, and because a one-word clause may be the surname it looks like (``Nee`` is Irish Ní/Nee and a Chinese romanization). A markerless clause stays a nickname too: ``"Cherice J. (Johnson) Williams"`` keeps nickname ``Johnson``, which is what ``Policy(maiden_delimiters=...)`` remains for -- the parenthesized birth surname is a real convention, and only a caller who knows the data can say that is what these brackets mean. One accepted consequence: the word after the marker is not tested for being a name word, so a bracketed clause does not stop at a credential where the bare form does -- ``"Jane Smith (née V)"`` reads maiden ``V`` while ``"Jane Smith née V"`` reads suffix ``V``. The clause goes to ``maiden`` whole and the bracket is its boundary, so the trailing-suffix rule never sees the word as trailing the name. One more shape moves with it, and it is the one to know about if you parse names that are a surname and a clause: a name that is ONLY a marker clause and one name word now reads that word as the given name rather than the family. ``"Smith (née Jones)"`` reads given ``Smith``, maiden ``Jones`` where every earlier version read family ``Smith``, nickname ``née Jones``. That is the bracketed form agreeing with the bare one -- ``"Smith née Jones"`` has read given ``Smith`` since 1.4.0 -- because a marker-led clause is not a nickname clause and so does not reach the rule that would have made the word a family name. The empty ``family`` it leaves behind is the bare form's behavior and is not new here, though it is not 1.4.0's either -- v1 read ``"Smith née Jones"`` as first ``Smith``, middle ``née``, last ``Jones``, and #445 tracks whether the family should be kept. Seven of the 1,080 differential corpus names move against the 2.1.0 and 2.0.0 baselines and six against 1.4.0: the Japanese name is the odd one, and against 1.4.0 it is not a new diff at all, since 1.4 read the fullwidth brackets as name text and had no nickname there to change hands. Eight corpus names have the shape and seven move (measured); the eighth is ``"Jane Smith (née Jr.)"``, which the suffix rule takes first. So the count says how few bracketed marker clauses the corpora hold rather than how narrow the change is -- six of the seven are hand-written rule or issue examples, only the Japanese name arriving from elsewhere, and the coverage that bounds the behavior is ``tests/v2/cases.py`` (closes #335) + - Fix a particle chain and a maiden name taking a trailing generational numeral as a name word: ``"John van der Berg V"`` read family ``van der Berg V`` and ``"John née Jones Smith V"`` read maiden ``Jones Smith V``, where ``"John Smith V"`` reads suffix ``V``. Both stops asked "is this a suffix?" with the test that vetoes a bare ``V`` as an initial -- the same question the bound given-name join asked until #401 -- and now ask the suffix rule's own walk, over the name as they would leave it: family ``van der Berg``, maiden ``Jones Smith``, suffix ``V``, for ``I`` and ``X`` alike; ``"John van der J. V"`` and ``"J. née Jones Smith V"`` keep their readings, the word before the numeral being an initial as written or as left. The chain stops before a bare credential with words to spare too, as ``"John Smith Ma"`` does, where the suffix rule will still find them once the chain has run: ``"John van der Berg Ma"`` reads suffix ``Ma``, which is how 1.4.0 read it, while ``"Freiherr von Berg MA"`` keeps family ``von Berg MA``; a maiden name keeps such a credential, since the words the marker takes are the very words that made it one. A particle that is also suffix vocabulary now ends the chain where it opens the trailing run: ``"John Smith Mc V"`` reads suffix ``Mc, V``. The chain also no longer swallows the given name behind an unlisted abbreviation: ``"Xyz. van Johnson"`` and ``"Esq. van Gogh"`` read given ``van``, as ``"Dr. van Johnson"`` has since #367, the chain and the bound given-name join now asking the same test that reads the abbreviation as a title (closes #424) - Fix a name losing its given/family split when a comma is followed only by an honorific: ``"John Smith, Mr."`` now gives given ``John``, family ``Smith``, title ``Mr.`` where it returned the whole of ``"John Smith"`` as the family name. The family-comma rule that the part before the comma is wholly the surname rests on the writer having marked where the surname ends, and a comma followed by nothing but titles marked no such thing -- ``"John Smith, Mr."`` is ``"Mr. John Smith"`` with the honorific moved to the end. A comma followed by an actual name still fixes the family (``"John Smith, Jones"`` keeps family ``John Smith``), a title followed by a postnominal is each read where it stands (``"Smith, Dr. Jr."`` is unchanged), and a single pre-comma piece has no split to keep, so ``"Smith, Dr."`` is unchanged. Because the pre-comma name is now read positionally, it also picks up the two things the merged read suppressed: the script-order table, the declared name order (``"de Mesnil Juan, Dr."`` keeps family ``de Mesnil`` under a family-first order) and the particle-or-given ambiguity report (``"Van Johnson, Mr."``). The same holds when the comma is followed by titles and postnominals with no name word between them: ``"John Smith, Mr. Jr."`` keeps its split, while ``"Smith Jr., Mr."`` -- one name word -- keeps its family diff --git a/docs/usage.rst b/docs/usage.rst index 5aedd40e..13017133 100644 --- a/docs/usage.rst +++ b/docs/usage.rst @@ -655,9 +655,36 @@ instead of relying on the default: >>> parse(text).maiden 'Jones' -Delimited content is not always a nickname. If what's inside is a known +Delimited content is not always a nickname. If it opens with a marker +word and has a word after it, the clause is a maiden name, whatever +pair encloses it and with nothing configured — the clause has said which +convention it means, so you do not have to: + +.. doctest:: + + >>> parse("Jane Smith (née Jones)").maiden + 'Jones' + >>> parse('Jane Smith "née Jones"').maiden + 'Jones' + >>> parse("Jane (née Jones) Smith").family + 'Smith' + +A marker with no name after it is just a word in brackets, and a clause +with no marker at all stays a nickname — the parenthesized birth surname +is a real convention, but nothing in the clause says so, and only you +can declare that with ``maiden_delimiters`` (see :doc:`customize`): + +.. doctest:: + + >>> parse("Jane Smith (née)").nickname + 'née' + >>> parse("Cherice J. (Johnson) Williams").nickname + 'Johnson' + +If what's inside is a known suffix, or simply ends in a period, it is read as a suffix instead — -parenthesized credentials and retired ranks are far more common than +that reading is taken before the maiden one, and parenthesized +credentials and retired ranks are far more common than parenthesized nicknames that happen to be credentials: .. doctest:: diff --git a/nameparser/_lexicon.py b/nameparser/_lexicon.py index fc6d2abe..5c0a2cf6 100644 --- a/nameparser/_lexicon.py +++ b/nameparser/_lexicon.py @@ -366,7 +366,7 @@ class Lexicon: #: :data:`~nameparser.config.bound_given_names.BOUND_GIVEN_NAMES`. bound_given_names: frozenset[str] = frozenset() #: Marker words introducing a birth surname, routed to the maiden - #: field ("née", "geb.", "roz.", ...). Full default list: + #: field ("née", "geb.", "rozená", ...). Full default list: #: :data:`~nameparser.config.maiden_markers.MAIDEN_MARKERS`. maiden_markers: frozenset[str] = frozenset() #: Family names for the unspaced-name segmentation stage (#271), diff --git a/nameparser/_pipeline/_extract.py b/nameparser/_pipeline/_extract.py index 3e33af62..da5f876c 100644 --- a/nameparser/_pipeline/_extract.py +++ b/nameparser/_pipeline/_extract.py @@ -7,11 +7,16 @@ A Role.MAIDEN region is the WHOLE inner span, marker word included -- nothing here strips one. classify tags a marker inside it like any other token, and group drops it from a multi-token clause (#329). -Reads: Policy.nickname_delimiters, Policy.maiden_delimiters, and +A region reaches that role two ways: the pair that matched sits in +Policy.maiden_delimiters (M1), or the content itself opens with a +marker word (M3), which reassigns the role after the match and so is +the one thing here that a bucket alone does not decide. +Reads: Policy.nickname_delimiters, Policy.maiden_delimiters, +Lexicon.maiden_markers, and Lexicon.suffix_words / suffix_acronyms / suffix_acronyms_ambiguous through _suffix_shaped. -Implements rules N1, N2, S1 and M1 of docs/design/rules.md (the #273 +Implements rules N1, N2, S1, M1 and M3 of docs/design/rules.md (the #273 matching mechanism); each is cited at its code below. One scan mechanic worth stating up front: matching is one left-to-right pass, no nesting, and delimiter characters inside a matched region are @@ -52,6 +57,24 @@ def _suffix_shaped(content: str, lexicon: Lexicon) -> bool: or content.endswith(".")) +# rules.md#M3: "a bracketed clause whose content opens with a +# recognized marker word and carries a word after it reads as the +# maiden name, whichever bucket the enclosing pair sits in" +def _maiden_marked(content: str, lexicon: Lexicon) -> bool: + """The clause says 'maiden' out loud, so the caller does not have to + say it in Policy. Requires a word AFTER the marker: a lone marker in + brackets is a word in brackets, and M1 deliberately keeps a one-word + clause's word (it may be the surname Nee). The word after is not + tested for anything -- M3's Accepted line, and the reason a + bracketed '(née V)' reads maiden 'V' where the bare 'née V' gives + M2 a suffix. Whitespace-split, so a marker the writer glued to + punctuation is not one here ('née,'); the tokenizer splits that + comma off and still tags the token, which is what keeps _group's + Role.MAIDEN filter reachable.""" + words = content.split() + return len(words) > 1 and _normalize(words[0]) in lexicon.maiden_markers + + # rules.md#N2: "a quote whose open and close are the same character # opens only at a word start and closes only at a word end, so an # apostrophe inside or at the end of a word is literal" @@ -203,6 +226,21 @@ def extract_delimited(state: ParseState) -> ParseState: masked.append(Span(j, j + len(close))) else: if inner.start < inner.end: + # M3 upgrades a nickname clause; a configured maiden + # pair is M1's and is left alone. The role test is + # False whenever a maiden pair matched, but it cannot + # change the OUTCOME, and no test can catch its + # removal: `order` above holds exactly two roles, so a + # role that is not NICKNAME is already MAIDEN and the + # assignment would be a no-op either way. It is kept + # for the day `order` gains a third bucket, when it + # becomes the difference between M3 claiming that + # bucket's clauses and leaving them. Measured + # 2026-08-26: dropping it leaves the suite and all + # three gates green. + if (role is Role.NICKNAME and _maiden_marked( + text[inner.start:inner.end], state.lexicon)): + role = Role.MAIDEN extracted.append((role, inner)) masked.append(Span(i, j + len(close))) # position-driven scanning makes overlapping matches diff --git a/nameparser/_policy.py b/nameparser/_policy.py index c8978274..a3000f04 100644 --- a/nameparser/_policy.py +++ b/nameparser/_policy.py @@ -623,9 +623,17 @@ class Policy: #: field instead; a pair listed here is dropped from the effective #: nickname set (maiden wins, see __post_init__), so #: maiden_delimiters=frozenset({("(", ")")}) is the whole recipe (#274). - #: A maiden_markers word opening the enclosed content is dropped - #: from the value, but only where that content holds more than one - #: token: a lone "(Nee)" is a maiden NAME, not a marker (#329). + #: Set this for a clause that says nothing about itself, which is + #: two kinds and not one: content with no marker word in it, and a + #: LONE marker word. Since #335 a clause that opens with a marker + #: word AND has a word after it reads as the maiden name whatever + #: pair encloses it, so "Jane Smith (née Jones)" needs no + #: configuration -- unless the content is suffix-shaped, which is + #: taken first, the brackets dropped and the content read as if + #: written bare. A maiden_markers word opening the enclosed + #: content is dropped from the value, but only where that content + #: holds more than one token: a lone "(Nee)" is a maiden NAME, + #: not a marker (#329). maiden_delimiters: frozenset[tuple[str, str]] = frozenset() #: Additional separators that split suffix groups (e.g. " - " for #: "Jane Smith, RN - CRNA"). Additive only: the comma always diff --git a/nameparser/config/maiden_markers.py b/nameparser/config/maiden_markers.py index d064333d..e9d26265 100644 --- a/nameparser/config/maiden_markers.py +++ b/nameparser/config/maiden_markers.py @@ -7,7 +7,6 @@ 'geb', 'geborene', 'geboren', - 'roz', 'rozená', 'født', 'fødd', @@ -22,13 +21,15 @@ """ Marker words that introduce a birth surname, e.g. "Jane Smith née Jones" (#274). French née/né/nee, German geb./geborene, Dutch geboren, -Czech/Slovak roz./rozená, Danish/Norwegian født (Nynorsk fødd), Swedish +Czech/Slovak rozená, Danish/Norwegian født (Nynorsk fødd), Swedish född, Russian урожд./урождённая/урождённый (both ё and е spellings — case normalization does not fold them, and running text routinely writes е). Both grammatical genders are listed where #274 or review attested them (née/né, урождённая/урождённый); Czech masculine rozený -awaits the same vetting. Entries are stored normalized: lowercase, no -periods. +awaits the same vetting, and would be vetted on its own merits rather +than following the feminine form in — the abbreviation roz. shared by +both genders is deliberately absent, below. Entries are stored +normalized: lowercase, no periods. Japanese 旧姓 is here rather than in locales.JA, on the rule that admitted the Cyrillic entries: a native-script marker cannot collide @@ -41,29 +42,51 @@ Matching being whole-token, the marker has to BE a token -- which for Japanese means something has to divide it from the name it marks. A -space does, and so does a configured delimiter: extract masks the -whole bracketed region, delimiter characters included, before tokenize -runs, so a bracket bounds a token exactly as a space does and -"山田（旧姓 佐藤）" needs no space in front of 旧姓 at all. The bare -"山田花子 旧姓 佐藤" and -- since #329 -- the bracketed -"山田 花子（旧姓 佐藤）" under Policy(maiden_delimiters=...) alike give -maiden 佐藤. What divides nothing is the fullwidth colon that the form +space does, and so does a delimiter: extract masks the whole bracketed +region, delimiter characters included, before tokenize runs, so a +bracket bounds a token exactly as a space does and "山田（旧姓 佐藤）" +needs no space in front of 旧姓 at all. The bare "山田花子 旧姓 佐藤" +and the bracketed "山田 花子（旧姓 佐藤）" alike give maiden 佐藤, and +since #335 the bracketed form needs no configuration to do it: the +fullwidth pair is a NICKNAME delimiter by default, and rules.md#M3 +reads a clause that opens with a marker word AND carries a word after +it as the maiden name, whichever bucket its pair sits in. The second +word is part of the condition, not a detail of it -- a lone "（旧姓）" +is a word in brackets and stays a nickname. #329, which drops the marker from inside the +clause, is what makes the value 佐藤 rather than "旧姓 佐藤". Declaring +the pair in Policy(maiden_delimiters=...) reaches the same reading by +M1's path. What divides nothing is the fullwidth colon that the form Japanese more often writes puts after the marker: "山田（旧姓：佐藤）" -still yields maiden "旧姓：佐藤" with the marker and its colon attached. -Not because delimited content escapes classification -- classify tags -a marker wherever it is a token -- but because ： is no separator -tokenize knows, so marker and name arrive as ONE token and there is -nothing to drop. The wholly unspaced "山田花子（旧姓佐藤）" reads as one -token for the same reason. Peeling a marker off the head of a token is -#317's job. +under Policy(maiden_delimiters=...) still yields maiden "旧姓：佐藤" +with the marker and its colon attached. Not because delimited content +escapes classification -- classify tags a marker wherever it is a +token -- but because ： is no separator tokenize knows, so marker and +name arrive as ONE token and there is nothing to drop. M3 does not +reach that form either, and by its own test rather than by tokenize's: +it splits the clause on WHITESPACE, and "旧姓：佐藤" is one whitespace +word, so there is no marker word for the clause to open with and by +default it stays a nickname. The two tests agree here and are +deliberately not the same test -- see decisions.md#M3. The wholly +unspaced "山田花子（旧姓佐藤）" reads as one token for the same reason. +Peeling a marker off the head of a token is #317's job. Consumed by the 2.0 parser's default lexicon. The 1.x parser does not read this module. Deliberately absent: Polish "z domu" (a two-token marker; pending the -2.0 pipeline's multi-token matching decision) and the Scandinavian +2.0 pipeline's multi-token matching decision), the Scandinavian abbreviation "f." (collides with the initial "F." — only the full -participles are safe). +participles are safe), and the Czech/Slovak abbreviation "roz." +(shipped through 2.1 and removed in 2.2). Roz is an ordinary English +diminutive of Rosalind, and matching is whole-token, case-folded and +period-insensitive, so "Roz", "roz" and "roz." are one string to this +set: with it shipped, "Rosalind Roz Smith" read maiden "Smith" and NO +family name at all. The collision is in the position the claim acts +on, which is the test decisions.md#vocabulary-collisions states, and +this set has no ambiguous subset to express a partial answer in — see +decisions.md's Excluded (MAIDEN_MARKERS) block. The full participle +rozená stays, being unambiguous; a caller who needs the abbreviation +adds it to their own Lexicon. """ diff --git a/tests/v2/cases.py b/tests/v2/cases.py index 82f13360..b78b278f 100644 --- a/tests/v2/cases.py +++ b/tests/v2/cases.py @@ -18,6 +18,17 @@ The v1 suite's full corpus is extracted into this table by the migration plan (facade runner consumes the same rows); this file seeds it with the pinned battery. + +What earns a row is a FORK: a branch taken for one input and not +another -- a rule's boundary, a precedence contest between two rules, +a policy that changes the answer. A row demonstrating one more member +of a vocabulary set pins nothing its other members do not, and one row +per entry grows this table without narrowing what can break +(mechanisms.md#VOCABULARY-EXERCISES-FORKS -- a restatement, and +nothing checks it against the entry: test_doc_citations verifies an +excerpt only where the reference is followed by a colon and a quoted +span, which running prose like this one is not. Read the entry, not +this paragraph, if the two ever disagree). """ from __future__ import annotations @@ -784,6 +795,50 @@ def __post_init__(self) -> None: {"given": "Jane", "family": "Smith", "maiden": "Jones"}, classification="fix(#274)", notes="v1 mangles to middle='Smith née'"), + Case("diminutive_that_was_a_marker_keeps_the_family", + "Rosalind Roz Smith", + {"given": "Rosalind", "middle": "Roz", "family": "Smith"}, + notes="'roz', the Czech/Slovak abbreviation, shipped in " + "MAIDEN_MARKERS through 2.1 and collided with the " + "English diminutive of Rosalind -- matching is " + "whole-token, case-folded and period-insensitive, so " + "Roz, roz and roz. are one string. This name read " + "maiden 'Smith' with NO family name at all (measured on " + "the pre-removal tree), because M2 hands the marker " + "every word after it. The entry is gone in 2.2, which " + "is what this row pins. Nothing to do with the " + "delimited path, though M3 would have widened it: " + "the defect is M2's and predates #335, and the one-word " + "'(Roz)' spelling was never affected since M3 declines " + "a lone marker -- but 'Jane Smith (Roz Jones)' reads " + "maiden 'Jones' with the entry restored, where 2.1.0 " + "read nickname 'Roz Jones' (measured 2026-08-26). " + "Parity, and it is " + "RESTORED parity rather than untouched -- 1.4.0 has no " + "maiden support and read first Rosalind / middle Roz / " + "last Smith (2026-08-26), which is where the removal " + "puts this name back"), + Case("full_participle_marker_still_consumes", + "Anna Nováková rozená Svobodová", + {"given": "Anna", "family": "Nováková", "maiden": "Svobodová"}, + classification="fix(#274)", + notes="the other half of the roz removal, and the reason it " + "was a removal and not a retreat from Czech: the full " + "participle rozená stays, being a word no one is " + "called. Pinned because deleting a vocabulary entry " + "invites deleting its neighbours, and because nothing " + "else in the suite reaches this entry: removing " + "rozená from MAIDEN_MARKERS fails exactly this row's " + "two tests, one per runner, and nothing else (measured " + "2026-08-26) -- the position " + "maiden_marker_delimited_unaccented holds for 'nee'. " + "The cost the removal accepts is the " + "abbreviation: 'Anna Nováková roz. Svobodová' now reads " + "middle 'Nováková roz.', family 'Svobodová' (measured), " + "which is exactly how 1.4.0 read it. 1.4.0 read this " + "row middle 'Nováková rozená' / last Svobodová " + "(2026-08-26) -- the marker inside the name, the " + "ordinary v1 reading of every marker"), Case("maiden_marker_after_particle_chain", "Ursula von der Leyen geb. Albrecht", {"given": "Ursula", "family": "von der Leyen", @@ -1209,7 +1264,15 @@ def __post_init__(self) -> None: "nickname_delimiters.pop('parenthesis'), it gave first " "Jane / last Smith / maiden 'née Jones' -- same name " "fields, marker still inside the value, which is the " - "single field this change moves"), + "single field this change moves. Since #335 the same " + "input reads identically with NO policy at all " + "(maiden_marked_clause_reads_maiden_by_default below), " + "which does not make this row redundant: the pair " + "sitting in the maiden bucket settles the role before " + "M3 is consulted, so this row exercises M1's path and " + "that one exercises M3's. Rewriting it to drop the " + "policy would delete the configured path's coverage " + "rather than move it"), Case("maiden_marker_delimited_unaccented", "Jane Smith (nee Jones)", {"given": "Jane", "family": "Smith", "maiden": "Jones"}, policy=Policy(maiden_delimiters=frozenset({("(", ")")})), @@ -1228,6 +1291,163 @@ def __post_init__(self) -> None: "maiden 'nee Jones' (2026-08-03) -- the same diff the " "accented row records, which is the point: the two " "spellings behave alike on both sides"), + Case("maiden_marked_clause_reads_maiden_by_default", + "Jane Smith (née Jones)", + {"given": "Jane", "family": "Smith", "maiden": "Jones"}, + classification="fix(#335)", + notes="rules.md#M3 -- the clause says 'maiden' out loud, so the " + "pair enclosing it does not have to be configured. " + "maiden_marker_delimited above is the same input under " + "Policy(maiden_delimiters=...) and reads identically -- " + "what M3 adds is the DEFAULT reading, where 1.4.0 and " + "2.1 alike gave nickname 'née Jones'"), + Case("maiden_marked_clause_interior_keeps_the_family", + "Jane (née Jones) Smith", + {"given": "Jane", "family": "Smith", "maiden": "Jones"}, + classification="fix(#335)", + notes="the row that decides the MECHANISM. Extracting the " + "clause as a Role.MAIDEN region keeps the closing " + "delimiter as the maiden name's right boundary; masking " + "the delimiters and letting M2's bare-marker rule " + "consume the content instead would read maiden 'Jones " + "Smith' with an empty family, because M2's take runs to " + "the end of the name. The parens say where it stops"), + Case("maiden_marked_clause_one_word_stays_a_nickname", + "Jane Smith (née)", + {"given": "Jane", "family": "Smith", "nickname": "née"}, + notes="M3's boundary: a marker with no word after it is not a " + "maiden clause. Without this condition the default " + "reading of a lone parenthesized marker would flip to " + "maiden 'née', and M1's own (Nee) boundary -- a " + "one-word clause keeps its word, which may be the " + "surname Nee -- would be contradicted on the " + "unconfigured path. Parity: 1.4.0 read nickname 'née'"), + Case("markerless_parenthesized_clause_stays_a_nickname", + "Cherice J. (Mary Johnson) Williams", + {"given": "Cherice", "middle": "J.", "family": "Williams", + "nickname": "Mary Johnson"}, + notes="M3's other boundary, and the reason the maiden " + "delimiters remain worth configuring: the parenthesized " + "birth surname without a marker is a real US convention " + "-- 'Cherice J. (Johnson) Williams' is the corpus name " + "(corpus_issues.jsonl) -- but nothing in the clause " + "says 'maiden', so it stays a nickname by default. Only " + "a caller who knows their data can say otherwise, which " + "is what Policy(maiden_delimiters=...) is for. The " + "clause is TWO words here, and that is the whole point " + "of the row: M3 tests the clause length before it tests " + "the vocabulary, so the corpus spelling's one-word " + "clause is refused by the length condition and never " + "reaches the vocabulary one -- it would duplicate " + "maiden_marked_clause_one_word_stays_a_nickname rather " + "than fence the other condition. Measured 2026-08-26: " + "with the vocabulary test dropped this reads maiden " + "'Mary Johnson' -- the WHOLE clause, because #329's " + "drop is gated on the first token carrying " + "vocab:maiden-marker and 'Mary' does not, so nothing is " + "dropped -- and with the vocabulary test in place the " + "one-word spelling reads nickname either way. Parity: " + "1.4.0 and 2.1.0 both read nickname 'Mary Johnson'"), + Case("markerless_one_word_clause_stays_a_nickname", + "Cherice J. (Johnson) Williams", + {"given": "Cherice", "middle": "J.", "family": "Williams", + "nickname": "Johnson"}, + notes="the corpus spelling (corpus_issues.jsonl) of the row " + "above, kept beside it rather than replaced by it. It " + "reaches M3's length condition and stops there, so it " + "cannot fence the vocabulary one -- which is why the " + "row above widens the clause to two words -- but it is " + "the name real US data actually carries, and a row for " + "the corpus name is worth its two lines. Parity"), + Case("maiden_marker_not_first_stays_a_nickname", + "Jane Smith (Jones née)", + {"given": "Jane", "family": "Smith", "nickname": "Jones née"}, + notes="the OPENS-WITH half of M3, which nothing else reaches: " + "a marker inside the clause but not first leaves the " + "clause a nickname. Measured 2026-08-26, widening the " + "predicate from the first word to any word left the " + "whole suite green and all three gates at 0 unexplained " + "-- this row is what closes that. The bracketed twin of " + "maiden_marker_delimited_trailing_marker, which pins the " + "same asymmetry one layer down, inside a clause already " + "routed to maiden by policy: no marker the shipped " + "vocabulary carries is written after the name it marks. " + "Parity: 1.4.0 and 2.1.0 both read nickname 'Jones née'"), + Case("marker_led_clause_with_one_name_word", + "Smith (née Jones)", + {"given": "Smith", "maiden": "Jones"}, + classification="fix(#335)", + notes="N3's shape meeting M3, and the row exists because the " + "two rules disagree about what a clause is. N3 reads a " + "name that is only a nickname plus one name word as " + "'that word is the family name' -- but a marker-led " + "clause is not a nickname clause, so N3 never sees this " + "one and 'Smith' keeps the given-name reading the bare " + "spelling gives it. Recorded as an N3 Accepted line. " + "The empty family is the bare path's, not this " + "change's: 'Smith née Jones' reads given 'Smith', " + "family '' on 2.1.0 too, and 1.4.0 read first 'Smith' / " + "middle 'née' / last 'Jones' -- the marker as a " + "middle name. The parallel evidence at 1.4.0 is not " + "the fix(#410) maiden-flavor rule, which is keyed on " + "the TITLED 'Dr. Smith née Jones', but fix(#274), " + "which claims the bare one-word names 'Janey née " + "Jones' and 'abd née Jones' and declares `family` for " + "exactly the reason this row records. The given side " + "has agreed since 1.4.0, " + "the emptying is 2.x's and is tracked separately as the " + "#410 analogue. What #335 moves is only the bracketed " + "spelling: 1.4.0, 2.0.0 and 2.1.0 all read it family " + "'Smith', nickname 'née Jones' (measured 2026-08-26)"), + Case("marker_led_clause_in_a_quote_pair", + 'Jane Smith "née Jones"', + {"given": "Jane", "family": "Smith", "maiden": "Jones"}, + classification="fix(#335)", + notes="M3 is keyed on the CONTENT, not on which pair matched, " + "and this is the row that says so in the commonest " + "spelling: a quote pair is how nicknames are usually " + "written, and the same clause inside one reads maiden " + "exactly as it does inside parentheses. Three of the " + "eleven shipped nickname pairs are exercised by a " + "marker-led clause anywhere in the suite -- this one, " + "the parenthesis, and the fullwidth pair below -- and " + "the other EIGHT have no row. Measured 2026-08-26 by " + "disabling the swap one pair at a time: those three " + "redden and the eight do not. That is a count over a " + "wordlist, so read it the way " + "mechanisms.md#VOCABULARY-EXERCISES-FORKS says to: the " + "eight are not eight gaps, since the pairs fork on " + "whether open and close are the same character and on " + "the apostrophe carve-out inside that, not on which " + "pair. This row is here because a quote pair is the " + "same-character branch, and because the release note " + "advertises the spelling. 1.4.0 and 2.1.0 both read " + "nickname 'née Jones'"), + Case("maiden_marked_clause_takes_the_suffix_reading_from_s1", + "Jane Smith (née Jr.)", + {"given": "Jane", "middle": "Smith", "family": "née", + "suffix": "Jr."}, + notes="S1 takes a suffix-shaped clause before M3 is " + "consulted, and the whole reading is here because the " + "surprising part is not the suffix: it is that the " + "MARKER becomes the family name. S1 drops the brackets " + "and lets the content read as if written bare, and " + "bare 'Jane Smith née Jr.' has no name word after the " + "marker for M2 to take, so 'née' stays an ordinary " + "word and lands in the family. rules.md#M3 carries the " + "same input as an example line, but the runner checks " + "one field per line; this row is the other three. " + "Parity, and unchanged by #335 -- 1.4.0 and 2.1.0 read " + "it the same way, which is why the corpus row it added " + "diffs against no baseline"), + Case("maiden_marked_clause_beside_a_nickname", + 'Jane "Janey" Smith (née Jones)', + {"given": "Jane", "family": "Smith", "nickname": "Janey", + "maiden": "Jones"}, + classification="fix(#335)", + notes="two clauses, two roles. Through 2.1 both were " + "nicknames and the facade joined them into one value, " + "'Janey née Jones' -- the merged-nickname half of #335"), Case("maiden_marker_delimited_unmarked_content", "Jane Smith (Mary Jones)", {"given": "Jane", "family": "Smith", "maiden": "Mary Jones"}, @@ -1286,19 +1506,27 @@ def __post_init__(self) -> None: "Smith / maiden 'Jones née'"), Case("maiden_marker_delimited_beside_a_nickname_clause", 'Jane "née Janie" Smith {née Jones}', - {"given": "Jane", "family": "Smith", "nickname": "née Janie", - "maiden": "Jones"}, + {"given": "Jane", "family": "Smith", "maiden": "Janie Jones"}, policy=Policy(maiden_delimiters=frozenset({("{", "}")})), - classification="fix(#329)", - notes="the pass is scoped to MAIDEN clauses, and this is the " - "row that says so: two extracted clauses, both opening " - "with a marker word, and only the maiden one loses it. " - "Braces route to maiden here precisely so the default " - "nickname set survives untouched -- the parenthesis " - "rows above cannot show this, since Policy's " - "maiden-wins canonicalization would take ( ) away from " - "nickname. Without the role filter the nickname reads " - "'Janie'. 1.4.0 cannot express a brace delimiter at " + classification="fix(#335)", + notes="#335 took this row's job away, and the row is kept to " + "record that. It was the pin for the #329 drop pass " + "being scoped to MAIDEN clauses -- two extracted " + "clauses, both opening with a marker word, only the " + "maiden one losing it, nickname 'née Janie' and maiden " + "'Jones'. M3 now reads the QUOTED clause as maiden too, " + "since it is marker-led like the braced one and M3 is " + "keyed on content rather than on which pair matched, so " + "there is no nickname left to contrast: both clauses " + "are maiden and M1's independence rule joins them into " + "one value. The role filter it used to discriminate " + "(the 'role is not Role.MAIDEN' branch of " + "_group.group's drop pass) is still reachable and " + "still pinned -- " + "that job moved to " + "marker_glued_to_punctuation_keeps_the_clause_a_nickname " + "below, which reaches a marker-led clause M3 declines. " + "1.4.0 cannot express a brace delimiter at " "all (its buckets hold the NAMES of compiled regexes " "and there is no brace one; measured 2026-08-03, " "maiden_delimiters['brace'] = ('{', '}') is accepted " @@ -1308,8 +1536,39 @@ def __post_init__(self) -> None: "first Jane / middle 'Smith {née' / last 'Jones}' / " "nickname 'née Janie' -- braces as name text, the same " "convention maiden_marker_kyusei_delimited uses for a " - "knob with no v1 spelling. The nickname agreed even " - "there"), + "knob with no v1 spelling (re-measured 2026-08-26, " + "unchanged)"), + Case("marker_glued_to_punctuation_keeps_the_clause_a_nickname", + 'Jane "née, Janie" Smith (née Jones)', + {"given": "Jane", "family": "Smith", "nickname": "née Janie", + "maiden": "Jones"}, + classification="fix(#335)", + notes="M3 and the #329 drop pass ask the marker question of " + "different things, and this row is where the two " + "answers differ. M3 splits the clause on WHITESPACE and " + "normalizes the first word: 'née,' normalizes to " + "'née,' -- _normalize strips a trailing period but not " + "a comma -- so M3 declines and the quoted clause stays " + "a nickname. tokenize splits the comma off as a " + "separator, so the clause's first TOKEN is 'née' and " + "carries vocab:maiden-marker, which is exactly what " + "the 'role is not Role.MAIDEN' branch of " + "_group.group's drop pass exists to refuse. Measured 2026-08-26: with that " + "branch removed this reads nickname 'Janie', the " + "marker dropped out of a nickname. BOTH clauses are " + "load-bearing -- the drop pass is gated on the name " + "holding a MAIDEN region at all, so the same quoted " + "clause alone ('Jane \"née, Janie\" Smith') leaves the " + "branch unexercised, removing it measurably changes " + "nothing there. The paren clause is what opens the " + "block, and M3 is what makes it maiden. The comma is " + "absent from the nickname VALUE because tokenize " + "treats COMMA_CHARS as a separator inside every region " + "including an extracted one, which predates #335 and " + "is not part of it. 1.4.0 gave first Jane / last Smith " + "/ nickname 'née, Janie née Jones' (2026-08-26) -- " + "comma kept, both clauses merged into the one field, " + "which is the merged-nickname half of #335"), Case("maiden_marker_delimited_two_clauses", "Jane Smith (Nee) (Jones)", {"given": "Jane", "family": "Smith", "maiden": "Nee Jones"}, @@ -1412,10 +1671,46 @@ def __post_init__(self) -> None: "ko_honorific_period_under_strict_comma_suffixes uses " "for a knob with no v1 spelling. That reading is also " "what the differential harness sees, since it runs the " - "corpus under the DEFAULT policy where （） is a #273 " - "NICKNAME delimiter and nothing in #329 is reachable; " - "the diff is classified there under " - "fix(cjk-fullwidth-paren-nickname)"), + "corpus under the DEFAULT policy. What the harness does " + "with it changed in 2.2: through 2.1 the （） pair was a " + "#273 NICKNAME delimiter and nothing in #329 was " + "reachable, so the diff classified under " + "fix(cjk-fullwidth-paren-nickname). Since #335 the marker " + "inside the clause is enough on its own, so this name " + "reads maiden under the default policy too -- see " + "maiden_marked_fullwidth_clause_by_default below -- and " + "the diff classifies under fix(#335) at 2.1.0 and 2.0.0 " + "while at 1.4.0 it moved to fix(cjk-maiden-marker), " + "leaving the fullwidth-paren rule dormant in that ledger. " + "This row keeps its policy because M1 still governs a " + "configured pair and settles the role before M3 is " + "consulted"), + Case("maiden_marked_fullwidth_clause_by_default", + "山田 花子（旧姓 佐藤）", + {"given": "花子", "family": "山田", "maiden": "佐藤"}, + classification="feat(#273) + fix(#271) + fix(#335)", + notes="the row above without its policy, and the one " + "that fences M3 across the delimiter SET rather " + "than at the parenthesis. Measured 2026-08-26 by " + "gating the swap to '(' and the double quote: this row " + "and its facade twin are the only two failures in the " + "suite, and of the gates only 1.4.0 and 2.0.0 redden " + "-- 2.1.0 stays green with its fix(#335) rule quietly " + "falling from six names to five, while the 2.0.0 " + "ledger catches it on the four-field rule it gives " + "this name. The fullwidth pair is the " + "one the maiden_markers docstring and the 2.2 " + "release note both advertise as newly working " + "without configuration, so it is the one that " + "most needs a row. Three changes compound in the " + "classification: #273 taught the parser the " + "fullwidth pair, #271 gives the wholly-Han " + "remainder its family-first reading, and #335 " + "makes the marker inside the clause enough on its " + "own. 1.4.0 read first 山田 / middle '花子（旧姓' / " + "last '佐藤）' with the brackets as name text; " + "2.1.0 read given 花子 / family 山田 / nickname " + "'旧姓 佐藤' (both measured 2026-08-26)"), Case("east_slavic", "Сидоров Иван Петрович", {"given": "Иван", "middle": "Петрович", "family": "Сидоров"}, policy=_ES), diff --git a/tests/v2/test_ledger_guards.py b/tests/v2/test_ledger_guards.py index 819c5c57..b5391d03 100644 --- a/tests/v2/test_ledger_guards.py +++ b/tests/v2/test_ledger_guards.py @@ -35,7 +35,7 @@ class declares, which members an alternation offers. Those are exact import pytest -from nameparser import _policy +from nameparser import DEFAULT_NICKNAME_DELIMITERS, _policy from nameparser._policy import Script # The parser's own fold, imported rather than reimplemented: a # hand-written one here stripped commas, parens, brackets and quotes, @@ -835,8 +835,8 @@ class _LatinCopy(NamedTuple): #: by a substring of the rule's `issue`. Kept apart from #: _HONORIFIC_SOURCES because the relationship is not set equality: #: these members are regex FRAGMENTS, not entries -- "n[ée]e" covers two -#: markers at once, "geb\.?" and "roz\.?" one each -- so there is no set -#: to compare against. +#: markers at once and "geb\.?" one -- so there is no set to compare +#: against. #: #: `covers` is recorded rather than equated to the whole vocabulary. #: Equality would force a rule to grow alternatives for markers it has @@ -845,13 +845,15 @@ class _LatinCopy(NamedTuple): #: removal: drop an entry a member covers and the snapshot shrinks. #: #: Three nearby counts differ and are easy to conflate, all for -#: fix(#274) specifically: MAIDEN_MARKERS ships 17 entries; that rule's -#: members reach 4 of them; the corpora contain 3 markers in total -#: (geb, née, 旧姓), only 2 of which it covers. +#: fix(#274) specifically, and all four numbers moved in 2.2 -- recount +#: rather than adjust them: MAIDEN_MARKERS ships 16 entries (roz left +#: the vocabulary); that rule's members reach 3 of them; the corpora +#: contain 4 markers in total (geb, nee, née, 旧姓 -- nee arrived with +#: #414's rules corpus), 3 of which it covers. _LATIN_ALTERNATION_SOURCES: dict[str, _LatinCopy] = { "fix(#274)": _LatinCopy( vocabulary=MAIDEN_MARKERS, - covers=frozenset({"geb", "nee", "née", "roz"})), + covers=frozenset({"geb", "nee", "née"})), "ambiguous-surname-acronym": _LatinCopy( vocabulary=SUFFIX_ACRONYMS_AMBIGUOUS, covers=frozenset({"do", "ma"})), @@ -1108,16 +1110,50 @@ def _carries(name: str, vocabulary: frozenset[str]) -> bool: marker like 旧姓 is written against the name it marks rather than spaced off it. - Note what the isascii() split actually covers: 12 of the 17 + Note what the isascii() split actually covers: 12 of the 16 entries, not only the CJK one. `né` is two characters, so the substring branch reads `René` as carrying a marker. Every over-match here SHRINKS the set of unexplained names and so weakens the guard -- the direction this module exists to close -- - but exactly one corpus name reaches that branch today, and it is - the 旧姓 one. Tighten this before admitting a vocabulary whose - short non-ASCII entries occur inside ordinary names. + but only two corpus names DEPEND on that branch today, meaning the + token test below says no and the substring test says yes, and both + are 旧姓 ones: the fullwidth-bracketed clause and the + fullwidth-colon spelling. Eight depended on it before the + delimiter strip below arrived (2026-08-26); that strip moved the + parenthesized née names onto the token branch, where the answer + does not rest on a substring. Both figures quantify over the + corpus and go stale on any row added to it, so recount rather than + adjust: + The body must sit flush left: `python -c` compiles it as a module, + so an indented first line raises IndentationError on paste. + +uv run python -c " +import glob, json +from nameparser import DEFAULT_NICKNAME_DELIMITERS as D +from nameparser.config.maiden_markers import MAIDEN_MARKERS as V +from nameparser._lexicon import _normalize +names = {json.loads(l) for f in glob.glob('tools/differential/corpus*.jsonl') for l in open(f, encoding='utf-8') if l.strip()} +strip = ''.join({c for p in D for c in p}) +sub = lambda n: any(e in n for e in V if not e.isascii()) +print(sum(not {_normalize(t.strip(strip)) for t in n.split()} & V and sub(n) for n in names), + sum(not {_normalize(t) for t in n.split()} & V and sub(n) for n in names))" + Tighten this before admitting a vocabulary whose short non-ASCII + entries occur inside ordinary names. + + Delimiter characters come off the token before the membership + test, because a marker glued to a bracket is still a marker to the + parser: rules.md#M3 reads '(geb. Schmidt)' as a maiden clause on + the strength of that very word, and _normalize strips the + abbreviating period but not the paren, so 'Anna Müller (geb. + Schmidt)' read as carrying no maiden vocabulary at all. This + direction is the safe one -- an unstripped token cannot be a + vocabulary entry, so the strip only ever finds markers that are + really there, and the names it rescues are exactly the ones a + maiden rule may legitimately claim. """ - tokens = {_normalize(token) for token in name.split()} + delimiters = "".join({ch for pair in DEFAULT_NICKNAME_DELIMITERS + for ch in pair}) + tokens = {_normalize(token.strip(delimiters)) for token in name.split()} return bool(tokens & vocabulary) or any( entry in name for entry in vocabulary if not entry.isascii()) @@ -1239,6 +1275,10 @@ def _claim(rule: dict) -> _Claim: #: both is growth into names the rule genuinely describes. _CORPUS_CLAIMS: dict[str, dict[str, _Claim]] = { "expected_since_1.4.0.toml": { + "fix(#335) a marker-led clause leaves the one name word its bare reading": + _Claim(1, ('family', 'given', 'maiden', 'nickname'), "c09cc7dba88b"), + "fix(#335) a marker-led bracketed clause reads as the maiden name whatever pair encloses it": + _Claim(5, ('maiden', 'nickname'), "a419f74143e3"), "fix(#410) a title and one name word name the family, whatever annotation stands beside it": _Claim(3, ('family', 'given'), "24d6223e472f"), "fix(#410) the maiden flavor, where 1.4.0 read the marker as a middle name": @@ -1276,7 +1316,7 @@ def _claim(rule: dict) -> _Claim: "fix(comma-precomma-family) pre-comma run reads as family, not given": _Claim(279, ('family', 'given'), "28a62b622a48"), "fix(suffix-routing) two-token name with unambiguous trailing suffix stays suffix": - _Claim(1075, ('family', 'given', 'suffix'), "97934f29bdc8"), + _Claim(1080, ('family', 'given', 'suffix'), "0cb2cda1ed6b"), "fix(suffix-delimiter-rendering) no-space delimiter core token kept whole": _Claim(0, ('suffix',), "e3b0c44298fc"), "ambiguous-surname-acronym data change: parenthesized (MA)/(DO) now stays nickname": @@ -1349,6 +1389,12 @@ def _claim(rule: dict) -> _Claim: _Claim(1, ('family', 'given'), "e62caedec864"), }, "expected_since_2.0.0.toml": { + "fix(#335) a marker-led clause leaves the one name word its bare reading": + _Claim(1, ('family', 'given', 'maiden', 'nickname'), "c09cc7dba88b"), + "fix(#335) a marker-led bracketed clause reads as the maiden name whatever pair encloses it": + _Claim(5, ('maiden', 'nickname'), "a419f74143e3"), + "fix(#335) a marker-led bracketed clause reads as the maiden name, compounding with the CJK order flip": + _Claim(1, ('family', 'given', 'maiden', 'nickname'), "cf370e856ae7"), "fix(#410) a title and one name word name the family, whatever annotation stands beside it": _Claim(4, ('family', 'given'), "da1dd1473145"), "fix(#430) a credential run does not end at the roman numeral describing it": @@ -1453,6 +1499,10 @@ def _claim(rule: dict) -> _Claim: _Claim(1, ('family', 'maiden'), "2150936a8c55"), }, "expected_since_2.1.0.toml": { + "fix(#335) a marker-led clause leaves the one name word its bare reading": + _Claim(1, ('family', 'given', 'maiden', 'nickname'), "c09cc7dba88b"), + "fix(#335) a marker-led bracketed clause reads as the maiden name whatever pair encloses it": + _Claim(6, ('maiden', 'nickname'), "d0e857deddb2"), "fix(#410) a title and one name word name the family, whatever annotation stands beside it": _Claim(4, ('family', 'given'), "da1dd1473145"), "fix(#430) a credential run does not end at the roman numeral describing it": @@ -1801,7 +1851,7 @@ class _Excluded(NamedTuple): ("fix(comma-family)", "fix(comma-precomma-family)", "fix(suffix-routing)")), '(^|[\\w.]\\s+)[("\'][^)"\']+[)"\'](\\s+\\w|\\s*$)': - _Excluded(46, "71eb2fa94553", ()), + _Excluded(51, "770738271273", ()), } diff --git a/tests/v2/test_parser.py b/tests/v2/test_parser.py index 5181f818..55e75da6 100644 --- a/tests/v2/test_parser.py +++ b/tests/v2/test_parser.py @@ -1310,15 +1310,62 @@ def test_stacked_activation_warns_only_for_uncovered_scripts() -> None: def _clause_free_latin_corpus_names() -> list[str]: + from nameparser import DEFAULT_NICKNAME_DELIMITERS from nameparser.config.maiden_markers import MAIDEN_MARKERS from ._differential_fixtures import _CORPUS_NAMES + # A marker glued to a delimiter character is still a marker, and + # the membership test is per WORD, so '(geb.' must lose the + # bracket as well as the abbreviating period before it is asked. + # Stripping only the period admitted every corpus name that + # brackets its marker, and once rules.md#M3 read such a clause as + # the maiden name, six of them had a maiden clause of their own -- + # two clauses, and the appended one no longer the only variable. + # The strip turns away TEN names in all: those six, plus four that + # M3 declines and that would have been safe to keep -- the + # one-word '(Nee)', '(Nee) (Jones)' and '(née)', and '(née Jr.)', + # which S1 takes before M3 sees it. Textual, and so deliberately + # conservative in exactly that direction. + # + # Every count in this comment quantifies over the corpus, so one + # added corpus row falsifies it silently. Recount rather than + # adjust: + # The body must sit flush left: `python -c` compiles it as a module, + # so an indented first line raises IndentationError on paste. + # uv run python -c " + # import re, sys; sys.path.insert(0, 'tests') + # from nameparser import DEFAULT_NICKNAME_DELIMITERS as D + # from nameparser.config.maiden_markers import MAIDEN_MARKERS as M + # from v2._differential_fixtures import _CORPUS_NAMES + # base = [n for n in _CORPUS_NAMES if re.match(r'^[\x00-\u024f]*$', n) and ',' not in n] + # strip = ''.join({c for p in D for c in p}) + '.' + # keep = lambda f: [n for n in base if not any(f(w) in M for w in n.split())] + # old = keep(lambda w: w.lower().rstrip('.')) + # new = keep(lambda w: w.lower().strip(strip)) + # print(len(old), len(new), sorted(set(old) - set(new)))" + # + # Delimiter characters come from the shipped set rather than a + # literal, so a pair added there cannot quietly reopen this. + strip = "".join({ch for pair in DEFAULT_NICKNAME_DELIMITERS + for ch in pair}) + "." return [name for name in _CORPUS_NAMES if _LATIN.match(name) and "," not in name - and not any(word.lower().rstrip(".") in MAIDEN_MARKERS + and not any(word.lower().strip(strip) in MAIDEN_MARKERS for word in name.split())] +def test_the_clause_free_corpus_is_not_empty() -> None: + """The invariant below is parametrized over a FILTERED corpus, and + an empty parametrization passes as a skip rather than failing -- + the shape #329 left behind. The filter has been widened once + already (the delimiter strip, 2026-08-26, which took it from 638 + names to 628 -- the one-liner in that filter's comment recounts + both), so the floor is what says a future widening emptied + it. Deliberately far below today's count: this asks whether the + filter still selects a corpus, not what the corpus holds.""" + assert len(_clause_free_latin_corpus_names()) > 100 + + @pytest.mark.parametrize("name", _clause_free_latin_corpus_names()) def test_a_maiden_clause_changes_nothing_else(name: str) -> None: """The grouping rules count and join only the words that remain diff --git a/tools/differential/README.md b/tools/differential/README.md index e2b5b749..6c2f45cf 100644 --- a/tools/differential/README.md +++ b/tools/differential/README.md @@ -412,13 +412,44 @@ The corpora run under the **default policy**, so any behavior gated behind a non-default `Policy` field is invisible here. Default *vocabulary* is a different matter: it is fully in EFFECT, never gated off the way a `Policy` field is, so a change to it can show up -here. That is not the same as coverage -- only 3 of the 17 shipped +here. That is not the same as coverage -- only 4 of the 16 shipped `maiden_markers` and 8 of the 15 `honorific_tails` appear anywhere in -the corpora (measured 2026-08-05), so an entry no corpus name -exercises is as invisible as an opt-in policy. +the corpora (re-measured 2026-08-26; the marker count was 3 until +#414's rules corpus brought in a parenthesized `Nee`, and the +denominator was 17 until `roz` left the vocabulary in 2.2 -- it +appeared in no corpus name, so only the denominator moved), so an +entry no corpus name exercises is as invisible as an opt-in policy. + +Those two numbers count WHOLE TOKENS, delimiters stripped. The strip +is what earns exactly one of the four: `nee`, which appears in the +corpora only inside brackets -- `Jane Smith (Nee)` and +`Jane Smith (Nee) (Jones)` -- where the token carries them until they +come off. `née` needs no strip, appearing bare in many names, and `né` +is not counted at all: it occurs only as a substring of `née`, never +as a token. The convention matters because the neighbouring guard +`tests/v2/test_ledger_guards.py::_carries` deliberately asks a wider +question -- it also matches a non-ASCII entry anywhere inside a name, +since 旧姓 is written flush against the name it marks -- and under that +reading the marker count is 5, not 4. Both are right about different +questions. Recompute: -Two independent mechanisms put a birth surname in `maiden`, and only -one is opt-in (measured 2026-08-05): +``` +uv run python -c " +import glob, json +from nameparser import Parser +from nameparser._lexicon import _normalize +L = Parser().lexicon +names = [json.loads(l) for f in glob.glob('tools/differential/corpus*.jsonl') for l in open(f, encoding='utf-8') if l.strip()] +toks = {_normalize(t.strip('()\'"«»“”„「」『』（）')) for n in names for t in n.split()} +for s in ('maiden_markers', 'honorific_tails'): + v = getattr(L, s) + print(s, len(toks & v), 'of', len(v), sorted(toks & v))" +``` + + +Two independent mechanisms put a birth surname in `maiden`, and what +is opt-in about them is narrower than it looks (rows measured +2026-08-05, re-measured 2026-08-26): | input | default policy | `maiden_delimiters={("(", ")")}` | |---|---|---| @@ -428,12 +459,36 @@ one is opt-in (measured 2026-08-05): Row 1 carries no marker word, so it isolates the delimiter: the brackets alone route their content to `maiden`, and only once the policy says they do. Row 2 carries no brackets, so it isolates the -marker: `Lexicon.maiden_markers` ships 17 entries by default, `nee` +marker: `Lexicon.maiden_markers` ships 16 entries by default, `nee` among them, and the bare form needs no configuration at all. -So what is opt-in is not the marker words — it is only the delimited -path. #329 changed what happens when both are in play (the marker -inside a delimited clause is now dropped from the value), and this -gate cannot see it, because the corpora never configure the delimiter. -Opt-in behavior is covered by `tests/v2/cases.py`, whose rows carry -their own `policy=`. +So what is opt-in is neither the marker words nor the delimited path +as a whole: it is the delimited path for content that does not +announce itself. Since #335 a clause of two words or more led by a +marker reads as the maiden name whichever bucket its pair sits in +(`rules.md#M3`), so `Jane Smith (née Jones)` needs no configuration +either. Row 1 is exactly the shape that still does — a markerless +clause — along with a one-word clause like `Jane Smith (Nee)`, where +nothing in the content says maiden and only a caller who knows the +data can. + +It does NOT put #329 within reach of this gate, and the reason +generalizes past this one change. #329 governs what a delimited maiden +clause CONTAINS -- the marker word is dropped from the value -- while +a ledger rule narrows by which FIELDS move, never by what they hold. +The six names the 2.1.0 ledger classifies under `fix(#335)` move +`{nickname, maiden}` whether the marker is dropped or not, so that rule +absorbs a #329 regression in silence. The field sets are per baseline +and only that ledger's are uniform: at 2.0.0 the CJK name declares four +fields and has a rule to itself, and at 1.4.0 it is not a `fix(#335)` +name at all. Measured 2026-08-26 by reverting the drop pass +in `_group.py`: `Jane Smith (née Jones)` reads maiden `née Jones`, and +all three gates still report 0 unexplained. #329 was out of reach +before #335 too, for a different reason -- under the default policy no +corpus name reached the drop at all, so the movement on +`山田 花子（旧姓 佐藤）` between 2.0.0 and 2.1.0 was the East Asian order +flip rather than #329 (`decisions.md#M1`'s 2026-08-05 entry calls that +change gate-visible; the 2026-08-26 entry beside it records the +correction). Value-level coverage for both is `tests/v2/cases.py`, +whose rows assert values, and whose opt-in rows carry their own +`policy=`. diff --git a/tools/differential/corpus_rules.jsonl b/tools/differential/corpus_rules.jsonl index b365291f..136ff033 100644 --- a/tools/differential/corpus_rules.jsonl +++ b/tools/differential/corpus_rules.jsonl @@ -43,9 +43,13 @@ "J.R. Smith" "Jack Ma." "Jack Wei Ma" +"Jane (née Jones) Smith" "Jane Smith (Nee)" "Jane Smith (Nee) (Jones)" "Jane Smith (née Jones)" +"Jane Smith (née Jr.)" +"Jane Smith (née V)" +"Jane Smith (née)" "Jane Smith née Jones" "Jane Smith née Jones PhD" "Jane Smith née V" @@ -130,6 +134,7 @@ "Sir John" "Sir abdul van der Berg" "Sir de Mesnil" +"Smith (née Jones)" "Smith Jr." "Smith Jr., Mr." "Smith, Abd" diff --git a/tools/differential/expected_since_1.4.0.toml b/tools/differential/expected_since_1.4.0.toml index 194b8a11..3b76a3f8 100644 --- a/tools/differential/expected_since_1.4.0.toml +++ b/tools/differential/expected_since_1.4.0.toml @@ -67,11 +67,19 @@ issue = "fix(#274) maiden markers consumed" # tests/v2/test_ledger_guards.py: every alternative must match at least # one entry the config ships, and none may match ordinary name text. # -# Its members reach 4 of the 17 entries -- geb, nee, née, roz -- which -# is not the same set as the markers the corpora contain (geb, née and -# 旧姓; only the first two are covered here). Growing it toward the -# other 12 buys nothing: none appears as a token in any corpus, so -# there is no diff for them to classify. +# Its members reach 3 of the 16 entries -- geb, nee, née -- which is +# not the same set as the markers the corpora contain (geb, nee, née +# and 旧姓; the first three are covered here). Thirteen entries go +# unreached, and the carve-out in that arithmetic is load-bearing: +# 旧姓 is one of the thirteen and DOES appear as a corpus token, which +# is why it has fix(cjk-maiden-marker) of its own below rather than an +# alternative here. Growing this rule toward the other TWELVE buys +# nothing: none of them appears as a token in any corpus, so there is +# no diff for them to classify. The roz alternative was +# dropped in 2.2 with the vocabulary entry it copied (see +# maiden_markers.py's Deliberately absent note): no corpus name +# contains the string in any case or position, so it classified +# nothing and its removal moved no name. # # 旧姓 is a different case, and it has been got wrong three times, so # state it precisely. Widening this rule to cover it changes no corpus @@ -126,7 +134,22 @@ issue = "fix(#274) maiden markers consumed" # would need the regex to ask a vocabulary question -- suffix or # connective membership -- a second alternation this rule cannot # spend. -name_regex = "(?i)(?