From 98a19d285e3f55a467b9c104d147225f4e6fea9c Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Sat, 11 Jul 2026 20:47:44 -0700 Subject: [PATCH] Add nameparser 2.0 design RFC The design document for 2.0: a new immutable core API (parse() -> ParsedName, Lexicon/Policy configuration) alongside HumanName as a compatibility layer through 2.x. Covers the compatibility promise, the new API, migration mechanics, locale plans, rejected alternatives, and the feedback questions. Discussion: #284. Co-Authored-By: Claude Fable 5 --- docs/design/nameparser-2.0-rfc.md | 294 ++++++++++++++++++++++++++++++ 1 file changed, 294 insertions(+) create mode 100644 docs/design/nameparser-2.0-rfc.md diff --git a/docs/design/nameparser-2.0-rfc.md b/docs/design/nameparser-2.0-rfc.md new file mode 100644 index 0000000..1e25799 --- /dev/null +++ b/docs/design/nameparser-2.0-rfc.md @@ -0,0 +1,294 @@ +# RFC: nameparser 2.0 — a new core API + +Status: **open for feedback** — no deadline. Implementation may proceed +in parallel; this document will be amended (as commits to this PR) as +implementation teaches us things. Comment on specific lines here, or on +the discussion issue (#284) for general reactions. + +## The short version, and the promise + +nameparser 2.0 introduces a new, immutable core API and keeps the +existing one working: + +```python +from nameparser import parse + +name = parse("Dr. Juan Q. Xavier de la Vega III (Doc Vega)") +name.given # "Juan" +name.family # "de la Vega" +name.family_base # "Vega" +name.title # "Dr." +``` + +**The compatibility promise:** code that runs warning-free on nameparser +1.4 keeps running on every 2.x release, with identical results, getting +at most `DeprecationWarning`s. `HumanName` continues to exist as a +compatibility layer over the new core. The exceptions to "identical +results" are (a) parse-output changes that are deliberate bug fixes, +each documented in the release log with its issue, and (b) three narrow +edge cases listed in the Migration section. Removal of the old API +happens in 3.0, not 2.x, and only after warnings have named every +replacement. + +If you only use `HumanName` and never customize configuration, 2.0 +should be a non-event for you. The rest of this document is for +everyone else — and for anyone who wants to push on the new design +while it is still cheap to change. + +## Why a new core + +Fifteen years of issues point at the same architectural limits: + +- **`HumanName` is four things in one mutable class** — parser, state, + vocabulary predicates, and formatter. Every feature grows all four at + once (it is currently ~1,600 lines), and mutation semantics are + inconsistent: assigning `full_name` re-parses, assigning `last` does + not, `capitalize()` mutates in place. +- **Parse state is `list[str]`**, so position information is lost and + re-derived by value lookups. That is the structural root cause of a + whole family of prefix-join bugs (#100 and relatives): when the same + string appears twice, the lookup answers the wrong question. +- **Configuration is a shared mutable module global** (`CONSTANTS`). + Most of `nameparser.config`'s ~850 lines exist to make shared + mutation survivable (custom set managers, descriptors, cache + invalidation), and it still causes test pollution and cross-thread + surprises. +- **Equality is broken by design** (#223): case-insensitive equality, + equality with plain strings, and hashability are mutually + inconsistent promises. 1.3.0 deprecated `==`/`hash()` and shipped + `matches()`/`comparison_key()`. +- **The roadmap needs structure v1 can't cheaply provide**: family-first + name order (#270), per-dataset configuration without global mutation, + and honest reporting of genuinely ambiguous parses. + +The rules-based approach is not changing. nameparser's niche is +deterministic, auditable, zero-dependency parsing — the same input +always parses the same way. The rewrite reorganizes *how the rules run*, +not what the library is. + +## The new API + +### Parsing produces an immutable value + +`parse()` returns a `ParsedName`: a frozen dataclass. There is no +mutation anywhere in the new API — editing is `replace()`, which +returns a new value: + +```python +name = parse("John Smith") +name2 = name.replace(given="Jane") # new object; `name` unchanged +``` + +`parse()` is **total over `str`**: any string — empty, emoji, +garbage — returns a `ParsedName` rather than raising. Content problems +are represented in the result; the only exceptions are type errors +(`bytes` raises `TypeError` — decode first). + +Equality is strict structural equality, and the semantic comparisons +are explicit and separate — `comparison_key()` for dedup/sorting and +`matches()` for component-wise case-insensitive comparison — resolving +the #223 trilemma by not conflating the three meanings of "equal". + +### Two access layers: strings for the 95% case, tokens for the rest + +The string properties behave like v1's: always `str`, `""` when empty. +Underneath, `ParsedName` carries the actual parse: a tuple of `Token`s, +each with its **span** (position in the original string), its role, and +tags recording *how it was classified*: + +```python +name = parse("Juan de la Vega") +name.tokens_for(Role.FAMILY) +# (Token('de' @5:7 FAMILY {particle}), +# Token('la' @8:10 FAMILY {particle}), +# Token('Vega' @11:15 FAMILY),) +``` + +This is the structural fix for the value-lookup bug family: pipeline +stages address tokens by index, so "which 'de' did you mean?" cannot +arise. It is also the debuggability story — "why is 'Van' in my family +name?" is answered by the token's tags, and `repr(name)` prints a +compact component-per-line breakdown like v1's. + +Derived views (`family_base`, `family_particles`, `surnames`, +`given_names`) are pure filters over tokens — they can never disagree +with the parse. + +### `given` and `family`, not `first` and `last` + +The v1 field names describe Western *position* — they date from before +the parser even supported comma formats, when position was all there +was. They read wrongly the moment name order varies: under family-first +order, `name.last` would return the first word. 2.0 adopts the names +the standards world already uses (HTML autocomplete +`given-name`/`family-name`, vCard, Unicode CLDR): + +- `first` → `given`, `last` → `family` +- `last_base`/`last_prefixes` → `family_base`/`family_particles` +- config: `prefixes` → `particles` (every web standard uses "prefix" + for honorifics like "Mr." — v1's usage collides; the linguistics term + for van/de la is *particle*) +- `title` and `suffix` **stay** — they have ecosystem gravity, and the + positional taxonomy ("pre-nominal"/"post-nominal") is documented + rather than spelled into identifiers. + +`HumanName` keeps every v1 spelling, including `string_format` keys, so +existing code is unaffected; only new-API adopters see the new names. + +### Configuration: `Lexicon` (vocabulary) and `Policy` (behavior) + +v1's `Constants` mixes three kinds of thing; 2.0 splits them by what +varies together: + +- **`Lexicon`** — vocabulary data (titles, suffixes, particles, + conjunctions, …). Immutable; composable by union; varies by + *language*. +- **`Policy`** — behavior switches (`name_order`, patronymic rules, + delimiter pairs, …). Immutable; varies by *data source*. +- **Rendering parameters** — `string_format`, initials style, + capitalization — move to render-time arguments; they vary by *output + destination* and are no longer configuration at all. + +```python +from nameparser import Parser, Lexicon, Policy, FAMILY_FIRST + +parser = Parser( + lexicon=Lexicon.default().add(titles={"dra"}), + policy=Policy(name_order=FAMILY_FIRST), +) +parser.parse("Wang Xiuying").family # "Wang" +``` + +A `Parser` is immutable, thread-safe, and does all its compilation once +at construction — build it at startup, share it everywhere. There is no +shared mutable configuration in the new API, which retires the entire +test-pollution/config-war problem class. `name_order` is an order-spec +tuple (`FAMILY_FIRST = (Role.FAMILY, Role.GIVEN, Role.MIDDLE)`) rather +than a boolean, because Vietnamese order — family, middle, *then* +given — needs a third arrangement (#270). + +### Honesty about ambiguity + +Some parses are genuinely undecidable by rules — "Kelly Michael" could +be either order; in "Van Johnson", "Van" is a Dutch particle *and* a +common given name. v1 silently picks a reading. 2.0 picks the same +reading **and says so**: + +```python +parse("Van Johnson").ambiguities +# (Ambiguity(PARTICLE_OR_GIVEN, "leading 'van' can be a particle or a +# given name; chose given", tokens=(...)),) +``` + +`Ambiguity.kind` values are stable API you can filter on. This is also +groundwork: a future `parse_all()` returning ranked alternative +readings, and `explain()` returning a stage-by-stage parse trace, are +designed for (the pipeline supports both) but deliberately not in 2.0. + +### Locales: opt-in packs, never auto-detection + +Language-specific behavior cannot be auto-detected — "Tham Jun Hoe" is +token-for-token indistinguishable from a Western three-token name, and +"Ali" is Arabic or Italian. So locale support is explicit: + +```python +from nameparser import parser_for, locales + +parser_for(locales.RU).parse("Ivanova Anna Sergeevna").given # "Anna" +``` + +A locale pack is pure data: a vocabulary fragment plus a partial policy +patch, folded in at parser construction. 2.0.0 ships `RU` and `TR_AZ` +(formalizing the patronymic support added in 1.3.0); Chinese/Korean +(bundled surname lists + segmentation of unspaced input), Vietnamese, +and Japanese (via a `nameparser[ja]` extra with a pluggable segmenter) +are designed and staged for 2.x minors. Vocabulary that cannot misfire +on other languages' names (e.g. Cyrillic honorifics) goes in the +default lexicon instead of packs. + +## Migration: what actually happens to existing code + +**`HumanName` in 2.0** is a wrapper over the new core with v1's exact +mutation semantics, field spellings, and formatting attributes. The v1 +test suite runs against it as the compatibility gate. v1 pickles load +throughout 2.x. + +**`CONSTANTS` keeps working** — mutations are honored exactly (the +facade rebuilds its config snapshot when the shared config changes) and +emit a `DeprecationWarning` pointing at the replacements. Mutating a +*private* `Constants` instance does not warn. + +**Already-scheduled removals land as warned** (every one was deprecated +in 1.3.0 or 1.4 first): `==`/`hash()`, bytes input, slice item access, +`empty_attribute_default`, `constants=None`, silent config-typo +fallbacks, legacy pickle blobs, `add_with_encoding()`. Python floor +becomes 3.11; the last dependency is dropped. + +**The three narrow behavior exceptions** (everything else identical): + +1. `*_list` attributes return snapshots — in-place + `name.first_list.append(...)` no longer mutates parse state + (assignment still does). +2. Subclasses overriding parsing hooks (`parse_pieces`, `is_title`, …) + are detected and warned at construction: the facade delegates to the + new core, so such overrides are no longer called (#280). If you do + this, we want to hear what your override does — that is a feature + request we would rather absorb than break. +3. Assigning to `CONSTANTS.regexes` raises with a pointer to the named + `Policy` flag that replaces it. Custom regex injection has no + equivalent in the new model; nothing is silently ignored. + +**Timeline:** 2.0.0's only *new* warning is on shared-`CONSTANTS` +mutation. A warning on `HumanName` construction itself comes in a later +2.x minor, once the new API has proven out — and 3.0 (which removes the +facade) does not ship before that warning has been out in a release. + +## Alternatives considered and rejected + +- **Auto-detecting name language/order** — impossible in principle, not + just hard; see the Locales section. +- **A statistical/ML core** — determinism and auditability are the + point of this library; the token/ambiguity model leaves room for + optional escalation later without changing the core. +- **Keeping `first`/`last`** — see the naming section; the facade keeps + them for v1 code, which serves the familiarity constituency without + wiring position-based names into the new API. +- **A boolean `family_name_first`** — cannot express Vietnamese; the + order-spec tuple can (#270). +- **Renaming `title`/`suffix` to `pre_nominal`/`post_nominal`** — more + precise, but no ecosystem uses those as field names; precision lives + in the docs instead. +- **Vocabulary entries as records with per-word metadata** — rejected + under "no vocabulary data without a consuming rule": a flag the + parser doesn't read is a promise it doesn't keep. Cross-category + words are handled by dual set membership (closed classes) or the two + `_ambiguous` sets (closed-vs-open), each of which has a consuming + rule. +- **A user-replaceable regex escape hatch** — replaced by named policy + flags; anything not expressible through them is a feature request we + want to see. + +## Feedback we are specifically looking for + +No deadline — comment whenever, on lines here or on the discussion +issue. The questions where answers would most change things: + +1. **Do you compare `HumanName` objects with `==` (especially against + plain strings), or use them in sets/dicts?** 2.0 changes this to + object identity (warned since 1.3.0; `matches()`/`comparison_key()` + are the replacements). This is the one silent behavior change in the + release. +2. **Do you subclass `HumanName` and override parsing methods?** Tell + us what your override accomplishes — we would rather absorb it as a + feature than break it silently (#280). +3. **Do you mutate `CONSTANTS` after startup** (not just at import + time)? Anything beyond startup-time customization we should know + about? +4. **Do you assign custom regexes to `CONSTANTS.regexes`?** What for? + Each real use case either becomes a named policy flag or stays + broken — we would like the list to be complete before 2.0. +5. **`given`/`family` instead of `first`/`last`** in the new API: does + this help or annoy you? +6. **Is there anything you build on top of nameparser** (wrappers, + pipelines) where the compatibility promise as stated would still + break you?