diff --git a/docs/design/decisions.md b/docs/design/decisions.md index 8412bc02..466f1028 100644 --- a/docs/design/decisions.md +++ b/docs/design/decisions.md @@ -694,11 +694,12 @@ The fourth ledger arc, and the first to change what the gate is FOR rather than Decisions that landed: - 2026-09-01 #468 — every corpus carries a TIER, and only the CONTRACT tier can fail the run. `corpus.jsonl` (scraped from v1's test banks) and `corpus_issues.jsonl` (harvested from the tracker) became RADAR: their diffs still classify against the ledger, so release notes keep their grouping, but an unmatched one prints under `UNCLASSIFIED (radar)` and the run still exits 0. `corpus_cjk.jsonl`, `corpus_rules.jsonl` and `corpus_shapes.jsonl` keep the full discipline. The reason is a VOCABULARY gap and not a volume one: the gate demanded a rule-precision explanation for names nobody chose, and the honest explanation for many of them was "this string exercises nothing" — which no ledger key can say: a rule means "this diff is intended and here is why", an exclusion means "this shape must never be explained", and neither is the sentence wanted. A rule invented to say it would be the #372 shape by construction, since the only thing it could narrow on is which file the name happens to sit in. `_CORPUS_TIERS` in compare.py is the roster and is fail-closed like `_CORPUS_FLOORS`: a corpus added without a tier is a hard error, so the split forces a decision when a corpus arrives rather than defaulting one. -- 2026-09-01 #468 — nothing is DELETED to quiet the gate; promotion is the answer to "then how does a radar name ever bind again". A name enters the contract by being CHOSEN — a tests/v2/cases.py row plus a shape tag, which puts it in `corpus_shapes.jsonl` at the next regeneration. Deleting the string was the alternative and is worse in both directions: it loses the radar signal, and it makes the gate's own population editable by whoever finds the gate inconvenient, which is the one edit a differential gate must not make cheap. A meaningless radar string costs one parse and one report line against a comparison of 1113 names today. +- 2026-09-01 #468 — nothing is DELETED to quiet the gate; promotion is the answer to "then how does a radar name ever bind again". A name enters the contract by being CHOSEN — a tests/v2/cases.py row plus a shape tag, which puts it in `corpus_shapes.jsonl` at the next regeneration. Deleting the string was the alternative and is worse in both directions: it loses the radar signal, and it makes the gate's own population editable by whoever finds the gate inconvenient, which is the one edit a differential gate must not make cheap. A meaningless radar string costs one parse and one report line against a comparison of 1120 names today (1113 when this bullet was written, hours earlier the same day; #486's bullet below carries the arithmetic). - 2026-09-01 (decided in review) — a `[[never]]` exclusion OUTRANKS the tier: a diff it refuses reports UNEXPLAINED and exits 1 on either tier. Not a carve-out but the tier's own principle applied — the split governs names nobody looked at, and an exclusion is chosen exactly as a rule is, someone having written its `why` and its `examples`. It is also what the shipped ledgers already promise, and the measurement is what settled it: the 1.4.0 ledger's trailing-`Ph. D.` exclusion, whose `why` says "a diff here is a regression", reaches exactly three corpus names (`John Smith Ph. D.`, `John Smith, Ph. D.`, `John Smith, Ph.D.`) and every one of them lives in `corpus.jsonl` or `corpus_issues.jsonl` alone. Reading the tier as outranking the exclusion would have retired that entry outright, silently, in the commit that split the corpora. Recompute by matching each `[[never]]` entry's `name_regex` over the corpus files and reading which files the matches come from. - 2026-09-01 (decided in review) — a ledger RULE may narrow by comparison order, through a new `orders` key; it is optional, and its absence is the order-blind behavior every rule written before shape-tagged entries has. Forced by the first order-scoped rule anyone wrote: `feat(#395)`'s fold on `de la Cruz Juan Carlos` moves `{family, given, middle}` under the two family-first orders, and the most plausible regression of that very change — the fold leaking into the DEFAULT order — moves those same three roles on that same string, which `corpus_rules.jsonl` carries as a bare string and therefore compares. Order-blind, the rule describing the feature would have absorbed the regression of the feature and called it intentional: #372's failure mode reached from a new direction, and on the one name best placed to hide it. The legal set is BORROWED from shapes.py rather than hand-copied (`_legal_orders`), the same call build_cjk_corpus.py makes for the script table — an order no shape declares is an order no comparison runs under, so a rule scoped to a typo could only ever be dormant. Amended 2026-09-01 (decided in review): the legal set gains a `DEFAULT` SENTINEL, and the two legacy rules that needed it were scoped. Found when the reviewer traced the absorption running the OTHER way from the one above — the order-blind `fix(#399) a maiden marker bounds the particle chain that swallowed it` rule reaches `de la Cruz née Vega`, which `corpus_shapes.jsonl` compares under both family-first orders, and it sorts AHEAD of the two rules written for that name, so a family-first-only `{family, maiden}` regression there would have classified as intentional. The fix a rule wants is `orders`, and no rule could say "default order only": the members are constant NAMES, the default order is the absence of one, and TOML has no null inside an array. So `"DEFAULT"` names it. `fix(#399) …that swallowed it` took `["DEFAULT"]` in both 2.x ledgers and `fix(#296) … 'John Smith, Dr.' keeps its split and its title` took `["DEFAULT", "FAMILY_FIRST"]`, those being the comparisons each one's diffs actually come from. Order-blind rules stay legal — every ledger written before shape tags is full of them — so the absorption is made VISIBLE rather than illegal: a run prints an `ORDER-BLIND` block for each (issue, name, order) where a rule with no `orders` key explained an order-bearing diff, informational and outside the exit code. Recompute by running the gate at each of the four baselines and reading that block; it is empty at all four today. - 2026-09-01 (decided in review) — EXCLUSIONS stay order-blind, and `validate_exclusions` rejects an `orders` key as unknown. The dormancy arc's monotone-refusal reasoning is the whole argument: an exclusion only ever removes a name from classification, so the widest thing an over-wide one can do is make a name report UNEXPLAINED and fail the run. There is no silent direction to protect against, and the narrowing a rule needs is the one an exclusion must not have — "never explain this, but only under FAMILY_FIRST" is a promise with a hole in it. - 2026-09-01 #469 — "some names are only supported under one order" is answered by the SHAPE rather than by a ledger exception. shapes.py records, per input shape, the `name_order` the shape is an input FOR and the oldest baseline whose worker can honor that order; `compare.py` parses a tagged entry under that order on both sides, compares it on the v2 surface alone (the facade is the v1-compat surface, and a family-first name is not a v1 contract), and leaves out an entry whose run predates its `min_baseline` — reported as `skipped N names tagged shape(s) [...]` and counted in the `corpora:` line, so a shrunken comparison is never silent. Structural, and that is the point: nothing has to be written into any ledger to say a family-first name is not a default-order promise. Today's arithmetic, all of it recomputable from the checked-in files and one gate run: 13 shape entries, of which 4 dedupe into names already compared under the default order, for 1113 comparisons; shapes 4 and 5 declare `min_baseline` 2.0.0, so 7 entries are skipped at `--baseline 1.4.0` and 1106 names compare there. +- 2026-09-01 #486 (later the same day, so read this bullet as superseding the arithmetic in the one above rather than the two figures disagreeing) — the shape 1-3 variation matrix filled the given-first half of the inventory, which the feature work that authored shapes 4 and 5 had left at whatever the pipeline PRs happened to tag. 14 existing case rows gained a tag and 3 rows were authored for slots no row instantiated, taking `corpus_shapes.jsonl` from 13 entries to 30 and shapes 1/2/3 from 1/4/1 names to 9/11/3. The arithmetic moves with it and the recipe is unchanged — run `uv run python tools/differential/compare.py` and read its `corpora:` and `corpus:` lines, adding `--baseline 1.4.0` for the skip figures: 30 shape entries, of which 14 dedupe into names already compared under the default order, for 1120 comparisons; shapes 4 and 5 still declare `min_baseline` 2.0.0, so 7 entries are skipped at `--baseline 1.4.0` and 1113 names compare there. The dedupe count is the half worth reading, because it is what the tier promotion looks like from this side: 8 of the newly tagged names were in a RADAR corpus only, so the tag moved them into the contract tier without adding a comparison, and 2 more were already contract through the rules corpus. None of the 17 names #486 tagged or authored needed a ledger rule — the seven of them that were in no corpus at all diff at no baseline, and intentional stayed 226/205/113/0 across the change. Not a claim about all 30 entries: the seven shape 4/5 entries are classified by ledger rules today, and scoping exactly those rules is what the `orders` bullet three above is about. - 2026-09-01 #469 — the CJK arrangement is deliberately OUTSIDE the shape inventory. Whether an unspaced CJK name is a third family-first shape is the open question on #469, and `corpus_cjk.jsonl` — contract-tier already, generated from the case table already — covers that ground meanwhile. Tagging it in would have to ANSWER the question first, since a shape row cannot be written without a `name_order` and a `min_baseline` for it, and the inventory is a table of arrangements rather than where that argument belongs. ### comma-suffix-arc — #291/#296/#316 (2026-07-26 → 2026-08-01) diff --git a/docs/design/mechanisms.md b/docs/design/mechanisms.md index 132e633d..358f95fe 100644 --- a/docs/design/mechanisms.md +++ b/docs/design/mechanisms.md @@ -77,7 +77,7 @@ Problem shape. A guard needs to know what the answer WAS, so it can detect the a ## LEDGER-RULE-SEPARATION — file order decides, fields narrow by subset -Problem shape. Two differential-ledger rules claim overlapping names. Contract statement. Every ledger rule must carry a `name_regex` — since #451 `validate_rules` REJECTS a rule with `fields` and no `name_regex`, and one with neither was already rejected — so every rule sits in ONE tier, the sort is stable, and FILE ORDER decides every contest: the narrower rule must be written first. `fields` narrows a rule by subset; it does not separate rules by sorting. Narrowing by subset is not the whole contract: since #452 a rule's `fields` must EQUAL the union of the diffs it explains, and `compare.py` reports OVER-DECLARED and exits non-zero otherwise — a declared role no diff moves is not inert, it lets the rule keep claiming a name whose diff SHRINKS into the excess (decisions.md#differential-ledger). Since #468 there is a THIRD narrowing key: `orders` admits only the comparison orders it lists, the key being optional and its absence the order-blind reading every earlier rule has — a name compared under two orders can move the same roles for opposite reasons, so a rule describing an order-scoped fold would otherwise absorb that fold leaking into the default order (decisions.md#differential-ledger carries the worked case, and the legal set is borrowed from tools/differential/shapes.py rather than copied — plus one member no shape can declare, the `DEFAULT` sentinel naming the comparison run under no declared order, TOML having no null to put in an array). Exclusions take no `orders` and stay order-blind, deliberately. The ban ends the SHAPE and not the property it enabled: a required `name_regex` bounds nothing by itself, since the only width check is the sentinel probe — measured, `[a-z]` validates and reaches 963 of 1113 comparisons (2026-09-01). What changed is that such a rule now carries a `_CORPUS_CLAIMS` reach and digest, so its breadth is visible once at recording time rather than never (#452). The two-tier sort in `_sorted_rules` is KEPT although the ban makes it the identity on every ledger that loads (four ledgers load today, measured 2026-09-01; the open cycle's carries no rules, so the identity holds trivially there): it is the defence for a reader that does not call `validate_rules` first — a future tool, a REPL, a test fixture — and its docstring in tools/differential/compare.py says so. How it works. Detail is owned by tools/differential/README.md. The file-order clause is measured, not theoretical: in the 1.4 ledger the comma-honorific-peel rule's fields are a strict subset of the comma-compound rule's, both carry a name_regex, and a pure reorder reattributes seven names — caught by _CROSS_RULE_WINNERS and by nothing else in the suite (#375's mutation). Whether that pair should be separated by a predicate instead of by order is +Problem shape. Two differential-ledger rules claim overlapping names. Contract statement. Every ledger rule must carry a `name_regex` — since #451 `validate_rules` REJECTS a rule with `fields` and no `name_regex`, and one with neither was already rejected — so every rule sits in ONE tier, the sort is stable, and FILE ORDER decides every contest: the narrower rule must be written first. `fields` narrows a rule by subset; it does not separate rules by sorting. Narrowing by subset is not the whole contract: since #452 a rule's `fields` must EQUAL the union of the diffs it explains, and `compare.py` reports OVER-DECLARED and exits non-zero otherwise — a declared role no diff moves is not inert, it lets the rule keep claiming a name whose diff SHRINKS into the excess (decisions.md#differential-ledger). Since #468 there is a THIRD narrowing key: `orders` admits only the comparison orders it lists, the key being optional and its absence the order-blind reading every earlier rule has — a name compared under two orders can move the same roles for opposite reasons, so a rule describing an order-scoped fold would otherwise absorb that fold leaking into the default order (decisions.md#differential-ledger carries the worked case, and the legal set is borrowed from tools/differential/shapes.py rather than copied — plus one member no shape can declare, the `DEFAULT` sentinel naming the comparison run under no declared order, TOML having no null to put in an array). Exclusions take no `orders` and stay order-blind, deliberately. The ban ends the SHAPE and not the property it enabled: a required `name_regex` bounds nothing by itself, since the only width check is the sentinel probe — measured, `[a-z]` validates and reaches 970 of 1120 comparisons (2026-09-01, re-measured the same day after #486 widened the shapes corpus; it read 963 of 1113 before that). What changed is that such a rule now carries a `_CORPUS_CLAIMS` reach and digest, so its breadth is visible once at recording time rather than never (#452). The two-tier sort in `_sorted_rules` is KEPT although the ban makes it the identity on every ledger that loads (four ledgers load today, measured 2026-09-01; the open cycle's carries no rules, so the identity holds trivially there): it is the defence for a reader that does not call `validate_rules` first — a future tool, a REPL, a test fixture — and its docstring in tools/differential/compare.py says so. How it works. Detail is owned by tools/differential/README.md. The file-order clause is measured, not theoretical: in the 1.4 ledger the comma-honorific-peel rule's fields are a strict subset of the comma-compound rule's, both carry a name_regex, and a pure reorder reattributes seven names — caught by _CROSS_RULE_WINNERS and by nothing else in the suite (#375's mutation). Whether that pair should be separated by a predicate instead of by order is [#382](https://github.com/derek73/python-nameparser/issues/382). The old #271/#272 slug taboo is RETIRED (#333): the canonical-rule selector that keyed on those substrings is deliberately deleted — rule authors are free to use them in compound slugs — and the surviving rosters select on their own explicit keys (_HONORIFIC_SOURCES and _LATIN_ALTERNATION_SOURCES by named issue strings, _SPAN_BEARING_RULES by exact leading fix(...) tag). Lives in. tools/differential/compare.py, the expected_since_*.toml ledgers. Reach for it when. A ledger rule's behavior seems to depend on where it sits in the file — it does, and the reorder mutation is the test (run twice in #375; it fails _CROSS_RULE_WINNERS). History: #372 (closed) measured the then-existing fields-only rule owning 1639 of 5257 name×field pairs as filed (2026-08-10); #375/#376 then cut its classifier-of-record share sharply, and the residual pair ownership was read as the last-resort tier working as designed rather than a defect — until #451 retired the shape outright (decisions.md#differential-ledger). #372's two proposed mechanical checks were DECLINED with measurements (see decisions.md#differential-ledger), not left open. diff --git a/tests/v2/cases.py b/tests/v2/cases.py index 8a73f280..77d852f6 100644 --- a/tests/v2/cases.py +++ b/tests/v2/cases.py @@ -134,22 +134,33 @@ def __post_init__(self) -> None: CASES: tuple[Case, ...] = ( Case("plain", "John Smith", {"given": "John", "family": "Smith"}, + notes="shape 1's bare Given Family arrangement, the floor the " + "other shape-1 rows vary from", shape=1), Case("family_comma", "Smith, John", - {"given": "John", "family": "Smith"}, shape=2), + {"given": "John", "family": "Smith"}, + notes="shape 2's bare Family, Given arrangement", + shape=2), Case("suffix_comma", "John Smith, PhD", - {"given": "John", "family": "Smith", "suffix": "PhD"}, shape=3), + {"given": "John", "family": "Smith", "suffix": "PhD"}, + notes="shape 3's bare Given Family, Suffix arrangement", + shape=3), Case("bound_given_pairwise_only", "Salem, Abdul Rahman Ahmed", {"given": "Abdul Rahman", "middle": "Ahmed", "family": "Salem"}, notes="the bound-given join is PAIRWISE (one merge, v1 " - "parity): the third piece stays a middle name", + "parity): the third piece stays a middle name. Shape " + "2's post-comma given slot, at the arity where the " + "join stops", shape=2), Case("family_comma_three_part_trailing_strict", "Smith, John V, Jr.", {"given": "John", "middle": "V", "family": "Smith", "suffix": "Jr."}, notes="the lenient trailing test applies only to TWO-part " "names; a third comma part makes the trailing token a " - "middle initial (v1 parity, pinned live 2026-07-17)", + "middle initial (v1 parity, pinned live 2026-07-17). " + "Shape 2's trailing suffix WITH the optional comma " + "written; family_comma_run_with_a_name_is_not_a_run is " + "the spelling without it", shape=2), Case("triple_trailing_commas", "Doe,,,", {"family": "Doe"}, @@ -411,7 +422,10 @@ def __post_init__(self) -> None: "'Beethoven', which is what #130 asked for. The " "textbook-correct Dutch listing reports the fork all " "the same (#405): the parser cannot tell it from " - "'Nguyen, Thi Van', which is the same string shape", + "'Nguyen, Thi Van', which is the same string shape. " + "Shape 2's particle slot in the tussenvoegsel " + "spelling, where the particle stands behind the given " + "name rather than before the family", shape=2), Case("tussenvoegsel_multiword", "Berg, Jan van der", {"given": "Jan", "family": "van der Berg"}, @@ -565,7 +579,9 @@ def __post_init__(self) -> None: classification="parity", notes="the boundary the row above needs: here the particles " "DO join a name word, so they stay particles -- base " - "'Vega', particles 'de la', initials 'J. V.'"), + "'Vega', particles 'de la', initials 'J. V.'. Shape " + "1's particle-bearing family slot", + shape=1), Case("suffix_word_title_ambiguous_particle", "Jr. Van Johnson", {"title": "Jr.", "given": "Van", "family": "Johnson"}, classification="fix(#367)", @@ -688,6 +704,19 @@ def __post_init__(self) -> None: notes="v1 renders each tail comma segment as ONE suffix " "entry; words within an entry space-join via the " "'joined' tag"), + Case("inline_suffix_then_comma_suffix", "John Smith Jr., PhD", + {"given": "John", "family": "Smith", "suffix": "Jr., PhD"}, + classification="parity", + notes="shape 3's optional inline suffix standing WITH a comma " + "suffix, which no other row writes: the two compose " + "rather than one displacing the other. C1 decides on " + "the part after the comma alone -- wholly suffix words, " + "more than one word before it -- so the trailing-suffix " + "mode fires with 'Jr.' already inside the name part, " + "and the written comma survives between the pieces. " + "'John Smith, PhD' is the comma half alone and " + "'John Smith Jr.' the inline half", + shape=3), Case("maiden_delimiters_win_when_shared", 'Baker (Johnson), Jenny', {"given": "Jenny", "family": "Baker", "maiden": "Johnson"}, @@ -767,7 +796,9 @@ def __post_init__(self) -> None: {"given": "John", "family": "Smith", "suffix": "Jr."}, notes="v1: the family part may have suffixes in it " "(parser.py:1368); the first piece is always the family " - "(pinned live 2026-07-17)"), + "(pinned live 2026-07-17). Shape 2's pre-comma " + "[Suffix] slot", + shape=2), Case("family_segment_multiple_suffixes", "Smith Jr. MD, John", {"given": "John", "family": "Smith", "suffix": "Jr., MD"}), Case("family_segment_particle_chain_suffix", "de la Vega III, Juan", @@ -796,7 +827,15 @@ def __post_init__(self) -> None: ambiguities=("comma-structure",), notes="only parts[1] decides the suffix-comma structure " "(v1 parser.py:1318); 'lutc' is not in the vocabulary " - "but rides along (v1 parity, pinned live 2026-07-16)"), + "but rides along (v1 parity, pinned live 2026-07-16). " + "Deliberately the COMPOUND shape-3 exemplar: it is the " + "only row filling Title, Middle and the repeated " + "[, Suffix] at once, and it carries the corpus's " + "hyphenated family besides, so a diff on it is not " + "attributable to any one slot -- read it as the " + "notation's fullest form rather than as a witness for " + "whichever slot the failure seems to be about", + shape=3), Case("suffix_comma_nonsuffix_tail_flagged", "John Smith, MD, Xyzzy", {"given": "John", "family": "Smith", "suffix": "MD, Xyzzy"}, ambiguities=("comma-structure",), @@ -870,10 +909,15 @@ def __post_init__(self) -> None: ambiguities=("comma-structure",), notes="post-comma segments land in suffix even when not " "suffix-shaped; the ambiguity flags the guess (v1 " - "parity, pinned live 2026-07-13)"), + "parity, pinned live 2026-07-13). Shape 2's repeated " + "[, Suffix] slot, the double trailing suffix", + shape=2), Case("delavega", "Dr. Juan de la Vega III", {"title": "Dr.", "given": "Juan", "family": "de la Vega", - "suffix": "III"}), + "suffix": "III"}, + notes="shape 1's Title and trailing Suffix slots paired, the " + "arrangement written end to end", + shape=1), Case("prefix_chain_to_end", "Juan de la Vega Martinez", {"given": "Juan", "family": "de la Vega Martinez"}), Case("van_johnson", "Van Johnson", @@ -881,7 +925,9 @@ def __post_init__(self) -> None: ambiguities=("particle-or-given",), notes="v2 surfaces #121's irreducible ambiguity"), Case("family_comma_particles", "de la Vega, Juan", - {"given": "Juan", "family": "de la Vega"}), + {"given": "Juan", "family": "de la Vega"}, + notes="shape 2's particle-bearing family, before the comma", + shape=2), Case("paren_suffix_escapes_nickname", "Andrew Perkins (MBA)", {"given": "Andrew", "family": "Perkins", "suffix": "MBA"}, notes="v1 parse_nicknames: suffix-shaped delimited content is " @@ -890,12 +936,36 @@ def __post_init__(self) -> None: Case("paren_period_escapes_nickname", "Andrew Perkins (Ret.)", {"given": "Andrew", "family": "Perkins", "suffix": "Ret."}), Case("nickname_quotes", 'John "Jack" Kennedy', - {"given": "John", "family": "Kennedy", "nickname": "Jack"}), + {"given": "John", "family": "Kennedy", "nickname": "Jack"}, + notes="shape 1's double-quoted Nickname slot, which is the " + "spelling its notation writes", + shape=1), Case("nickname_parens", "John (Jack) Kennedy", {"given": "John", "family": "Kennedy", "nickname": "Jack"}), Case("sir_bob", "Sir Bob Andrew Dole", {"title": "Sir", "given": "Bob", "middle": "Andrew", - "family": "Dole"}), + "family": "Dole"}, + notes="shape 1's first Middle slot; middle_run_at_two_words " + "below is the second", + shape=1), + Case("middle_run_at_two_words", "John Jack Andrew Kennedy", + {"given": "John", "middle": "Jack Andrew", "family": "Kennedy"}, + classification="parity", + notes="shape 1's SECOND Middle slot, which sir_bob above " + "leaves unwritten: everything standing between the " + "given name and the family is middle, at any arity, and " + "the pieces render space-joined. The row that fails if " + "the middle run is ever capped at one word", + shape=1), + Case("family_comma_paren_nickname", "Kennedy, John (Jack)", + {"given": "John", "family": "Kennedy", "nickname": "Jack"}, + classification="parity", + notes="shape 2's (Nickname) slot: the clause is lifted out " + "before the comma structure is read, so what reaches " + "C1 is the bare 'Kennedy, John' and the listing form " + "still wins. nickname_parens above is the same clause " + "in the medial position of shape 1", + shape=2), Case("long_title", "President of the United States Barack Obama", {"title": "President of the United States", "given": "Barack", "family": "Obama"}), @@ -903,7 +973,10 @@ def __post_init__(self) -> None: {"title": "The Secretary of State", "given": "Hillary", "family": "Clinton"}), Case("comma_middle_initial", "Doe, John A.", - {"given": "John", "middle": "A.", "family": "Doe"}), + {"given": "John", "middle": "A.", "family": "Doe"}, + notes="shape 2's post-comma Middle slot, in the form it is " + "usually written after a family comma -- an initial", + shape=2), Case("single", "John", {"given": "John"}), Case("title_only", "Dr.", {"title": "Dr."}), Case("double_comma_suffix", "Smith, John, Jr.", @@ -921,11 +994,17 @@ def __post_init__(self) -> None: "roman numeral; V/X/I are also ordinary middle initials, " "so the reading is reported"), Case("initial_not_suffix", "John V. Smith", - {"given": "John", "middle": "V.", "family": "Smith"}), + {"given": "John", "middle": "V.", "family": "Smith"}, + notes="shape 1's Middle slot filled by an initial-shaped " + "word, which is the branch a numeral spelling would " + "otherwise take to the suffix", + shape=1), Case("lenient_after_comma", "John Ingram, V", {"given": "John", "family": "Ingram", "suffix": "V"}), Case("comma_then_title", "Smith, Dr. John", - {"title": "Dr.", "given": "John", "family": "Smith"}), + {"title": "Dr.", "given": "John", "family": "Smith"}, + notes="shape 2's post-comma Title slot", + shape=2), Case("nickname_single_name", "John (Jack)", {"family": "John", "nickname": "Jack"}), Case("nickname_only", "(Jack)", {"nickname": "Jack"}), @@ -2214,7 +2293,14 @@ def __post_init__(self) -> None: "Muhammad (#343/#345's corpus rows), a prenominal use " "the 'postnominal only' disposition did not consider; " "'MD' after the name is the degree. Position decides, " - "as for 'sr'"), + "as for 'sr'. The shape-1 tag rides on that disposition " + "-- 'md' is the ONE deviation from the approved " + "2026-07-30 audit table (decisions.md#comma-suffix-arc, " + "where #291 is still unshipped), so if the deviation is " + "ever reversed this row stops being a Title Given " + "Family arrangement and the tag has to move to another " + "row rather than the expectations being edited under it", + shape=1), Case("audit_md_after_comma_is_the_degree", "Smith, MD", {"family": "Smith", "suffix": "MD"}, classification="fix(#296)", @@ -2241,7 +2327,10 @@ def __post_init__(self) -> None: Case("audit_phd_trailing_unchanged", "John Smith PhD", {"given": "John", "family": "Smith", "suffix": "PhD"}), Case("audit_jr_trailing_unchanged", "John Smith Jr.", - {"given": "John", "family": "Smith", "suffix": "Jr."}), + {"given": "John", "family": "Smith", "suffix": "Jr."}, + notes="shape 1's trailing Suffix slot, written without a " + "comma", + shape=1), Case("audit_lt_leading_stays_a_title", "Lt. Smith", {"title": "Lt.", "family": "Smith"}, notes="'lt' KEPT its dual membership -- a prenominal rank with " @@ -2547,7 +2636,11 @@ def __post_init__(self) -> None: Case("family_comma_run_with_a_name_is_not_a_run", "Smith, John Jr.", {"given": "John", "family": "Smith", "suffix": "Jr."}, notes="the non-flip: a name word in the run makes it the " - "given-and-suffix walk v1 had"), + "given-and-suffix walk v1 had. Shape 2's trailing " + "suffix with the optional comma OMITTED; " + "family_comma_three_part_trailing_strict is the " + "spelling that writes it", + shape=2), Case("family_comma_title_then_suffix", "Smith, Dr. Jr.", {"title": "Dr.", "family": "Smith", "suffix": "Jr."}, classification="fix(comma-family)", diff --git a/tests/v2/test_ledger_guards.py b/tests/v2/test_ledger_guards.py index 43d225a9..77d6f90f 100644 --- a/tests/v2/test_ledger_guards.py +++ b/tests/v2/test_ledger_guards.py @@ -1591,8 +1591,22 @@ def _claim(rule: dict) -> _Claim: # regex matches one more corpus name. The name does not diff at # this baseline (v1 and HEAD both read first 'John', last # 'Ph. D.'), so nothing was absorbed. + # + # 284 -> 286 with the shape 1-3 variation matrix (#486), and + # the growth is corpus again: 'Smith, John Jr.' and + # 'Smith, John, Extra, Jr.' were case-table rows in no corpus + # until their shape tags put them in corpus_shapes.jsonl. Both + # are parity rows, so neither diffs at this baseline and this + # rule absorbs nothing new -- the same reach-not-behavior + # growth the paragraph above records, from the other source. + # It moves this rule and fix(comma-precomma-family) below by + # the same two names, both matching on the bare comma. 286 -> + # 288 in the commit after it, for the two comma-bearing rows + # #486 had to AUTHOR -- 'John Smith Jr., PhD' and + # 'Kennedy, John (Jack)' -- and neither diffs at this baseline + # either, so all four names are reach without absorption. "fix(comma-family) lone post-comma piece routes to suffix/title, not first": - _Claim(284, ('given', 'suffix', 'title'), "8b046fbdb1a1", None), + _Claim(288, ('given', 'suffix', 'title'), "10c78dd0f2d2", None), "fix(comma-family) a comma followed only by titles keeps the given/family split": _Claim(2, ('family', 'given'), "5bd9c6d96c38", None), "fix(comma-family) a comma followed only by titles keeps the given/family split, the C1 example": @@ -1612,7 +1626,7 @@ def _claim(rule: dict) -> _Claim: "fix(#367) an inferred title no longer displaces a leading particle either": _Claim(1, ('family', 'given'), "d8ee9cd5da5f", None), "fix(comma-precomma-family) pre-comma run reads as family, not given": - _Claim(284, ('family', 'given'), "8b046fbdb1a1", None), + _Claim(288, ('family', 'given'), "10c78dd0f2d2", None), "fix(#342) NOT WANTED: a bare trailing 'Rai' is read as a post-nominal suffix and the family is lost": _Claim(1, ('family', 'suffix'), "694fd06a2e9a", None), "fix(#397) NOT WANTED: a trailing Catalan/Polish linking 'i' is read as a generation marker and the family is lost": @@ -2402,7 +2416,20 @@ class _Excluded(NamedTuple): # no diff to silence. Growth in the corpus, not in the # exclusion -- its regex is untouched -- and `absorbed_by` # stayed empty, so no rule reaches the protected shape. - _Excluded(55, "7ad8ff289eb2", ()), + # 55 -> 56 for the shape-1 variation matrix (#486): tagging + # 'John "Jack" Kennedy' put this entry's own FIRST `examples` + # string into a corpus for the first time. It was in none + # before -- the radar corpora hold the smart-quote spelling + # 'John “Jack” Kennedy' and not this one, which is why the + # two read differently at 1.4.0 (feat(#273) classifies the + # typographic pair; the ASCII pair is what this entry promises + # was already recognized). It costs the entry nothing either -- + # 1.4.0 reads the quoted clause as a nickname exactly as the + # tree does, so there is no diff to silence. 56 -> 57 for the + # shape-2 slot the same matrix opened, 'Kennedy, John (Jack)', + # which is the paren spelling of that clause after a family + # comma and costs the entry nothing for the same reason. + _Excluded(57, "35ac9a8c4195", ()), } @@ -2512,7 +2539,9 @@ def test_a_fields_narrowing_actually_narrows_something() -> None: Measured: deleting `fields = ["nickname", "middle"]` from the ASCII-pairs entry passes every other check in this tree. The entry - then refuses ANY diff on the 46 corpus names it captures -- + then refuses ANY diff on every corpus name it captures -- the + count is _EXCLUSION_EFFECT's `captures` for that pattern, above, + which is where it is checked and where it stays current -- including 'Jenny (Johnson) Baker' and 'Lon (Jr.) Williams', whose parens are a maiden name and a suffix, both under active development. Nothing failed, because none of those names diffs diff --git a/tools/differential/README.md b/tools/differential/README.md index 9226c3ae..5350ab24 100644 --- a/tools/differential/README.md +++ b/tools/differential/README.md @@ -570,7 +570,10 @@ ambiguity, which is why `feat(#273)`'s own rule can be a bare character class and its exclusion cannot. The ASCII-pairs entry is narrowed by role ALONE, and covers a delimited -run in any position -- 34 corpus names. It was medial-only for three +run in any position; how many corpus names that reaches is pinned as +its `captures` in `tests/v2/test_ledger_guards.py`'s +`_EXCLUSION_EFFECT`, which is the one place the number is checked and +so the one place it is worth reading. It was medial-only for three rounds on the theory that trailing parens are credentials to be kept out; 1.4.0 says otherwise, reading a trailing `(JD)` as a nickname just as it reads `(Ben)`. Where 1.4 did read parens as a credential it put diff --git a/tools/differential/compare.py b/tools/differential/compare.py index af6b092d..817599f9 100644 --- a/tools/differential/compare.py +++ b/tools/differential/compare.py @@ -506,7 +506,7 @@ def _legal_orders() -> frozenset[str]: "corpus_cjk.jsonl": 95, # 98 today, generated from the case table "corpus_issues.jsonl": 370, # 381 today, harvested and append-only "corpus_rules.jsonl": 150, # 252 today, generated from rules.md - "corpus_shapes.jsonl": 11, # 13 today, generated from shape-tagged + "corpus_shapes.jsonl": 27, # 30 today, generated from shape-tagged # case rows } diff --git a/tools/differential/corpus_shapes.jsonl b/tools/differential/corpus_shapes.jsonl index 3e4f9b3a..f5a9af0b 100644 --- a/tools/differential/corpus_shapes.jsonl +++ b/tools/differential/corpus_shapes.jsonl @@ -1,8 +1,25 @@ +{"name": "Dr. Juan de la Vega III", "shape": 1} +{"name": "John \"Jack\" Kennedy", "shape": 1} +{"name": "John Jack Andrew Kennedy", "shape": 1} {"name": "John Smith", "shape": 1} +{"name": "John Smith Jr.", "shape": 1} +{"name": "John V. Smith", "shape": 1} +{"name": "Juan de la Vega", "shape": 1} +{"name": "Md Abdul Karim", "shape": 1} +{"name": "Sir Bob Andrew Dole", "shape": 1} {"name": "Beethoven, Ludwig van", "shape": 2} +{"name": "Doe, John A.", "shape": 2} +{"name": "Kennedy, John (Jack)", "shape": 2} {"name": "Salem, Abdul Rahman Ahmed", "shape": 2} +{"name": "Smith Jr., John", "shape": 2} +{"name": "Smith, Dr. John", "shape": 2} {"name": "Smith, John", "shape": 2} +{"name": "Smith, John Jr.", "shape": 2} {"name": "Smith, John V, Jr.", "shape": 2} +{"name": "Smith, John, Extra, Jr.", "shape": 2} +{"name": "de la Vega, Juan", "shape": 2} +{"name": "Dr. John P. Doe-Ray, CLU, CFP, LUTC", "shape": 3} +{"name": "John Smith Jr., PhD", "shape": 3} {"name": "John Smith, PhD", "shape": 3} {"name": "John Smith, Dr.", "shape": 4} {"name": "de Mesnil Jean, Dr.", "shape": 4} diff --git a/tools/differential/expected_since_1.4.0.toml b/tools/differential/expected_since_1.4.0.toml index 706bd973..a7d4be43 100644 --- a/tools/differential/expected_since_1.4.0.toml +++ b/tools/differential/expected_since_1.4.0.toml @@ -1599,13 +1599,35 @@ fields = ["given", "family"] # reports UNEXPLAINED however many rules would claim it. # # `examples` is not decoration. A protected shape need not appear in -# any corpus -- 'John "Jack" Kennedy' does not -- so the entry carries +# any corpus -- 'John (Jack) Kennedy' does not -- so the entry carries # its own test data, which tests/v2/test_ledger_guards.py runs against # every non-empty subset of the fields a rule may name: the seven # roles and the `_ambiguities` pseudo-field. An entry carrying # `fields` is asked only about the subsets those cover -- 3 for the # ASCII pairs, not 255 -- which is where the 387 in that file comes # from. +# +# The illustration was 'John "Jack" Kennedy' until #486's shape 1-3 +# variation matrix gave that string's existing case row a shape tag -- +# the row (tests/v2/cases.py 'nickname_quotes') long predates that +# work; only the tag is new -- which put it in corpus_shapes.jsonl. +# The sentence is about the KEY, not about that name, so it moved to +# the sibling example still in no corpus at all; of the four, the +# remaining two ('"Rick" Edmonds', 'Franklin, Benjamin (Ben)') are +# corpus names in radar files. That an example can be promoted this +# way is the point of the key rather than a problem with it -- +# nothing about the entry depends on which side of the corpus its +# test data sits on. +# +# The replacement is one tag from the same fate, and whoever writes +# that tag should read this: 'John (Jack) Kennedy' is itself a case +# row's text ('nickname_parens', the medial-paren sibling of the row +# above), so tagging it with shape 1 puts it in corpus_shapes.jsonl +# and falsifies this sentence exactly as the last tag did. There is +# no example left that is not a corpus name or a case row, which is +# the honest state of it: repoint the sentence at whichever example +# is still outside, or restate it as "examples are not required to be +# corpus names" and stop naming one. [[never]] why = "trailing 'Ph. D.' split-token healing is PARITY, not a 2.0 change: v1 healed the adjacent pair too, so a diff here is a regression" @@ -1671,12 +1693,16 @@ why = "feat(#273) recognizes TYPOGRAPHIC nickname delimiters; the ASCII pairs we # ('"Rick" Edmonds'), medial ('John (Jack) Kennedy') or trailing # ('Franklin, Benjamin (Ben)'), with a word character or a period as # the inner flank so a middle initial counts ('Cherice J. (Johnson) -# Williams'). 51 corpus names (recounted 2026-08-26). The figure is -# the one test_ledger_guards.py pins as this entry's `captures`, so it -# is not merely stale when it drifts, it contradicts a machine-checked -# constant: read it there rather than trusting this line, and grep -# _EXCLUSION_EFFECT for the pattern. It was 46 for three rounds and -# moved twice in one delta, once per rules-corpus row added. +# Williams'). HOW MANY corpus names that is has a single home, and it +# is not this line: tests/v2/test_ledger_guards.py pins it as this +# entry's `captures` -- grep _EXCLUSION_EFFECT for the pattern above +# and read the number there, where a change fails CI. This comment +# carried a copy of it for several rounds (34, then 46, then 51) and +# the copy was stale more often than not, since the figure moves +# whenever the CORPORA move and not only when the entry does -- it +# never caught up to 55, which the pin was already holding when the +# copy still read 51, and that lag is the whole argument for deleting +# the copy rather than correcting it a fourth time. # # It was medial-only for three rounds, on the theory that the trailing # position is where credentials live and that widening would silence