From 900c00ba1af826a9a20d3fe807b8683f6faacf11 Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Tue, 1 Sep 2026 16:10:45 -0700 Subject: [PATCH 1/7] feat(tests): shapes 6 and 7 join the inventory as pure CJK arrangements, and rows can declare tolerated input MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Shape 6 (family-first CJK) and shape 7 (source-order transcription listing) close #469's third-shape question. Both carry order=None: the family-first/source-order reading is script-carried, not declared, so there is no name_order for a shape to assert -- and min_baseline (2.1.0) is documentary, recording when the reading shipped, rather than a skip trigger the way shapes 4/5's is. Case.__post_init__ enforces PURITY on shapes 6/7: a classified CJK codepoint is required, and a comma or any ASCII letter is refused -- that ground belongs to the new tolerated=True flag, not a shape tag. Shape 6 additionally refuses 间隔号 U+00B7 and wholly-katakana text; shape 7 requires one of those two. The fullwidth nakaguro U+30FB is NOT part of that vocabulary on its own: decisions.md#T3 scopes its source-order reading to katakana content specifically, so U+30FB on non-katakana text (e.g. '高橋・一郎', Han) is an ordinary separator and the text reads family-first -- pinned by cases.py's own ja_nakaguro_han_takes_the_han_order and zh_interpunct_nakaguro_typed_stays_roster rows. A wholly-katakana text already carries U+30FB when it has one, so _wholly_katakana alone covers the katakana-transcription case without testing the nakaguro as a divider in its own right. Neither shape carries policy= or locale=: a zh-pack row exercises a locale fork, not an input shape. Shapes 1-5's existing CJK refusal now names 6/7 as CJK's home instead of pointing only at corpus_cjk.jsonl. tolerated: bool = False marks a row's text as best-effort, contract- exempt input, mutually exclusive with shape and restricted to CJK-bearing text; its docstring states this is the row-level declaration a later corpus-generator task will read, not a routing that exists yet. TDD: seventeen parametrized probes in test_cases.py pin each refusal to its own diagnosis (twelve raising, five valid constructions -- among them a spaced wholly-katakana shape-7 admission, a Han+nakaguro shape-6 admission proving U+30FB does not force source order outside katakana, and a tolerated construction built from the same comma text a shape probe refuses, read as a deliberate boundary pair), added before the validator and watched red before green. The two existing cross-file guards (test_case_shape_ids_exist_in_the_inventory, test_every_shape_orders_resolve_and_bound_sanely) needed no edits and pass unmodified now that both sides of the shape-id mirror have grown to {1..7}. Fixed, not just documented: compare.py's min-baseline skip was keyed on `shape is not None`, not on `shape.order is not None`, so it would have skipped shape-6/7 corpus_shapes entries at 1.4.0/2.0.0 once a later task tags case rows with them -- contradicting this design's intent that those diffs stay compared as the already-classified East Asian arc. The skip loop in compare.py's main() now gates on `shapes_by_id[shape].order is not None` too, matching shapes.py's stated design (an order-None shape's min_baseline is documentary; an order-bearing shape's is a real skip trigger). A new pin test, test_an_order_none_shapes_later_minimum_does_not_skip_the_entry, mirrors the existing skip test with a shape-6 fixture at baseline 1.4.0 and asserts the entry reaches the fake worker with no skip line printed. shapes.py's docstring was rewritten from a known-bug caveat into a statement of the now-true semantics, and the Shape NamedTuple's min_baseline field comment no longer claims "oldest baseline that supports the order" for shapes whose order is None. Verified the fix changes nothing today (no shape-6/7 corpus entries exist yet): the gate at 1.4.0 and 2.2.0 reproduces Task 0's reference exactly -- 226 intentional diffs at 1.4.0 (0 at 2.2.0), 0 unexplained, 0 radar unclassified at both, and corpus_shapes.jsonl still reads "(30, 7 skipped)" at 1.4.0 (the 7 being shapes 4/5, order-bearing) and "(30)" at 2.2.0. Also: _has_latin_letter renamed to _has_ascii_letter with a docstring stating what it does and does not test (neither commas nor non-ASCII Latin, deliberately); _has_katakana_only rebuilt on the parser's own _script_matcher(Script.KATAKANA, whole=True) instead of a hand-copied codepoint range, its docstring quoting _policy.py's own halfwidth- katakana span (U+FF65-U+FF9F) rather than a guessed one; the two `assert self.shape ...` comments corrected from "narrows for mypy" (they also state a caller contract) to name both roles; _SHAPE_IDS restored to an explicit {1, 2, 3, 4, 5, 6, 7} literal rather than range(1, 8), matching the table's original style. Post-review correction (same commit): the first pass over-unified the divider vocabulary, treating U+30FB as equivalent to U+00B7 in both directions -- which would have made shape 6 wrongly REFUSE '高橋・一郎' (a genuine family-first Han name) and shape 7 wrongly ADMIT it under a source-order reading the parser does not give it. Caught against the case table's own pins before landing; fixed by dropping U+30FB from the `has_divider` test (U+30FB sits inside the KATAKANA span, so `_wholly_katakana` already carries the legitimate katakana-transcription case on its own) and rewording both shapes' messages to state the resulting asymmetry plainly instead of asserting a shared two-codepoint vocabulary. Co-Authored-By: Claude Fable 5 --- tests/v2/cases.py | 229 +++++++++++++++++++++++++++------- tests/v2/test_cases.py | 83 +++++++++++- tests/v2/test_differential.py | 53 ++++++++ tools/differential/compare.py | 31 +++-- tools/differential/shapes.py | 49 +++++++- 5 files changed, 388 insertions(+), 57 deletions(-) diff --git a/tests/v2/cases.py b/tests/v2/cases.py index 77d852f6..28645030 100644 --- a/tests/v2/cases.py +++ b/tests/v2/cases.py @@ -37,15 +37,43 @@ from nameparser import (FAMILY_FIRST, FAMILY_FIRST_GIVEN_LAST, GIVEN_FIRST, Policy) # Not in nameparser.__all__: _order_repr renders a name_order for an -# error message, and _SCRIPT_RANGES/_script_matcher build the same -# borrowed predicate build_cjk_corpus.py uses to find CJK text. -from nameparser._policy import (PatronymicRule, _SCRIPT_RANGES, _order_repr, - _script_matcher) +# error message, Script/_SCRIPT_RANGES/_script_matcher build the same +# borrowed predicates build_cjk_corpus.py uses to find CJK text. +from nameparser._policy import (PatronymicRule, Script, _SCRIPT_RANGES, + _order_repr, _script_matcher) #: mirrors tools/differential/shapes.py's SHAPES keys; #: test_case_shape_ids_exist_in_the_inventory (test_ledger_guards.py) #: holds the two equal, since this file cannot import tools/. -_SHAPE_IDS = frozenset({1, 2, 3, 4, 5}) +_SHAPE_IDS = frozenset({1, 2, 3, 4, 5, 6, 7}) + + +def _has_ascii_letter(text: str) -> bool: + """True when text contains an ASCII a-z/A-Z letter. Shapes 6/7's + purity check calls this ALONGSIDE a separate comma test -- this + function tests neither a comma nor a non-ASCII Latin letter on its + own. The ASCII restriction is deliberate: a diacritic or a letter + outside a-z/A-Z is not what a Latin WRAPPER around CJK text looks + like in the corpus today (title/credential vocabulary is ASCII), + and widening this is a call for whichever future row needs it.""" + return any(c.isascii() and c.isalpha() for c in text) + + +#: Shape 7's other admission besides an explicit divider: a +#: transcription written wholly in katakana with no dividing +#: punctuation at all (e.g. "マイケルジャクソン" or the spaced +#: "マイケル ジャクソン"). Built on the parser's own predicate -- +#: _script_matcher(Script.KATAKANA, whole=True) -- rather than a +#: hand-copied codepoint range, so the KATAKANA span lives in exactly +#: one place (nameparser._policy._SCRIPT_RANGES). That table's choice, +#: not this file's: halfwidth katakana (a different Unicode block, +#: U+FF65-U+FF9F, per _policy.py's own comment) is out of scope. +#: Applied to the text with whitespace stripped, so a spaced +#: transcription still counts as wholly katakana; a whitespace-only +#: string never reaches this predicate in practice, since the purity +#: check's _has_cjk gate (a real classified codepoint) has already +#: run by the time shape 7 consults it. +_wholly_katakana = _script_matcher(Script.KATAKANA, whole=True) #: Whether a text carries a codepoint the parser's script table #: classifies -- built once, same idiom as build_cjk_corpus.py's @@ -73,6 +101,21 @@ class Case: #: exercising a policy fork rather than an input shape stays #: untagged. shape: int | None = None + #: Marks a row's text as TOLERATED input (2026-09-01 CJK demotion): + #: parsed best-effort and contract-exempt, the opposite of a shape + #: tag -- mutually exclusive with `shape`, since a shape ADMITS a + #: text to the contract and tolerated deliberately does not. Every + #: composed/wrapped CJK form (a comma listing, a Latin title or + #: credential around a CJK name) is this table's ground for it, + #: not shapes 6/7's. Restricted to CJK-bearing text (`_has_cjk`): + #: it exists to demote composed/wrapped CJK forms specifically, and + #: a Latin row asking for it is a smell until some future arc + #: argues otherwise. Intent, not yet current behavior: the + #: generator split that actually routes a tolerated row to its own + #: radar-tier corpus file (rather than today's corpus_cjk.jsonl) + #: lands with a later task in the 2026-09-01 plan -- this flag is + #: the row-level declaration that split will read. + tolerated: bool = False def __post_init__(self) -> None: if self.policy is not None and self.locale is not None: @@ -84,48 +127,146 @@ def __post_init__(self) -> None: if self.shape is not None: if self.shape not in _SHAPE_IDS: raise ValueError(f"{self.id}: unknown shape {self.shape}") - # A locale carries an order too (script_orders), but as a - # LOOKUP this table cannot see -- cases.py stays - # import-light and stores only the locale CODE. Faking - # "declared" as GIVEN_FIRST for a locale row would let a - # tag validate against an order nobody here can name. - if self.locale is not None: + if self.shape in (6, 7): + self._check_cjk_shape_purity() + else: + self._check_latin_shape_order() + # tolerated is the opposite of a shape tag: a reviewed act + # admitting a composed/wrapped CJK form to the radar corpus + # rather than the contract one. Checked regardless of which + # branch above ran (or whether shape was tagged at all), so a + # row cannot smuggle both declarations onto one text. + if self.tolerated: + if self.shape is not None: raise ValueError( - f"{self.id}: a shape tag needs the row's own " - f"policy; a locale carries an order this table " - f"cannot see") - # corpus_cjk.jsonl already claims this ground: _has_cjk is - # the same predicate build_cjk_corpus.py selects with, so - # a shape tag would double-admit the text (shapes 1-5 are - # the Latin-order arrangements; CJK is deliberately absent - # from shapes.py, #469's open question). Order alone - # cannot stand in for this check -- DEFAULT_SCRIPT_ORDERS - # forces HAN/HANGUL/HIRAGANA to FAMILY_FIRST but leaves - # KATAKANA unmapped, so a pure-katakana text can carry a - # GIVEN_FIRST name_order and still be CJK ground, not a - # shape. - if _has_cjk(self.text): + f"{self.id}: tolerated is mutually exclusive with " + f"shape; a tolerated row is the opposite of admitted") + if not _has_cjk(self.text): + raise ValueError( + f"{self.id}: tolerated requires CJK text (a " + f"classified codepoint _has_cjk recognizes); it " + f"exists for the CJK comma demotion, and a Latin " + f"row asking for it is a smell until some future " + f"arc argues otherwise") + + def _check_latin_shape_order(self) -> None: + """Shapes 1-5: the Latin-order arrangements, each implying a + name_order the row's own policy (or its absence) must agree + with, and each refusing CJK text outright.""" + # Contract: only called from the `if self.shape is not None` + # branch above -- restated here (not just implied by the call + # site) because it also narrows the type for mypy, which + # cannot see across the method boundary on its own. + assert self.shape is not None + # A locale carries an order too (script_orders), but as a + # LOOKUP this table cannot see -- cases.py stays + # import-light and stores only the locale CODE. Faking + # "declared" as GIVEN_FIRST for a locale row would let a + # tag validate against an order nobody here can name. + if self.locale is not None: + raise ValueError( + f"{self.id}: a shape tag needs the row's own " + f"policy; a locale carries an order this table " + f"cannot see") + # corpus_cjk.jsonl already claims this ground: _has_cjk is + # the same predicate build_cjk_corpus.py selects with, so + # a shape tag would double-admit the text (shapes 1-5 are + # the Latin-order arrangements; shapes 6/7 are the CJK + # arrangements, #469's now-settled third-shape question). + # Order alone cannot stand in for this check -- + # DEFAULT_SCRIPT_ORDERS forces HAN/HANGUL/HIRAGANA to + # FAMILY_FIRST but leaves KATAKANA unmapped, so a pure- + # katakana text can carry a GIVEN_FIRST name_order and + # still be CJK ground, not a shape. + if _has_cjk(self.text): + raise ValueError( + f"{self.id}: shape {self.shape} cannot tag CJK " + f"text; that ground belongs to shapes 6/7 " + f"(corpus_cjk.jsonl), not this shape") + declared = (self.policy.name_order if self.policy is not None + else GIVEN_FIRST) + wanted = {4: FAMILY_FIRST, 5: FAMILY_FIRST_GIVEN_LAST}.get( + self.shape, GIVEN_FIRST) + if declared != wanted: + if self.policy is not None: + declared_desc = ( + f"the row's policy declares {_order_repr(declared)}") + else: + declared_desc = ("this row declares no policy, so it " + "is GIVEN_FIRST") + raise ValueError( + f"{self.id}: shape {self.shape} implies name_order " + f"{_order_repr(wanted)}, but {declared_desc}; add " + f"policy=Policy(name_order={_order_repr(wanted)}) or " + f"drop the tag") + + def _check_cjk_shape_purity(self) -> None: + """Shapes 6/7 (2026-09-01): the CJK arrangements, admitted + wholly classified-script text only -- no comma, no Latin + letter. Every composed/wrapped form is tolerated=True's + ground, not a shape tag's, so this REFUSES rather than + requires a particular arrangement beyond that purity test + (plus shape 7's divider/katakana requirement, and shape 6's + interpunct refusal, below).""" + # Contract: only called from the `if self.shape in (6, 7)` + # branch above -- restated here (not just implied by the call + # site) because it also narrows the type for mypy, which + # cannot see across the method boundary on its own. + assert self.shape in (6, 7) + # A zh-pack row exercises a locale FORK (the segmenter, an + # opt-in policy choice), not an input shape: the default- + # policy reading of the same string is what the shape admits, + # so shape 6/7 rows carry neither. (Nothing separately checks + # self.policy here because shapes 6/7's order is None -- + # there is no order for a policy to agree or disagree with -- + # so a stray policy would silently do nothing; refusing both + # together keeps the row's intent legible.) + if self.policy is not None or self.locale is not None: + raise ValueError( + f"{self.id}: shape {self.shape} rows carry neither " + f"policy nor locale; a zh-pack row exercises a locale " + f"fork, not an input shape") + if not _has_cjk(self.text): + raise ValueError( + f"{self.id}: shape {self.shape} requires a classified " + f"codepoint (CJK text); {self.text!r} carries none") + if "," in self.text: + raise ValueError( + f"{self.id}: shape {self.shape} refuses a comma; " + f"composed comma forms belong under tolerated=True, " + f"not a shape tag") + if _has_ascii_letter(self.text): + raise ValueError( + f"{self.id}: shape {self.shape} refuses a Latin " + f"letter; Latin-wrapped compositions belong under " + f"tolerated=True, not a shape tag") + # U+00B7 (间隔号) marks a name transcription in SOURCE order + # (W1 Accepted) -- shape 7's ground, not shape 6's family- + # first one. The fullwidth nakaguro U+30FB is NOT a source- + # order marker on its own: decisions.md#T3 scopes that reading + # to the codepoint, so U+30FB on non-katakana text is an + # ordinary Han/Hangul separator and the text reads family- + # first (cases.py's own ja_nakaguro_han_takes_the_han_order, + # '高橋・一郎', pins exactly this). A wholly-katakana text + # DOES read as a transcription regardless of whether it + # happens to contain U+30FB internally -- that admission comes + # from being wholly katakana, not from the nakaguro -- so + # _wholly_katakana already covers the katakana case and U+30FB + # is not tested as a divider here at all. + has_divider = "·" in self.text + stripped = "".join(self.text.split()) + is_transcription = has_divider or _wholly_katakana(stripped) + if self.shape == 6: + if is_transcription: raise ValueError( - f"{self.id}: shape {self.shape} cannot tag CJK " - f"text; that ground is corpus_cjk.jsonl's, and " - f"whether a family-first CJK shape exists is " - f"#469's open question") - declared = (self.policy.name_order if self.policy is not None - else GIVEN_FIRST) - wanted = {4: FAMILY_FIRST, 5: FAMILY_FIRST_GIVEN_LAST}.get( - self.shape, GIVEN_FIRST) - if declared != wanted: - if self.policy is not None: - declared_desc = ( - f"the row's policy declares {_order_repr(declared)}") - else: - declared_desc = ("this row declares no policy, so it " - "is GIVEN_FIRST") + f"{self.id}: shape 6 refuses U+00B7 and wholly-" + f"katakana text; that reads source order and " + f"belongs to shape 7") + else: + if not is_transcription: raise ValueError( - f"{self.id}: shape {self.shape} implies name_order " - f"{_order_repr(wanted)}, but {declared_desc}; add " - f"policy=Policy(name_order={_order_repr(wanted)}) or " - f"drop the tag") + f"{self.id}: shape 7 requires U+00B7 or wholly-" + f"katakana text; {self.text!r} has neither") _ES = Policy(patronymic_rules=frozenset({PatronymicRule.EAST_SLAVIC})) diff --git a/tests/v2/test_cases.py b/tests/v2/test_cases.py index 165fa0f7..89511b31 100644 --- a/tests/v2/test_cases.py +++ b/tests/v2/test_cases.py @@ -97,7 +97,7 @@ def test_the_family_partitions_into_particles_and_base( "the row's policy declares FAMILY_FIRST_GIVEN_LAST", id="shape-order-disagrees-with-the-rows-own-policy"), pytest.param( - dict(text="John Smith", shape=6), + dict(text="John Smith", shape=8), "unknown shape", id="shape-id-outside-the-inventory"), pytest.param( @@ -105,8 +105,89 @@ def test_the_family_partitions_into_particles_and_base( policy=Policy(name_order=FAMILY_FIRST)), "cannot tag CJK text", id="cjk-refusal-survives-a-matching-order"), + pytest.param( + dict(text="김민준, 지훈", shape=6), + "refuses a comma", + id="shape-6-refuses-a-comma"), + pytest.param( + dict(text="김민준 V", shape=6), + "refuses a Latin letter", + id="shape-6-refuses-a-latin-letter"), + pytest.param( + dict(text="김민준·지훈", shape=6), + "belongs to shape 7", + id="shape-6-refuses-the-interpunct"), + pytest.param( + dict(text="マイケル・ジャクソン", shape=6), + "belongs to shape 7", + id="shape-6-refuses-the-nakaguro"), + pytest.param( + dict(text="マイケルジャクソン", shape=6), + "belongs to shape 7", + id="shape-6-refuses-wholly-katakana-text"), + pytest.param( + dict(text="John Smith", shape=6), + "requires a classified codepoint", + id="shape-6-requires-cjk-text"), + pytest.param( + dict(text="김민준", shape=7), + r"requires U\+00B7 or wholly-katakana", + id="shape-7-requires-a-divider-or-katakana"), + pytest.param( + dict(text="김민준, 지훈", shape=7), + "refuses a comma", + id="shape-7-refuses-a-comma-too"), + pytest.param( + dict(text="高橋・一郎", shape=7), + r"requires U\+00B7 or wholly-katakana", + id="shape-7-refuses-a-nakaguro-on-non-katakana-text"), + pytest.param( + dict(text="김민준", shape=6, locale="zh"), + "carry neither policy nor locale", + id="shape-6-refuses-a-locale"), + pytest.param( + dict(text="김민준", shape=6, tolerated=True), + "mutually exclusive with shape", + id="tolerated-and-shape-are-mutually-exclusive"), + pytest.param( + dict(text="John Smith", tolerated=True), + "tolerated requires CJK text", + id="tolerated-requires-cjk-text"), ]) def test_case_construction_rejects_a_bad_shape_tag( kwargs: dict[str, Any], match: str) -> None: with pytest.raises(ValueError, match=match): Case(id="probe", expect={}, **kwargs) + + +#: The constructions the battery above proves nothing rejects: a pure +#: shape-6 row, a Han shape-6 row divided by a nakaguro that does NOT +#: mark it as source order (U+30FB is not a divider outside katakana +#: -- decisions.md#T3; this is family-first per +#: ja_nakaguro_han_takes_the_han_order), an interpunct-divided shape-7 +#: row, a SPACED wholly-katakana shape-7 row (the subtler admission -- +#: a transcription with no U+00B7 at all is still a shape, not a +#: demotion, as long as every non-space character is katakana), and a +#: tolerated row built from the SAME text a shape probe above refuses +#: as a comma -- the boundary reading the pair as intended: what a +#: shape tag refuses, tolerated=True admits. Each must construct +#: cleanly -- the purity rule is a REFUSAL rule, not a requirement +#: that admits nothing. +@pytest.mark.parametrize("kwargs", [ + pytest.param(dict(text="김민준", shape=6), id="pure-shape-6-constructs"), + pytest.param(dict(text="高橋・一郎", shape=6), + id="han-nakaguro-shape-6-constructs"), + pytest.param(dict(text="威廉·莎士比亚", shape=7), + id="interpunct-shape-7-constructs"), + pytest.param(dict(text="マイケル ジャクソン", shape=7), + id="spaced-wholly-katakana-shape-7-constructs"), + pytest.param(dict(text="김민준, 지훈", tolerated=True), + id="tolerated-accepts-the-comma-text-a-shape-tag-refuses"), +]) +def test_case_construction_accepts_a_valid_shape_or_tolerated_tag( + kwargs: dict[str, Any]) -> None: + case = Case(id="probe", expect={}, **kwargs) + if "shape" in kwargs: + assert case.shape == kwargs["shape"] + else: + assert case.tolerated is True diff --git a/tests/v2/test_differential.py b/tests/v2/test_differential.py index b7e92079..091c2bc1 100644 --- a/tests/v2/test_differential.py +++ b/tests/v2/test_differential.py @@ -191,6 +191,59 @@ def _fake(v: str, w: bool, assert "corpus_x.jsonl (2, 1 skipped)" in out +def test_an_order_none_shapes_later_minimum_does_not_skip_the_entry( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + """The companion to test_entries_below_their_shapes_min_baseline_ + are_skipped, pinning the other half of the same branch: shape 6's + min_baseline (2.1.0) is DOCUMENTARY, not a skip trigger, because + `order` is None -- the default policy already exists at 1.4.0, so + there is no order for that baseline's worker to fail to honor. + Without the `shapes_by_id[shape].order is not None` gate in + compare.py's skip loop, this entry would be silently dropped the + same way an order-bearing one correctly is above -- this proves + the gate actually distinguishes the two rather than reverting to + shape-blind or, worse, always-skip behavior.""" + import contextlib + import io + import json as _json + import sys + from nameparser import HumanName + name = "김민준" + corpus = tmp_path / "corpus_x.jsonl" + corpus.write_text( + _json.dumps({"name": name, "shape": 6}, ensure_ascii=False) + "\n", + encoding="utf-8") + (tmp_path / "expected_since_1.4.0.toml").write_text("", encoding="utf-8") + monkeypatch.setitem(compare._CORPUS_FLOORS, corpus.name, 1) + monkeypatch.setitem(compare._CORPUS_TIERS, corpus.name, "contract") + monkeypatch.setattr(compare, "HERE", tmp_path) + # The tree's own facade reading, used as the "1.4.0" side too -- + # this test is about the skip decision, not about what the parse + # produces, so an old/new facade that agree by construction keeps + # a real diff from muddying the assertion. + old_facade = {k: (v or "") for k, v in HumanName(name).as_dict().items()} + sent: dict = {} + + def _fake(v: str, w: bool, + entries: list[dict[str, object]]) -> tuple[dict, list[dict]]: + sent["entries"] = list(entries) + return ({"__version__": v, + "__file__": "/wheel/nameparser/__init__.py"}, + [{"facade": old_facade}]) + + monkeypatch.setattr(compare, "_run_worker", _fake) + monkeypatch.setattr(sys, "argv", ["compare.py", "--baseline", "1.4.0", + "--corpus", str(corpus)]) + buf = io.StringIO() + with contextlib.redirect_stdout(buf): + code = compare.main() + assert code == 0 + assert [e["name"] for e in sent["entries"]] == [name] + out = buf.getvalue() + assert "skipped" not in out + assert "corpus_x.jsonl (1)" in out + + def _tree_v2_row(name: str, order: str) -> dict: """The tree's own v2 reading of `name` under `order`, built the same way main()'s tree side and the worker template's _v2_row both diff --git a/tools/differential/compare.py b/tools/differential/compare.py index 817599f9..b3112a64 100644 --- a/tools/differential/compare.py +++ b/tools/differential/compare.py @@ -1300,17 +1300,30 @@ def main() -> int: # order alone (true today, since every order-bearing shape's # minimum is 2.0.0) -- a future order-None shape carrying a higher # minimum than an order-bearing duplicate would make survival, and - # so the skip decision, depend on which file loaded first. + # so the skip decision, depend on which file loaded first. That + # future is now real -- shapes 6/7 are order-None with a later + # minimum (2.1.0) than shapes 1-3's (1.4.0) -- but still + # unreachable as an actual duplicate: shapes 1-5's purity check + # refuses CJK text and shapes 6/7's requires it, so no string can + # ever carry two order-None shape tags with different minimums, + # whichever file loaded first. by_key: dict[tuple[str, str | None], dict[str, object]] = {} for e in entries: by_key.setdefault((e["name"], e.get("order")), e) entries = list(by_key.values()) - # an order-bearing entry must never reach a worker whose baseline + # an ORDER-BEARING entry must never reach a worker whose baseline # cannot honor it (no Policy below 2.0.0) -- skip it and say so, - # rather than shrink the comparison silently. Skips are also - # counted PER FILE: per_file above records pre-skip counts, so a - # shapes corpus fully skipped at an old baseline would otherwise - # print at full size while contributing nothing. + # rather than shrink the comparison silently. An order-NONE + # shape's min_baseline is documentary, not a skip trigger: the + # default policy exists at every baseline, so a name tagged with + # such a shape compares just fine below its min_baseline -- the + # resulting diff (if any) is an ordinary classified one, not a gap + # the skip needs to hide. Only an unhonorable ORDER forces a skip + # (shapes.py's docstring states the same asymmetry against shapes + # 4/5). Skips are also counted PER FILE: per_file above records + # pre-skip counts, so a shapes corpus fully skipped at an old + # baseline would otherwise print at full size while contributing + # nothing. kept = [] dropped = 0 dropped_by_file: dict[str, int] = {} @@ -1318,8 +1331,10 @@ def main() -> int: dropped_minimums: set[str] = set() for e in entries: shape = e.get("shape") - if shape is not None and _parse_version(baseline) \ - < _parse_version(shapes_by_id[shape].min_baseline): + if (shape is not None + and shapes_by_id[shape].order is not None + and _parse_version(baseline) + < _parse_version(shapes_by_id[shape].min_baseline)): dropped += 1 dropped_by_file[e["file"]] = dropped_by_file.get(e["file"], 0) + 1 dropped_shape_ids.add(shape) diff --git a/tools/differential/shapes.py b/tools/differential/shapes.py index 5d7294d7..568890ef 100644 --- a/tools/differential/shapes.py +++ b/tools/differential/shapes.py @@ -17,9 +17,39 @@ its declared order -- not that every word-level reading the notation could admit is pinned here. -The CJK arrangement is deliberately absent: whether it is a third -family-first shape is #469's open question, and corpus_cjk.jsonl -covers that ground meanwhile. +Shapes 6-7 are the CJK arrangements (#469's third-shape question, +settled 2026-09-01): shape 6 is family-first CJK, shape 7 is a +source-order transcription listing. Both carry `order=None` -- not +because they are order-less, but because the family-first (resp. +source-order) reading is SCRIPT-carried rather than declared: a pure +shape 6/7 string parses correctly under the DEFAULT policy already +(rules.md#W4, T2/T3), the same way DEFAULT_SCRIPT_ORDERS routes Han/ +Hangul/Hiragana to family-first without a Policy saying so. A shape +whose reading depends on the string's own script has nothing for +`order` to name. + +Their `min_baseline` (2.1.0, when East Asian support shipped) is +DOCUMENTARY rather than a skip trigger, and that is a real asymmetry +with shapes 4/5: those two skip below their baseline because Policy +itself (and the order it carries) did not exist yet -- an order- +bearing shape sent to a pre-Policy worker has nothing to apply. +Shapes 6/7 need no such protection: `order` is None here, so at +1.4.0/2.0.0 these strings are compared as opaque tokens rather than +skipped -- the old baselines DO parse them, just without the script +rules, and the resulting diffs are already the classified East Asian +arc (fix(cjk-*) et al.), not an unhandled gap. compare.py's +min-baseline skip (main(), the `dropped`/`kept` split) is gated on +`shapes_by_id[shape].order is not None` precisely so this holds: it +drops an order-bearing entry below its shape's minimum, and leaves an +order-None entry -- shapes 1-3 as much as 6/7 -- to compare at every +baseline regardless of min_baseline. (Verified 2026-09-01: before +that gate existed the skip was keyed on `shape is not None` alone, +which happened to be harmless only because shapes 1-3 all carry +min_baseline "1.4.0", the earliest baseline the gate ever runs, so no +order-None shape had ever actually triggered it. Shapes 6/7, the +first order-None shapes with a later minimum, would have -- silently +dropping the "already classified" comparison this paragraph +describes. The `order is not None` gate closes that.) `order` is the PUBLIC constant name on the nameparser package, as a string, because the consumer that matters is the generated baseline @@ -33,7 +63,12 @@ class Shape(NamedTuple): order: str | None # public constant name on nameparser, None = default notation: str - min_baseline: str # oldest baseline that supports the order + # For an order-bearing shape: the oldest baseline that can honor + # the order (compare.py skips an earlier one). For an order-None + # shape: documentary only, recording when the reading shipped -- + # see the module docstring's shapes-6/7 paragraph for why that is + # not a skip trigger. + min_baseline: str SHAPES: dict[int, Shape] = { @@ -53,4 +88,10 @@ class Shape(NamedTuple): 5: Shape("FAMILY_FIRST_GIVEN_LAST", "Title Family Middle Middle Given [, Suffix]", "2.0.0"), + 6: Shape(None, + "Family Given [Honorific]", + "2.1.0"), + 7: Shape(None, + "Given[·Given]·Family / katakana transcription (source order)", + "2.1.0"), } From d562164080494df5256541993f2fdd1c14c67a72 Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Tue, 1 Sep 2026 16:43:13 -0700 Subject: [PATCH 2/7] feat(tests): the comma and wrapper CJK rows declare tolerated; pure exemplars take shapes 6 and 7 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The 2026-09-01 CJK demotion's sweep. Every CJK-bearing row whose text carries a comma or an ASCII letter was reviewed one by one; 28 of the 32 candidates now declare tolerated=True. The line is FORMAT PURITY, not comma class: a family-comma listing, an honorific-comma probe and a Latin title or credential wrapped around a CJK name are all forms native CJK writing does not contain, so the contract should not promise how they parse. No text and no expected value moves here -- the rows still pin current behavior at HEAD; only the differential tier they will land on changes (the generator split is a later task). Tolerated, family-comma listings (a comma naming the family is not a CJK convention): ko_family_comma_stays_whole 남궁민수, 지훈 ko_honorific_glued_given_after_family_comma 김, 민준씨 ja_honorific_glued_given_after_family_comma 田中, 太郎さん ja_honorific_glued_family_comma_no_site 田中さん, 太郎 ko_honorific_glued_given_nickname_family_comma 김, 민준씨 (Jimmy) Tolerated, the honorific-comma probe zoo (agent-authored comma spellings of a glued honorific; the peel's answer on them is best-effort, not promised): ko_honorific_after_comma 김민준, 씨 ko_honorific_written_with_a_period 김민준, 씨. ko_honorific_period_under_strict_comma_suffixes 김민준, 씨. ja_honorific_period_does_not_stop_the_peel 田中さん, 様. ja_honorific_glued_family_comma_suffixy_second_run 田中さん, V. ja_honorific_glued_family_comma_strict_knob 田中さん, V. ko_honorific_glued_family_comma_suffixy_second_run 김민준씨, V. zh_honorific_glued_family_comma_suffixy_second_run 王先生, V. ko_honorific_glued_family_comma_site_only_beyond_the_comma 이, J.씨 ko_honorific_glued_family_comma_site_in_both_runs 김민준씨, J.씨 ko_honorific_glued_family_comma_lone_post_nominal_before_it 선생님, J.씨 Tolerated, Latin wrappers around a CJK name -- a leading title, a trailing credential or generational suffix, a parenthesized English nickname, written with a comma or without: ja_honorific_glued_family_comma_title_only 田中さん, Dr. ja_honorific_glued_family_comma 田中さん, PhD ja_honorific_glued_family_comma_credential_pair 田中さん, Ph. D. ja_honorific_glued_family_comma_credential_pair_strict_knob 田中さん, Ph. D. zh_interpunct_with_suffix_comma 威廉·莎士比亚, PhD ko_suffix_comma_name_part_splits Dr 김민준, Jr. ko_honorific_glued_given_suffix_comma Dr 김민준씨, Jr. ko_honorific_glued_given_suffix_comma_initial Dr 김민준씨, V. ko_honorific_glued_given_trailing_suffix 김민준씨 Jr. ko_honorific_glued_given_nickname 김민준씨 (Jimmy) ja_honorific_glued_before_a_roman_suffix 田中さん II ja_honorific_glued_before_an_initial 田中さん V. Swept and deliberately NOT marked, four rows that are CJK-plus-ASCII for reasons outside this demotion's scope: mixed_script_untouched_by_script_orders John 王 -- not a wrapper but a genuinely mixed-script name, and the row pins script_orders DECLINING (effective_script is None), which is normative. zh_interpunct_half_flanked_stays 王·Smith -- the interpunct guard's negative case: one classified neighbour is not enough. Also a decline, also normative. latin_stem_glued_kana_honorific Andersonさん -- the name is LATIN and the CJK is honorific vocabulary; "Japanese text about a foreigner" is a real form, and the row's own measurement records that the script machinery never touches it. latin_stem_glued_hangul_honorific Anderson선생님 -- its hangul twin, for the same reason. Per-text consistency (Task 3's generator will hard-error on a text marked on one row and not another): the only CJK texts on more than one row are 田中さん, V. / 田中さん, Ph. D. / 김민준, 씨. -- each a default-policy row plus its knob row, and both halves of each pair are marked. Nothing needed resolving. 98 CJK texts today: 73 contract, 25 tolerated. Shape exemplars, all default-configuration rows, each with a slot clause added to its notes (the #487 convention): shape 6 ko_unspaced_default 김민준 bare Family Given shape 6 ko_spaced_family_first_default 김 민준 the space written shape 6 ko_two_syllable_surname_default 남궁민수 Family at two syllables shape 6 ko_honorific_glued_given 김민준씨 Honorific glued shape 6 ja_kana_spaced_family_first 高橋 みなみ the kana spelling shape 7 zh_interpunct_transcription_source_order 威廉·莎士比亚 the 间隔号 shape 7 ja_nakaguro_divides_the_transcription マイケル・ジャクソン katakana Three of the plan's candidates are NOT tagged. Two have no row and none is authored here: nothing holds the text 山田 太郎 (only 山田 太郎 (マイケル・ジャクソン) and 山田 エミ), and nothing holds 毛·泽东 at all. The third, zh_honorific_suffix_spaced (王小明 先生), has a row but does not instantiate the shape: no DEFAULT segmenter divides 王小明, so its fields are an undivided name plus an honorific and the Given slot of "Family Given [Honorific]" is never filled. The notation's arrangement-not-grammar clause covers which BUCKET a written word lands in, not a missing SLOT, and the deliberately untagged han_unspaced_unsegmented_default (毛泽东) is the precedent. Its note now says why it carries no tag; the zh pack is what splits the token, and a pack row exercises a locale fork, not an input shape. Plan amendment: Task 2's file list named cases.py only, but shape tags are projected by build_shapes_corpus.py, so corpus_shapes.jsonl is regenerated here (30 -> 37 entries) and its floor ratcheted 27 -> 35 with the count comment updated. The seven new entries are pure CJK texts already in corpus_cjk.jsonl and carry order None, so compare.py's (name, order) dedup collapses them: the file grew, the comparison did not. Measured at all four baselines -- intentional 226/205/113/0, unexplained 0, radar unclassified 0, 1120 names compared (1113 at 1.4.0, with the same 7 shape-4/5 skips) -- byte-identical to the plan's reference, which is the arc's stop condition. Co-Authored-By: Claude Fable 5 --- tests/v2/cases.py | 128 ++++++++++++++++++------- tools/differential/compare.py | 9 +- tools/differential/corpus_shapes.jsonl | 7 ++ 3 files changed, 106 insertions(+), 38 deletions(-) diff --git a/tests/v2/cases.py b/tests/v2/cases.py index 28645030..9dc39c29 100644 --- a/tests/v2/cases.py +++ b/tests/v2/cases.py @@ -2892,7 +2892,8 @@ def _check_cjk_shape_purity(self) -> None: "in script_segment on the other structures only, before " "group or assign can say this comma fixed nothing -- so " "the honorific stays glued, joining master's '田中さん, " - "Mr.'. Master peeled it through the suffix-comma route"), + "Mr.'. Master peeled it through the suffix-comma route", + tolerated=True), Case("title_word_trailing_is_not_a_title_mr", "John Smith Mr.", {"given": "John", "middle": "Smith", "family": "Mr."}), @@ -2902,13 +2903,19 @@ def _check_cjk_shape_purity(self) -> None: classification="fix(#271)", notes="hangul is unambiguously Korean: census surnames ship " "as default vocabulary and HANGUL segmentation is " - "default-on"), + "default-on. Shape 6's bare Family Given arrangement, " + "written unspaced, the floor the other shape-6 rows " + "vary from", + shape=6), Case("ko_two_syllable_surname_default", "남궁민수", {"family": "남궁", "given": "민수"}, classification="fix(#271)", ambiguities=("segmentation",), notes="남 is itself a shipped surname; longest-first takes " - "남궁 and records the decided fork"), + "남궁 and records the decided fork. Shape 6's Family " + "slot at two syllables, where the arrangement's own " + "boundary is what has to be found", + shape=6), Case("ko_bare_two_syllable_surname", "남궁", {"family": "남궁"}, classification="fix(#271)", @@ -2922,18 +2929,23 @@ def _check_cjk_shape_purity(self) -> None: notes="the comma already decided the family: segmentation " "is inert under FAMILY_COMMA (comma doctrine -- see " "the script_segment stage docstring, which uses this " - "exact example)"), + "exact example)", + tolerated=True), Case("ko_suffix_comma_name_part_splits", "Dr 김민준, Jr.", {"title": "Dr", "family": "김", "given": "민준", "suffix": "Jr."}, classification="fix(#271)", notes="the one comma structure where segmentation still " "fires: a second word before the comma makes it " "SUFFIX_COMMA, and the name part is a full positional " - "name"), + "name", + tolerated=True), Case("ko_spaced_family_first_default", "김 민준", {"family": "김", "given": "민준"}, classification="fix(#271)", - notes="script_orders, no segmentation involved"), + notes="script_orders, no segmentation involved. Shape 6's " + "Family Given with the space written, the spelling " + "ko_unspaced_default reaches by segmenting instead", + shape=6), Case("han_spaced_family_first_default", "毛 泽东", {"family": "毛", "given": "泽东"}, classification="fix(#271)", @@ -2999,7 +3011,10 @@ def _check_cjk_shape_purity(self) -> None: classification="fix(#272)", notes="hiragana identifies Japanese as certainly as hangul " "identifies Korean; kana-licensed names read " - "family-first by default"), + "family-first by default. Shape 6's Family Given in " + "kana rather than hangul -- the arrangement is one " + "shape across the scripts that carry it", + shape=6), Case("ja_kanji_katakana_pieces", "山田 エミ", {"family": "山田", "given": "エミ"}, classification="fix(#272)", @@ -3022,7 +3037,10 @@ def _check_cjk_shape_purity(self) -> None: notes="the katakana middle dot is the transcription's own " "part divider: it separates like whitespace, the " "license declines each katakana token, and the " - "positional default keeps the source-language order"), + "positional default keeps the source-language order. " + "Shape 7's katakana-transcription half, the arrangement " + "admitted by the script rather than by the 间隔号", + shape=7), Case("ja_nakaguro_han_takes_the_han_order", "高橋・一郎", {"family": "高橋", "given": "一郎"}, classification="fix(#272)", @@ -3084,7 +3102,9 @@ def _check_cjk_shape_purity(self) -> None: "keeps source order -- the B7 is the transcription " "marker, playing the role pure katakana plays in the " "kana license; it divides only between classified " - "characters"), + "characters. Shape 7's Given·Family with the 间隔号 " + "itself written, the divider half of the notation", + shape=7), Case("zh_interpunct_nakaguro_typed_stays_roster", "威廉・莎士比亚", {"given": "莎士比亚", "family": "威廉"}, classification="fix(#272)", @@ -3127,7 +3147,8 @@ def _check_cjk_shape_purity(self) -> None: {"given": "威廉", "family": "莎士比亚", "suffix": "PhD"}, classification="fix(#298)", notes="the transcription reading composes with a suffix " - "comma: the marker is structure-independent"), + "comma: the marker is structure-independent", + tolerated=True), Case("zh_interpunct_half_flanked_stays", "王·Smith", {"given": "王·Smith"}, notes="one classified neighbor is not enough: the guard " @@ -3139,7 +3160,15 @@ def _check_cjk_shape_purity(self) -> None: notes="CJK honorifics FOLLOW the name; a spaced 先生 (Mr.) is " "a suffix, and recognizing it must come before the " "family-first order hands it a role -- unrecognized it " - "read as the GIVEN name under the 2.1 defaults"), + "read as the GIVEN name under the 2.1 defaults. NOT " + "tagged shape 6, though it looks like the Han spelling " + "of one: no DEFAULT segmenter divides 王小明, so the " + "fields here are an undivided name plus an honorific " + "and the arrangement's Given slot is never filled -- " + "the same reason han_unspaced_unsegmented_default " + "(毛泽东) carries no tag. The zh pack is what splits " + "the token (zh_honorific_glued_given), and a pack row " + "exercises a locale fork rather than an input shape"), Case("ko_honorific_ssi", "김민준 씨", {"family": "김", "given": "민준", "suffix": "씨"}, classification="fix(#307) + fix(#271)", @@ -3184,7 +3213,8 @@ def _check_cjk_shape_purity(self) -> None: "the token behind it. Half of the pair that pins " "_is_post_nominal's use of is_suffix_STRICT -- the " "other half is the row below, and swapping in " - "is_suffix_lenient changes that one and not this one"), + "is_suffix_lenient changes that one and not this one", + tolerated=True), Case("ja_honorific_glued_before_an_initial", "田中さん V.", {"given": "田中さん", "family": "V."}, notes="the strict/lenient discriminator, and the reason " @@ -3198,7 +3228,8 @@ def _check_cjk_shape_purity(self) -> None: "'V.', which is these fields under the 2.0 names. " "Classification agrees with what classify does with " "the same token downstream -- 'V.' is a middle " - "initial, not a post-nominal"), + "initial, not a post-nominal", + tolerated=True), Case("ja_honorific_with_a_period_no_comma", "田中さん 様.", {"family": "田中", "suffix": "さん, 様."}, classification="fix(#320)", @@ -3270,7 +3301,8 @@ def _check_cjk_shape_purity(self) -> None: "1.4.0 read " "this first '様.' / last 田中さん, which is exactly what " "2.0 produced before this change -- the row sat at " - "parity until #320 moved it"), + "parity until #320 moved it", + tolerated=True), Case("ja_sama_glued", "山田太郎様", {"family": "山田太郎", "suffix": "様"}, classification="fix(#308) + fix(#271)", @@ -3298,7 +3330,8 @@ def _check_cjk_shape_purity(self) -> None: "'씨.' carried both, so the suffix-shaped piece went to " "the given. 1.4.0 read this first '씨.' / last 김민준 -- " "the same fields 2.0 gave before this change, so the row " - "was at parity and #320 is what moves it"), + "was at parity and #320 is what moves it", + tolerated=True), Case("ko_honorific_period_under_strict_comma_suffixes", "김민준, 씨.", {"family": "김민준", "suffix": "씨."}, policy=Policy(lenient_comma_suffixes=False), @@ -3328,7 +3361,8 @@ def _check_cjk_shape_purity(self) -> None: "(the facade runner skips this row), so the " "classification compares against 1.4.0's single " "reading, first '씨.' / last 김민준 -- the same fields " - "2.0 gave under EITHER setting before this change"), + "2.0 gave under EITHER setting before this change", + tolerated=True), Case("ko_honorific_with_a_period_no_comma", "김민준 씨.", {"given": "민준", "family": "김", "suffix": "씨."}, classification="fix(#320)", @@ -3504,7 +3538,8 @@ def _check_cjk_shape_purity(self) -> None: "also why this row stays single-issue while the rest of " "the block is compound with fix(#271): measured, the " "order table and the segmenter both leave it alone, " - "because the comma already decided the family"), + "because the comma already decided the family", + tolerated=True), Case("ko_honorific_glued_given", "김민준씨", {"family": "김", "given": "민준", "suffix": "씨"}, classification="fix(#308) + fix(#271)", @@ -3512,7 +3547,10 @@ def _check_cjk_shape_purity(self) -> None: "replaces (ko_honorific_glued_given_stays) pinned the " "old boundary: 씨 peels off the last token first, and " "the remainder 김민준 then segments as usual -- peel " - "and split compose, in that order"), + "and split compose, in that order. Shape 6's optional " + "Honorific slot glued, the everyday spelling next to " + "zh_honorific_suffix_spaced's spaced one", + shape=6), Case("ko_honorific_glued_given_trailing_suffix", "김민준씨 Jr.", {"family": "김", "given": "민준", "suffix": "씨, Jr."}, classification="fix(#308) + fix(#271)", @@ -3520,7 +3558,8 @@ def _check_cjk_shape_purity(self) -> None: "post-nominal, so an unrelated trailing suffix cannot " "hide it -- this now agrees with the comma-written " "'Dr 김민준씨, Jr.', where the suffix comma had " - "already put 씨 within reach"), + "already put 씨 within reach", + tolerated=True), Case("ko_honorific_glued_given_suffix_comma", "Dr 김민준씨, Jr.", {"title": "Dr", "family": "김", "given": "민준", "suffix": "씨, Jr."}, @@ -3532,7 +3571,8 @@ def _check_cjk_shape_purity(self) -> None: "name across two runs, #312). Pairs with " "ko_honorific_glued_given_trailing_suffix, whose " "comma-less spelling of the same name reaches the same " - "answer by the scan-back instead"), + "answer by the scan-back instead", + tolerated=True), Case("ko_honorific_glued_given_nickname", "김민준씨 (Jimmy)", {"family": "김", "given": "민준", "suffix": "씨", "nickname": "Jimmy"}, @@ -3544,7 +3584,8 @@ def _check_cjk_shape_purity(self) -> None: "the site -- it is no post-nominal -- and lose the peel " "entirely, with 씨 back in the given name. Nothing else " "pins that choice: under NO_COMMA the two are otherwise " - "the same run"), + "the same run", + tolerated=True), Case("ko_honorific_glued_given_nickname_family_comma", "김, 민준씨 (Jimmy)", {"family": "김", "given": "민준", "suffix": "씨", @@ -3556,7 +3597,8 @@ def _check_cjk_shape_purity(self) -> None: "agree. Here the peel has to cross the comma AND still " "not reach Jimmy, so declining to cross whenever " "extract_delimited claimed something passes the row " - "above and fails only here"), + "above and fails only here", + tolerated=True), Case("ja_honorific_glued_family_comma", "田中さん, PhD", {"family": "田中", "suffix": "さん, PhD"}, classification="fix(#312)", @@ -3568,7 +3610,8 @@ def _check_cjk_shape_purity(self) -> None: "since #296's audit took 'phd' out of TITLES -- this row " "carried title 'PhD' until then, which was the title " "peel claiming a credential because v1's lists put it " - "where v1's parser needed it"), + "where v1's parser needed it", + tolerated=True), Case("ja_honorific_glued_family_comma_suffixy_second_run", "田中さん, V.", {"given": "V.", "family": "田中", "suffix": "さん"}, @@ -3603,7 +3646,8 @@ def _check_cjk_shape_purity(self) -> None: "(first V., last 田中さん) until this change, which is " "what moves it; the 1.4.0 fields are still reachable " "through Policy(lenient_comma_suffixes=False), pinned " - "by ja_honorific_glued_family_comma_strict_knob below"), + "by ja_honorific_glued_family_comma_strict_knob below", + tolerated=True), Case("ja_honorific_glued_family_comma_credential_pair", "田中さん, Ph. D.", {"family": "田中", "suffix": "さん, Ph. D."}, @@ -3634,7 +3678,8 @@ def _check_cjk_shape_purity(self) -> None: "ja_honorific_glued_family_comma above the expectation " "carries TWO deviations -- the peel is #319's, first -> " "family is comma-family's, which 2.0 already had before " - "this change (family 田中さん / suffix 'Ph. D.')"), + "this change (family 田中さん / suffix 'Ph. D.')", + tolerated=True), Case("ja_honorific_glued_family_comma_strict_knob", "田中さん, V.", {"family": "田中さん", "given": "V."}, policy=Policy(lenient_comma_suffixes=False), @@ -3668,7 +3713,8 @@ def _check_cjk_shape_purity(self) -> None: "text, as the other exercise of the knob named above " "does: measured, 1.4.0 gave first 'V.' / last " "田中さん, which is field for field what the knob holds " - "here -- parity, and the point of the knob"), + "here -- parity, and the point of the knob", + tolerated=True), Case("ja_honorific_glued_family_comma_credential_pair_strict_knob", "田中さん, Ph. D.", {"family": "田中", "suffix": "さん, Ph. D."}, @@ -3693,7 +3739,8 @@ def _check_cjk_shape_purity(self) -> None: "'Ph. D.'), which is also why the knob cannot be judged " "against a v1 spelling here -- there is none, so the " "facade runner skips this row as it does the other two " - "knob rows"), + "knob rows", + tolerated=True), Case("ko_honorific_glued_family_comma_suffixy_second_run", "김민준씨, V.", {"given": "V.", "family": "김민준", "suffix": "씨"}, @@ -3708,7 +3755,8 @@ def _check_cjk_shape_purity(self) -> None: "stays 김민준 undivided -- the FAMILY comma gates the " "surname split off, so hangul segmentation never runs " "here and only the peel acts. 1.4.0 gave first 'V.' / " - "last 김민준씨, peeling nothing"), + "last 김민준씨, peeling nothing", + tolerated=True), Case("zh_honorific_glued_family_comma_suffixy_second_run", "王先生, V.", {"given": "V.", "family": "王", "suffix": "先生"}, @@ -3720,7 +3768,8 @@ def _check_cjk_shape_purity(self) -> None: "Policy.segment_scripts, and HAN is not activated here " "(the family is what the peel left behind, not a " "vocabulary split). 1.4.0 gave first 'V.' / last " - "王先生"), + "王先生", + tolerated=True), Case("ko_honorific_glued_family_comma_site_only_beyond_the_comma", "이, J.씨", {"given": "J.", "family": "이", "suffix": "씨"}, @@ -3747,7 +3796,8 @@ def _check_cjk_shape_purity(self) -> None: "settings call this run wholly suffix. 1.4.0 gave first " "'J.씨' / last 이 -- it peels nothing, so the deviation " "here is #312's crossing, which is what puts the site on " - "'J.씨' in the first place"), + "'J.씨' in the first place", + tolerated=True), Case("ko_honorific_glued_family_comma_site_in_both_runs", "김민준씨, J.씨", {"family": "김민준", "suffix": "씨, J.씨"}, @@ -3768,7 +3818,8 @@ def _check_cjk_shape_purity(self) -> None: "junk-tail reach " "ko_honorific_glued_given_suffix_comma_initial's note " "names under a suffix comma. 1.4.0 gave first 'J.씨' / " - "last 김민준씨, peeling neither"), + "last 김민준씨, peeling neither", + tolerated=True), Case("ko_honorific_glued_family_comma_lone_post_nominal_before_it", "선생님, J.씨", {"given": "J.", "family": "선생님", "suffix": "씨"}, @@ -3787,18 +3838,21 @@ def _check_cjk_shape_purity(self) -> None: "shape of that divergence is reachable from the gate, " "which is why this row is one token before the comma; " "_peel_site's docstring derives the bound. " - "1.4.0 gave first 'J.씨' / last 선생님"), + "1.4.0 gave first 'J.씨' / last 선생님", + tolerated=True), Case("ko_honorific_glued_given_after_family_comma", "김, 민준씨", {"family": "김", "given": "민준", "suffix": "씨"}, classification="fix(#312)", notes="under a family comma the name spans both segments and " "the honorific is on the GIVEN side, where the peel " "never looked before #312. Agrees with the spaced " - "김 민준씨"), + "김 민준씨", + tolerated=True), Case("ja_honorific_glued_given_after_family_comma", "田中, 太郎さん", {"family": "田中", "given": "太郎", "suffix": "さん"}, classification="fix(#312)", - notes="the Han twin of the row above"), + notes="the Han twin of the row above", + tolerated=True), Case("zh_interpunct_transcription_glued_honorific", "威廉·莎士比亚さん", {"given": "威廉", "family": "莎士比亚", "suffix": "さん"}, classification="fix(#312)", @@ -3813,7 +3867,8 @@ def _check_cjk_shape_purity(self) -> None: "the site is the last NON-POST-NOMINAL token, which is " "太郎, so nothing peels -- exactly as in the spaced " "田中さん 太郎. #312 was originally filed naming this " - "pair as a disagreement; it never was one"), + "pair as a disagreement; it never was one", + tolerated=True), Case("ko_honorific_glued_given_suffix_comma_initial", "Dr 김민준씨, V.", {"title": "Dr", "family": "김", "given": "민준", "suffix": "씨, V."}, @@ -3841,7 +3896,8 @@ def _check_cjk_shape_purity(self) -> None: "row is still the only SUFFIX comma among the three. " "Its comma-less twin ja_honorific_glued_before_an_initial " "shows the same veto from the other side, where 'V.' is " - "in the name's own run and so IS the site"), + "in the name's own run and so IS the site", + tolerated=True), Case("zh_honorific_glued_surname", "王先生", {"family": "王", "suffix": "先生"}, locale="zh", diff --git a/tools/differential/compare.py b/tools/differential/compare.py index b3112a64..acaee282 100644 --- a/tools/differential/compare.py +++ b/tools/differential/compare.py @@ -506,8 +506,13 @@ def _legal_orders() -> frozenset[str]: "corpus_cjk.jsonl": 95, # 98 today, generated from the case table "corpus_issues.jsonl": 370, # 381 today, harvested and append-only "corpus_rules.jsonl": 150, # 252 today, generated from rules.md - "corpus_shapes.jsonl": 27, # 30 today, generated from shape-tagged - # case rows + "corpus_shapes.jsonl": 35, # 37 today, generated from shape-tagged + # case rows. Ratcheted 27 -> 35 on + # 2026-09-01 with the shape 6/7 + # exemplars (7 CJK names, already in + # corpus_cjk.jsonl and deduped against + # it by (name, order) -- the file grew, + # the comparison did not) } #: Tier per corpus file, fail-closed like the floors above. CONTRACT diff --git a/tools/differential/corpus_shapes.jsonl b/tools/differential/corpus_shapes.jsonl index f5a9af0b..9eba6d93 100644 --- a/tools/differential/corpus_shapes.jsonl +++ b/tools/differential/corpus_shapes.jsonl @@ -28,3 +28,10 @@ {"name": "de la Cruz Juan Carlos", "shape": 5} {"name": "de la Cruz Juan Carlos, Dr.", "shape": 5} {"name": "de la Cruz née Vega", "shape": 5} +{"name": "高橋 みなみ", "shape": 6} +{"name": "김 민준", "shape": 6} +{"name": "김민준", "shape": 6} +{"name": "김민준씨", "shape": 6} +{"name": "남궁민수", "shape": 6} +{"name": "マイケル・ジャクソン", "shape": 7} +{"name": "威廉·莎士比亚", "shape": 7} From eb8f3b6ffc2e530479ea26c1a6af964c477cd0c4 Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Tue, 1 Sep 2026 17:00:14 -0700 Subject: [PATCH 3/7] feat(differential): the tolerated CJK forms move to a radar corpus MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit build_cjk_corpus.py's one sweep now writes two files. The 98 distinct CJK-bearing texts of the case table partition by the `tolerated` flag Task 2 placed on their rows: 73 stay in corpus_cjk.jsonl (contract, unchanged in kind) and 25 go to the new corpus_cjk_tolerated.jsonl, registered as RADAR in compare.py's _CORPUS_TIERS. A demoted name is still parsed at every baseline and still classified against the ledger -- what it can no longer do is fail the run as UNEXPLAINED or demand a rule be written for it. The flag moves the FILE, which is not yet the tier for every text in it, and the prose here says so rather than promising the fix: five of the 25 sit byte-identical in corpus_rules.jsonl as rules.md examples ('田中さん, Dr.' and '田中さん, PhD' under C1, '김, 민준씨' and '田中さん, V.' under W2, '남궁민수, 지훈' under W3), contract files load first, and the (name, order) dedup keeps the contract reading -- so those five stay contract-tier until the rules.md example sweep removes them. build_cjk_corpus.py's docstring, the _CORPUS_TIERS comment and the README paragraph each carry that scope. Per-file counts moved, nothing else did: corpus_cjk.jsonl 98 -> 73 corpus_cjk_tolerated.jsonl 0 -> 25 (new) The floors follow. corpus_cjk.jsonl's is LOWERED 95 -> 70, deliberately and with the demotion named in its comment (the README's lower-the-floor-deliberately clause is the license); the new file takes 22, a little under its 25. The flag is read PER TEXT, not per row, and a text marked on one row while unmarked on another is a hard error rather than a silent choice. A corpus line is a name string, so a text carried by both a default row and a policy/locale fork of it is one line in one file. The failure the raise prevents is not a name on the wrong tier but a name on NO tier: the halves select flags == {False} and {True}, so a split text matches neither and would be dropped from both corpora, watched at no baseline and invisible to both pins. Three texts do sit on two rows each today ('김민준, 씨.', '田中さん, V.', '田中さん, Ph. D.') and all three are consistently marked. A guard test pins the refusal on a fixture table built to be split, since the checked-in table is and must stay consistent; both files also gain the usual staleness pin against the generator's own selection. The stop condition holds. Intentional counts are byte-identical at all four baselines -- 226 / 205 / 113 / 0, unexplained 0, radar unclassified 0, 1113 names compared at 1.4.0 and 1120 at the rest -- because a classified diff counts the same on either tier. The two 1.4.0 rules whose names all demoted, fix(cjk-comma-compound) (11) and fix(cjk-comma-honorific-peel) (7), keep those exact counts from radar sources and neither goes dormant; no EXPLAINED NOTHING, NO LONGER DORMANT, OVER-DECLARED or ORDER-BLIND output at any baseline. Beyond the corpora line, the only movement in the gate's output is which ten names each block SAMPLES: the sample is corpus order, not sorted, and the new file loads after corpus.jsonl. Set equality checked rather than assumed: the new pair's union is the old corpus_cjk.jsonl exactly, with no overlap. That is also why _CORPUS_CLAIMS does not move -- _CORPUS_NAMES pools every corpus*.jsonl, radar included, so a name changing files stays pooled (1116 distinct names, unchanged). A moved claim here would have meant a dropped name, not a demoted one. tools/differential/README.md gains the file's row in the corpora table, a tier-paragraph sentence for the direction this arc travels (a chosen name demoted, and how to promote it back), and the generator's own description under corpus provenance. Wording throughout says "the differential stops enforcing it", not "changeable without notice": the case rows still assert every one of these parses, and a behavior change still edits a classification. The radar tier's own description stops reading "scraped/harvested" now that one radar file holds reviewed names, and the promotion sentence stops offering a shape tag to a name shapes 6/7 structurally refuse -- clearing the flag is that name's road back. Four sweep candidates were declined and now record why on their rows: 'Andersonさん' and 'Anderson선생님' put Latin at the name's CORE with a CJK honorific glued on (the form #308 promises), and 'John 王' and '王·Smith' make the Latin a name PART. The demoted forms wrap Latin around a CJK name; that is the boundary. Co-Authored-By: Claude Fable 5 --- tests/v2/cases.py | 65 ++++++++++---- tests/v2/test_ledger_guards.py | 79 +++++++++++++++-- tools/differential/README.md | 32 ++++++- tools/differential/build_cjk_corpus.py | 86 ++++++++++++++++--- tools/differential/compare.py | 63 +++++++++++--- tools/differential/corpus_cjk.jsonl | 25 ------ tools/differential/corpus_cjk_tolerated.jsonl | 25 ++++++ 7 files changed, 299 insertions(+), 76 deletions(-) create mode 100644 tools/differential/corpus_cjk_tolerated.jsonl diff --git a/tests/v2/cases.py b/tests/v2/cases.py index 9dc39c29..df7d70a3 100644 --- a/tests/v2/cases.py +++ b/tests/v2/cases.py @@ -97,12 +97,19 @@ class Case: #: Tagging a row admits its text to the differential's CONTRACT #: corpus (see tools/differential/shapes.py, projected into #: corpus_shapes.jsonl by build_shapes_corpus.py beside it) under - #: the shape's name_order. Optional: a row + #: the shape's name_order. For shapes 6/7 that admission is + #: NOMINAL: a pure CJK text is already in corpus_cjk.jsonl, also + #: contract, and compare.py's (name, order) dedup collapses the + #: two -- the tag buys the coverage answer, not a comparison. + #: Optional: a row #: exercising a policy fork rather than an input shape stays #: untagged. shape: int | None = None #: Marks a row's text as TOLERATED input (2026-09-01 CJK demotion): - #: parsed best-effort and contract-exempt, the opposite of a shape + #: read best-effort, exempt from the DIFFERENTIAL's contract tier + #: -- not from this table, whose expectations are asserted by the + #: suite either way, and a behavior change on a tolerated row still + #: edits its classification. The opposite of a shape #: tag -- mutually exclusive with `shape`, since a shape ADMITS a #: text to the contract and tolerated deliberately does not. Every #: composed/wrapped CJK form (a comma listing, a Latin title or @@ -110,11 +117,14 @@ class Case: #: not shapes 6/7's. Restricted to CJK-bearing text (`_has_cjk`): #: it exists to demote composed/wrapped CJK forms specifically, and #: a Latin row asking for it is a smell until some future arc - #: argues otherwise. Intent, not yet current behavior: the - #: generator split that actually routes a tolerated row to its own - #: radar-tier corpus file (rather than today's corpus_cjk.jsonl) - #: lands with a later task in the 2026-09-01 plan -- this flag is - #: the row-level declaration that split will read. + #: argues otherwise. build_cjk_corpus.py reads the flag: a + #: tolerated row's text goes to the radar-tier + #: corpus_cjk_tolerated.jsonl instead of the contract + #: corpus_cjk.jsonl, and clearing the flag promotes it back at the + #: next regeneration. Mark every row of a text, or none: the + #: generator reads the flag per TEXT (a corpus line is a name + #: string) and HARD-ERRORS on a split declaration, which + #: __post_init__ cannot catch from inside one row. tolerated: bool = False def __post_init__(self) -> None: @@ -168,11 +178,16 @@ def _check_latin_shape_order(self) -> None: f"{self.id}: a shape tag needs the row's own " f"policy; a locale carries an order this table " f"cannot see") - # corpus_cjk.jsonl already claims this ground: _has_cjk is - # the same predicate build_cjk_corpus.py selects with, so - # a shape tag would double-admit the text (shapes 1-5 are - # the Latin-order arrangements; shapes 6/7 are the CJK - # arrangements, #469's now-settled third-shape question). + # Shapes 1-5 are the LATIN-ORDER arrangements -- a title + # slot, a comma listing, a suffix run -- and CJK text does + # not instantiate one: the arrangement is what the shape id + # names, so tagging a CJK string shape 1 asserts a form the + # string does not have. The CJK arrangements are shapes 6/7 + # (#469's now-settled third-shape question), where the same + # text is admitted under a purity test of its own. (That + # shapes 6/7 do double-admit into corpus_cjk.jsonl is fine + # and deliberate -- the dedup collapses it; double admission + # is not what this check is about.) # Order alone cannot stand in for this check -- # DEFAULT_SCRIPT_ORDERS forces HAN/HANGUL/HIRAGANA to # FAMILY_FIRST but leaves KATAKANA unmapped, so a pure- @@ -2961,7 +2976,10 @@ def _check_cjk_shape_purity(self) -> None: Case("mixed_script_untouched_by_script_orders", "John 王", {"given": "John", "family": "王"}, notes="effective_script is None for a mixed name: script_orders " - "declines and the positional default governs"), + "declines and the positional default governs. Swept as a " + "2026-09-01 tolerated candidate and declined for the " + "reason the kana-stem rows were: the Latin is a name " + "PART here, not a wrapper around a CJK name"), Case("two_han_scripts_untouched_by_script_orders", "毛 김", {"given": "毛", "family": "김"}, notes="two scripts also decline -- the rule is one script, or " @@ -3011,9 +3029,11 @@ def _check_cjk_shape_purity(self) -> None: classification="fix(#272)", notes="hiragana identifies Japanese as certainly as hangul " "identifies Korean; kana-licensed names read " - "family-first by default. Shape 6's Family Given in " - "kana rather than hangul -- the arrangement is one " - "shape across the scripts that carry it", + "family-first by default. Shape 6's Family Given with " + "a kanji family and a hiragana given rather than the " + "hangul of the Korean rows -- the arrangement is one " + "shape across the scripts that carry it, and across a " + "mix of them within one name", shape=6), Case("ja_kanji_katakana_pieces", "山田 エミ", {"family": "山田", "given": "エミ"}, @@ -3153,7 +3173,10 @@ def _check_cjk_shape_purity(self) -> None: {"given": "王·Smith"}, notes="one classified neighbor is not enough: the guard " "requires both, so the undivided dot remains part of " - "the word -- declining, not deciding"), + "the word -- declining, not deciding. Swept as a " + "2026-09-01 tolerated candidate and declined on the " + "same boundary as 'John 王': the Latin is a name part, " + "not a wrapper"), Case("zh_honorific_suffix_spaced", "王小明 先生", {"family": "王小明", "suffix": "先生"}, classification="fix(#307) + fix(#271)", @@ -3402,7 +3425,13 @@ def _check_cjk_shape_purity(self) -> None: "Single-issue on purpose where the block around it is " "compound: measured, disabling script_orders and " "segment_scripts leaves this row unchanged, because a " - "Latin remainder never reaches either"), + "Latin remainder never reaches either. Swept as a " + "2026-09-01 tolerated candidate (CJK text, ASCII " + "letters) and DECLINED, this row and its hangul twin " + "below: the demoted forms wrap Latin AROUND a CJK " + "name, while here the Latin is the name's own core " + "with a CJK honorific glued on -- a form the glued " + "peel is written for, and one #308 promises"), Case("latin_stem_glued_hangul_honorific", "Anderson선생님", {"given": "Anderson", "suffix": "선생님"}, classification="fix(#308)", diff --git a/tests/v2/test_ledger_guards.py b/tests/v2/test_ledger_guards.py index 77d6f90f..a3192531 100644 --- a/tests/v2/test_ledger_guards.py +++ b/tests/v2/test_ledger_guards.py @@ -595,12 +595,14 @@ def test_the_emoji_boundary_rule_copies_the_dividing_ranges() -> None: def test_cjk_corpus_matches_the_case_table() -> None: """corpus_cjk.jsonl is GENERATED, not curated (#295): every distinct case-table text bearing a codepoint the script table - classifies, sorted -- see build_cjk_corpus.py for why the other - two corpora cannot carry these names. The checked-in file must - equal what the generator would write, so a CJK case row added - without regenerating fails HERE instead of silently narrowing - the differential gate back toward the blind spot #295 closed. - Same promise as the toml pin above, aimed at a generated artifact + classifies -- minus the rows that declare `tolerated` (the + 2026-09-01 demotion; they are the twin test below) -- sorted. + See build_cjk_corpus.py for why the other two corpora cannot + carry these names. The checked-in file must equal what the + generator would write, so a CJK case row added without + regenerating fails HERE instead of silently narrowing the + differential gate back toward the blind spot #295 closed. Same + promise as the toml pin above, aimed at a generated artifact instead of a hand copy. """ module = load_tool("build_cjk_corpus") @@ -612,10 +614,73 @@ def test_cjk_corpus_matches_the_case_table() -> None: "`uv run python tools/differential/build_cjk_corpus.py`") +def test_tolerated_cjk_corpus_matches_the_case_table() -> None: + """The radar half of the same generated projection: the texts + whose case rows declare `tolerated`. Pinned for the same reason + as the contract half one function up -- one command writes both + files, so a row marked without regenerating leaves the demoted + name in NEITHER file and its diffs invisible at every baseline, + which is the failure the radar tier exists to prevent.""" + module = load_tool("build_cjk_corpus") + checked_in = [json.loads(line) for line in + (_TOOLS / "corpus_cjk_tolerated.jsonl") + .read_text(encoding="utf-8").splitlines()] + assert checked_in == module.tolerated_names(), ( + "corpus_cjk_tolerated.jsonl is stale: regenerate with " + "`uv run python tools/differential/build_cjk_corpus.py`") + + +def test_a_text_tolerated_on_one_row_only_is_a_hard_error( + monkeypatch: pytest.MonkeyPatch) -> None: + """The two pins above compare files to a selection; neither can + see the one input that has no right answer. A corpus carries name + STRINGS, so a text on two rows -- a default row and a + policy/locale fork of it, which several CJK texts have -- is one + line in one file, and marking one of those rows and not the other + declares both tiers for it. + + What makes the raise load-bearing rather than tidy is the + consequence with it removed, which the negative control below + measures instead of asserting in prose: the two halves select + `flags == {False}` and `flags == {True}`, so a split text matches + NEITHER and would be dropped from both files -- gone from the + harness, watched at no baseline, and invisible to every guard + here (the pins would agree with the degraded selection, and the + floors bound the loss to a few names). The message must name the + offending text for the same reason: an author who has to go + hunting for which row is split is an author who marks the other + one at random. + """ + from tests.v2.cases import Case + module = load_tool("build_cjk_corpus") + split = [ + Case(id="split_a", text="田中さん, PhD", expect={"family": "田中"}, + tolerated=True), + Case(id="split_b", text="田中さん, PhD", expect={"family": "田中"}), + Case(id="pure", text="김민준", expect={"family": "김"}), + ] + monkeypatch.setattr(module, "CASES", split) + with pytest.raises(SystemExit, match=r"'田中さん, PhD' on \['split_a', " + r"'split_b'\]"): + module._partition() + + # The negative control: the same selections the generator runs, + # with the raise conceptually deleted. The split text is in + # neither half -- which is the dropped-name failure the docstring + # above describes, reproduced rather than asserted from reading. + by_text: dict[str, set[bool]] = {} + for case in split: + by_text.setdefault(case.text, set()).add(case.tolerated) + contract = {t for t, f in by_text.items() if f == {False}} + tolerated = {t for t, f in by_text.items() if f == {True}} + assert "田中さん, PhD" not in contract | tolerated + assert contract | tolerated == {"김민준"} + + def test_shapes_corpus_matches_the_case_table() -> None: """corpus_shapes.jsonl is GENERATED from the shape-tagged case rows -- the same promise test_cjk_corpus_matches_the_case_table - makes one function up, for the tag predicate instead of the + makes above, for the tag predicate instead of the codepoint one: a row tagged without regenerating fails HERE instead of silently keeping the contract tier narrower than the table says it is.""" diff --git a/tools/differential/README.md b/tools/differential/README.md index 5350ab24..140ec7b9 100644 --- a/tools/differential/README.md +++ b/tools/differential/README.md @@ -194,7 +194,8 @@ that silently shrinks to nothing would otherwise report a green run. |---|---|---|---| | `corpus.jsonl` | v1's own test suite at a pinned ref | radar | anything 2.0 added — v1's authors had no reason to test a typographic nickname delimiter or a Cyrillic title | | `corpus_issues.jsonl` | name-like strings harvested from the GitHub issue tracker | radar | anything nobody ever reported | -| `corpus_cjk.jsonl` | the CJK-bearing rows of `tests/v2/cases.py`, via `build_cjk_corpus.py` (#295) | contract | anything the case table itself missed — it re-witnesses reviewed expectations at the baseline boundary rather than discovering new shapes | +| `corpus_cjk.jsonl` | the CJK-bearing rows of `tests/v2/cases.py` that do not declare `tolerated`, via `build_cjk_corpus.py` (#295) | contract | anything the case table itself missed — it re-witnesses reviewed expectations at the baseline boundary rather than discovering new shapes | +| `corpus_cjk_tolerated.jsonl` | the `tolerated` CJK rows of `tests/v2/cases.py`, via the same `build_cjk_corpus.py` run (2026-09-01) | radar | any composed or wrapped CJK form nobody wrote a case row for — it holds demoted names, and demotion presupposes admission | | `corpus_rules.jsonl` | every example in `docs/design/rules.md`, via `build_rules_corpus.py` (#414) | contract | anything the rules doc has no example for — it re-witnesses the normative examples at the baseline boundary | | `corpus_shapes.jsonl` | shape-tagged rows of `tests/v2/cases.py`, via `build_shapes_corpus.py` (#468) | contract | anything no one has tagged a row for | @@ -209,6 +210,19 @@ fail the run or demand a ledger rule. Nothing is deleted to keep the gate quiet -- a meaningless string in radar costs one parse and a report line. To promote a radar name, give it a tests/v2/cases.py row and a shape tag: it enters the contract by being chosen. A +name can also travel the other way, and one file records it: +`corpus_cjk_tolerated.jsonl` holds names that WERE chosen and were +then demoted, one reviewed row at a time, by the `tolerated` flag on +their case rows (the 2026-09-01 CJK demotion — composed and wrapped +CJK forms, read best-effort). Radar is what "we still watch it, we no +longer enforce it" costs: the diffs still classify and still group in +the release notes, the case rows still assert every one of these +parses in the suite, and clearing the flag promotes the name back. +What the flag moves is the file, which is not quite the same as the +tier: contract files load first and the dedup keeps the contract +reading, so a text another contract corpus also holds stays contract +until it leaves there too — five of these are `rules.md` examples in +`corpus_rules.jsonl` today. A `[[never]]` exclusion outranks the tier either way: it was chosen too -- someone wrote its `why` and its `examples` -- so a name it refuses stays UNEXPLAINED and fails the run even when the name itself sits in @@ -383,6 +397,22 @@ as new issues arrive. Over-collection is fine in both builders: the comparator just parses more names, and junk like `Bridge (1.4)` costs one parse and produces no diff. +`corpus_cjk.jsonl` and `corpus_cjk_tolerated.jsonl` are the two halves +of ONE projection of the case table, written by one run: + +``` +uv run python tools/differential/build_cjk_corpus.py +``` + +Every distinct CJK-bearing `text` in `tests/v2/cases.py` goes to the +first file, or to the second when its rows declare `tolerated`. The +flag is read per TEXT, not per row, because a corpus line is a name +string: a text carried by a default row and a policy/locale fork of it +is one line in one file. A text marked on one of its rows and not +another is a hard error rather than a silent choice — the generator +refuses the run, since either answer puts a name on a tier half its +rows deny. + ## The ledgers (`expected_since_.toml`) One ledger per baseline, so each release's classified changes stay as diff --git a/tools/differential/build_cjk_corpus.py b/tools/differential/build_cjk_corpus.py index 38b2c860..06a20bae 100644 --- a/tools/differential/build_cjk_corpus.py +++ b/tools/differential/build_cjk_corpus.py @@ -1,4 +1,4 @@ -"""Regenerate corpus_cjk.jsonl from the CJK rows of the case table. +"""Regenerate the two CJK corpora from the CJK rows of the case table. The third corpus, with the third provenance (#295): corpus.jsonl regenerates from v1's test banks at an immutable ref and @@ -19,13 +19,37 @@ the table everyone already reviews (#298's 间隔号 forms arrived exactly this way). +That harvest is SPLIT in two by the row's `tolerated` flag (the +2026-09-01 CJK demotion): an unmarked text goes to +corpus_cjk.jsonl, which stays a CONTRACT corpus, and a text marked +tolerated on its rows goes to corpus_cjk_tolerated.jsonl, a RADAR one +(compare.py's _CORPUS_TIERS). Both files are written by one run over +one sweep, so the two halves cannot drift apart or double-count a +text. Nothing about the harvest predicate changed: a composed or +wrapped CJK form -- a comma listing, a Latin title or credential +around a CJK name -- is still compared at every baseline and still +classified against the ledger. What the flag moves is which of THESE +TWO FILES a text is written to, and nothing else: a text that another +CONTRACT corpus also holds (corpus_rules.jsonl carries five of them +today, as rules.md examples) keeps the contract tier until it leaves +there too, because compare.py loads contract files first and its +(name, order) dedup keeps the contract reading. + +The flag is read PER TEXT, not per row: the corpora carry name +strings, so a text on two rows (a default row and a policy/locale +fork of it) is one line in one file. A text marked tolerated on one +of its rows and unmarked on another is therefore a hard error rather +than a silent choice: neither half would select it -- contract takes +the texts whose flags are all False, tolerated the ones all True -- +so the name would leave the harness entirely. + Regenerate after editing CJK case rows: uv run python tools/differential/build_cjk_corpus.py -tests/v2/test_ledger_guards.py pins the checked-in file against this -module's selection, so a stale corpus fails the suite rather than -silently narrowing the differential gate. +tests/v2/test_ledger_guards.py pins both checked-in files against +this module's selection, so a stale corpus fails the suite rather +than silently narrowing the differential gate. """ from __future__ import annotations @@ -41,22 +65,60 @@ from tests.v2.cases import CASES # noqa: E402 OUT = HERE / "corpus_cjk.jsonl" +OUT_TOLERATED = HERE / "corpus_cjk_tolerated.jsonl" _has_cjk = _script_matcher(*_SCRIPT_RANGES) +def _partition() -> tuple[list[str], list[str]]: + """(contract, tolerated) texts, each sorted for a deterministic + file. A text marked tolerated on ANY row while unmarked on + another is a hard error, and note WHICH failure the raise + prevents: the two selections below are `flags == {False}` and + `flags == {True}`, so a split text matches NEITHER and would + vanish from both files -- dropped from the harness, unwatched at + every baseline, rather than landing on one tier or the other.""" + by_text: dict[str, set[bool]] = {} + ids_by_text: dict[str, list[str]] = {} + for case in CASES: + if _has_cjk(case.text): + by_text.setdefault(case.text, set()).add(case.tolerated) + ids_by_text.setdefault(case.text, []).append(case.id) + split = sorted(text for text, flags in by_text.items() if len(flags) > 1) + if split: + rows = "; ".join( + f"{text!r} on {sorted(ids_by_text[text])}" for text in split) + raise SystemExit( + f"texts marked tolerated on one row and not another: {rows}. " + f"A split text is selected by neither half, so it would be " + f"dropped from both corpora and watched at no baseline. " + f"Resolve it deliberately: every row of the text marked " + f"demotes it to the radar file, every row clear keeps it " + f"in the contract one") + contract = sorted(t for t, flags in by_text.items() if flags == {False}) + tolerated = sorted(t for t, flags in by_text.items() if flags == {True}) + return contract, tolerated + + def selected_names() -> list[str]: - """Every distinct case-table text bearing a classified codepoint, - sorted for a deterministic file.""" - return sorted({case.text for case in CASES if _has_cjk(case.text)}) + """The contract half: every distinct case-table text bearing a + classified codepoint whose rows do not declare it tolerated.""" + return _partition()[0] + + +def tolerated_names() -> list[str]: + """The radar half: the same harvest, for texts whose rows declare + tolerated.""" + return _partition()[1] def main() -> None: - names = selected_names() - with OUT.open("w", encoding="utf-8") as fh: - for name in names: - fh.write(json.dumps(name, ensure_ascii=False) + "\n") - print(f"wrote {len(names)} names to {OUT.name}") + contract, tolerated = _partition() + for path, names in ((OUT, contract), (OUT_TOLERATED, tolerated)): + with path.open("w", encoding="utf-8") as fh: + for name in names: + fh.write(json.dumps(name, ensure_ascii=False) + "\n") + print(f"wrote {len(names)} names to {path.name}") if __name__ == "__main__": diff --git a/tools/differential/compare.py b/tools/differential/compare.py index acaee282..2864fc7b 100644 --- a/tools/differential/compare.py +++ b/tools/differential/compare.py @@ -503,7 +503,20 @@ def _legal_orders() -> frozenset[str]: #: decision when a corpus is added, the way the Script tables do. _CORPUS_FLOORS = { "corpus.jsonl": 480, # 486 today, from v1's banks at a pinned ref - "corpus_cjk.jsonl": 95, # 98 today, generated from the case table + "corpus_cjk.jsonl": 70, # 73 today, generated from the case table. + # LOWERED 95 -> 70 on 2026-09-01, + # deliberately: the CJK comma demotion + # moved 25 tolerated texts out of this + # file into corpus_cjk_tolerated.jsonl + # below. Nothing left the harness -- + # the names are compared and classified + # exactly as before, on the radar tier + "corpus_cjk_tolerated.jsonl": 22, # 25 today, the tolerated half of + # the same generator: composed and + # wrapped CJK forms (comma listings, + # Latin titles and credentials) whose + # handling the contract stopped + # promising on 2026-09-01 "corpus_issues.jsonl": 370, # 381 today, harvested and append-only "corpus_rules.jsonl": 150, # 252 today, generated from rules.md "corpus_shapes.jsonl": 35, # 37 today, generated from shape-tagged @@ -518,22 +531,45 @@ def _legal_orders() -> frozenset[str]: #: Tier per corpus file, fail-closed like the floors above. CONTRACT #: corpora hold names someone chose -- an unmatched diff on one is #: UNEXPLAINED and fails the run, today's discipline. RADAR corpora -#: hold scraped/harvested names (#468): their diffs still classify -#: against the ledger (release notes want the grouping) but an -#: unmatched one prints under UNCLASSIFIED (radar) and cannot fail -#: the run or demand a rule. Promotion is a cases.py row plus a -#: shape tag -- a name enters the contract by being chosen. +#: hold the names the contract does not answer for (#468) -- scraped +#: and harvested ones, and since 2026-09-01 the deliberately demoted +#: ones too: their diffs still classify against the ledger (release +#: notes want the grouping) but an unmatched one prints under +#: UNCLASSIFIED (radar) and cannot fail the run or demand a rule. +#: Promotion is a cases.py row plus a shape tag -- a name enters the +#: contract by being chosen -- or, for a demoted name that already +#: has rows, clearing `tolerated` on them (a shape tag is not +#: available to it: shapes 6/7 refuse composed and wrapped text, and +#: that refusal is why the name was demoted). #: #: A `[[never]]` exclusion is the same kind of choice, and stays fatal #: on a name in a radar file for exactly that reason: someone wrote #: the entry, its `why`, and its `examples`, so the shape it refuses -#: was chosen the same way a rule is -- unlike the rest of a radar -#: file, which nobody has looked at name by name. The tier split -#: governs names nobody chose; a [[never]] entry is the opposite of -#: that, so it outranks the tier the name happens to sit in. +#: was chosen the same way a rule is -- unlike most of a radar file, +#: which nobody has looked at name by name. The tier split governs +#: what the contract answers for; a [[never]] entry declares a shape +#: nobody may explain away, so it outranks the tier the name happens +#: to sit in. _CORPUS_TIERS = { "corpus.jsonl": "radar", "corpus_cjk.jsonl": "contract", + # The one radar file whose names WERE looked at one by one: the + # 2026-09-01 demotion moved them here by a reviewed flag on their + # case rows, not by scraping. The tier still fits, and for the + # reason the flag was written -- these are composed and wrapped + # CJK forms (a comma listing, a Latin title or credential around a + # CJK name) that native CJK writing does not contain, so the + # differential stops answering for them. Radar is what "we still + # watch it, we no longer enforce it" costs; the case rows still + # assert every one of these parses in the suite. + # + # This entry demotes the FILE, which is not the same as demoting + # every text in it: the dedup above loads contract files first and + # keeps the contract reading, so a text some contract corpus also + # holds reads contract no matter what this says. Five do today, + # as rules.md examples in corpus_rules.jsonl; the demotion of + # those texts is complete only when no contract corpus holds them. + "corpus_cjk_tolerated.jsonl": "radar", "corpus_issues.jsonl": "radar", "corpus_rules.jsonl": "contract", "corpus_shapes.jsonl": "contract", @@ -1553,9 +1589,10 @@ def _tree_parse(name: str, order: str | None) -> object: print(f"UNEXPLAINED {name!r}{_order_tag(order)}") _print_field_diffs(old_facade, new, old_v2, new_v2, order) if radar: - print("\nRadar tier (scraped/harvested names, #468): shown, " - "never blocking. Promote a name that matters via a " - "cases.py row + shape tag.\n") + print("\nRadar tier (names the contract does not answer for, " + "#468): shown, never blocking. Promote a name that " + "matters via a cases.py row + shape tag, or -- for a " + "demoted one -- by clearing `tolerated` on its rows.\n") for entry, (name, old_facade, new, old_v2, new_v2, order) in radar: labels = entry.get("tests") tag = f" [v1: {', '.join(labels)}]" if labels else "" diff --git a/tools/differential/corpus_cjk.jsonl b/tools/differential/corpus_cjk.jsonl index 55f8a00f..b45a00d7 100644 --- a/tools/differential/corpus_cjk.jsonl +++ b/tools/differential/corpus_cjk.jsonl @@ -1,8 +1,5 @@ "Andersonさん" "Anderson선생님" -"Dr 김민준, Jr." -"Dr 김민준씨, Jr." -"Dr 김민준씨, V." "John 王" "〆木 ひろ" "〆木 太郎" @@ -17,7 +14,6 @@ "司马相如" "夏侯惇" "威廉·莎士比亚" -"威廉·莎士比亚, PhD" "威廉·莎士比亚さん" "威廉・莎士比亚" "山田 エミ" @@ -33,7 +29,6 @@ "毛泽东" "王·Smith" "王先生" -"王先生, V." "王君" "王小明 先生" "王小明先生" @@ -41,18 +36,9 @@ "田中 さん" "田中 太郎 様" "田中 殿" -"田中, 太郎さん" "田中『ハナ』花子" "田中さん" -"田中さん II" -"田中さん V." "田中さん 様." -"田中さん, Dr." -"田中さん, Ph. D." -"田中さん, PhD" -"田中さん, V." -"田中さん, 太郎" -"田中さん, 様." "田中博士" "諸葛亮" "阿明" @@ -63,8 +49,6 @@ "高橋一郎" "鵜殿" "김 민준" -"김, 민준씨" -"김, 민준씨 (Jimmy)" "김민준" "김민준 님" "김민준 박사" @@ -74,25 +58,16 @@ "김민준 씨." "김민준 양" "김민준 양." -"김민준, 씨" -"김민준, 씨." "김민준님" "김민준박사님" "김민준씨" -"김민준씨 (Jimmy)" -"김민준씨 Jr." -"김민준씨, J.씨" -"김민준씨, V." "김선생님" "김씨" "김지양" "남궁" "남궁민수" -"남궁민수, 지훈" "마이클·잭슨" "선생님" -"선생님, J.씨" "씨" "양 미선" "양 지훈" -"이, J.씨" diff --git a/tools/differential/corpus_cjk_tolerated.jsonl b/tools/differential/corpus_cjk_tolerated.jsonl new file mode 100644 index 00000000..94f2e456 --- /dev/null +++ b/tools/differential/corpus_cjk_tolerated.jsonl @@ -0,0 +1,25 @@ +"Dr 김민준, Jr." +"Dr 김민준씨, Jr." +"Dr 김민준씨, V." +"威廉·莎士比亚, PhD" +"王先生, V." +"田中, 太郎さん" +"田中さん II" +"田中さん V." +"田中さん, Dr." +"田中さん, Ph. D." +"田中さん, PhD" +"田中さん, V." +"田中さん, 太郎" +"田中さん, 様." +"김, 민준씨" +"김, 민준씨 (Jimmy)" +"김민준, 씨" +"김민준, 씨." +"김민준씨 (Jimmy)" +"김민준씨 Jr." +"김민준씨, J.씨" +"김민준씨, V." +"남궁민수, 지훈" +"선생님, J.씨" +"이, J.씨" From f046d5df26a2ba8ffd1874b90372c025858e4ab1 Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Tue, 1 Sep 2026 18:02:52 -0700 Subject: [PATCH 4/7] docs(design): W3 demotes to a tolerated note, and the rules corpus stops enforcing it MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit rules.md gains a rule class: a `tolerated: ` line takes a rule out of the document's normative reading. The rule keeps its ID, statement, precedence, examples and `implemented:` pointer -- the examples still parse and test_rules_doc.py still executes them -- but build_rules_corpus.py harvests nothing from a marked rule, so its names stop being enforced against released baselines. The preamble states the two obligations the mechanism does NOT discharge on its own: a clause reaching past the tolerated shape belongs in a normative rule, and keeping a demoted name WATCHED is a separate act. W3 takes the marker. Native CJK writing has no family-comma convention, so every input W3 reads is a listing form wrapped around a name its own script has already arranged; the statement now describes that behavior rather than promising it, and absorbs the "crosses a family comma" clause out of W2's normative statement. W2 keeps its rule status and swaps its two comma examples for pure ones that pin the same peel: "김, 민준씨" -> suffix="씨" becomes "김민준씨" -> suffix="씨" "田中さん, V." -> suffix="さん" becomes "田中さん 様." -> suffix="さん, 様." Both replacements were already in corpus_cjk.jsonl, so the swap adds no name to the pool. C1's second Accepted clause was to be WIDENED to a script-independent statement carried by Latin examples. Measurement refused it: over seven pre-comma texts, the fate of the part before the comma differs between a title-only part and a credential part for exactly the two glued-honorific ones (田中さん, 김민준씨) and is invariant for every Latin name tried, so a Latin example cannot witness the fork and the clause would have shipped with inert witnesses. The clause and its two examples MOVED to W3 instead -- the honest home, since what they state is precedence over a tolerated composition -- and C1 keeps a pointer saying why it carries no example there. W3 states the fork on the SHAPE of the part after the comma, which is the criterion decisions.md#W2 (#319) already records. A first draft attributed it to which of C1's two readings that part selects, and the reviewer's second round measured that false twice over: `田中さん, Dr.` and `田中さん, PhD` both read FAMILY_COMMA (C1's credential reading also wants more than one pre-comma word, and 田中さん is one), so the reading is not the discriminator; and where the credential reading IS selected -- `田中さん 太郎, PhD`, a suffix comma -- the honorific is not freed at all. What decides is whether the part after the comma is nothing but suffix words: such a part is declined as the name's end, so the split-off falls back to the part before the comma and takes the honorific there, provided that part offers a site of its own; a title or a name word after the comma IS the name's end and the honorific stays glued. It is C1's own vocabulary question without C1's word-count condition -- #319 lifted the predicate so the two stages could share vocabulary without becoming the same test. The example pair witnesses it exactly, so no example changed. Pointer lines: W3 gains `interacts: W1, W2, C1`, W2 `interacts: W3`, C1 `W3` alongside H2 and P6, and W1 `interacts: W3` beside the decisions.md#W3 citation its relocated clause carries. Two clauses moved for scope while the marker was being placed. W3's segmenter sentence ("consulted only for a name whose written form is wholly undivided, a spaced honorific counting as a written division") is about comma-free native input, which the demotion had no business sweeping in, so it moved to W1 with decisions.md#W3 cited inline for its provenance. The W Background gains the writing-system fact the demotion rests on. The five-texts caveat is discharged. corpus_cjk_tolerated.jsonl landed holding five texts that corpus_rules.jsonl also held, so compare.py's contract-first dedup kept them reading CONTRACT and their demotion was incomplete. All five left corpus_rules.jsonl here, so every text in the radar file now reads radar; the three sites that carried the caveat (build_cjk_corpus.py, compare.py's _CORPUS_TIERS, the README) now state the standing rule instead of the exception. `지훈, 남궁민수` was W3's doc-only example and lived in no other corpus, so the skip alone would have dropped it from the harness rather than demoting it. It gets a tolerated case row with measured expectations (family 지훈, given 남궁민수; 1.4.0 gives the same fields, so parity), which puts it on the radar tier and pins it at HEAD. Counts, all recomputable by regenerating and running the gate: corpus_rules.jsonl 252 -> 248 (six comma texts out, two pure ones in), corpus_cjk_tolerated.jsonl 25 -> 26. The pool the claims record sweeps is unchanged -- every departing text has a home in another corpus file -- so _CORPUS_CLAIMS does not move. Floors left where they are (150 and 22), with the "N today" comments updated and the demotion cited. Gate at 1.4.0/2.0.0/2.1.0/2.2.0: intentional 226/205/113/0, unexplained 0, radar unclassified 0, 1113/1120/1120/1120 compared -- byte-identical to the pre-change reference apart from the `corpora:` line. The two cjk-comma ledger rules still explain 11 and 7 diffs from the radar tier; nothing went dormant. New guards: the grammar reads the marker (with a negative control that a normative rule has none), and test_ledger_guards.py asserts a tolerated rule's own example texts are absent from corpus_rules.jsonl -- the regeneration equality would pass with the skip deleted. Reviewed by the design-docs-reviewer over three rounds. Its first pass measured C1's Latin clause false and found the segmenter clause swept in; its second measured the fork's stated CAUSE false, over five probes read at ParseState.structure. Both corrections are in this commit. Co-Authored-By: Claude Fable 5 --- docs/design/rules.md | 71 +++++++++++++------ nameparser/_pipeline/_script_segment.py | 10 ++- tests/v2/cases.py | 17 +++++ tests/v2/rules_doc.py | 21 ++++++ tests/v2/test_ledger_guards.py | 44 ++++++++++++ tests/v2/test_rules_doc_grammar.py | 35 +++++++++ tools/differential/README.md | 11 ++- tools/differential/build_cjk_corpus.py | 13 +++- tools/differential/build_rules_corpus.py | 25 ++++++- tools/differential/compare.py | 38 ++++++++-- tools/differential/corpus_cjk_tolerated.jsonl | 1 + tools/differential/corpus_rules.jsonl | 8 +-- 12 files changed, 251 insertions(+), 43 deletions(-) diff --git a/docs/design/rules.md b/docs/design/rules.md index 2ebfbe46..98177b67 100644 --- a/docs/design/rules.md +++ b/docs/design/rules.md @@ -13,6 +13,10 @@ Every example line is EXECUTABLE. The grammar (its executable definition is `tes An `annotation` names a policy, locale (`[ru]`), or extras gate (`[ja+segmenter]`) in the registry beside the test. `· boundary` marks the non-firing example every rule must carry: an input shaped like the rule's subject where its effect does NOT occur. That is usually the rule's OWN stated exception — H1's given-name title, P3's single-letter carve-out — not an input the rule never reaches, so the exception is executable rather than merely asserted. Or the rule declares `no-boundary: ` instead, so skipping the boundary is a recorded decision. `deviates:` states the INTENDED output on the example line while the marker records TODAY's output and the tracking issue; the runner asserts today's output strictly, so a parser change that closes the gap fails the suite until the marker is removed in the same PR. `grep deviates:` on this file is the deviation backlog (deviations from statable rules — coverage gaps are a separate, larger category no grep can see, and contested vocabulary memberships a third, tracked as Open blocks keyed to the vocabulary set in decisions.md). +One class of rule stands outside the normative reading and declares it on a `tolerated: ` line beside its pointer line. Such a rule is ABOUT an input the writing systems it speaks about do not produce — a shape no convention writes, read best-effort because parsing is total over strings — and it describes what the parser does with that input instead of promising it. Its statement is still precedence-bearing and its examples are still executable and still run, but they illustrate rather than promise, and the behavior is changeable without notice. W3 is the only such rule today (the 2026-09-01 comma demotion). + +The marker's unit is the whole rule, because that is the unit `tools/differential/build_rules_corpus.py` skips: it harvests no example from a marked rule, so none of them is enforced against released baselines. Two obligations follow, and neither is automatic. A clause that reaches past the tolerated shape does not belong in a marked rule — say it in a normative one. And the skip only STOPS enforcing a name; keeping it watched is a separate act, discharged here by giving each demoted example a `tolerated` row in `tests/v2/cases.py`, which projects it onto the differential's radar tier. + ## Not in scope - **Language detection.** The parser never infers a language from Latin-script text: transliteration destroys the signal ("Ali", @@ -866,18 +870,22 @@ C1. Rationale: a credential run after the comma means the name is in duals, and the slot is postnominal. "Smith, Ms." → suffix="Ms." "Smith, Ms. Jane" → title="Ms." - Accepted: a title-only part after a one-word family keeps the - family whole, and a glued honorific in it stays glued — the - honorific peel (W3) runs on the other structures, before the - comma is read; a credential after the comma still frees it. - "田中さん, Dr." → family="田中さん" - "田中さん, PhD" → suffix="さん, PhD" + Accepted: the vocabulary question above — is the part after the + comma nothing but suffix words — is asked a second time, without + the word-count condition, to decide whether a glued East Asian + honorific standing before a family comma comes off. That + composition is tolerated input rather than contract — a comma + between a family name and a given name is no part of CJK writing + — so the precedence is stated at W3, with its examples, and not + here. It is also the one consequence of this question Latin + script cannot witness, nothing there being glued to the end of a + name — which is why this clause carries no example of its own. Accepted: a delimiter core the policy names (T1) is a word here, not structure — v1 applied the delimiter to the suffix-comma form alone, and that limitation is kept as parity: "Smith, RN - CRNA" reads given "RN" under the policy as without it. "John Smith, LEED AP" → family="Smith" deviates: #291 (today: family="John Smith") - history: decisions.md#C1 · interacts: H2, P6 · implemented: nameparser/_pipeline/_segment.py, nameparser/_pipeline/_assign.py, nameparser/_pipeline/_group.py + history: decisions.md#C1 · interacts: H2, P6, W3 · implemented: nameparser/_pipeline/_segment.py, nameparser/_pipeline/_assign.py, nameparser/_pipeline/_group.py C2. Rationale: text beyond the recognized comma parts should be taken in without silent guessing. @@ -986,7 +994,7 @@ O5. Rationale: O4 reads a name by comparing where its words stand, ## Scripts & writing systems (W) -Background: script-conditional behavior is permitted exactly where the writing system itself — not statistics about it — settles the convention; a language can never be inferred from Latin-script text, because transliteration destroys the signal. The facts this section builds on: Chinese and Japanese both write the family name first in native script, so the script settles the order without knowing the language. Hangul is written by exactly one language and Korean family names are a small closed census set. Han text does not identify its language — a Chinese surname list would divide Japanese 高橋一郎 as 高 + 橋一郎 — which is why Han division is opt-in and there is no Korean pack to opt into. Hiragana never transcribes a foreign name (transcriptions are katakana alone), so kanji-plus-kana is a Japanese name in Japanese order, while wholly-katakana is predominantly a transcribed foreign name already in given-first order. Real Chinese text is unspaced (毛泽东); the spaced 毛 泽东 is an artifact. A fuller narrative lives in docs/usage.rst's East Asian section. +Background: script-conditional behavior is permitted exactly where the writing system itself — not statistics about it — settles the convention; a language can never be inferred from Latin-script text, because transliteration destroys the signal. The facts this section builds on: Chinese and Japanese both write the family name first in native script, so the script settles the order without knowing the language. Hangul is written by exactly one language and Korean family names are a small closed census set. Han text does not identify its language — a Chinese surname list would divide Japanese 高橋一郎 as 高 + 橋一郎 — which is why Han division is opt-in and there is no Korean pack to opt into. Hiragana never transcribes a foreign name (transcriptions are katakana alone), so kanji-plus-kana is a Japanese name in Japanese order, while wholly-katakana is predominantly a transcribed foreign name already in given-first order. Real Chinese text is unspaced (毛泽东); the spaced 毛 泽东 is an artifact. A fuller narrative lives in docs/usage.rst's East Asian section. One fact carries its own consequence: none of the three writing systems marks the family name with a comma — position in the written form is what identifies it, so a comma standing between the family name and the given name is a listing convention carried in from elsewhere rather than a form the script produces. That is why the rule reading one (W3) is tolerated rather than normative. W1. Rationale: hangul is monoglot Korean and its surnames are a closed census set, so an unspaced hangul name divides at a @@ -997,7 +1005,11 @@ W1. Rationale: hangul is monoglot Korean and its surnames are a recognizes nothing, an optional segmenter may divide instead, and with neither the word stays whole rather than divide in a wrong place. Korean division is active by default; Han division - is opt-in. + is opt-in. A segmenter — unlike the vocabulary, which ignores + spacing but not the interpunct (the Accepted below) — is + consulted only for a name whose written form is wholly + undivided, a spaced honorific counting as a written division + (decisions.md#W3 records what that trade keeps and costs). "김민준" → family="김" "남궁민수" → family="남궁" "남궁민수 지훈" → family="남궁" @@ -1008,7 +1020,7 @@ W1. Rationale: hangul is monoglot Korean and its surnames are a the sense that matters — division stands down entirely there, vocabulary and segmenter alike (#298; decisions.md#T3). "安东尼·陈志明" [zh] → family="陈志明" - history: decisions.md#W1 · implemented: nameparser/_pipeline/_script_segment.py + history: decisions.md#W1 · interacts: W3 · implemented: nameparser/_pipeline/_script_segment.py W2. Rationale: some East Asian honorifics glue directly onto the end of the name (田中さん); a glued word peels off only if it could @@ -1016,31 +1028,48 @@ W2. Rationale: some East Asian honorifics glue directly onto the end own license and needs no other gate. A listed honorific glued to the end of the name's last name word splits off once and reads as a suffix. The split-off - crosses a family comma and ignores surrounding punctuation, but - never treats a part that is not name text as the name's end. + ignores surrounding punctuation, but never treats a part that is + not name text as the name's end. "田中さん" → suffix="さん" - "김, 민준씨" → suffix="씨" - "田中さん, V." → suffix="さん" + "김민준씨" → suffix="씨" + "田中さん 様." → suffix="さん, 様." "马丁·路德·金씨" → suffix="씨" "김지양" → suffix="" · boundary "王君" → family="王君" · boundary - history: decisions.md#W2 · implemented: nameparser/_pipeline/_script_segment.py + history: decisions.md#W2 · interacts: W3 · implemented: nameparser/_pipeline/_script_segment.py W3. Rationale: a family name declared by a comma is the writer's own division, and re-dividing it would invent a boundary nobody - drew. + drew — but none of the East Asian writing systems declares a + family name that way (the Background above), so every input this + rule reads is a listing convention wrapped around a name whose + own script has already arranged it. What follows describes what + the parser does with such input; it does not promise it. Under a family comma the pre-comma text is the family by declaration and never divides, and the post-comma side is given text with no family to find; only the honorific split-off (W2) crosses the comma, an honorific being no part of the name on - either side. A segmenter — unlike the vocabulary, which ignores - spacing but not the interpunct (W1's Accepted) — is consulted - only for a name whose written form is wholly undivided, a - spaced honorific counting as a written division. + either side — the crossing is stated here rather than in W2 + because it is a claim about the comma, and the comma is the part + nobody's writing system produces. Which side the split-off takes + it from is decided by the SHAPE of the part after the comma, not + by which of C1's two readings that part selects — a one-word + family reads the listing form either way. A part that is nothing + but suffix words is not the name's end: it is declined as the + site, and the split-off falls back to the part before the comma + and takes the honorific there, provided that part offers a site + of its own. Any other part — a title, a name word — IS the + name's end, and a glued honorific before the comma stays glued. + The vocabulary question is C1's own, asked without C1's + word-count condition: the two differ in what else they require, + not in what they ask of the words. "남궁민수" → family="남궁" "지훈, 남궁민수" → given="남궁민수" "남궁민수, 지훈" → family="남궁민수" · boundary - history: decisions.md#W3 · implemented: nameparser/_pipeline/_script_segment.py + "田中さん, Dr." → family="田中さん" + "田中さん, PhD" → suffix="さん, PhD" + tolerated: native CJK writing has no family-comma convention, so the four comma lines above illustrate current behavior — changeable without notice — rather than promise it; the comma-free line beside them is W1's claim, which is normative. All four comma names stay watched at every released baseline on the differential's radar tier (tools/differential/corpus_cjk_tolerated.jsonl, projected from the `tolerated` rows of tests/v2/cases.py) instead of its contract tier, and those rows pin them at HEAD. + history: decisions.md#W3 · interacts: W1, W2, C1 · implemented: nameparser/_pipeline/_script_segment.py W4. Rationale: Chinese, Japanese and Korean all write the family name first in native script — the script settles the order diff --git a/nameparser/_pipeline/_script_segment.py b/nameparser/_pipeline/_script_segment.py index 2ceeb1c3..a28e6f03 100644 --- a/nameparser/_pipeline/_script_segment.py +++ b/nameparser/_pipeline/_script_segment.py @@ -273,9 +273,13 @@ def _peel_site(state: ParseState, flat: Sequence[int], # rules.md#W2: "a listed honorific glued to the end of the name's # last name word splits off once and reads as a suffix. The -# split-off crosses a family comma and ignores surrounding -# punctuation, but never treats a part that is not name text as the -# name's end." (history: decisions.md#W2) +# split-off ignores surrounding punctuation, but never treats a +# part that is not name text as the name's end." (history: +# decisions.md#W2) +# That the peel also reaches ACROSS a family comma is stated at +# rules.md#W3 instead, which is a tolerated rule since the +# 2026-09-01 comma demotion -- the crossing is what the parser does +# today, not something W2 promises. def _peel_honorific_tail(state: ParseState) -> ParseState: """#308: split a listed honorific off the END of the name's last NON-POST-NOMINAL token -- 田中さん -> 田中 + さん -- and let diff --git a/tests/v2/cases.py b/tests/v2/cases.py index df7d70a3..89ddc68d 100644 --- a/tests/v2/cases.py +++ b/tests/v2/cases.py @@ -2946,6 +2946,23 @@ def _check_cjk_shape_purity(self) -> None: "the script_segment stage docstring, which uses this " "exact example)", tolerated=True), + Case("ko_family_comma_given_side_stays_whole", "지훈, 남궁민수", + {"family": "지훈", "given": "남궁민수"}, + notes="the mirror of the row above, and one of " + "rules.md#W3's comma illustrations: with the " + "comma naming 지훈 the family, the post-comma side is " + "given text with no family to find, so 남궁민수 is " + "not segmented THERE either -- the same inertness " + "reached from the other side. Added 2026-09-01 with " + "the W3 demotion: the rules corpus stopped harvesting " + "W3's examples, and this text lived in no other " + "corpus, so without a row it would have left the " + "differential harness entirely rather than moving to " + "the radar tier. Classification measured for this row " + "rather than copied from the row above: 1.4.0 gives " + "first 남궁민수 / last 지훈, field for field what 2.3 " + "gives, so parity", + tolerated=True), Case("ko_suffix_comma_name_part_splits", "Dr 김민준, Jr.", {"title": "Dr", "family": "김", "given": "민준", "suffix": "Jr."}, classification="fix(#271)", diff --git a/tests/v2/rules_doc.py b/tests/v2/rules_doc.py index 1b80b2a0..acf78e5d 100644 --- a/tests/v2/rules_doc.py +++ b/tests/v2/rules_doc.py @@ -13,6 +13,17 @@ two is required, so a normative rule always points either at the code that honors it or at the work that will. +A rule may also carry ``tolerated: ``, which takes it out of +the document's normative reading: its statement DESCRIBES what the +parser does with input the writing systems it speaks about do not +produce. The line changes nothing here — the examples parse, and +test_rules_doc.py still executes them — but +tools/differential/build_rules_corpus.py skips a rule carrying it, so +a tolerated rule's examples illustrate current behavior without +entering the contract corpus that enforces it at released baselines. +The reason is free text and unvalidated; what is machine-read is that +the line is present. + Inside a rule block, any line whose first non-space character is a double quote (or an opening bracket, the D-section subject form) is an example line and MUST parse — a silent skip would un-execute a claim, @@ -39,6 +50,7 @@ rf"(?P[a-z_]+)=(?P{_VALUE})\))?" r"\s*$") _NO_BOUNDARY_RE = re.compile(r"^\s*no-boundary:\s+(?P\S.*)$") +_TOLERATED_RE = re.compile(r"^\s*tolerated:\s+(?P\S.*)$") _POINTER_RE = re.compile(r"^\s*(history|interacts|implemented|tracked):") _POINTER_PART_RE = re.compile( r"(history|interacts|implemented|tracked):\s*([^·]+)") @@ -73,6 +85,11 @@ class Rule: #: exclusive with ``implemented:`` -- a rule points at code or at #: the issues that will produce it, never at neither. tracked: tuple[str, ...] = () + #: The ``tolerated:`` line's reason, or None for a normative rule. + #: Read by build_rules_corpus.py, which harvests no example from a + #: rule carrying it -- a tolerated rule's examples illustrate what + #: the parser does today, they do not promise it. + tolerated: str | None = None def has_boundary_or_waiver(self) -> bool: return self.no_boundary is not None or any( @@ -188,6 +205,10 @@ def parse_rules_doc(text: str) -> list[Rule]: if nb: current.no_boundary = nb.group("reason") continue + tol = _TOLERATED_RE.match(line) + if tol: + current.tolerated = tol.group("reason") + continue if _POINTER_RE.match(line): for key, val in _POINTER_PART_RE.findall(line): items = tuple(v.strip() for v in val.split(",") if v.strip()) diff --git a/tests/v2/test_ledger_guards.py b/tests/v2/test_ledger_guards.py index a3192531..5d72a470 100644 --- a/tests/v2/test_ledger_guards.py +++ b/tests/v2/test_ledger_guards.py @@ -932,6 +932,50 @@ def test_rules_corpus_matches_the_rules_doc() -> None: f"that never had any") +def test_a_tolerated_rule_puts_no_name_in_the_rules_corpus() -> None: + """A rule marked `tolerated:` in rules.md contributes no example + to the CONTRACT corpus (2026-09-01, the CJK comma demotion). + + The equality above would pass with the skip deleted -- regenerate + and the file agrees with whatever the generator now selects. This + reads the doc's marker directly and asks the file. Scoped to the + marked rule's OWN texts: an example some normative rule also + carries is in the corpus on that rule's account, and demoting one + rule cannot take another's name away, so the check subtracts them + rather than failing on them. + + What it does NOT check, deliberately: that the demoted names are + still watched somewhere. That is a different promise, and it is + the tolerated case rows plus corpus_cjk_tolerated.jsonl that keep + it -- pinned by test_tolerated_cjk_corpus_matches_the_case_table + above, and measured by the gate's own claims record. + """ + from .rules_doc import parse_rules_doc + rules = parse_rules_doc( + (_TOOLS.parents[1] / "docs" / "design" / "rules.md") + .read_text(encoding="utf-8")) + tolerated = [r for r in rules if r.tolerated is not None] + assert tolerated, ( + "no rule in rules.md carries a `tolerated:` marker; W3 has " + "carried one since 2026-09-01. If a demotion was reversed, " + "delete this guard in that commit rather than leaving it " + "asserting nothing") + normative_texts = {e.text for r in rules if r.tolerated is None + for e in r.examples if e.text} + in_corpus = {json.loads(line) for line in + (_TOOLS / "corpus_rules.jsonl") + .read_text(encoding="utf-8").splitlines()} + leaked = sorted({e.text for r in tolerated for e in r.examples + if e.text and e.text not in normative_texts} + & in_corpus) + assert not leaked, ( + f"corpus_rules.jsonl carries example texts belonging only to " + f"tolerated rules {[r.rule_id for r in tolerated]}: {leaked}. " + f"A tolerated rule's examples illustrate current behavior; " + f"enforcing them against released baselines is the promise " + f"the marker withdrew") + + #: Which vocabulary constant each ledger rule's alternation is a hand #: copy of. A roster rather than an inference: GLUED_HONORIFICS is a #: SUBSET of SUFFIX_WORDS (asserted at the bottom of diff --git a/tests/v2/test_rules_doc_grammar.py b/tests/v2/test_rules_doc_grammar.py index 2e34491c..a348b14d 100644 --- a/tests/v2/test_rules_doc_grammar.py +++ b/tests/v2/test_rules_doc_grammar.py @@ -83,6 +83,41 @@ def test_nested_pieces_value_parses() -> None: assert ex.value == [["de"], ["Mesnil"]] +def test_tolerated_marker_is_read_off_a_rule() -> None: + """The 2026-09-01 comma demotion's vehicle, at the grammar layer. + + A rule may declare its examples ILLUSTRATIVE with a + ``tolerated: `` line. The grammar's whole job is to make + the line visible: the examples still parse and are still executed + by test_rules_doc.py, and it is build_rules_corpus.py that acts + on the flag by harvesting nothing from a marked rule. The other + half of that promise -- that a tolerated rule really contributes + no name to corpus_rules.jsonl -- is + test_a_tolerated_rule_puts_no_name_in_the_rules_corpus in + tests/v2/test_ledger_guards.py, which can see the corpus file. + + The fixture is synthetic, like DOC above, and stays local to this + test so the shared fixture keeps describing a normative doc. + """ + doc = ('X1. Statement the parser only describes.\n' + ' "Smith, John" → family="Smith"\n' + ' tolerated: nobody writes this shape; current behavior only.\n' + ' no-boundary: illustrative, not normative.\n') + rule = parse_rules_doc(doc)[0] + assert rule.tolerated == ( + "nobody writes this shape; current behavior only.") + assert len(rule.examples) == 1, ( + "a tolerated rule keeps its examples -- the marker demotes the " + "claim, it does not un-write the lines") + + +def test_a_normative_rule_carries_no_tolerated_marker() -> None: + """The negative control for the test above: without the line the + field is None, so `rule.tolerated is None` is a real question and + not a constant the builder's skip could never fail on.""" + assert all(r.tolerated is None for r in parse_rules_doc(DOC)) + + def test_registry_resolves_policies_and_gates() -> None: from tests.v2.rules_doc import resolve_annotation kind, obj = resolve_annotation("family-first") diff --git a/tools/differential/README.md b/tools/differential/README.md index 140ec7b9..53e557b2 100644 --- a/tools/differential/README.md +++ b/tools/differential/README.md @@ -196,7 +196,7 @@ that silently shrinks to nothing would otherwise report a green run. | `corpus_issues.jsonl` | name-like strings harvested from the GitHub issue tracker | radar | anything nobody ever reported | | `corpus_cjk.jsonl` | the CJK-bearing rows of `tests/v2/cases.py` that do not declare `tolerated`, via `build_cjk_corpus.py` (#295) | contract | anything the case table itself missed — it re-witnesses reviewed expectations at the baseline boundary rather than discovering new shapes | | `corpus_cjk_tolerated.jsonl` | the `tolerated` CJK rows of `tests/v2/cases.py`, via the same `build_cjk_corpus.py` run (2026-09-01) | radar | any composed or wrapped CJK form nobody wrote a case row for — it holds demoted names, and demotion presupposes admission | -| `corpus_rules.jsonl` | every example in `docs/design/rules.md`, via `build_rules_corpus.py` (#414) | contract | anything the rules doc has no example for — it re-witnesses the normative examples at the baseline boundary | +| `corpus_rules.jsonl` | every example of every NORMATIVE rule in `docs/design/rules.md`, via `build_rules_corpus.py` (#414) — a rule carrying that doc's `tolerated:` marker is skipped whole, its examples illustrating rather than promising | contract | anything the rules doc has no example for, and anything a tolerated rule's examples are the only witness of | | `corpus_shapes.jsonl` | shape-tagged rows of `tests/v2/cases.py`, via `build_shapes_corpus.py` (#468) | contract | anything no one has tagged a row for | Since the v2.3 tier split (#468), a corpus is CONTRACT or RADAR -- @@ -221,8 +221,13 @@ parses in the suite, and clearing the flag promotes the name back. What the flag moves is the file, which is not quite the same as the tier: contract files load first and the dedup keeps the contract reading, so a text another contract corpus also holds stays contract -until it leaves there too — five of these are `rules.md` examples in -`corpus_rules.jsonl` today. A +until it leaves there too. Five of these were `rules.md` examples in +`corpus_rules.jsonl` when the file was created, and the same day's +rules.md edits took all five out of it — W3 marked `tolerated:`, +W2's two comma examples swapped for pure ones, C1's two moved into +W3 — so every text in the file reads radar today, and the sentence +above is the standing rule the next demotion has to satisfy rather +than a note about those five. A `[[never]]` exclusion outranks the tier either way: it was chosen too -- someone wrote its `why` and its `examples` -- so a name it refuses stays UNEXPLAINED and fails the run even when the name itself sits in diff --git a/tools/differential/build_cjk_corpus.py b/tools/differential/build_cjk_corpus.py index 06a20bae..b28511db 100644 --- a/tools/differential/build_cjk_corpus.py +++ b/tools/differential/build_cjk_corpus.py @@ -30,10 +30,17 @@ around a CJK name -- is still compared at every baseline and still classified against the ledger. What the flag moves is which of THESE TWO FILES a text is written to, and nothing else: a text that another -CONTRACT corpus also holds (corpus_rules.jsonl carries five of them -today, as rules.md examples) keeps the contract tier until it leaves +CONTRACT corpus also holds keeps the contract tier until it leaves there too, because compare.py loads contract files first and its -(name, order) dedup keeps the contract reading. +(name, order) dedup keeps the contract reading. That was live when +this split landed -- corpus_rules.jsonl carried five of these texts +as rules.md examples -- and the rules.md edits later the same day +removed all five: W3 marked tolerated, W2's two comma examples +swapped for pure ones, C1's two moved into W3. No text written here +is held contract anywhere today. The sentence stays because it is the +mechanism, not a note about those five: a demotion is complete only +when no contract corpus holds the text, and the next demotion has to +check that for itself. The flag is read PER TEXT, not per row: the corpora carry name strings, so a text on two rows (a default row and a policy/locale diff --git a/tools/differential/build_rules_corpus.py b/tools/differential/build_rules_corpus.py index 62006c00..2e4eaf6a 100644 --- a/tools/differential/build_rules_corpus.py +++ b/tools/differential/build_rules_corpus.py @@ -40,6 +40,27 @@ so a family-first-scoped example is simply one more name to diff -- build_cjk_corpus.py makes the same call for its zh-scoped rows. +A TOLERATED rule is skipped whole. A rule carrying rules.md's +`tolerated: ` line has stepped out of the document's +normative reading: its statement describes what the parser does with +input the writing systems it speaks about do not produce, and it says +so on that line. A tolerated rule's examples ILLUSTRATE, they do not +promise -- so they stay in the doc, stay executed by +test_rules_doc.py, and stay pinned at HEAD, while this corpus (which +is what enforces a name against RELEASED baselines, in a contract +tier where an unclassified diff fails the run) stops carrying them. +Skipping the rule rather than filtering its examples is the point: +the unit the doc marks is a rule, so a tolerated rule gaining an +example does not quietly re-enter the contract. + +That is the whole of the demotion's mechanism here, and it leaves the +name watched: rules.md#W3's subjects are also tolerated rows in +tests/v2/cases.py, so build_cjk_corpus.py writes them to the +radar-tier corpus_cjk_tolerated.jsonl and every baseline still parses +and classifies them. A tolerated rule whose examples land in NO +corpus would be a name dropped from the harness, which is a different +act and not one this skip performs on its own. + The one example form that carries no name is skipped. A `[subject]` example names a policy or locale rather than a name string, so its `text` is empty -- rules.md has three, and without the filter all @@ -81,10 +102,12 @@ def selected_names() -> list[str]: """Every distinct example text in the rules doc, sorted for a - deterministic file.""" + deterministic file. Rules marked `tolerated:` contribute nothing: + their examples illustrate, they do not promise.""" rules = parse_rules_doc(RULES_DOC.read_text(encoding="utf-8")) return sorted({example.text for rule in rules + if rule.tolerated is None for example in rule.examples if example.text}) diff --git a/tools/differential/compare.py b/tools/differential/compare.py index 2864fc7b..4e86872e 100644 --- a/tools/differential/compare.py +++ b/tools/differential/compare.py @@ -511,14 +511,33 @@ def _legal_orders() -> frozenset[str]: # below. Nothing left the harness -- # the names are compared and classified # exactly as before, on the radar tier - "corpus_cjk_tolerated.jsonl": 22, # 25 today, the tolerated half of + "corpus_cjk_tolerated.jsonl": 22, # 26 today, the tolerated half of # the same generator: composed and # wrapped CJK forms (comma listings, # Latin titles and credentials) whose # handling the contract stopped - # promising on 2026-09-01 + # promising on 2026-09-01. 25 on the + # day it was created; the 26th is + # '지훈, 남궁민수', which had no case + # row until rules.md#W3 was demoted + # and the rules corpus stopped + # carrying it -- the row was written + # so the text moved tiers instead of + # leaving the harness "corpus_issues.jsonl": 370, # 381 today, harvested and append-only - "corpus_rules.jsonl": 150, # 252 today, generated from rules.md + "corpus_rules.jsonl": 150, # 248 today, generated from rules.md. + # 252 until 2026-09-01, when W3 took + # rules.md's `tolerated:` marker and + # build_rules_corpus.py stopped + # harvesting a marked rule: six comma + # texts left (W3's two, W2's two, C1's + # two) and two pure ones arrived with + # the W2 swap. Every one of the six is + # still compared and classified, from + # corpus_cjk_tolerated.jsonl above. + # Floor left at 150: it guards against + # the file emptying, and a demotion + # this size is nowhere near it "corpus_shapes.jsonl": 35, # 37 today, generated from shape-tagged # case rows. Ratcheted 27 -> 35 on # 2026-09-01 with the shape 6/7 @@ -566,9 +585,16 @@ def _legal_orders() -> frozenset[str]: # This entry demotes the FILE, which is not the same as demoting # every text in it: the dedup above loads contract files first and # keeps the contract reading, so a text some contract corpus also - # holds reads contract no matter what this says. Five do today, - # as rules.md examples in corpus_rules.jsonl; the demotion of - # those texts is complete only when no contract corpus holds them. + # holds reads contract no matter what this says. Five did on the + # day this file was created -- rules.md examples, so + # corpus_rules.jsonl held them too and they went on reading + # contract. The rules.md edits later the same day took those five + # out of corpus_rules.jsonl -- W3 marked tolerated, W2's two comma + # examples swapped for pure ones, C1's two moved into W3 -- so + # NONE do today and every text in this file reads radar. The rule + # is stated as a rule, not as a caveat about the five: whatever + # lands here next is demoted only once no contract corpus holds + # it. "corpus_cjk_tolerated.jsonl": "radar", "corpus_issues.jsonl": "radar", "corpus_rules.jsonl": "contract", diff --git a/tools/differential/corpus_cjk_tolerated.jsonl b/tools/differential/corpus_cjk_tolerated.jsonl index 94f2e456..64630834 100644 --- a/tools/differential/corpus_cjk_tolerated.jsonl +++ b/tools/differential/corpus_cjk_tolerated.jsonl @@ -23,3 +23,4 @@ "남궁민수, 지훈" "선생님, J.씨" "이, J.씨" +"지훈, 남궁민수" diff --git a/tools/differential/corpus_rules.jsonl b/tools/differential/corpus_rules.jsonl index 5836a586..e341ac40 100644 --- a/tools/differential/corpus_rules.jsonl +++ b/tools/differential/corpus_rules.jsonl @@ -235,18 +235,14 @@ "王君" "田中、太郎" "田中さん" -"田中さん, Dr." -"田中さん, PhD" -"田中さん, V." +"田中さん 様." "马丁·路德·金씨" "高橋 みなみ" "高橋・一郎" "高橋一郎" "김 민준" -"김, 민준씨" "김민준" +"김민준씨" "김지양" "남궁민수" "남궁민수 지훈" -"남궁민수, 지훈" -"지훈, 남궁민수" From de8c57361d5e13ed4ebe135f34a52a25947b6f91 Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Tue, 1 Sep 2026 18:17:45 -0700 Subject: [PATCH 5/7] docs: forms 6 and 7, and the comma-CJK sections say tolerated out loud MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit usage.rst's Input shapes list gains forms 6 (`Family Given [Honorific]`) and 7 (`Given[·Given]·Family / katakana transcription (source order)`), quoted verbatim from tools/differential/shapes.py. Form 6 is the native family-first arrangement (spaced or unspaced, honorific spaced or glued into `suffix`) and has no title slot and no comma, because native CJK writing has neither convention. Form 7 is the transcription listing, kept in source order and never segmented. A comma or Latin wrapper around a CJK name is now stated as tolerated input -- parsed best-effort, changeable without notice -- right where the shapes are introduced, and the old East-Asian pointer paragraph reworks to point at forms 6/7 instead of restating them. "Commas and dots" and "Credentials after a comma" collapse into one "Commas and Latin wrappers around a CJK name" passage under that same tolerated framing: three sentences sketching current behavior (a comma still names the family and stops the split; a glued honorific still peels, falling back to the pre-comma run when what follows the comma is nothing but suffix words; credentials after a comma read best-effort) with no doctests -- a doctest is a promise, and this zone makes none; tests/v2/cases.py's tolerated rows pin the details at HEAD. The examples used (남궁민수, 지훈 / 田中さん, PhD / 田中さん, Dr. / 田中さん, V. / 田中さん, Ph. D.) are checked against their case rows, not invented. The lone comma sentence under "When the script rules don't apply" folds into the same passage. Everything about pure forms is untouched: the honorific peel sections, spacing/division, decomposed text, and the segmenter material all stay. AGENTS.md's design-philosophy paragraph loses its promise voice for the same clause: "it also crosses the FAMILY comma ... so `김, 민준씨` reads exactly as the spaced `김 민준씨` does" is now scoped to "tolerated rather than settled", pointing at rules.md#W3 and noting the reading is watched on the differential's radar tier rather than pinned as contract. The 间隔号-crossing half of that sentence stays normative, since only the comma half was demoted. The segments[:2] site mechanics, the #319 fork narrative and every example past that sentence are untouched -- they describe today's code, which the demotion didn't change. One retained mechanics parenthetical was false on this tree regardless of the demotion (pre-existing on master): `PhD` lands in `suffix`, not `title` -- measured live under both the default and the strict (`lenient_comma_suffixes=False`) policy, and rules.md:1070 pins the same. Corrected to the two-way landing usage.rst already states elsewhere: `V.` -> `given`, `PhD` and `Ph. D.` -> `suffix`. docs/release_log.rst is deliberately untouched: it is the shipped historical record, and recording the comma demotion as a doctrine change is Task 6's decisions.md entry, not a release-note rewrite. Verification: `sphinx-build -b doctest` 244 -> 239 tests (5 removed, all from the two collapsed sections), 0 failures before and after. Full suite unchanged: 6400 passed, 156 skipped, 9 xfailed. ruff clean. tests/v2/test_rules_doc.py + test_doc_citations.py: 419 passed. Co-Authored-By: Claude Fable 5 --- AGENTS.md | 2 +- docs/usage.rst | 98 +++++++++++++++++++------------------------------- 2 files changed, 37 insertions(+), 63 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index e210185e..9a1ddbbc 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -215,7 +215,7 @@ logging.getLogger('HumanName').setLevel(logging.DEBUG) The library has two layers: `nameparser/config/` (data) and `nameparser/parser.py` (logic). -**Design philosophy — positional and language-agnostic.** The parser assigns parts by *position* plus small sets of words that join to neighbors; it never detects language. A name's language can't be reliably inferred from Latin-script transliteration ("Ali" is Arabic or Italian; "Van"/"Della"/"Bin" are first names in some cultures, particles in others), so language-specific rules belong in opt-in `Constants` config, never global defaults. Many "wrong for language X" reports (#133, #150, #130, #85, #103, #146, #83) are irreducible ambiguities — e.g. `de Mesnil` (want last name) vs `Van Johnson` (want first name) are the same `[prefix][word]` shape. Before adding a rule, confirm it doesn't break the opposite case (run the full suite — Portuguese and "Van Johnson" tests are the usual canaries). **The one scoped exception (2.1, #271/#272): script-conditional behavior is permitted exactly where the SCRIPT ITSELF — not statistics about it — determines the convention.** The never-detect-language rule above is about Latin *transliteration*, where the signal genuinely is destroyed; native script is a different question, and it is answered per behavior rather than per script. Five defaults fall out of it, plus a sixth that applies the same not-a-guess standard to specific WORDS rather than to a script (#308's honorific peel, below). Wholly-Han, wholly-Hangul and kana-licensed names read family-first (`Policy.script_orders`) — no language detection needed, because zh and ja both write family-first in native script, so order cannot be misread even though the language is unknowable. Unspaced hangul splits into surname + given name (`config/surnames.py` ships the Korean census list as DEFAULT vocabulary) — nothing but Korean is written in hangul and the surnames are a closed census set, and the vocabulary is self-selecting besides: a hangul entry can only ever match hangul text. Hiragana licenses Japanese (#272) — a name whose characters stay inside Han∪kana while carrying at least one kana cannot be Chinese (the kana rules it out) and is not a transcription (foreign names are transcribed in katakana ALONE, マイケル has no kanji), so 高橋みなみ and 山田 エミ read family-first too; mechanically they resolve to the HIRAGANA entry, the license's carrier key. PURE katakana is excluded and keeps the positional default: マイケル・ジャクソン is a transcribed foreign name in its source order. And the 间隔号 U+00B7 (#298) is the transcription marker for scripts that HAVE no transcription script: a name it divides (威廉·莎士比亚 — flanked by classified characters on both sides, so Catalan's Gal·la is untouched) keeps its source order and never segments — the orthography names the convention, exactly as pure katakana does, with the divider carrying the signal instead of the script. And a listed CJK honorific glued to the END of a name token is split off it (#308) — 田中さん is 田中 plus さん — on the same orthography-settles-it test, narrowed for the glued position: an entry peels only where it can never end a name, so 씨/님/さん/様/先生 peel while 양/군/氏/博士/殿 stay spaced-only (김지양 and 田中博士 are names, and ~90 Japanese surnames end in 殿) and 君 is in NEITHER set (王君 is a complete Chinese name), though its kana spelling くん peels. Like the nakaguro's tokenize-level separation described next, it is reached by neither policy opt-out — but for its own reason: the vocabulary carries the license itself rather than borrowing the script's, so `segment_scripts` has nothing to say about it. Since #312 it also crosses the FAMILY comma and the 间隔号, which is the more surprising fact, since both of those still stop the surname split standing right beside it: each answers where a name DIVIDES into surname and given, and the peel never asks that question — so `김, 민준씨` reads exactly as the spaced `김 민준씨` does (family 김, given 민준, suffix 씨) while the split stands down as before. Its site is accordingly the name-bearing segment runs — `segments[:2]` under a family comma, and `segments[0]` as before otherwise, the family comma being the one structure that splits the name itself across two runs, with the honorific as often glued to the given side as to the family. That is the whole reach and nothing past it (`김, 민준 지훈씨` peels; `김, 민준, 지훈씨` and `김,, 민준씨` do not, both landing in a third run), and whether `segments[1]` is name text at all is now ASKED rather than inferred from the structure — `segment` does not guarantee it, since a one-word part before the comma reads as FAMILY_COMMA even when the part after it is entirely suffix-shaped, and the peel walking into such a run took `V.` for its site, found no listed tail and abandoned (#319). The question is `segment`'s own suffix-comma predicate, lifted into `_vocab.is_wholly_suffix` so the two stages cannot drift: a wholly suffix-shaped second run is declined and the scan stays in `segments[0]`, so `田中さん, V.` and `田中さん, Ph. D.` give さん up as `田中さん, PhD` always did. The test is necessary but NOT sufficient, and the second condition is not decoration: every honorific tail is also a suffix word, so a glued honorific is itself part of what makes its run read as suffix-shaped, and declining a run that holds the ONLY site loses the peel outright. `segments[0]` must therefore offer a peel site of its own before the second run is declined — `이, J.씨` and `선생님, J.씨` pass the suffix test and are scanned anyway, keeping the pre-#319 reading, while `김민준씨, J.씨` has a site on both sides and peels the person's own 씨 rather than the junk one behind the comma. Uniform in the PEEL, that is — where the credential itself lands is `assign`'s question and still differs by spelling (`PhD` → `title`, `V.` → `given`, `Ph. D.` → `suffix`). Not `_is_post_nominal` pluralized: the run predicate says yes both to what the token predicate vetoes (`V.`, `V`, `I` — the class the defect was reported as) and to what the token predicate never sees at all, since `period_joined_vocab` and the delimiter routes are the run predicate's alone (`Msc.Ed.` and `J.씨` reach it that way, and `田中さん, Msc.Ed.` moves with the rest). `Policy(lenient_comma_suffixes=False)` drops this call to the strict token test too — so those three read as name text again and keep the pre-#319 answer, while `Ph. D.` peels under the knob regardless, its merged `phd` passing the strict test. `田中さん, 太郎` is unchanged, and not because of its comma — the honorific there is not at the end of the name, 太郎 is. The nakaguro belongs to the same doctrine but is decided a layer down: U+30FB and its halfwidth twin U+FF65 separate tokens like whitespace, unconditionally and in tokenize, so neither policy opt-out (`script_orders={}`, `segment_scripts=()`) reaches it — the codepoints are CJK-only and appear in no other script's names, which is what licenses a tokenize-level rule where U+00B7 (also the Catalan punt volat, interior to Gal·la) needs the flanked-by-classified-script guard `_tokenize_region` gives it (#298). Han segmentation stays OPT-IN (`locales.ZH` for Chinese, `locales.JA` for Japanese) — a zh surname list corrupts Japanese kanji names, since 高 is a common Chinese surname and 高橋一郎 would split 高+橋一郎 where the correct reading is 高橋+一郎; no surname list divides a kanji name at all, so `locales.JA` activates the stage and a pluggable `Parser(segmenter=...)` does the dividing. Latin-script input is never touched by any of this: "Kim Min-jun" is genuinely order-ambiguous and stays governed by `name_order` and opt-in packs. Before adding a script-conditional rule, work out which of the three it is — certain, certain for this one behavior only, or a statistical guess wearing a script's clothes. +**Design philosophy — positional and language-agnostic.** The parser assigns parts by *position* plus small sets of words that join to neighbors; it never detects language. A name's language can't be reliably inferred from Latin-script transliteration ("Ali" is Arabic or Italian; "Van"/"Della"/"Bin" are first names in some cultures, particles in others), so language-specific rules belong in opt-in `Constants` config, never global defaults. Many "wrong for language X" reports (#133, #150, #130, #85, #103, #146, #83) are irreducible ambiguities — e.g. `de Mesnil` (want last name) vs `Van Johnson` (want first name) are the same `[prefix][word]` shape. Before adding a rule, confirm it doesn't break the opposite case (run the full suite — Portuguese and "Van Johnson" tests are the usual canaries). **The one scoped exception (2.1, #271/#272): script-conditional behavior is permitted exactly where the SCRIPT ITSELF — not statistics about it — determines the convention.** The never-detect-language rule above is about Latin *transliteration*, where the signal genuinely is destroyed; native script is a different question, and it is answered per behavior rather than per script. Five defaults fall out of it, plus a sixth that applies the same not-a-guess standard to specific WORDS rather than to a script (#308's honorific peel, below). Wholly-Han, wholly-Hangul and kana-licensed names read family-first (`Policy.script_orders`) — no language detection needed, because zh and ja both write family-first in native script, so order cannot be misread even though the language is unknowable. Unspaced hangul splits into surname + given name (`config/surnames.py` ships the Korean census list as DEFAULT vocabulary) — nothing but Korean is written in hangul and the surnames are a closed census set, and the vocabulary is self-selecting besides: a hangul entry can only ever match hangul text. Hiragana licenses Japanese (#272) — a name whose characters stay inside Han∪kana while carrying at least one kana cannot be Chinese (the kana rules it out) and is not a transcription (foreign names are transcribed in katakana ALONE, マイケル has no kanji), so 高橋みなみ and 山田 エミ read family-first too; mechanically they resolve to the HIRAGANA entry, the license's carrier key. PURE katakana is excluded and keeps the positional default: マイケル・ジャクソン is a transcribed foreign name in its source order. And the 间隔号 U+00B7 (#298) is the transcription marker for scripts that HAVE no transcription script: a name it divides (威廉·莎士比亚 — flanked by classified characters on both sides, so Catalan's Gal·la is untouched) keeps its source order and never segments — the orthography names the convention, exactly as pure katakana does, with the divider carrying the signal instead of the script. And a listed CJK honorific glued to the END of a name token is split off it (#308) — 田中さん is 田中 plus さん — on the same orthography-settles-it test, narrowed for the glued position: an entry peels only where it can never end a name, so 씨/님/さん/様/先生 peel while 양/군/氏/博士/殿 stay spaced-only (김지양 and 田中博士 are names, and ~90 Japanese surnames end in 殿) and 君 is in NEITHER set (王君 is a complete Chinese name), though its kana spelling くん peels. Like the nakaguro's tokenize-level separation described next, it is reached by neither policy opt-out — but for its own reason: the vocabulary carries the license itself rather than borrowing the script's, so `segment_scripts` has nothing to say about it. Since #312 it also crosses the 间隔号, which still stops the surname split standing right beside it: it answers where a name DIVIDES into surname and given, and the peel never asks that question. Whether it also crosses the FAMILY comma is tolerated rather than settled: the 2026-09-01 demotion (rules.md#W3) narrowed that half from contract to best-effort, since no CJK writing system's own convention puts a comma between family and given at all — so `김, 민준씨` reads today exactly as the spaced `김 민준씨` does (family 김, given 민준, suffix 씨) while the split stands down as before, but that reading is watched on the differential's radar tier rather than pinned as contract. Its site is accordingly the name-bearing segment runs — `segments[:2]` under a family comma, and `segments[0]` as before otherwise, the family comma being the one structure that splits the name itself across two runs, with the honorific as often glued to the given side as to the family. That is the whole reach and nothing past it (`김, 민준 지훈씨` peels; `김, 민준, 지훈씨` and `김,, 민준씨` do not, both landing in a third run), and whether `segments[1]` is name text at all is now ASKED rather than inferred from the structure — `segment` does not guarantee it, since a one-word part before the comma reads as FAMILY_COMMA even when the part after it is entirely suffix-shaped, and the peel walking into such a run took `V.` for its site, found no listed tail and abandoned (#319). The question is `segment`'s own suffix-comma predicate, lifted into `_vocab.is_wholly_suffix` so the two stages cannot drift: a wholly suffix-shaped second run is declined and the scan stays in `segments[0]`, so `田中さん, V.` and `田中さん, Ph. D.` give さん up as `田中さん, PhD` always did. The test is necessary but NOT sufficient, and the second condition is not decoration: every honorific tail is also a suffix word, so a glued honorific is itself part of what makes its run read as suffix-shaped, and declining a run that holds the ONLY site loses the peel outright. `segments[0]` must therefore offer a peel site of its own before the second run is declined — `이, J.씨` and `선생님, J.씨` pass the suffix test and are scanned anyway, keeping the pre-#319 reading, while `김민준씨, J.씨` has a site on both sides and peels the person's own 씨 rather than the junk one behind the comma. Uniform in the PEEL, that is — where the credential itself lands is `assign`'s question and still differs by spelling (`V.` → `given`, `PhD` and `Ph. D.` → `suffix`). Not `_is_post_nominal` pluralized: the run predicate says yes both to what the token predicate vetoes (`V.`, `V`, `I` — the class the defect was reported as) and to what the token predicate never sees at all, since `period_joined_vocab` and the delimiter routes are the run predicate's alone (`Msc.Ed.` and `J.씨` reach it that way, and `田中さん, Msc.Ed.` moves with the rest). `Policy(lenient_comma_suffixes=False)` drops this call to the strict token test too — so those three read as name text again and keep the pre-#319 answer, while `Ph. D.` peels under the knob regardless, its merged `phd` passing the strict test. `田中さん, 太郎` is unchanged, and not because of its comma — the honorific there is not at the end of the name, 太郎 is. The nakaguro belongs to the same doctrine but is decided a layer down: U+30FB and its halfwidth twin U+FF65 separate tokens like whitespace, unconditionally and in tokenize, so neither policy opt-out (`script_orders={}`, `segment_scripts=()`) reaches it — the codepoints are CJK-only and appear in no other script's names, which is what licenses a tokenize-level rule where U+00B7 (also the Catalan punt volat, interior to Gal·la) needs the flanked-by-classified-script guard `_tokenize_region` gives it (#298). Han segmentation stays OPT-IN (`locales.ZH` for Chinese, `locales.JA` for Japanese) — a zh surname list corrupts Japanese kanji names, since 高 is a common Chinese surname and 高橋一郎 would split 高+橋一郎 where the correct reading is 高橋+一郎; no surname list divides a kanji name at all, so `locales.JA` activates the stage and a pluggable `Parser(segmenter=...)` does the dividing. Latin-script input is never touched by any of this: "Kim Min-jun" is genuinely order-ambiguous and stays governed by `name_order` and opt-in packs. Before adding a script-conditional rule, work out which of the three it is — certain, certain for this one behavior only, or a statistical guess wearing a script's clothes. **A constant's membership is a question you may reopen.** Proposing that a word be ADDED, REMOVED or MOVED between vocabulary sets is ordinary design work — a shipped entry is not evidence that anyone judged it. `SUFFIX_ACRONYMS` arrived in `af5bdab` as a bulk Wikipedia import never reviewed against surname collisions: 575 of its 579 alphabetic entries leave `family` empty in `"John "` against four ambiguous-gated exceptions, and `rai` (#342), `ba`, `cha`, `sa`, `se`, `om` and `mc` are all borne as surnames (measured 2026-08-23). When a fix starts to look like new machinery, check the vocabulary first. Criterion: `decisions.md#vocabulary-collisions`, with #360's positional qualifier. diff --git a/docs/usage.rst b/docs/usage.rst index 060e85b0..026e2ced 100644 --- a/docs/usage.rst +++ b/docs/usage.rst @@ -62,9 +62,25 @@ displaced from the family name it belongs to; form 5's trailing word is the given name by the caller's declaration, so there is nothing there to reinterpret. -Names written in Han or Hangul, and Japanese names written in kanji -and kana, are the exception that needs no setting at all: see `East -Asian names`_ below. +Two more arrangements are native East Asian forms and need no +``name_order`` at all — the script itself carries the reading: + +6. ``Family Given [Honorific]`` +7. ``Given[·Given]·Family / katakana transcription (source order)`` + +Form 6 is the native family-first arrangement written in Han or +Hangul — spaced or unspaced, with the honorific spaced or glued and +landing in ``suffix``. It has no title slot and no comma, because +native CJK writing has neither convention. Form 7 is a transcription +listing — Han divided by the 间隔号, or katakana joined by the +nakaguro — kept in the order it was written and never segmented. + +A comma or a Latin wrapper around a CJK name — a listing comma, a +Latin honorific or credential set beside it — is tolerated input: +parsed best-effort, its handling changeable without notice. + +Forms 6 and 7, and how a Latin wrapper around either is handled, are +covered in full under `East Asian names`_ below. Words that attach to their neighbors -------------------------------------- @@ -357,10 +373,6 @@ dot carries its own script's convention, the nakaguro's is Japanese roster formatting rather than transcription, so only the Chinese dot rescues the source order. -A comma disables the script behaviors that decide where a name -divides, on the reasoning ``name_order`` already follows: whoever -wrote the comma has already said where the family name ends. - Honorifics come off first ^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -381,61 +393,23 @@ without it. That is why ``김민준씨`` still divides into family 김 and given 민준, and why a configured Japanese segmenter is handed 山田太郎 rather than 山田太郎様. -Commas and dots -^^^^^^^^^^^^^^^ - -Neither a comma nor a 间隔号 switches the peel off. Both say where a -name divides: the comma that the writer has already given the family -name, the dot that the pieces are a transcription's syllable groups. -An honorific is not part of the name in either reading. - -.. doctest:: - - >>> parse("김, 민준씨").given == parse("김 민준씨").given - True - -What a comma does instead is say which runs are the name: the two -around a family comma, an honorific being as often glued to the given -name as to the family. Nothing past those two is in reach. - -.. doctest:: - - >>> parse("김, 민준 지훈씨").suffix # second run, still in reach - '씨' - >>> parse("김, 민준, 지훈씨").suffix # a third run, left whole - '지훈씨' - -What those marks do stop is the *split*, which is a different question -and still theirs to answer. ``田中さん, 太郎`` is unchanged, and not -because of its comma: the honorific there is not at the end of the -name, 太郎 is. - -Credentials after a comma -^^^^^^^^^^^^^^^^^^^^^^^^^ - -The second of those two runs counts only where it is name text. A run -that is nothing but credentials is not, whatever the comma looked -like, and the same test that decides the comma structure decides this -too: the run is left alone and the honorific is found in the name. - -Being credentials is necessary but not enough, because the test cannot -help counting the honorific itself. A glued honorific is a suffix -word, so it is part of what makes its own run read as credentials. The -run is passed over only when the part *before* the comma has an -honorific of its own to give up; otherwise passing it over would -discard the only one there is. - -.. doctest:: - - >>> credential = parse("田中さん, V.") - >>> credential.family, credential.suffix - ('田中', 'さん') - -So ``田中さん, V.`` and ``田中さん, Ph. D.`` give さん up exactly as -``田中さん, PhD`` does. Where the credential itself lands is the -comma's business rather than the peel's, and still differs by -spelling: ``given`` for ``V.``, while ``PhD`` and ``Ph. D.`` join さん -in ``suffix``. +Commas and Latin wrappers around a CJK name +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +A comma or a Latin credential set wrapped around a CJK name is +tolerated input rather than contract: no native CJK writing uses +either convention, so nameparser reads it best-effort and the +handling can change without notice. Today a comma still names the +family and stops the split — ``남궁민수, 지훈`` reads family +``남궁민수`` whole, where the bare ``남궁민수`` alone would split into +family ``남궁`` and given ``민수`` — and a glued honorific still peels +off before the comma when what follows it is nothing but suffix +words, as in ``田中さん, PhD`` (suffix ``さん, PhD``), but stays glued +when the comma is followed by a title or another name word, as in +``田中さん, Dr.`` (family ``田中さん``). Credentials after the comma are +read the same best-effort way, landing in ``given`` or ``suffix`` by +spelling: ``田中さん, V.`` and ``田中さん, Ph. D.`` give up さん exactly +as ``田中さん, PhD`` does. Spacing, and where the name divides ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ From a55f5034c2511fca58ba77085350df2eedfe4fa8 Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Tue, 1 Sep 2026 18:35:47 -0700 Subject: [PATCH 6/7] docs(design): the CJK comma demotion's decision record, with CLDR pulled verbatim MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The doctrine narrowing gets its entry: W3 goes contract -> tolerated for CJK script, every wrapper composition (family comma, honorific-comma probes, Latin titles, credential commas) is tolerated input, and shapes 6/7 admit only pure classified-script text. The line is FORMAT PURITY, not comma class. Three supersessions written the way the file writes them, none by editing what they supersede: - 3-0-reevaluations' FAMILY_COMMA bullet ("correct on its own terms") is SCOPED: it stands for Latin script, and for CJK the comma reading is description rather than contract. - The in-session comma-class fork, which kept credential commas contract while demoting the family comma, is recorded and replaced — both are wrappers, and purity states the line without a taxonomy. - The 2026-08-23 C1 snapshot attributing the glued honorific of '田中さん, Dr.' to "(C1 Accepted)" now points at W3, where the clause lives; the snapshot stays as written, true of when it was taken. The corpus-tier arc gains two bullets: shapes 6/7 supersede its own "CJK is deliberately OUTSIDE the inventory" bullet (the question it could not answer split into two shapes plus a tolerated class), and the `tolerated` flag is recorded there rather than in mechanisms.md — one arc's device, no second instance, no comment citing it, and the tier mechanics it rides on are already README-owned. CLDR pulled verbatim 2026-09-01, failures included: the plan's common/personNames/ paths 404 (the data is a element in common/main/.xml), read at unicode-org/cldr main 8e4fb0f. Each of ko/zh/ja holds 42 namePattern elements and none of the 126 carries a comma of any width; the surname-first referring patterns separate surname from given with a single space. The one comma in reach is the locale-neutral root's SORTING pattern, which ko and zh override comma-free and ja all but one inherited slot — recorded so the entry does not overclaim. The pull confirms, so rules.md's W Background gains the citation sentence; it adds no example and corpus_rules.jsonl is byte-identical at 248. Counts re-verified against the tree, each with its recompute: 26 tolerated texts over 29 rows, corpus_cjk 98 -> 73 with the radar file at 26 (73 + 25 = 98 says the split moved names and dropped none), corpus_rules 252 -> 248, corpus_shapes 37, coverage 5/2 for shapes 6/7. Intentional counts unmoved at 226/205/113/0, unexplained 0, radar unclassified 0, exit 0 at all four baselines. The shipped 2.2.0 release notes are left untouched by choice, and the entry says so: they described the comma-crossing as a fix, which was true of those releases. Co-Authored-By: Claude Fable 5 --- docs/design/decisions.md | 26 ++++++++++++++++++++++++++ docs/design/rules.md | 2 +- 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/docs/design/decisions.md b/docs/design/decisions.md index 466f1028..4e928bfc 100644 --- a/docs/design/decisions.md +++ b/docs/design/decisions.md @@ -405,6 +405,29 @@ Excluded (SUFFIX_ACRONYMS / SUFFIX_WORDS — the esq dual membership, deliberate - esq is in BOTH sets and must not be "deduplicated". The load-bearing membership is the acronym one (it carries the multi-dot spellings: removing it costs "John Smith E.S.Q." its family name); the word membership is inert as shipped but is what keeps "Esq" matching for a caller who edits suffix_acronyms themselves. esq is the ONLY member of SUFFIX_ACRONYMS ∩ SUFFIX_WORDS — that singleton is why the two sets cannot carry a disjointness assert, which is the standing cost this entry defends. Deliberately no changed-parse count — the count is a property of the measuring grid, not of the code. +### cjk-comma-demotion — the script shapes are pure, the wrappers are tolerated (2026-09-01, #469) + +Closes #469, and continues the corpus-tier arc below rather than standing apart from it: the tier split gave the differential somewhere to WATCH a name without promising it, and this is the first doctrine narrowing to spend that. No parser behavior moves anywhere in it. Counts are this session's and every one is recomputable from the checked-in tree — `wc -l` over tools/differential/, `uv run python tools/differential/build_shapes_corpus.py --coverage`, and one `uv run python tools/differential/compare.py --baseline X` run per baseline, read off its `corpora:` and `corpus:` lines. + +- 2026-09-01 #469 — THE LINE IS FORMAT PURITY, and not comma class. Shapes 6 ("Family Given [Honorific]") and 7 (the interpunct/katakana transcription in source order) admit wholly classified-script text and refuse a comma and every ASCII letter; each composed form around a CJK name — the family-comma listing W3 reads, the honorific-comma probe zoo, a leading Latin title, a trailing credential comma — is TOLERATED input instead: parsed best-effort, contract-exempt, changeable without notice. What the inventory is FOR is the whole argument. It tabulates written ARRANGEMENTS, and a comma-wrapped or Latin-wrapped CJK name is not an arrangement any of the three writing systems produces — it is a listing convention from elsewhere with a CJK name inside it. So the admission test is a property of the FORMAT, and a rule reading such input describes rather than promises. + Supersedes an in-session fork of this same day's design, recorded because it is a line the next reader would redraw the same wrong way. The first cut kept trailing CREDENTIAL commas contract while demoting the family comma, on the theory that the two comma classes differ in kind — a credential comma being a Latin convention wrapped around anything, where a family comma makes a claim about the name's own division. The shape lens replaced it: both are wrappers, neither is a form the script writes, and a doctrine promising one while tolerating the other has to keep explaining a distinction its own admission test cannot state. Purity states the line in one sentence and needs no comma taxonomy at all. + +- 2026-09-01 #469 — SCOPES the FAMILY_COMMA bullet under 3-0-reevaluations below, which reads "inherited from v1's lastname-comma but correct on its own terms — an explicit comma is stronger evidence than script". It stands unchanged for LATIN script, which is the script it was argued about. For CJK script it is narrowed to a description: an explicit comma still beats script and the parser still reads it that way, but every input carrying one is a form no East Asian writing system produces, so what the parser makes of it is current behavior and not a promise. The bullet is not edited — it was right about what it was reasoning over, and this entry is the scope it never had to state while Latin was the only script in the room. + +- 2026-09-01 #469 — the 2026-08-23 (PR #428) snapshot under decisions.md#C1 below attributes `田中さん, Dr.` keeping its honorific glued to "(C1 Accepted)". That clause now lives in rules.md#W3's `tolerated:` note: the BEHAVIOR is unchanged and only its home moved, W3 being where the CJK comma forms are set down now that they are described rather than promised. The snapshot stays as written — it was true when it was taken, and this is the note that says where to read the clause instead. + +- CLDR, PULLED VERBATIM 2026-09-01 — #469 asked for the citation and asked that it be a pull rather than a recollection, so the failures are recorded with the findings. The paths named in the plan (`https://raw.githubusercontent.com/unicode-org/cldr/main/common/personNames/ko.xml`, `.../zh.xml`, `.../ja.xml`) all 404: there is no `common/personNames/` directory in the repository, and the data is a `` element inside `common/main/.xml`. Read there instead, from unicode-org/cldr at `main`, commit `8e4fb0fa1e287c4612b512cf2ef77e890467da28`; the grammar from https://www.unicode.org/reports/tr35/tr35-personNames.html. The grammar first, verbatim, because it decides whether absence is evidence: "A namePattern is composed of a sequence of field IDs, each enclosed in curly braces, and separated by zero or more literal characters (eg, space or comma + space)." A comma between surname and given is therefore expressible, and its absence below is a choice the locale data makes rather than a limit of the format. The `order="surnameFirst" usage="referring" formality="formal"` patterns, verbatim, long then medium then short: + ko {surname} {given} {given2} {credentials} · {surname} {given} {credentials} · {surname} {given} + zh {generation}{surname} {given} {given2} {credentials}{title} · {generation}{surname} {given} {given2-initial}{credentials} · {surname} {given} {given2} + ja {surname} {given2} {given}{title} · {surname} {given}{title} · {surname} {given}{title} + Counted rather than eyeballed: each of the three files holds 42 `namePattern` elements, and not one of the 126 contains a comma of any width (U+002C, U+FF0C, U+3001) or a semicolon. The delimiter between surname and given is a single space throughout. + THE ONE COMMA IN REACH, recorded because the record must not overclaim. It is in the locale-neutral root, whose sorting pattern reads verbatim `{surname} {surname2}, {title} {given} {given2} {credentials}`. `sorting` is a list-ordering format and not a referring one — UTS #35 verbatim: "Used to format names for a sorted list. example: “Brown, William” [medium, informal]" — and ko and zh override all six of their sorting slots comma-free (`{surname} {given}` and kin), while ja overrides five and leaves `order="sorting" length="medium" usage="referring" formality="formal"` as `↑↑↑`, inheriting root's comma-bearing pattern. So the honest reading, and it is the stronger one for being stated with its exception: no native pattern in any of the three writes a comma between surname and given, and the sole comma any of them can reach is INHERITED from the locale-neutral default, in the format whose entire job is sorted lists. That is the "listing convention carried in from elsewhere" the W Background already named, arriving from CLDR rather than from this project's own reasoning — the pull CONFIRMS the writing-system rationale, and complicates it only by showing the exact seam where a comma does get in. + Two findings worth keeping past the comma question. zh writes `{surname}{title}` with no delimiter at all (and ja `{surname} {given}{title}`), which corroborates rules.md#W2's glued honorific from the formatter's side, independently of the peel's own vocabulary argument. And ja declares `・` — a foreign name's spaces become the nakaguro — which is shape 7's divider reached from the formatting direction rather than the parsing one. + +- 2026-09-01 — the SHIPPED 2.2.0 release notes are left untouched, deliberately, and this bullet is the record of that choice. docs/release_log.rst describes the comma-crossing as a fix in two 2.1.0 entries ("Fix a comma or a 间隔号 stopping the glued-honorific peel", and #319's `田中さん, V.` entry). Both were true statements about what those releases did, made while the crossing was contract and #312 had argued it. Demoting the rule does not make a released note false about its release, and editing shipped notes to agree with a later doctrine would cost the log the one property it exists for. The demotion is 2.3's news and belongs in 2.3's notes; the log is a history, not a mirror of the current contract. + +- COUNTS, 2026-09-01, with the recompute beside each. 29 case rows carry `tolerated=True` over 26 distinct texts — the flag is per ROW and the corpus is per TEXT, which is why the two numbers differ (recompute: read `CASES` and count `c.tolerated`). The 26th text arrived with the W3 demotion itself: the given-side listing `지훈, 남궁민수` was a rules.md example with no case row, and would have left the harness altogether when the rules corpus stopped carrying W3, so it was given a tolerated row rather than dropped. `corpus_cjk.jsonl` went 98 → 73 and the radar-tier `corpus_cjk_tolerated.jsonl` was created at 25, then 26; the identity is the check worth keeping, since 73 + 25 = 98 exactly says the split MOVED names and dropped none, and the 99th is that one rescued row. `corpus_rules.jsonl` went 252 → 248: six comma-bearing example texts left (`田中さん, Dr.`, `田中さん, PhD`, `남궁민수, 지훈`, `지훈, 남궁민수` when the builder skipped W3 whole, plus `김, 민준씨` and `田中さん, V.` from W2's swap) and two pure ones arrived (`김민준씨`, `田中さん 様.`). `corpus_shapes.jsonl` stands at 37, with the new shapes covered 5 names (shape 6) and 2 (shape 7). And the stop condition held: intentional diffs are UNMOVED at 226/205/113/0 across 1.4.0/2.0.0/2.1.0/2.2.0, with unexplained 0, radar unclassified 0 and exit 0 at each — 1113 names compare at 1.4.0 (7 skipped, shapes 4/5) and 1120 above it. That the counts do not move is the substance and not a formality: a classified diff counts identically on either tier, so demoting a name changes which file it loads from and nothing about what the gate makes of it. + ### P5 — bound given names - 2026-06-30 (first-name-prefix-join design; v1-era, carried into the v2 port) — the join is vocabulary-driven and deliberately tiny. @@ -702,6 +725,9 @@ Decisions that landed: - 2026-09-01 #486 (later the same day, so read this bullet as superseding the arithmetic in the one above rather than the two figures disagreeing) — the shape 1-3 variation matrix filled the given-first half of the inventory, which the feature work that authored shapes 4 and 5 had left at whatever the pipeline PRs happened to tag. 14 existing case rows gained a tag and 3 rows were authored for slots no row instantiated, taking `corpus_shapes.jsonl` from 13 entries to 30 and shapes 1/2/3 from 1/4/1 names to 9/11/3. The arithmetic moves with it and the recipe is unchanged — run `uv run python tools/differential/compare.py` and read its `corpora:` and `corpus:` lines, adding `--baseline 1.4.0` for the skip figures: 30 shape entries, of which 14 dedupe into names already compared under the default order, for 1120 comparisons; shapes 4 and 5 still declare `min_baseline` 2.0.0, so 7 entries are skipped at `--baseline 1.4.0` and 1113 names compare there. The dedupe count is the half worth reading, because it is what the tier promotion looks like from this side: 8 of the newly tagged names were in a RADAR corpus only, so the tag moved them into the contract tier without adding a comparison, and 2 more were already contract through the rules corpus. None of the 17 names #486 tagged or authored needed a ledger rule — the seven of them that were in no corpus at all diff at no baseline, and intentional stayed 226/205/113/0 across the change. Not a claim about all 30 entries: the seven shape 4/5 entries are classified by ledger rules today, and scoping exactly those rules is what the `orders` bullet three above is about. - 2026-09-01 #469 — the CJK arrangement is deliberately OUTSIDE the shape inventory. Whether an unspaced CJK name is a third family-first shape is the open question on #469, and `corpus_cjk.jsonl` — contract-tier already, generated from the case table already — covers that ground meanwhile. Tagging it in would have to ANSWER the question first, since a shape row cannot be written without a `name_order` and a `min_baseline` for it, and the inventory is a table of arrangements rather than where that argument belongs. +- 2026-09-01 #469 (later the same day, so read this bullet as SUPERSEDING the one above rather than the two disagreeing) — the CJK arrangement is IN the inventory after all, as shapes 6 and 7, and the bullet above is right about why it could not be until the question was answered: a shape row cannot be written without settling what it admits. What settled it was splitting the question the bullet treats as one. There is no single "unspaced CJK name" shape — there is a pure family-first arrangement (6) and a source-order transcription listing (7), and the composed forms are not arrangements at all. Both take `order=None`, which is the part that had looked impossible: the family-first reading is SCRIPT-carried rather than declared, so a pure shape 6/7 string already parses correctly under the DEFAULT policy and there is nothing for `order` to name. `min_baseline` is then documentary rather than a skip trigger, an asymmetry with shapes 4/5 that tools/differential/shapes.py's docstring states so it reads as designed. The doctrine half is its own entry (cjk-comma-demotion above); what belongs here is that `corpus_cjk.jsonl` no longer "covers that ground meanwhile" — the ground is split between it and a radar file now. +- 2026-09-01 #469 — the `tolerated` flag on a case row is the demotion's VEHICLE, and it is recorded here rather than as a mechanisms.md entry. It is an explicit declaration, reviewed one row at a time exactly as a `shape=` tag is, mutually exclusive with one, and restricted to rows whose text bears a classified codepoint; `build_cjk_corpus.py` partitions on it, and a text marked on one row and not another is a hard error, the flag being per-text in effect. Why not a mechanisms entry: that catalog is keyed by RECURRING problem shape and its contract statements are citable verbatim from code comments, and this is one arc's device with no second instance and no comment needing to cite it — the tier mechanics it rides on are already owned by tools/differential/README.md, so an entry would restate a source rather than say something the source does not (the #473 lesson under decisions.md#review-agent-single-source). If a second arc ever needs "record the behavior, promise nothing", that is the point to promote it and this bullet is the first instance to cite. + ### comma-suffix-arc — #291/#296/#316 (2026-07-26 → 2026-08-01) #291 was filed 2026-07-26 out of the 2.0 vocabulary cleanup, with diff --git a/docs/design/rules.md b/docs/design/rules.md index 98177b67..1a515558 100644 --- a/docs/design/rules.md +++ b/docs/design/rules.md @@ -994,7 +994,7 @@ O5. Rationale: O4 reads a name by comparing where its words stand, ## Scripts & writing systems (W) -Background: script-conditional behavior is permitted exactly where the writing system itself — not statistics about it — settles the convention; a language can never be inferred from Latin-script text, because transliteration destroys the signal. The facts this section builds on: Chinese and Japanese both write the family name first in native script, so the script settles the order without knowing the language. Hangul is written by exactly one language and Korean family names are a small closed census set. Han text does not identify its language — a Chinese surname list would divide Japanese 高橋一郎 as 高 + 橋一郎 — which is why Han division is opt-in and there is no Korean pack to opt into. Hiragana never transcribes a foreign name (transcriptions are katakana alone), so kanji-plus-kana is a Japanese name in Japanese order, while wholly-katakana is predominantly a transcribed foreign name already in given-first order. Real Chinese text is unspaced (毛泽东); the spaced 毛 泽东 is an artifact. A fuller narrative lives in docs/usage.rst's East Asian section. One fact carries its own consequence: none of the three writing systems marks the family name with a comma — position in the written form is what identifies it, so a comma standing between the family name and the given name is a listing convention carried in from elsewhere rather than a form the script produces. That is why the rule reading one (W3) is tolerated rather than normative. +Background: script-conditional behavior is permitted exactly where the writing system itself — not statistics about it — settles the convention; a language can never be inferred from Latin-script text, because transliteration destroys the signal. The facts this section builds on: Chinese and Japanese both write the family name first in native script, so the script settles the order without knowing the language. Hangul is written by exactly one language and Korean family names are a small closed census set. Han text does not identify its language — a Chinese surname list would divide Japanese 高橋一郎 as 高 + 橋一郎 — which is why Han division is opt-in and there is no Korean pack to opt into. Hiragana never transcribes a foreign name (transcriptions are katakana alone), so kanji-plus-kana is a Japanese name in Japanese order, while wholly-katakana is predominantly a transcribed foreign name already in given-first order. Real Chinese text is unspaced (毛泽东); the spaced 毛 泽东 is an artifact. A fuller narrative lives in docs/usage.rst's East Asian section. One fact carries its own consequence: none of the three writing systems marks the family name with a comma — position in the written form is what identifies it, so a comma standing between the family name and the given name is a listing convention carried in from elsewhere rather than a form the script produces. That is why the rule reading one (W3) is tolerated rather than normative. CLDR's own locale data says the same where a contrary convention would have had to appear: across its ko, zh and ja personName patterns not one of the 126 pattern strings carries a comma of any width, the surname-first referring patterns separating surname from given by a single space, and the only comma in reach belongs to the locale-neutral root's sorting format — a list-ordering format, which ko and zh override comma-free and ja all but one inherited slot (decisions.md#cjk-comma-demotion carries the pull verbatim, with its URLs, its commit and its date). W1. Rationale: hangul is monoglot Korean and its surnames are a closed census set, so an unspaced hangul name divides at a From 7441383d1d43fca36d9e5b76d94138ff84952ef9 Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Tue, 1 Sep 2026 19:58:12 -0700 Subject: [PATCH 7/7] test,docs(differential): the review round's walls -- tolerated examples stay watched, the tiers stay disjoint Six fixes from the PR #488 review round, each closing a way the 2026-09-01 CJK comma demotion could mean less than it says. - A tolerated rule's example texts, minus those a normative rule also carries, must be in some corpus the harness loads. Closes the sixth W3 example with no case row: executed at HEAD, compared at no baseline, every suite green. The injection a reviewer used runs as the test's own negative control. - corpus_cjk_tolerated.jsonl is disjoint from every CONTRACT corpus, the tier list read from compare._CORPUS_TIERS rather than copied. The dedup keeps the contract reading, so an overlap re-enforces a demoted name while its rows, the README and the release notes all call it demoted. - rules.md's parser raises on a near-miss `tolerated:` marker instead of reading past it. A typo'd marker leaves the rule NORMATIVE and its examples under contract -- the opposite of the line's intent. An empty reason raises too: a demotion nobody justified is the one nobody reviews. - The differential README's tier paragraph takes compare.py's own wording: radar holds the names the contract does not answer for -- scraped, harvested, and since 2026-09-01 deliberately demoted. - Two prose corrections: customize.rst's lenient_comma_suffixes row says the CJK reading it shows is tolerated input and can change; the 1.4.0 ledger's provenance comment describes the two-file harvest split (both files load in every run, so the never-dormant conclusion survives). - A shape-6 row carrying a policy is refused, the arm the locale probe left untested. Co-Authored-By: Claude Fable 5 --- docs/customize.rst | 4 +- tests/v2/rules_doc.py | 31 +++- tests/v2/test_cases.py | 9 + tests/v2/test_ledger_guards.py | 168 +++++++++++++++++-- tests/v2/test_rules_doc_grammar.py | 42 +++++ tools/differential/README.md | 9 +- tools/differential/expected_since_1.4.0.toml | 9 + 7 files changed, 254 insertions(+), 18 deletions(-) diff --git a/docs/customize.rst b/docs/customize.rst index 397a618b..8a027897 100644 --- a/docs/customize.rst +++ b/docs/customize.rst @@ -354,7 +354,9 @@ listed below. glued-honorific peel asks before crossing a family comma (#319), so the setting reaches CJK names too: ``"田中さん, V."`` gives family ``田中``, suffix ``さん`` when ``True``, and - family ``田中さん``, given ``V.`` when ``False``. + family ``田中さん``, given ``V.`` when ``False`` — though a + comma around a CJK name is tolerated input + (``rules.md#W3``) and this reading can change. * - ``strip_emoji`` - ``bool`` - Excludes emoji from tokenization — they appear in no field or diff --git a/tests/v2/rules_doc.py b/tests/v2/rules_doc.py index acf78e5d..20d1c5dc 100644 --- a/tests/v2/rules_doc.py +++ b/tests/v2/rules_doc.py @@ -21,8 +21,12 @@ tools/differential/build_rules_corpus.py skips a rule carrying it, so a tolerated rule's examples illustrate current behavior without entering the contract corpus that enforces it at released baselines. -The reason is free text and unvalidated; what is machine-read is that -the line is present. +The reason is free text, but it must be there: a marker with no reason +is a hard error, the same way a malformed example is. What is +machine-read is that the line is present, so a near miss on the +spelling — ``Tolerated:``, ``tolerated :``, a bare ``tolerated:`` — +fails loudly too, rather than leaving the rule normative and its +examples under contract. Inside a rule block, any line whose first non-space character is a double quote (or an opening bracket, the D-section subject form) is an @@ -51,6 +55,14 @@ r"\s*$") _NO_BOUNDARY_RE = re.compile(r"^\s*no-boundary:\s+(?P\S.*)$") _TOLERATED_RE = re.compile(r"^\s*tolerated:\s+(?P\S.*)$") +#: A line that was MEANT to be the marker above. Everything the strict +#: form rejects -- any casing of the word, a space before the colon, a +#: reason that is empty or all whitespace -- lands here and raises, +#: because the silent alternative is the dangerous one (see the raise +#: below). Case-insensitive rather than just `[Tt]`: an author +#: shouting the word is making the same mistake as one capitalizing +#: it, and no line in a rule block legitimately opens with it. +_NEAR_TOLERATED_RE = re.compile(r"^\s*tolerated\b", re.IGNORECASE) _POINTER_RE = re.compile(r"^\s*(history|interacts|implemented|tracked):") _POINTER_PART_RE = re.compile( r"(history|interacts|implemented|tracked):\s*([^·]+)") @@ -209,6 +221,21 @@ def parse_rules_doc(text: str) -> list[Rule]: if tol: current.tolerated = tol.group("reason") continue + if _NEAR_TOLERATED_RE.match(line): + raise ValueError( + f"{current.rule_id}: line {lineno} looks like a " + f"`tolerated:` marker but does not parse: {stripped!r}. " + f"The spelling is `tolerated: ` -- lowercase, " + f"no space before the colon, and a reason that is not " + f"empty. A near miss fails in the worst direction if it " + f"is allowed through: the rule stays NORMATIVE, so " + f"build_rules_corpus.py harvests its examples into " + f"corpus_rules.jsonl and enforces them at every " + f"released baseline -- the opposite of what the line " + f"was written to say, and green in every suite. The " + f"reason is unvalidated free text but required, because " + f"a demotion nobody had to justify is the one nobody " + f"reviews") if _POINTER_RE.match(line): for key, val in _POINTER_PART_RE.findall(line): items = tuple(v.strip() for v in val.split(",") if v.strip()) diff --git a/tests/v2/test_cases.py b/tests/v2/test_cases.py index 89511b31..f1e74101 100644 --- a/tests/v2/test_cases.py +++ b/tests/v2/test_cases.py @@ -145,6 +145,15 @@ def test_the_family_partitions_into_particles_and_base( dict(text="김민준", shape=6, locale="zh"), "carry neither policy nor locale", id="shape-6-refuses-a-locale"), + pytest.param( + # The other arm of the same `or`. The locale row above passes + # with `self.policy is not None` deleted, so without this one + # a shape-6 row could carry a policy fork -- the refusal's own + # comment says a stray policy on shapes 6/7 would silently do + # nothing, which is exactly why it must not be admitted. + dict(text="김민준", shape=6, policy=Policy(middle_as_family=True)), + "carry neither policy nor locale", + id="shape-6-refuses-a-policy"), pytest.param( dict(text="김민준", shape=6, tolerated=True), "mutually exclusive with shape", diff --git a/tests/v2/test_ledger_guards.py b/tests/v2/test_ledger_guards.py index 5d72a470..127e1467 100644 --- a/tests/v2/test_ledger_guards.py +++ b/tests/v2/test_ledger_guards.py @@ -54,7 +54,7 @@ class declares, which members an alternation offers. Those are exact from ._differential_fixtures import ( _CORPUS_NAMES, _LEDGERS, _TOOLS, _UNCLASSIFIED_NAMES, _claimed, - _exclusions, _rules, _unclassified_names, load_tool) + _entry_name, _exclusions, _rules, _unclassified_names, load_tool) # The one sanctioned divergence between the differential rules' @@ -932,6 +932,24 @@ def test_rules_corpus_matches_the_rules_doc() -> None: f"that never had any") +def _tolerated_only_texts(rules: list) -> set[str]: + """Example texts a `tolerated:` rule carries and no normative rule + does -- the subtraction the demotion guards below both scope by. + + Shared rather than written twice because the two guards must agree + on WHICH texts a demotion is about: one asserts they left the + contract corpus, the other that they are still watched somewhere, + and a pair disagreeing about the set would leave a text neither + enforced nor watched while both stayed green. An example some + normative rule also carries is in the contract on that rule's + account -- demoting one rule cannot take another's name away. + """ + normative = {e.text for r in rules if r.tolerated is None + for e in r.examples if e.text} + return {e.text for r in rules if r.tolerated is not None + for e in r.examples if e.text} - normative + + def test_a_tolerated_rule_puts_no_name_in_the_rules_corpus() -> None: """A rule marked `tolerated:` in rules.md contributes no example to the CONTRACT corpus (2026-09-01, the CJK comma demotion). @@ -944,11 +962,12 @@ def test_a_tolerated_rule_puts_no_name_in_the_rules_corpus() -> None: rule cannot take another's name away, so the check subtracts them rather than failing on them. - What it does NOT check, deliberately: that the demoted names are - still watched somewhere. That is a different promise, and it is - the tolerated case rows plus corpus_cjk_tolerated.jsonl that keep - it -- pinned by test_tolerated_cjk_corpus_matches_the_case_table - above, and measured by the gate's own claims record. + What it does NOT check: that the demoted names are still watched + somewhere. That is the opposite promise and it has its own guard, + the next one down -- this pair is a floor and a ceiling on the + same set of texts, and each is vacuous without the other. Deleting + an example satisfies this one; adding an unwatched one satisfies + that one; only together do they say what a demotion is. """ from .rules_doc import parse_rules_doc rules = parse_rules_doc( @@ -960,14 +979,10 @@ def test_a_tolerated_rule_puts_no_name_in_the_rules_corpus() -> None: "carried one since 2026-09-01. If a demotion was reversed, " "delete this guard in that commit rather than leaving it " "asserting nothing") - normative_texts = {e.text for r in rules if r.tolerated is None - for e in r.examples if e.text} in_corpus = {json.loads(line) for line in (_TOOLS / "corpus_rules.jsonl") .read_text(encoding="utf-8").splitlines()} - leaked = sorted({e.text for r in tolerated for e in r.examples - if e.text and e.text not in normative_texts} - & in_corpus) + leaked = sorted(_tolerated_only_texts(rules) & in_corpus) assert not leaked, ( f"corpus_rules.jsonl carries example texts belonging only to " f"tolerated rules {[r.rule_id for r in tolerated]}: {leaked}. " @@ -976,6 +991,137 @@ def test_a_tolerated_rule_puts_no_name_in_the_rules_corpus() -> None: f"the marker withdrew") +def test_a_tolerated_rules_examples_are_still_watched_somewhere() -> None: + """Every text a tolerated rule alone carries is in SOME corpus the + harness loads, so it is still compared at every released baseline + -- the half of the demotion the marker did not withdraw. + + The failure this closes, verified by injection rather than + reasoned about (the negative control below runs it): a sixth comma + example added to W3 with no `tolerated` row in tests/v2/cases.py. + Nothing in the suite notices. The doc parses, test_rules_doc.py + executes the example and it passes, build_rules_corpus.py skips + the whole rule so the contract pin above stays equal, and + build_cjk_corpus.py projects the CASE TABLE -- it never sees a + name nobody wrote a row for. The text is executed at HEAD and + compared at no baseline, with every suite green. That is exactly + the state the demotion was written to avoid: "we still watch it, + we no longer enforce it" costs the watching, or it costs nothing + and means nothing. + + Membership is asked of the UNION of the corpora rather than of + corpus_cjk_tolerated.jsonl by name, because the union is what this + can honestly promise: the requirement is that the name reaches the + comparison, and which file carries it is the projection's business + (a future tolerated rule outside the CJK sections would have no + business in a CJK file at all). Today every one of them arrives + through corpus_cjk_tolerated.jsonl -- W3's four comma texts, its + fifth example being W1's normative one and subtracted here -- and + that file's equality with the case table is pinned separately by + test_tolerated_cjk_corpus_matches_the_case_table. + """ + from .rules_doc import parse_rules_doc + doc = (_TOOLS.parents[1] / "docs" / "design" / "rules.md").read_text( + encoding="utf-8") + rules = parse_rules_doc(doc) + demoted = _tolerated_only_texts(rules) + # Anti-vacuity, the same shape the guard above carries: with no + # marked rule, or with the subtraction eating everything, the + # assertion below is a truth about the empty set. + assert demoted, ( + "no rule in rules.md carries a `tolerated:` marker with an " + "example text of its own; W3 has carried four since " + "2026-09-01. If a demotion was reversed, delete this guard in " + "that commit rather than leaving it asserting nothing") + unwatched = sorted(demoted - set(_CORPUS_NAMES)) + assert not unwatched, ( + f"tolerated rules in rules.md carry example texts no corpus " + f"holds: {unwatched}. Each is executed at HEAD and compared " + f"at no baseline. Give it a `tolerated=True` row in " + f"tests/v2/cases.py and regenerate " + f"(`uv run python tools/differential/build_cjk_corpus.py`), or " + f"take the example out of the doc") + + # The negative control, run rather than described: the doc with + # one extra example on the tolerated rule -- a text no case row + # produces -- and the same question asked again. It must find it. + # Without this, a subtraction that quietly emptied `demoted` would + # leave the assertion above green forever. + probe = "조, 은우" + assert probe not in _CORPUS_NAMES, ( + f"{probe!r} was chosen because no corpus holds it; a row was " + f"added for it, so pick another string for the injection") + lines = doc.splitlines() + # Anchored on the marker line, not on one of the rule's example + # texts: W3's first example is a text W1 also carries, so a search + # by text lands in W1's block and injects a NORMATIVE example -- + # which the subtraction then removes, and the control passes by + # doing nothing. The marker is in the tolerated rule's block by + # definition. + at = next(i for i, line in enumerate(lines) + if re.match(r"^\s*tolerated:", line)) + lines.insert(at, f' "{probe}" → family="조"') + injected = _tolerated_only_texts(parse_rules_doc("\n".join(lines))) + assert sorted(injected - set(_CORPUS_NAMES)) == [probe] + + +def _corpus_texts(filename: str) -> set[str]: + """One corpus file's names, in whichever line format it uses.""" + return {_entry_name(json.loads(line)) for line in + (_TOOLS / filename).read_text(encoding="utf-8").splitlines() + if line.strip()} + + +def test_the_tolerated_corpus_is_disjoint_from_the_contract_ones() -> None: + """No text is in corpus_cjk_tolerated.jsonl and in a CONTRACT + corpus at the same time. + + Overlap does not error anywhere -- it reads as a demotion that did + not happen. compare.py's dedup loads contract files first and + keeps the contract reading, so a text both tiers hold is still + enforced: an unmatched diff on it fails the run, exactly as before + the flag was set. Meanwhile everything that describes it says + otherwise -- the case row says `tolerated`, README's tier table + says radar, the release notes group it as watched-not-enforced. + The name is enforced and documented as demoted, which is the one + combination nobody is reading for. + + _CORPUS_TIERS's own comment states the rule this asserts: a file + entry demotes the file, and a text some contract corpus also holds + reads contract no matter what the entry says. Five texts did on + the day the file was created; the same day's rules.md edits took + all five out of corpus_rules.jsonl. This keeps that true instead + of leaving it a fact about one afternoon. + + Either resolution is deliberate and neither is this test's to + pick: clear the `tolerated` flag (the name was never demoted), or + take the text out of the contract corpus that still holds it (it + was). The tier list is read from compare.py rather than copied, + so a new contract corpus is covered by existing here, not by + someone remembering to add it. + """ + tiers = load_tool("compare")._CORPUS_TIERS + contract = sorted(name for name, tier in tiers.items() + if tier == "contract") + assert len(contract) >= 3, ( + f"_CORPUS_TIERS lists {contract} as contract; corpus_cjk, " + f"corpus_rules and corpus_shapes have been contract since the " + f"#468 tier split. A shorter list means this guard is asking " + f"about fewer files than it reads as") + demoted = _corpus_texts("corpus_cjk_tolerated.jsonl") + assert demoted, "corpus_cjk_tolerated.jsonl is empty" + for filename in contract: + overlap = sorted(demoted & _corpus_texts(filename)) + assert not overlap, ( + f"{filename} (contract) and corpus_cjk_tolerated.jsonl " + f"(radar) both hold {overlap}. compare.py's dedup keeps " + f"the contract reading, so these names are still enforced " + f"at released baselines while their case rows, the README " + f"tier table and the release notes all call them demoted. " + f"Clear the `tolerated` flag, or take the text out of " + f"{filename}") + + #: Which vocabulary constant each ledger rule's alternation is a hand #: copy of. A roster rather than an inference: GLUED_HONORIFICS is a #: SUBSET of SUFFIX_WORDS (asserted at the bottom of diff --git a/tests/v2/test_rules_doc_grammar.py b/tests/v2/test_rules_doc_grammar.py index a348b14d..cf16b53d 100644 --- a/tests/v2/test_rules_doc_grammar.py +++ b/tests/v2/test_rules_doc_grammar.py @@ -118,6 +118,48 @@ def test_a_normative_rule_carries_no_tolerated_marker() -> None: assert all(r.tolerated is None for r in parse_rules_doc(DOC)) +#: Spellings an author reaching for `tolerated:` can plausibly write, +#: none of which _TOLERATED_RE accepts. Each is a near miss, and the +#: point of the parametrization is that every one of them RAISES: the +#: silent reading leaves the rule normative, which means +#: build_rules_corpus.py harvests its examples into the contract +#: corpus and enforces at released baselines exactly the claim the +#: line was written to withdraw -- with the whole suite green, since +#: the doc still parses and the examples still pass. +#: +#: The empty-reason row is a decision, not an oversight: the reason is +#: unvalidated free text, but a demotion nobody had to justify is the +#: one nobody reviews, so the validators' house rule (a reason is the +#: whole safeguard) applies here too. +@pytest.mark.parametrize("marker", [ + pytest.param(" Tolerated: capital T, otherwise well formed.", + id="capitalized-marker"), + pytest.param(" tolerated : a space before the colon.", + id="space-before-the-colon"), + pytest.param(" tolerated:", id="no-reason-at-all"), + pytest.param(" tolerated: ", id="whitespace-only-reason"), + pytest.param(" toleraTED: shouting the middle.", + id="mixed-case-marker"), +]) +def test_a_near_miss_tolerated_marker_is_an_error(marker: str) -> None: + doc = ('X1. Statement the parser only describes.\n' + ' "Smith, John" → family="Smith"\n' + f'{marker}\n' + ' no-boundary: illustrative, not normative.\n') + with pytest.raises(ValueError, match="X1"): + parse_rules_doc(doc) + + +def test_the_near_miss_lint_leaves_the_real_marker_alone() -> None: + """The negative control for the battery above: the accepted + spelling still parses to a reason, so the lint is rejecting near + misses rather than the marker itself.""" + doc = ('X1. Statement the parser only describes.\n' + ' "Smith, John" → family="Smith"\n' + ' tolerated: reason.\n') + assert parse_rules_doc(doc)[0].tolerated == "reason." + + def test_registry_resolves_policies_and_gates() -> None: from tests.v2.rules_doc import resolve_annotation kind, obj = resolve_annotation("family-first") diff --git a/tools/differential/README.md b/tools/differential/README.md index 53e557b2..7c4e76a6 100644 --- a/tools/differential/README.md +++ b/tools/differential/README.md @@ -203,10 +203,11 @@ Since the v2.3 tier split (#468), a corpus is CONTRACT or RADAR -- the roster is `_CORPUS_TIERS` in compare.py, fail-closed like the floors. Contract corpora hold names someone chose, and an unmatched diff on one is UNEXPLAINED and fails the run. Radar corpora hold the -scraped and harvested names: their diffs still classify against the -ledger, so intended changes keep their release-note grouping, but an -unmatched radar diff prints under UNCLASSIFIED (radar) and cannot -fail the run or demand a ledger rule. Nothing is deleted to keep the +names the contract does not answer for -- the scraped and harvested +ones, and since 2026-09-01 the deliberately demoted ones too: their +diffs still classify against the ledger, so intended changes keep +their release-note grouping, but an unmatched radar diff prints under +UNCLASSIFIED (radar) and cannot fail the run or demand a ledger rule. Nothing is deleted to keep the gate quiet -- a meaningless string in radar costs one parse and a report line. To promote a radar name, give it a tests/v2/cases.py row and a shape tag: it enters the contract by being chosen. A diff --git a/tools/differential/expected_since_1.4.0.toml b/tools/differential/expected_since_1.4.0.toml index a7d4be43..e35d704b 100644 --- a/tools/differential/expected_since_1.4.0.toml +++ b/tools/differential/expected_since_1.4.0.toml @@ -67,6 +67,15 @@ issue = "fix(#271/#272/#298) native-script CJK: family-first order, hangul segme # two corpora have by construction: v1's banks had no reason to test # CJK, and build_issues_corpus.py requires an internal space, which # unspaced names never have. +# +# Since 2026-09-01 that one harvest writes TWO files, not one: rows +# carrying `tolerated` go to corpus_cjk_tolerated.jsonl and the rest +# to corpus_cjk.jsonl. The never-dormant conclusion above survives +# the split unchanged, because both files load in every real run -- +# what the split changed is the TIER (corpus_cjk_tolerated.jsonl is +# radar, so an unmatched diff on one of its names reports instead of +# failing), and a rule fires on names it is given regardless of what +# an unmatched diff would then cost. name_regex = "[\\u3005-\\u3006\\u3040-\\u309F\\u30A0-\\u30FF\\u3400-\\u4DBF\\u4E00-\\u9FFF\\uF900-\\uFAFF\\uAC00-\\uD7A3\\uFF65-\\uFF65]" fields = ["given", "middle", "family"]