Merge pull request #455 from derek73/claude/452-declared-reach

Fail the gate when a ledger rule declares a role it never explains

Author: derek73

AGENTS.md

@@ -99,7 +99,10 @@ uv run sphinx-build -b html docs dist/docs
 # 1. Run the differential gate at EVERY baseline that has a ledger with rules,
 #    and confirm each exits 0. The dynamic check only runs for the ledger
 #    matching the baseline being run, so a ledger left out here gets no
-#    dynamic dormancy check at all -- see tools/differential/README.md:
+#    dynamic dormancy check AND no over-declaration check (#452) at all
+#    -- and neither is hypothetical: #452 found 3, 5 and 6 stale
+#    declarations sitting in the three files. See
+#    tools/differential/README.md:
 #    uv run python tools/differential/compare.py --baseline 1.4.0   # the v1 compat contract
 #    uv run python tools/differential/compare.py                    # the previous minor
 #    uv run python tools/differential/compare.py --baseline 2.0.0   # 2.0.0's ledger has rules too

docs/design/decisions.md

@@ -629,14 +629,17 @@ Decisions that landed:
 - 2026-08-27 #451 — `validate_rules` REJECTS a rule carrying `fields` and no `name_regex`. Such a rule claims every name whose diff fits those fields, and `_CORPUS_CLAIMS` records a regexless rule's reach as the WHOLE corpus — already at its maximum, so arrivals never move it and `test_every_rule_claims_the_recorded_share_of_the_corpus` passes whether it explains 4 names or 40. The one rule with the shape drifted from the four names its prose described to 25 before #372 cut it back, and then to SEVENTEEN after — decisions.md#H1 records that 14 -> 17 step and #410 taking three back — standing at fourteen across six unrelated behaviour families when this bundle opened, every guard green throughout, absorbing two open v2.2 bugs (#342, #397) as intentional 2.0 changes while AGENTS.md names the 1.4 summary as the release log's source for Behavior Changes. Free to state only because the fourteen were given rules first; a role-count floor was the earlier proposal and stays declined (#372/#373, vacuous).
 - 2026-08-27 #451 — a rule whose declared `fields` EXCEED every diff it explains is a defect in its own right, not untidiness. Measured on the catch-all: it declared `{given, family, suffix}`, and by the time this bundle's first three commits had given its `{given, family}` names rules of their own, nothing it still explained moved `given`. Note WHEN the excess appeared, because it is the whole mechanism — the declaration was earned when the rule was written and became excess as its explained set SHRANK under it, which is exactly `fix(#424)` at decisions.md#H1, where #410 narrowed a diff from three roles to two and the rule kept matching by superset with no run naming it. A rule does not have to widen to become over-declared; the diffs beneath it only have to leave. The two together are why #452 was filed — nothing counts what a rule actually explains, only whether it explains anything.
 - 2026-08-27 #451 — a ledger rule may classify a reading NOBODY WANTS, provided its `issue` says so. `fix(#342)` and `fix(#397)` carry NOT WANTED in the issue text, the cause-not-consent sentence decisions.md#P5 established for the #369 regression, and a delete-when-fixed instruction the dormancy check enforces once the fix lands.
-- 2026-08-27 #451 — one vocabulary per alternation, FORCED rather than chosen. `_LatinCopy` carries one `vocabulary` and `test_latin_alternations_mean_something_the_vocabulary_ships` asserts exactly one roster key per alternation, so `jr` (SUFFIX_WORDS) and `mc`/`mp` (SUFFIX_ACRONYMS) cannot share one. That constraint forces ONE of the three splits. The other two are forced separately and are recorded below under the declined single-rule tail: `m\.?a\.?` cannot be an alternation member at all, and the numeral alternation copies no wordlist. Four rules, three independent reasons, none of them taste.
-- 2026-08-27 #451 — a ledger rule's character class must be a copy of the range the IMPLEMENTATION acts on, and is pinned to it. The emoji rule's first draft spanned `U+1F300-U+1FAFF` while `_tokenize.py`'s `_EMOJI_RANGES` divides on two narrower blocks: it claimed 1072 codepoints where the parser leaves the token whole, so its own prose ("an emoji inside a token divides it") was false for every one of them and a `{given, family}` diff with another cause would have classified as intended. `test_the_emoji_boundary_rule_copies_the_dividing_ranges` holds it, and fails on the draft it replaced.
+- 2026-08-27 #451 — one vocabulary per alternation, FORCED rather than chosen. `_LatinCopy` carries one `vocabulary` and `test_latin_alternations_mean_something_the_vocabulary_ships` asserts exactly one roster key per alternation, so `jr` (SUFFIX_WORDS) and `mc`/`mp` (SUFFIX_ACRONYMS) cannot share one. That constraint forces one of the three splits, and forces it only given the roster as written: `_LatinCopy.vocabulary` is a hand-supplied frozenset, so declaring `SUFFIX_WORDS | SUFFIX_ACRONYMS` would let `(jr\.?|v|mp|mc)` pass the member checks — measured. Declining to write a union vocabulary is a judgement, that a rule's alternation should name one wordlist a reader can go and check, and it is recorded here as one rather than as a constraint. The other two are forced separately and are recorded below under the declined single-rule tail: `m\.?a\.?` cannot be an alternation member at all, and the numeral alternation copies no wordlist. Four rules, three independent reasons, none of them taste.
+- 2026-08-27 #451 — a ledger rule's character class must not reach a codepoint the IMPLEMENTATION does not act on, and is pinned to that. A SUBSET, not a copy: `test_the_emoji_boundary_rule_copies_the_dividing_ranges` asserts `claimed - divides == set()` and nothing in the other direction, deliberately — `_EMOJI_RANGES` has FOUR blocks and the rule copies the two astral ones, leaving the BMP pair out because no corpus name reaches them through its token anchor. Reading it as equality would instruct the next author to pre-claim the BMP half, which is the #372 shape this same section leans on. The emoji rule's first draft spanned `U+1F300-U+1FAFF` while the two ASTRAL blocks of `_tokenize.py`'s `_EMOJI_RANGES` are narrower than that span: it claimed 1072 codepoints where the parser leaves the token whole, so its own prose ("an emoji inside a token divides it") was false for every one of them and a `{given, family}` diff with another cause would have classified as intended. `test_the_emoji_boundary_rule_copies_the_dividing_ranges` holds it, and fails on the draft it replaced.
 - 2026-08-27 #451 — `_sorted_rules` is KEPT although the ban makes it the identity on every ledger that loads. Verified against all three: the ban removes the only shape that could occupy its second tier, so file order settles every tie there is and the function reorders nothing. Kept because it is what makes the ban safe to state for any reader that does not call `validate_rules` first — a future tool, a REPL, a test fixture. Deleting it would move the guarantee from the code into a convention, which is the trade #451 was filed to undo. Recorded as a decision rather than as a consequence precisely so a later dead-code sweep finds the reasoning before the `return`.
 - 2026-08-27 #451 — past-tense counts stay in a ledger comment; live ones do not. "It claimed 25 until #372" is a fixed fact. "Four corpus names, and the prose fits all four" was written when the rule explained four and still read that way at fourteen, because nothing recomputes it. Where a count is owned by another comment, point at the owner rather than restating it — two copies means one gets updated and the other does not.
+- 2026-08-28 #452 — a rule's declared `fields` must EQUAL the union of the diffs it explains, checked by `compare.py` at the end of every run and failing it like an unexplained diff. The statement is exact rather than heuristic, which is what makes it cheap: `classify()` already requires `declared >= union` for the rule to match the names it matches, so the only possible error is the other direction, and the union is simultaneously the check and the repair. Narrowing to it cannot orphan a name, since every name a rule explains contributed to it. Measured before landing: 3 of 67 explaining rules over-declared at 1.4.0, 5 of 58 at 2.0.0, 6 of 51 at 2.1.0 — all fourteen narrowed first, so the check was silent the day it arrived.
+- 2026-08-28 #452 — over-declaration is BASELINE-RELATIVE, and each ledger is measured on its own run rather than copied. `fix(#296) a lone post-comma credential is a suffix` declared `{family, given, suffix, title}` in all three ledgers: exactly exercised at 1.4.0, where v1 reads the pre-comma word as `first` and all four roles move, and over-declared at both 2.x baselines, where the same behaviour moves only `{suffix, title}` — which is all it declares in those two files now. A reader comparing the three sees one rule with TWO different field lists and should read that as correct rather than as drift. Two and not three, measured: the 2.x pair narrowed to the same set, so the split is 1.4.0 against both 2.x ledgers, not one list per file.
+- 2026-08-28 #452 — NO escape hatch, decided rather than deferred. `dormant` already covers the explains-nothing case in both directions, a rule with no `fields` has nothing to over-declare, and the ledger's own doctrine — "a rule that pre-claims shapes it has never seen is the #372 failure mode" — makes strictness the existing principle. Accepted cost, stated so it is not rediscovered as a surprise: the first rule that genuinely needs a wider declaration has to argue for a key the way `dormant` was argued for in #373, rather than reaching for one that already exists.
 
 Found rather than decided, and worth as much:
 
-- **A fields-only rule silently falsifies OTHER comments.** The 2.x ledgers' idiom "such a name should arrive UNEXPLAINED and be read once" is true there because they have no fields-only rule. In the 1.4.0 ledger it was false for anything of the catch-all's shape, and nothing said so. Measured while the rule still lived, by driving `compare.classify()` directly over the seven `_MUST_NOT_MATCH` probes of the two rules #451 ported, at diff shape `{given, family}`: ZERO arrived unexplained. Four went to the catch-all ('Mcintyre Smith Jr.', 'Los Santos', 'John Sir de Mesnil', 'Sir Smith'), two to `fix(comma-precomma-family)` ('McDonald, Ronald', 'Smith, Sir de Vaux'), and one ('Ste Marie') to the neighbouring `fix(#360) ste` rule that exists to claim it. Driven through `classify()` rather than read off a gate run, deliberately: six of the seven are in no corpus, so a run would never reach them — which is also why the promise they carried could go false unnoticed. The blast radius of the shape was never just its own heading.
+- **A fields-only rule silently falsifies OTHER comments.** The 2.x ledgers' idiom "such a name should arrive UNEXPLAINED and be read once" holds there because no rule above the one making the promise claims that diff shape. A fields-only rule is the extreme way to break it, not the only way: measured while fixing #452, a `{family, maiden}` diff on `fix(#367)`'s own probe goes to `fix(#274)` in the 1.4.0 ledger, which has no fields-only rule anywhere. Note the shape, not the role: `{given, family, maiden}` on the same probe is claimed by nothing, because `fix(#274)`'s `fields` stop at maiden/middle/family. That is the point — the question is only ever answerable per shape. The reusable check is "does anything above me claim this shape", which is `classify()` driven directly — NOT "is there a catch-all". In the 1.4.0 ledger it was false for anything of the catch-all's shape, and nothing said so. Measured while the rule still lived, by driving `compare.classify()` directly over the seven `_MUST_NOT_MATCH` probes of the two rules #451 ported, at diff shape `{given, family}`: ZERO arrived unexplained. Four went to the catch-all ('Mcintyre Smith Jr.', 'Los Santos', 'John Sir de Mesnil', 'Sir Smith'), two to `fix(comma-precomma-family)` ('McDonald, Ronald', 'Smith, Sir de Vaux'), and one ('Ste Marie') to the neighbouring `fix(#360) ste` rule that exists to claim it. Driven through `classify()` rather than read off a gate run, deliberately: six of the seven are in no corpus, so a run would never reach them — which is also why the promise they carried could go false unnoticed. The blast radius of the shape was never just its own heading.
 - **Copy-paste is how a ledger comment goes false without being edited.** All three ledgers carried the same `fix(#379)` comment; four of its sentences were true only in the 1.4.0 file they were written for, including one citing `fix(suffix-routing)`, which exists in no 2.x ledger, and one saying a name "falls to the fields-only catch-all" twenty lines after the same file says no fields-only rule exists there. Corrected in `8237b54`. A cross-ledger claim needs re-deriving per file, not copying.
 
 Declined:
@@ -645,6 +648,7 @@ Declined:
 - One replacement rule with a `(jr|v|mp|x|m\.?a\.?|mc)` tail (2026-08-27) — measured, not argued: `m\.?a\.?` matches the corpus fragment `'M.A.'`, which `_normalize` leaves as `'m.a'` and is not a `SUFFIX_ACRONYMS` entry, so the member guard rejects it — the same finding the `ambiguous-surname-acronym` rule already records. `x` is in no vocabulary set at all, reaching `suffix` through the trailing-roman-numeral fork, so it could not be a member either.
 - A literal alternation of all 14 absorbed names (2026-08-27), on #413's literal-anchored precedent — kept as the fallback if a reading turned out contested, and not needed: every family had an existing rule in another ledger, a decision to cite, or an open issue.
 - Extending #451 to the trailing-numeral ambiguity (2026-08-27) — `'Mohamad X'` loses its family to `_ROMAN`, and by C-i's "under uncertainty, default to AMBIGUOUS" a trailing `X` IS borne as an ordinary name in that position (Malcolm X). Numerals reach `suffix` through a pattern rather than a vocabulary set, so there is no ambiguous SUBSET for `x` to join — but the fork does not go unreported: measured, `parse('Mohamad X')` and `parse('John V')` both carry `AmbiguityKind.SUFFIX_OR_NAME`, which `_assign` raises from the fork itself because "V/X/I are ordinary middle initials, so taking it as a suffix is a call, not a fact". The contrast worth keeping is with the acronym half: `parse('Aishwarya Rai')` loses its family and reports NOTHING, because `rai` is unambiguous vocabulary. So the numeral fork already flags its own uncertainty and the vocabulary path does not — which is an argument about `SUFFIX_ACRONYMS_AMBIGUOUS`, not about `_ROMAN`, and it belongs to #342 rather than here. `'Malcolm X'` is in no corpus (`'Mohamad X'` is, and is what this bundle classified); no open issue on the numeral half.
+- `explains = N` per rule (2026-08-28, #452's own first sketch) — catches a rule explaining fewer NAMES even while its fields stay exercised, which the field check does not. Declined on maintenance cost rather than merit: 179 numbers across the three ledgers, every one of them liable to move whenever a corpus does rather than only when a rule does. Measured on the precedent — #414's harvest moved 17 of the 30 `_CORPUS_CLAIMS` entries that existed then and added 14 more, parsing the roster at `7a10689^` and `7a10689`. (That roster's own comment said "Twenty of these moved" and was wrong; measured three ways — that commit, the whole of PR #415, and the backtick-harvest commit after it — the figure is 17 every time, and the comment now says so with the recipe.) The field check needs no number and cannot go stale. This is the design to revisit if a name-count shrink ever bites.
 - Extending it to `'Donald mc'` (2026-08-27) — `mc` is `SUFFIX_ACRONYMS` AND `PARTICLES`, the collision C-ii decided for `vd`, and C-ii reaches this shape only through P6's attachment, which is scoped to the comma form. The authority is rules.md#S2's Accepted — "an unambiguous suffix is consumed even when that leaves no family name at all" — and NOT P6's scope note, which was this entry's first citation and does not support it. P6 promises the comma-less shapes "keep their positional reading", and for the words that are both particle and suffix vocabulary it does not hold: measured, `'Jong de'` and `'Smith do'` keep theirs (family 'de', family 'do') while `'Donald mc'` and `'Smith vd'` lose the family entirely to `suffix`. So `'Donald mc'` is out of #451's scope on S2's statement, and P6's own Accepted clause is falsified by two of the three words it sweeps in — its only example, `"Jong Anke de"`, is `de`-shaped, which is why no test catches it. Recorded here rather than fixed; it is a rules.md defect, not a ledger one.
 
 ### comma-suffix-arc — #291/#296/#316 (2026-07-26 → 2026-08-01)