[derek73]

Adds local_corpora.py pairs SOURCE, the follow-up planned in #841. It reads a paired source (tse-donors, whose records carry the tax registry's name beside the filed one) as two spellings of one person.

What it does

• Shapes: each pair takes the first of 15 shapes it fits. In order: the same, no reference name, no filed name, case only, accents only (case unchanged), case and accents, punctuation or spacing only (words run together included), the same words reordered, words left out, words added, words shortened, words misspelled (at most two words, difflib ratio ≥ 0.75), words left out and shortened, mostly the same words, few words in common. pair_shape is a pure function with one invented pair per shape and rows at each edge.
• Words: it lists the words most often left out and added. Each pair counts a word once, and a word shared by fewer than 10 pairs (PAIR_WORD_FLOOR) is not listed, since it could be one family's.
• --agreement: parses both spellings of each pair with a reference name that differs and counts, per shape, the pairs whose readings end the family name in the same word (surname()). Two readings with no family name share none.
  • It also lists the accents-only pairs read with two surnames, by the two spellings' last words, under the same floor.
  • It parses each distinct name once; 39s for tse-donors on 2026-10-22.
• Refusal: an empty corpus, or a record with no reference name, exits with a message.

Findings on tse-donors (fetch of 2026-10-22)

205,925 of 854,531 pairs differ:

shape	pairs	share	one surname
words left out	45,382	22.0%	44%
accents only	43,552	21.1%	97%
words shortened	32,447	15.8%	85%
words misspelled	30,704	14.9%	71%
words left out and shortened	24,307	11.8%	63%
mostly the same words	11,906	5.8%	36%
few words in common	6,338	3.1%	31%
words added	5,024	2.4%	36%
punctuation or spacing only	3,523	1.7%	85%
case only	2,218	1.1%	100%
the same words reordered	352	0.2%	47%
case and accents	172	0.1%	97%

• Words left out: most often DE (left out of 13,412 pairs), DA (5,906) and DOS (3,033), then second surnames such as SILVA (2,869).
• Reading the agreement: read it per shape. A pair that leaves out the last surname ends the family name in another word for a real reason, which is likely most of why words left out agrees in only 44%. That cause has not been measured apart from the others.
• Accents only cannot change a surname, so its 1,304 disagreements are the parser reading one name two ways. The command lists them by the two spellings' last words:
  • JÚNIOR / JUNIOR in 866 pairs, which is Should José Silva Júnior read Júnior as a suffix, as José Silva Junior does, and where do Filho, Neto and Sobrinho belong? #843.
  • SÁ / SA in 255 pairs, a new finding (now also a comment on Should José Silva Júnior read Júnior as a suffix, as José Silva Junior does, and where do Filho, Neto and Sobrinho belong? #843). sa is a listed credential with no may-be-a-name marking, so MARIA DE SA reads family DE, suffix SA, where MARIA DE SÁ reads family DE SÁ. JOAO SA reads no family at all.

Docs

• tools/differential/README.md records the command and the breakdown.
• docs/design/mechanisms.md#NAME-DATA-JOBS records the reusable move: a source that pairs two spellings of one person scores without fields, and accents-only disagreements isolate the parser's own inconsistency.

Review

One round with three reviewers (code, tests, comments), addressed in 93a1999:

• Two readings with no family name had counted as agreeing.
• Words left out had counted per occurrence, not per pair, against what the report and the floor say.
• "Accents only" had held pairs whose case differed too.
• The JÚNIOR and SÁ figures had no recipe; the command now prints them.
• Twelve missing boundary rows.

Verification

• 299 tests in tests/v2/test_local_corpora.py.
• 32 branch mutations, each failing a test.
• ruff, mypy and the full suite are green, as are the module's tests on 3.15.

The branch's first commit was squashed onto a moved master and reverted #846; the second restores those files, and the PR's diff against master is only the four files above.

No parser behavior or vocabulary changes.

🤖 Generated with Claude Code

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 99.05%. Comparing base (5a11532) to head (93a1999).

Additional details and impacted files

@@           Coverage Diff           @@
##           master     #852   +/-   ##
=======================================
  Coverage   99.05%   99.05%           
=======================================
  Files          47       47           
  Lines        4669     4669           
=======================================
  Hits         4625     4625           
  Misses         44       44           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.