fix(pdf): rejoin every printed word break, deciding the hyphen from the document (#650)

[willhea]

Closes #650. Reviewed against the amended contract in that issue.

What this fixes

A word GPO's printer broke across a line reached full_text broken. _merge_print_lines required .islower() on the continuation, so INTEL- / LIGENCE and McKinney- / Vento were never joined at all.

Measured against each version's own XML, the defect is larger than the uppercase case it was filed for:

	sites	today
Uppercase continuation	1,031	never joined, the filed defect
Lowercase continuation, compound hyphen	423	joined but welded: project-based → projectbased
Break at a page seam	1,542	never joined; _merge_print_lines runs per page

Whether the hyphen belongs to the word is not decidable at the break, but it is decidable from how the rest of the document spells that word.

Measured with the shipped aligned oracle and the gate's own accounting (_account), evaluating develop's parser and this branch's over the same population:

	judged	undecided	wrong	of which left unjoined
today	50,785	224	3,027	2,611
this branch	50,785	224	165	0
fixed			2,862

Split sites are enumerated from the printed lines, upstream of any join, using #650's definition (a line ending in an alphanumeric, or the period of an abbreviation compound, then a hyphen, including page-seam breaks continued by running-header chrome). The oracle then judges each site or counts it undecided, whether or not the merger joined it. So the population and the judged/undecided split depend only on the printed lines and the XML, not on the parser, and the two rows are measured over the identical 50,785 judged sites.

The 165 decomposes exactly: 165 occurrences over 159 distinct per-version residual keys (version, left, right), which is the residual fixture row-for-row, and zero judged sites left unjoined. Those 159 keys cover 96 globally distinct fragment pairs, since the same pair recurs across versions.

An earlier revision of this PR claimed 2,980 → 118 with 1,336 fixed. Those came from the vocabulary oracle the contract rejects and are withdrawn, not restated: the numbers above are re-derived on the population every other figure here is now stated over.

All six named examples reflow correctly, each against its own version's XML: INTELLIGENCE, McKinney-Vento, E-Verify, Non-Dedicated, Climate-Related, Child-Rescue.

Three false greens found in review, now fixed

The first version of this PR passed its own gate while being wrong in three ways. Each fix is pinned by a negative control, and each control was confirmed to go red when the mistake is reintroduced.

1. The repair joined broken words to running-header chrome. PDFium floats H. R. 3547—61 to the top of the next page, where it lands between a page-seam break and its continuation. It starts with an alphanumeric, so the guard let it through, producing evidence-H., project-H., Hart-Scott-H. — and all 16 were then blessed in the residual fixture, so the gate certified them. In the corpus this shape outnumbers the genuine cross-page uppercase breaks 16 to 5, so the dominant cross-page case was corrupted while the named examples passed.

2. Pooling let one version overrule the other about its own text. Both sides were merged into one index and the majority won. A version writing Non-Dedicated once and never NonDedicated was rendered NonDedicated when the other version wrote the closed form more often — corrupting a version that was never ambiguous, and erasing a real spelling change so the diff reports no change where there is one. Evidence is now ordered tiers: own text first, sibling only where silent, then the fallback.

The earlier "0 cross-version disagreements after pooling" measurement was vacuous. Pooling applies identical evidence to both sides, so agreement was guaranteed by construction. It measured a tautology and was reported as a safety property.

3. The gate used the vocabulary oracle the contract rejects. It asked whether a produced form appears anywhere in the version's XML, which certifies a reconstruction from an unrelated occurrence and cannot choose at all when a bill uses both spellings. XmlOracle now matches against PDF neighbours, trigram first then bigram, comparing both candidates within a tier before falling to the next. Writing that control caught a further bug in my own oracle: mixing tiers reads a bigram for one candidate as competing with a trigram for the other. Sites the oracle cannot decide are now counted per version and asserted, so silent growth of the uncovered set is itself a failure.

Scope

The join_points schema 2.1 change and the in-browser search rewrite have been removed from this PR and preserved on issue653-join-points-wip for a separate PR against #653. The amended #650 states search is out of scope: it reads the print-faithful DOM and normalizes independently, so repairing full_text does not change it, and the two fail differently.

The net diff against the base now touches no file under schema/ or src/deltatrack/formatters/. schema_version stays 2.0.

Verification

• New gate tests/test_pdf_word_break_recall.py, the only hyphen-sensitive check in the suite. normalize_for_cross_format deletes hyphens before matching and says so; normalize_for_recall rewrites Child- Rescue back at compare time. Both are right for a recall question and both make this defect invisible.
• Clause 1 is oracle-driven, asked as "would joining these fragments make a word the bill has in that position", so it cannot pass by restating the merger's own rule.
• Residuals asserted by set equality, not a ceiling: 159 per-version keys, plus 224 undecided sites asserted per version across 52 versions. A missing budget entry fails rather than disabling the check for that version, and a companion test pins the budget's key set to the collected versions. A join at a printed line outside the site population also fails, so no join can escape the count.
• Red→green proven by mutation on the original defect, and five controls proven red on theirs: chrome join, pooled majority, vocabulary oracle, deleted budget key, and an undecidable unjoined site added to a passing input.
• All five gates run locally on d482576a: ruff check . and ruff format --check . (255 files), fast tier 2,011 passed, browser tier 41 passed, slow tier (--deselect tests/test_govinfo_corpus_parity.py) 1,807 passed. See Base below for hosted status.

Review repairs at d482576a

Two findings from the review of 8415084c, one commit each.

1. Abbreviation compounds lost their hyphen (e59f7963). _is_break_tail accepted U.S.- as a break, but _BREAK_TAIL required an alphanumeric before the hyphen, so _break_left("U.S.-") returned an empty fragment. The lookup then missed the document's own U.S.-FSM and produced U.S.FSM. The fragment pattern now accepts the same terminal period.

• A new parameterized test, U.S.- / based → U.S.-based and U.S.- / FSM → U.S.-FSM, supplies the unbroken spelling as own-document evidence. It was red on the pre-fix pattern (U.S.based, U.S.FSM) and is green now.
• 8 residual keys resolve. Each was checked against its version's XML, which writes the hyphenated form: U.S.-FSM, U.S.-RMI, U.S.-Palau / U.S.-PALAU, U.S.-Cuba, U.S.-Israeli.
• One canonical baseline pair moves, 118-hr-4366 4→5: identical change count and summary, +27 bytes. Its canonical output was diffed with and without the fix, and the whole difference is 27 hyphens restored after U.S., with no other text edit.
• The round-1 sentinel moves only parser_revision. No pairing-stream digest changed on any of the 27 pairs.

2. Undecidable sites the merger did not join sat outside the budget (d482576a). The budget counted only among _joins, and _unjoined skipped undecidable candidates without counting them. _account now enumerates sites from the printed lines first and records each one as judged or undecided, joined or not. scripts/regen_word_break_residuals.py calls the same function.

• Judged sites are unchanged at 50,785. Undecided goes from 53 to 224, and each of the 171 added sites is an unjoined candidate: 165 running-header chrome, 5 non-alphanumeric continuations (enumeration dash, suspended hyphen), 1 end of document. No residual moved as a result of this change.
• New negative control: adding an undecidable, unjoined chrome boundary to a passing input fails the unchanged budget. It went red under the mutation that counts undecided sites only among joins.

ADR 0020 evidence

Of 23 pinned pairs, 6 changed and all 6 emit fewer changes (−32 total, almost entirely spurious added); none emits more. That is the #31 defect class clearing. One round-1 revocation moved with it, 115-hr-5895/3→4 (72 → 71); the other 34 pairs are identical under both extractors, and test_pdf_round1_revocation's pinned figure is updated 224 → 223 with that attribution recorded.

The U.S.- repair in this round moves one of those six pairs again, 118-hr-4366 4→5, by text only: its change count (1,272) and summary are unchanged, so the six-pair, −32 figure stands.

Known limits

• 159 per-version residual keys still reconstruct wrongly (165 occurrences of 50,785 judged, 0.32%; 96 globally distinct fragment pairs). These counts come from the gate, which runs the single-document path and does not consult sibling evidence, so they are an upper bound on what a comparison's reader sees. Two causes, and an earlier revision of this text wrongly said all were the first:
  • 148 keys fall to the case-shape rule: a lowercase-continuation compound with no evidence in the document (government- / driven).
  • 11 keys are decided by the document's own evidence, where the bill uses both spellings and its majority form differs from the one at this position (118-hr-4366 writes multifamily 20 times and multi-family 7; this site's XML has multi-family). The fixture's per-row reason string still says "decided by case shape" for these 11; see follow-ups.
• The case-shape rule's observed error of 5.9% (2,001 sites, 118 wrong) is carried over from the earlier measurement and was not re-derived in this round. A probe with this round's site definition does not reproduce its totals, so the population definition behind 2,001 differs. What the probe does show is the effect of the U.S.- repair on it: 2 fewer fallback sites, no change in case-shape errors. It fails single-directionally, never wrongly keeping a hyphen. Draft PDFs are unmeasured; 5.9% is not a prediction for them.
• Page-seam breaks continued by running-header chrome stay split: H. R. 3547—61 on enrolled prints, † HR 4366 EAS on Senate engrossed amendments. That is Running page header/footer is read as bill text and splices into the middle of a word #535, untouched here. The oracle cannot judge them because the chrome token is not a word, and they are now counted in the undecided budget rather than skipped: 165 of the 224 undecided sites. That is why they sit outside the "zero left unjoined" result above rather than contradicting it.

Base and hosted status

origin/develop has been merged into this branch and pushed. GitHub now reports the
pull request mergeable; the three conflicts were confined to derived artifacts
(examples/hr8752_pdf_diff.html, tests/data/pdf_canonical_baseline.json,
web/webapp/sample/example.html) and were resolved by regenerating them on the new
base rather than hand-merged. No non-derived implementation code conflicted:
develop changed nothing under src/deltatrack/parsers/ or src/deltatrack/compare/,
which is the code this branch touches.

Two things to know when reading the artifact diff:

• develop reworked src/deltatrack/formatters/ in the same span, so the regenerated
  examples and canonical baseline carry those changes as well as this branch's. That
  movement is not word-break drift.
• The round-1 sentinel's parser_revision fields move whenever the parser's bytes
  change, including for comments and formatting. Every regeneration on this branch has
  moved those fields and no pairing stream, which is what makes those edits
  demonstrably behaviour-neutral.

Hosted CI is green on d482576a: all 27 checks, including corpus-gates and remaining-slow on 3.12.0, 3.12, 3.13 and 3.14. The per-tier test counts quoted above are from the local run on the same commit.

🤖 Generated with Claude Code