Recover PDF heading structure for the bill ledger, measured by agreement with the XML twin

.github/workflows/ci.yml

@@ -182,6 +182,7 @@ jobs:
           tests/test_manifest_report_fixtures.py
           tests/test_canonical_baseline.py
           tests/test_pdf_canonical_baseline.py
+          tests/test_pdf_ledger_location.py
           tests/test_pdf_matching_boundary.py
           tests/test_pdf_observation_emission.py
           tests/test_pdf_observation_identity.py

.gitignore

@@ -144,3 +144,6 @@ docs/research/pdf-backend-bakeoff/validation/external-validity/results/x*.json
 dist/
 build/
 *.egg-info/
+
+# Jupyter autosave copies, written next to any notebook that is opened (docs/research/).
+.ipynb_checkpoints/

CONTRIBUTING.md

@@ -299,7 +299,7 @@ What to look at, roughly in priority order:
 
 - **Correctness of the diff itself.** This is the product. Passing tests are necessary, not sufficient: a diff can be green and still wrong. For any change that affects diff output, **run the tool on a real bill and eyeball the report** rather than trusting the suite alone. To check the PDF and XML pipelines against each other, render each to its own HTML file (see [TESTING.md](TESTING.md#comparing-the-two-pipelines-by-eye)).
 - **The risk hotspots**, where a bug does the most damage:
-  - **Parser accuracy** (`src/deltatrack/bill_tree.py`, `src/deltatrack/parsers/`) -- does the bill's structure come through intact? A missing or mis-nested section corrupts everything downstream. See [docs/parser-validation.md](docs/parser-validation.md).
+  - **Parser accuracy** (`src/deltatrack/bill_tree.py`, `src/deltatrack/parsers/`) -- does the bill's structure come through intact? A missing or mis-nested section corrupts everything downstream. See [docs/parser-validation.md](docs/parser-validation.md). For PDF headings, the ledger checks in [TESTING.md](TESTING.md#the-ledger-location-pin-and-when-you-may-regenerate-it) show where each amount moved.
   - **Financial diff** (`src/deltatrack/diff_bill.py` and its financial filtering) -- dollar amounts and their changes must be exact.
   - **The canonical schema contract** (`src/deltatrack/formatters/canonical.py`) -- both pipelines and the renderer depend on it, so a breaking change there ripples everywhere.
 - **Tests for the change.** New behavior should come with a test that would fail without the fix. Judge that by the red-green delta on your own machine rather than by the totals the author reported, and compare like-for-like selections — [TESTING.md](TESTING.md#reading-test-counts) explains what a count does and does not tell you, including which differences are a fail-open signal rather than an environment difference.

TESTING.md

@@ -19,7 +19,7 @@ comparison needs no key and no internet connection.
 
 ## How accuracy is checked
 
-Accuracy is checked in six ways. Each one answers a different question, and
+Accuracy is checked in seven ways. Each one answers a different question, and
 each has limits worth being honest about. There is no single accuracy
 percentage that would be truthful across all of appropriations, so we describe
 what each layer does and does not establish.
@@ -172,6 +172,26 @@ position, so it confirms a passage is present somewhere in the version, not that
 it appears in the right place. It also cannot cover draft bills at all, which have
 no official version to compare against; that is check 4's job.
 
+### 7. Cross-checking *where* the PDF files each dollar amount
+
+Check 5 confirms every dollar amount in the official version turns up somewhere in
+the PDF. This one asks whether it turns up under the right heading. For every bill we
+have in both forms, each amount in the official version is lined up with the same
+amount in the PDF reading, in document order, and the two locations (the headings the
+amount sits under) are compared. Each amount lands in one of a few grades: same place;
+same place under a slightly different label; the right place but with a parent heading
+missing; the right heading under the wrong parent; or filed under a different heading
+entirely. The first three are acceptable, since a heading left out is visible, while a
+wrong one is not. The count in each grade is frozen per bill version, so a change that
+files money under the wrong heading more often fails, and one that improves it has to
+be locked in on purpose.
+
+**Limit:** the official version's own reader is the answer key, and it leaves out some
+headings the print shows (a heading with no text of its own, such as an umbrella agency
+over its accounts). Where the PDF keeps such a heading the two disagree even though the
+PDF is right, so the "wrong parent" grade overstates PDF errors. Draft bills have no
+official version, so this check cannot cover them.
+
 ## Known soft spots
 
 We keep these in the open rather than papering over them:
@@ -324,6 +344,75 @@ Note what the sentinel cannot see. Two corpus-invisible behaviours move zero of
 pairs and are bound only by synthetic fixtures in `tests/test_round1_stages.py`, so "the corpus is
 still green" is not evidence about them.
 
+### The ledger-location pin, and when you may regenerate it
+
+`tests/test_pdf_ledger_location.py` measures where the PDF pipeline files each dollar amount,
+against the XML twin of the same version ([ADR 0022](docs/decisions/0022-pdf-heading-convergence.md)).
+Every amount in the XML ledger is aligned with the same amount in the PDF ledger and its two
+locations are compared level by level, the whole path, not just the account and its parent:
+
+| tier | the PDF path, against the XML's | counted as |
+|---|---|---|
+| `T0` | identical | true hit |
+| `T1` | the same levels in the same order, a label differs (a joined, tail or near-variant name) | tolerated |
+| `T2` | the XML's ancestors in order, with some left out | tolerated |
+| `T3` | an ancestor that is wrong, extra or out of order | not tolerated |
+| `T4` | a different account or section | not tolerated |
+| `MISS` | no PDF amount to pair with | not tolerated |
+
+`tests/test_ledger_location_scorer.py` pins each case on hand-built paths. The tier counts for
+every committed dual-format version (enrolled excluded) are pinned in
+`tests/data/ledger_location_baseline.json`.
+
+The pin is exact in both directions. More not-tolerated amounts (`T3`, `T4`, `MISS`) or fewer
+true hits (`T0`) is a regression. An improvement also fails, so it gets locked in:
+
+```bash
+UPDATE_LEDGER_BASELINE=1 uv run pytest tests/test_pdf_ledger_location.py
+```
+
+#### Two readings: the tier totals, and the per-amount check
+
+The **tier totals** are the formal result. They are what the pin holds, what a heading change is
+optimized toward, and what an ADR reports. The **per-amount check** is informal: it follows each
+amount from one parser to another and lists every amount whose tier got worse, with the XML
+path and the path before and after. It is not pinned and is not a gate. It exists because a
+total hides a regression whenever another amount in the same version improves: one amount
+moving `T2 → T3` and another `T3 → T2` leaves every count as it was. Use it to find and explain
+regressions; report the totals as the result.
+
+Both come from `tests/ledger_location.py`, which prints the totals for whichever parser is
+importable, so the "before" is the base branch's `src/` on `PYTHONPATH`. `--save` writes every
+amount's locations; `--against` compares a later run with them. The comparison grades both runs
+with the scorer that is checked out, so a scorer change never passes for a parser change.
+`--extra bills` adds your locally fetched versions to both readings (they are not pinned):
+
+```bash
+git archive origin/develop src | tar -x -C /tmp/before
+PYTHONPATH=/tmp/before/src uv run python -m tests.ledger_location --save /tmp/before.json   # before
+uv run python -m tests.ledger_location --against /tmp/before.json                          # after
+```
+
+For a change to PDF headings or breadcrumbs, put both in the pull request: the before and after
+totals, and the check's count of amounts better and worse, with each group of worse amounts
+explained or fixed.
+
+#### What the answer key gets wrong
+
+The answer key is DeltaTrack's own XML reader, which leaves heading-only elements (an agency
+heading with no text of its own) out of the breadcrumb, so where the PDF keeps such a heading
+it is graded "wrong parent" although it matches the page. Read `T3` as an upper bound on PDF
+errors. The reference for headings is the raw XML file's heading tags, not the reader
+([ADR 0022](docs/decisions/0022-pdf-heading-convergence.md)): a change to the reader that
+restores them shows up here as fewer `T3`. The same holds for two headings the XML sets on one
+level where the PDF nests one under the other, and for a file whose own tags are misplaced
+(in division G of 117-hr-4502 the `TITLE I` element is empty and its accounts sit in an unnamed
+title after it, while the print nests them correctly). `T1` is lenient by design: a label that is a
+near-variant of the XML's, or ends with the same words, counts as the same place. A fragment
+passes too: a PDF heading read as just `ADMINISTRATION` grades `T1` against `FEDERAL HIGHWAY
+ADMINISTRATION`, and reading the full printed `ADMINISTRATIVE PROVISIONS—FEDERAL HIGHWAY
+ADMINISTRATION` instead (a heading the reader drops) shows in the per-amount check as `T1 → T3`. The XML is read only by these tests, never by the product.
+
 ### The rest of the slow suite runs in CI too
 
 A further CI step runs the remaining slow modules (`CI_SLOW_MODULES` in

docs/bill-structure.md

@@ -218,12 +218,15 @@ Closing the gap (DeltaTrack#54 and beyond — applies to both pipelines):
 
 Re-checked against the corpus 2026-06-27. Glyph size yields **two** bands, not
 three: a body band (~14pt) and a single heading band (~11.2pt small-caps) that holds
-*both* `appropriations-intermediate` and `-small`. Casing does **not** rescue the
-split — an agency header that is title-case in the XML `header`
+*both* `appropriations-intermediate` and `-small`. The *text* does not rescue the
+split: an agency header that is title-case in the XML `header`
 (`Management directorate`) extracts **ALL-CAPS** in heading position
-(`MANAGEMENT DIRECTORATE`); small-caps flattens to caps in PDF text, so it is
-lexically indistinguishable from an all-caps account header. `Line` carries no
-x-position, so indentation/centering is not a usable signal either.
+(`MANAGEMENT DIRECTORATE`), because small capitals flatten to capitals in PDF text.
+The *letters* do. GPO sets an agency in title case in small caps, each word's first
+letter printed larger (11.2 pt over 14.0 pt initials), and an account in even small
+caps. The line's median glyph size hides that; `LineGeom.initial_caps` reads it per
+word ([ADR 0022](decisions/0022-pdf-heading-convergence.md)). A department heading is
+set in capitals at body size throughout (`LineGeom.size_min`/`size_max`).
 
 What that leaves recoverable from PDF-only input, keyed on *what follows* a
 heading-band line:
@@ -233,18 +236,41 @@ heading-band line:
 | Major (department), `appropriations-major` | body-size + uppercase, sits above the band | Yes, with a dedicated detector |
 | Intermediate **carry-over** (one agency over ≥2 accounts) | another heading-band line | **Yes** — today silently dropped |
 | Intermediate **grouping header** (`ADMINISTRATIVE PROVISIONS`) | a `SEC.` line | **Yes** — today mislabeled `account`; the `SEC.` is detectable |
-| Intermediate **prose-leading** (agency → "For necessary expenses…") | appropriation prose | **No** — identical to an account by size, case, and position |
+| Intermediate **prose-leading** (agency → "For necessary expenses…") | appropriation prose | **Partly**: same size and position as an account, but printed agency-style. The case pattern ends the previous agency's reach; the heading itself is still emitted as an account |
 | Account, `appropriations-small` | appropriation prose | Yes |
 
 The unrecoverable row is not an edge case: prose-leading intermediates are the
 **majority** of the agency level in many bills (H.R. 8774 Defense: 59 of 67;
 H.R. 5895 Energy-Water: 37 of 44). Only H.R. 8752 and S. 2625 in the working corpus
 have none, which is why H.R. 8752 alone can look like full agency recovery. So on
-PDF-only input the agency level is recoverable only in its carry-over and
-grouping-header forms; the prose-leading form is not separable from an account by any
-signal currently extracted. On the XML side all three levels are present in the path
-tuples (`display_path` / `match_path`), even though they are dropped as standalone
-nodes.
+PDF-only input the agency level is recovered in its carry-over and grouping-header
+forms; the prose-leading form is recognized only by its case pattern, which is used for
+scope, not to re-label the heading. On the XML side all three levels are present in
+the path tuples (`display_path` / `match_path`), even though they are dropped as
+standalone nodes.
+
+### Reading heading structure across lines (ADR 0022)
+
+The **ledger** of a bill is every dollar amount it contains, each with its location:
+the breadcrumb of headings it sits under. It is the same idea for an appropriations
+account and a reconciliation section, and it is how PDF heading quality is measured:
+for a published bill, each amount's PDF location is compared with its location in the
+XML twin ([ADR 0022](decisions/0022-pdf-heading-convergence.md)). Forms that only show
+across lines:
+
+| Form | Example | How the print shows it |
+|---|---|---|
+| Wrapped name | `SALARIES AND EXPENSES, FOREIGN CLAIMS` / `SETTLEMENT COMMISSION` | letters alike and the upper line full: one heading |
+| Stacked headings | `BUREAU OF PRISONS` / `BUILDINGS AND FACILITIES` | letters differ, or the upper line broke early: two headings |
+| Hanging-indent heading | first line at the margin, the rest at the paragraph indent | one heading, whatever the letters |
+| Department mid-title | `UNITED STATES SECRET SERVICE` after an account's prose | capitals at body size, centered |
+| Umbrella heading | `ATOMIC ENERGY DEFENSE ACTIVITIES` over `NATIONAL NUCLEAR SECURITY ADMINISTRATION` | same style as the heading below it, nothing between: **not separable** (read as one) |
+| Quoted heading | `‘‘CHAPTER 16—…` inside an amendment to another law | inside an open quotation: not a heading of this bill |
+
+The XML reader leaves heading-only elements (an umbrella agency, `FOOD AND DRUG
+ADMINISTRATION` over its accounts, a topical general-provisions heading) out of
+`display_path`, so where the PDF keeps such a heading the two disagree even though the
+PDF matches the print.
 
 ## Why not USLM?
 

docs/decisions/0012-pdf-heading-levels.md

@@ -48,18 +48,27 @@ discarded — so the probe added zero PDFium calls). The findings:
   (split line sums 250–315 pt, wrap sums 354–418 pt), robust to the small constants
   and to the one bill whose column measured 311 pt rather than 339.
 - **No geometric signal separates boundary 1.** A prose-leading agency is a single
-  body-size line followed by prose: there is no second heading line to test for
-  fullness, and x-centering and leading are null as above.
+  heading-band line followed by prose: there is no second heading line to test for
+  fullness, and x-centering and leading are null as above. The spike did not measure
+  the letters themselves: an agency prints in title case in small caps, an account in
+  even small caps ([0022](0022-pdf-heading-convergence.md)).
 
 ## Decision
 
 We will recover the PDF heading levels from **deterministic geometry and structural
 position only**, and accept the one boundary that no such signal separates.
 
-- **Boundary 1 (prose-leading agency vs account): accept the gap.** A lone
-  body-size heading followed by prose is emitted as an account. We do not guess it is
-  an agency. The level is genuinely not present in the print as a distinguishable
-  signal, so inventing one would trade an honest gap for unauditable noise.
+- **Boundary 1 (prose-leading agency vs account): accept the gap, narrowed.** A lone
+  heading followed by prose is emitted as an account; we do not guess it is an agency
+  from its print alone. Its case pattern ([0022](0022-pdf-heading-convergence.md)) ends
+  the reach of an earlier agency, and re-labels the heading only with one more fact: a
+  title-case heading whose text carries no dollar amount, followed by an account-style
+  heading, introduces accounts rather than holding money, and is an agency
+  (`BUREAU OF RECLAMATION`, `GREAT LAKES ST. LAWRENCE SEAWAY DEVELOPMENT CORPORATION`).
+  Measured on 118 PDFs: 183 amounts toward the XML, none away. Title case alone was
+  measured and rejected (265 amounts the wrong way against 8), because account names
+  print title case too. The dollar sign is read as a fact about the text, not as
+  appropriations wording ([0018](0018-text-triggers-are-financial-only.md)).
 - **Boundary 2 (stacked vs wrapped major): split on line-fullness.** The major
   detector splits a post-`TITLE` body-size run at each line-fullness hard break:
   `w_i + space + first_word_width(line_{i+1}) ≤ column_width − slack` means line *i*
@@ -95,8 +104,8 @@ Alternatives considered:
   level fully separated (the 4/12 stacked residue from #105 is retired) and the
   carry-over agency level recovered where it is followed by another heading (#104).
 - Defense-class bills keep a known, documented agency gap: prose-leading agencies
-  surface as accounts. This is recorded as an accepted limitation, not a bug to chase
-  with a heuristic. Where the agency level matters for those bills, it comes from the
+  whose own text carries money surface as accounts. This is recorded as an accepted
+  limitation, not a bug to chase with a heuristic. Where the agency level matters for those bills, it comes from the
   XML pipeline (which the user already supplies for a published bill,
   [0010](0010-pdf-pipeline-pre-publication.md)).
 - The line-fullness rule carries one residual edge, absent from the FY2025 corpus:

docs/decisions/0014-leveled-heading-tree-scope.md

@@ -85,9 +85,9 @@ not an equality it would have to fudge:
 
 | bill (v1→v2) | XML | PDF | gap | attributed cause |
 |--------------|----:|----:|----:|------------------|
-| 118-hr-8752  |  39 |  37 |  −2 | clean; XML sees 2 bare subsections of added SEC.s the PDF folds into the section block (#188) |
+| 118-hr-8752  |  39 |  38 |  −1 | clean; XML sees 2 bare subsections of added SEC.s the PDF folds into the section block (#188); the PDF sees the added SPENDING REDUCTION ACCOUNT heading as its own block ([0022](0022-pdf-heading-convergence.md)) |
 | 118-hr-8774  |  31 |  33 |  +2 | PDF over-segments a few blocks |
-| 117-hr-4502  | 1415 | 1500 | +85 | PDF over-segments a large added block; XML +304 from subsection nodes (#188) |
+| 117-hr-4502  | 1415 | 1456 | +41 | PDF over-segments a large added block; XML +304 from subsection nodes (#188) |
 | 115-hr-5895  | 300 | 336 | +36 | division-collapse + segmentation; XML +54 from subsection nodes (#188) |
 
 Snapshot 2026-06-29; the 117-hr-4502 and 115-hr-5895 PDF rows re-measured
@@ -100,7 +100,10 @@ the #96 PDF recalibration), and the XML↔PDF gap narrows sharply (4502 +389→+
 5895 +90→+36) — much of what was attributed to PDF over-segmentation was
 subsection granularity the XML tree previously lacked. 118-hr-8752's exact parity
 gives way to a −2 by design: XML emits bare subsections the PDF cannot detect
-(the all-subsections scope decision recorded on #188). The live numbers and the bands that gate them are in
+(the all-subsections scope decision recorded on #188). Two PDF rows re-measured
+2026-09-27 after the heading passes of [0022](0022-pdf-heading-convergence.md): 4502
+1500→1456 (wrapped account names now one block), 8752 37→38 (a heading the PDF
+now reads). The live numbers and the bands that gate them are in
 `tests/test_pipeline_parity.py` (regenerate the table:
 `.venv/bin/python scripts/parity_table.py`). The Senate #89
 residual `118-s-4795` recovers its heading hierarchy at a size-band ratio of 1.02