Show every heading the XML tags in its breadcrumb, and measure the ledger on each kind of bill that appropriates

[mattzamora]

Related issue

Closes #733
Closes #521

Depends on #736, which depends on #734. This branch is built on #736's branch, so its first six commits are those two pull requests'; review from the seventh. It will be rebased onto develop as they merge.

Related: #557 (a display-only change to a heading can alter the structure tree, and no test would catch it), #474 (an account whose name and money sit in separate elements), #468 (a display change that rewired diff pairing), #519 (the PDF subsection detector misses catchline subsections on engrossed-amendment prints, and leaks quoted-block ones), #115 (financial semantics).

What does this change?

The XML reader now shows every heading the bill's XML tags in its breadcrumb. GPO often marks a heading up as an element of its own, a heading and no text, with the text it heads in the elements after it. The reader kept such a heading only while it sat in one of two slots, so it dropped 2,744 of the 10,172 on the committed corpus: North atlantic treaty organization over its security investment program, Food and drug administration over its first account, ADMINISTRATIVE PROVISIONS—FEDERAL HIGHWAY ADMINISTRATION over its sections, a reconciliation bill's fourth level of nesting. They now appear on change cards, in the sidebar and the tree, as heading lines in the Full bill view, and as locations in the financial views.

The key the diff pairs sections on (match_path) is built exactly as before. The breadcrumb (display_path) is built beside it from the same walk, so a heading shown in the breadcrumb cannot re-pair a section. Where the XML does not say how far a heading reaches, the breadcrumb follows the page's layout: a section sits under the nearest heading printed above it, until the next heading. Decision record: ADR 0024 (Proposed).

The corpus now has one bill of each kind that appropriates money, and the ledger check reports each kind separately. Added: a continuing resolution (118-hr-5860), a large supplemental (118-hr-815), a second reconciliation bill (the Tax Cuts and Jobs Act, 115-hr-1) and an authorizing law with an appropriations division (the Bipartisan Safer Communities Act, 117-s-2938). Every bill in tests/corpus_manifest.toml now names its vehicle.

Evidence

Diff pairing did not move. Every field the reader emits except display_path is byte-identical before and after, on every node of 116 XML files (the committed corpus and the fetched bills); display_path only gains headings. The round-1 pairing sentinel, re-stamped, changes only its parser-revision field on all 27 existing pairs: no pairing digest moved.

Every tagged heading reaches the breadcrumb. A new corpus gate, tests/test_corpus_properties.py::test_every_heading_the_file_tags_reaches_a_breadcrumb, checks the reader against the raw file's heading tags: it passes on every committed XML file, and with the previous reader 45 of the 63 fail it.

#521 (provision-group and agency-container headings reach no node in the parsed tree), by its own measure. Distinct headers in the raw XML that appear in no node's heading or breadcrumb, on the paired bills #521 measured: 118-hr-4820 9 → 0, 118-hr-4366 4 → 0, 114-hr-2029 3 → 0, 115-hr-5895 2 → 0, 117-hr-4502, 118-hr-8774 and 118-s-4795 1 → 0. All nine of 118-hr-4820's reach the leveled tree as nodes of level heading, not account, and the account vocabulary in scripts/heading_precision.py is unchanged: it is built from the headings nodes carry, which this does not touch. That is #521's verification as written; if a different node type is wanted, drop the Closes #521 line.

Where the PDF files each dollar amount, against the XML (the ledger check from #734), per kind of bill. "OK" is same place, same place relabeled, or true but shallower:

The versions pinned before this change, before → after (only the XML answer key moved; the PDF reader is unchanged):

kind of bill	versions	amounts	same location	OK	wrong parent	different heading
regular	18	4,824	89.2% → 95.1%	93.6% → 99.5%	295 → 10	16 → 16
omnibus	21	8,598	86.7% → 93.0%	92.2% → 98.6%	649 → 93	25 → 25
supplemental	1	2	100% → 100%	100% → 100%	0 → 0	0 → 0
reconciliation	1	783	0% → 0%	100% → 100%	0 → 0	0 → 0

The versions this change adds:

version	kind of bill	amounts	same location	OK	wrong parent	different heading
118-hr-5860 v3	continuing resolution	54	55.6%	100%	0	0
118-hr-815 v4	supplemental	101	98.0%	100%	0	0
118-hr-815 v5	supplemental	127	37.8%	52.0%	3	58
115-hr-1 v5	reconciliation	251	2.0%	100%	0	0
117-s-2938 v5	authorizing	59	6.8%	23.7%	0	45

Reconciliation is "OK" but rarely in the same location: its print has no subtitle or part level, so the PDF files its amounts true but shallower (a known residual in ADR 0022). No pinned version got worse; 23 improved. The regular and omnibus rows are where the XML answer key changed: the PDF was being graded "wrong parent" for headings it printed and the reader dropped.

The new kinds expose a PDF gap, now pinned rather than fixed here: on the supplemental's House version and the authorizing law, the PDF reader misses agency and account headings, so an account's money is filed under another account or under a title with no account (58 of 127 and 45 of 59 amounts). Filed as #738 (on a supplemental and an authorizing law, the PDF reader misses account headings, so money is filed under the wrong account).

Choices a reviewer should check

• How far a heading reaches. A heading over the sections below it covers them until the next heading, as the page lays them out (ADMINISTRATIVE PROVISIONS—… over SEC. 120 to 126). The known cost: a section with no heading of its own after a single-topic heading is filed under that topic, 71 sections on the committed corpus (SEC. 421 of 118-hr-4366 v2, a floor-added restriction, under SPENDING REDUCTION ACCOUNT), exactly as the PDF reads it. The alternative, one section per heading, never does that but shows every provisions block one level shallower than the page (98.0% against 98.9% OK). A heading over the next account of its own level heads only that account (FDA over Salaries and expenses, not over the accounts after it): the XML carries none of the typography the PDF uses to bound it.
• Which bills. The issue proposed the Infrastructure Investment and Jobs Act and the CHIPS and Science Act for the authorizing kind, and the American Rescue Plan and Inflation Reduction Act for reconciliation. Each was measured and left out because its PDF misses catchline-bearing subsections, or reads quoted ones as its own, which the subsection gates assert absolutely (PDF subsection detector misses some catchline subsections on engrossed-amendment prints, and anchors subsections inside quoted law #519, above). The Infrastructure Act's Senate amendment misses 3 subsections, reads 2 that are not there and takes in 6 quoted ones; the CHIPS Act's second Senate amendment misses 14 and reads 4 that are not there; the Rescue Plan misses 4 on its Senate amendment and 2 on its House-reported print; the Inflation Reduction Act's Senate amendment misses 1. The Tax Cuts and Jobs Act and the Bipartisan Safer Communities Act pass every gate. With the continuing resolution and the supplemental, the 12 new files add 4.0 MB to the corpus; the Infrastructure Act's one version would have added 8.5 MB alone.
• vehicle is the bill's, not the version's. 117-hr-2471 and 118-hr-2882 are omnibus although their introduced text is something else: both became an omnibus.

Pins regenerated, each with its cause

• tests/data/canonical_baseline.json: 22 of 27 XML pairs change digest and byte count, every one with its change count and summary identical; 2 pairs added.
• tests/data/pdf_canonical_baseline.json: all 23 pairs unchanged; 2 pairs added (the continuing resolution's is declined: its enrolled print has no line numbers).
• tests/data/round1_pairing_sentinel.json: parser_revision only on the 27 existing pairs; 2 pairs added.
• tests/data/ledger_location_baseline.json: 23 versions improved, none worse; 5 versions added.
• tests/test_round1_stages.py: similarity-call table gains the two pairs (0 and 12; total 1,108 → 1,120).
• tests/test_pdf_round1_revocation.py: split population 225 + 6 → 263 + 7, all of it the two new pairs (38 accepted, 1 declined).
• tests/test_pdf_subsection_recall.py EXPECTED, the zero-anchor registries, _KNOWN_DUPLICATE_COUNTS (118-hr-815 v5: its twenty divisions each number sections from 1) and ALLOWED_CORPUS_SKIPS (the continuing resolution and the reconciliation bill carry no appropriations headings): entries for the new bills, each with its reason.
• examples/hr4366_*_xml_diff.html: re-rendered; the breadcrumbs gain the headings. The PDF and H.R. 8752 examples and the web sample are unchanged.

Docs

ADR 0024 (new, Proposed) and its index row; ADR 0022 points to it where it deferred the reader change; docs/bill-structure.md (the reader shows header-only headings; the tree nests sections under their provisions heading on both pipelines); TESTING.md (check 7's limit, the ledger pin's answer key, totals per kind of bill); tests/corpus_manifest.toml (the vehicle field and the new bills' notes).

How to test

uv run ruff check . && uv run ruff format --check .
uv run pytest -m "not slow and not browser"
uv run pytest tests/test_bill_tree.py -k TestHeadingsWithNoTextOfTheirOwn -v   # one test per rule
uv run pytest -m slow --deselect tests/test_govinfo_corpus_parity.py

# The ledger per kind of bill, before and after, from the same scorer
git archive <base> src | tar -x -C /tmp/before
PYTHONPATH=/tmp/before/src uv run python -m tests.ledger_location
uv run python -m tests.ledger_location

To see it on a page: compare tests/corpus/118-hr-4366/1_reported-in-house.xml with 2_engrossed-in-house.xml. The change card for SEC. 420 reads TITLE IV > GENERAL PROVISIONS > SPENDING REDUCTION ACCOUNT > sec. 420 (was TITLE IV > GENERAL PROVISIONS > sec. 420), and the sidebar groups those sections under SPENDING REDUCTION ACCOUNT.

Test results (local, WSL)

• ruff check . and ruff format --check .: clean.
• Fast suite (-m "not slow and not browser"): 2,142 passed. The 8 failures are the checks that every fixture is committed, which pass once the 12 new files are staged, and test_every_cli_command_appears_in_the_readme_command_reference, which fails only because this machine's Windows mount reports every file as executable (git records tools/fetch_govinfo.py as 100644; the test passes on a git archive export with real modes).
• CI corpus-gates step: 1,564 passed, 29 skipped; the 9 failures are the same fixture-committed checks.
• CI remaining slow suites: 455 passed, 7 skipped (versions with no amounts, all in bills already in the corpus).
• CI external validation: 35 passed.
• Browser tier (-m browser --run-browser), run on Windows: 48 passed. Its scroll and jump-link tests build their report from the committed H.R. 4366 XML with this branch's reader.
• Fast suite on Windows as well: 2,128 passed; the 22 failures are the same 7 fixture-committed checks plus the 15 that fail identically on an untouched develop on Windows (backslash paths, executable bit, CI-report checks, line endings). A fresh render of every committed example on Windows equals the committed bytes once CR is removed (On Windows, regenerating the example reports fails, because the renderer writes OS-dependent encoding and line endings #737).
• Packaging gate (tests/test_engine_installs.py), run on Windows: 7 passed, against a wheel freshly built from this branch.

Checklist

• Linked the issue above (Closes #733)
• Ran the CI gates locally and they pass (see What CI checks); the failures listed above are uncommitted-fixture or Windows-environment only
• New or changed behavior has tests
• For a bug fix: the test fails without the fix, and I ran it both ways to check

AI assistance

Claude Code (Claude Opus 5.5) wrote the code, tests and docs under the contributor's direction; the contributor decided how far a heading reaches after reviewing the three cases on the printed pages.

🤖 Generated with Claude Code

[willhea]

Hey @mattzamora! Thank you for this work. Preserving headings that the XML explicitly tags is a valuable improvement for change cards, navigation, and the bill tree. I’d like to move that forward, with the following changes. Reviewed with GPT Sol 6.1 High.

1. Separate this work from #736.

This branch includes #736’s commits, so merging it would also introduce the financial displays and data changes discussed there. Please separate the heading improvements from that integration and make any remaining dependency on #734 explicit.

2. Revise ADR 0024’s approach to ambiguous heading scope.

My preference is to recover the structure the input supports and accept documented limitations where it does not. PDFs will often produce less complete structure than XML. We should improve that with defensible general rules, while allowing some gaps to remain.

That approach is already reflected in accepted ADRs 0012, 0014, and 0018. ADR 0018 states: “A shallower breadcrumb that is true beats a deeper one that is invented.”

Proposed ADR 0024 makes a different tradeoff. It acknowledges that the XML does not establish where some headings stop, then carries them over subsequent sections. For example, in tests/corpus/118-hr-4366/2_engrossed-in-house.xml, SEC. 421 changes from:

TITLE IV → GENERAL PROVISIONS → sec. 421

to:

TITLE IV → GENERAL PROVISIONS → SPENDING REDUCTION ACCOUNT → sec. 421

SEC. 421 concerns Veterans Affairs restrictions involving state marijuana programs. The additional parent relationship is misleading.

Please revise the ambiguous-scope decision and its implementation. Preserving a heading should not require making it the parent of unrelated content. I would accept shallower breadcrumbs where the relationship cannot be established.

3. Keep agreement metrics subordinate to source correctness.

ADR 0024 reports better PDF/XML agreement under the broader scope rule, but agreement does not establish correctness when both readers make the same assumption. Please also qualify the claim that the remaining disagreement is necessarily the PDF’s error.

Similarly, I support ADR 0022’s general typography and reading-order improvements, but the named wording exceptions and associated relaxation of ADR 0018 remain a separate decision. They should not be assumed accepted through this PR.

4. Strengthen the tests around placement and repeated headings.

The new rule tests are meaningful: in focused verification, all 14 passed with the new reader, and 10 failed with the previous reader.

The corpus coverage check only establishes that each heading name appears somewhere. It can pass after incorrect placement or loss of a repeated occurrence. Please add focused checks that fail when unrelated content inherits a heading, including SEC. 421, or when a repeated heading occurrence disappears. Retain the existing matching controls without duplicating them.

I’m requesting changes because these scope decisions need resolution, while the underlying heading-preservation work is worth keeping.

[willhea]

@mattzamora, the class renames, schema-version folding and stylesheet move described on #736 (#736 (comment)) apply here too, since this branch is stacked on it: the same financial_views.py badges and change-header check, the .removed-block rule, and the two .sidebar-toc / .change-card browser-test selectors. The one extra "toc" this branch adds, in bill_tree.py, is GPO's <toc> element rather than a class name, so it is fine as it is. Rebasing onto the updated #736 should be enough.

[willhea]

@mattzamora, a note on overlap with #525 (a section with only a heading is dropped by the XML reader, and the provision after it loses its account and chapter headings). Its rewritten body has the cause and a synthetic repro.

• Same functions. XML reader drops a section that has only a heading, so the provision after it loses its account and chapter headings #525's fix lands in the same parts of src/deltatrack/bill_tree.py this PR rewrites: _walk_structural_children, _process_section_element, and the current_intermediate slot in _process_appro_element.
• This PR doesn't fix it. At this PR's head (e7229c7), XML reader drops a section that has only a heading, so the provision after it loses its account and chapter headings #525's repro gives the same output as develop.
• Suggested order. If anyone picks up XML reader drops a section that has only a heading, so the provision after it loses its account and chapter headings #525, it's probably easiest after this PR lands, or alongside it, so the two changes don't conflict.

On your 2026-09-28 request: #732 is now typed Bug and assigned to you. We couldn't assign #524 too. GitHub only lets us assign someone without repository access who has commented on that issue. A short "claiming" comment on #524 is enough, and we'll assign it.

---

Generated by Claude Code