diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 3229c3ee..582830b5 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -182,6 +182,8 @@ jobs: tests/test_manifest_report_fixtures.py tests/test_canonical_baseline.py tests/test_pdf_canonical_baseline.py + tests/test_pdf_ledger_location.py + tests/test_financial_corpus.py tests/test_pdf_matching_boundary.py tests/test_pdf_observation_emission.py tests/test_pdf_observation_identity.py diff --git a/.gitignore b/.gitignore index ce47748b..519e49a1 100644 --- a/.gitignore +++ b/.gitignore @@ -144,3 +144,6 @@ docs/research/pdf-backend-bakeoff/validation/external-validity/results/x*.json dist/ build/ *.egg-info/ + +# Jupyter autosave copies, written next to any notebook that is opened (docs/research/). +.ipynb_checkpoints/ diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 39a3077d..d8dad346 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -299,7 +299,7 @@ What to look at, roughly in priority order: - **Correctness of the diff itself.** This is the product. Passing tests are necessary, not sufficient: a diff can be green and still wrong. For any change that affects diff output, **run the tool on a real bill and eyeball the report** rather than trusting the suite alone. To check the PDF and XML pipelines against each other, render each to its own HTML file (see [TESTING.md](TESTING.md#comparing-the-two-pipelines-by-eye)). - **The risk hotspots**, where a bug does the most damage: - - **Parser accuracy** (`src/deltatrack/bill_tree.py`, `src/deltatrack/parsers/`) -- does the bill's structure come through intact? A missing or mis-nested section corrupts everything downstream. See [docs/parser-validation.md](docs/parser-validation.md). + - **Parser accuracy** (`src/deltatrack/bill_tree.py`, `src/deltatrack/parsers/`) -- does the bill's structure come through intact? A missing or mis-nested section corrupts everything downstream. See [docs/parser-validation.md](docs/parser-validation.md). For PDF headings, the ledger checks in [TESTING.md](TESTING.md#the-ledger-location-pin-and-when-you-may-regenerate-it) show where each amount moved. - **Financial diff** (`src/deltatrack/diff_bill.py` and its financial filtering) -- dollar amounts and their changes must be exact. - **The canonical schema contract** (`src/deltatrack/formatters/canonical.py`) -- both pipelines and the renderer depend on it, so a breaking change there ripples everywhere. - **Tests for the change.** New behavior should come with a test that would fail without the fix. Judge that by the red-green delta on your own machine rather than by the totals the author reported, and compare like-for-like selections — [TESTING.md](TESTING.md#reading-test-counts) explains what a count does and does not tell you, including which differences are a fail-open signal rather than an environment difference. diff --git a/README.md b/README.md index 117a62cd..3fe57815 100644 --- a/README.md +++ b/README.md @@ -147,15 +147,15 @@ The index is read from BILLSTATUS ZIPs in `bills/`, which are **not** part of a *internal* diff dictionary, not the versioned canonical JSON that is the published interchange contract between the engine and its consumers. Use the canonical document instead: [`schema/canonical-diff.md`](schema/canonical-diff.md) specifies it, and the HTML -report's **Export and share → Download `diff.json`** button produces one. +report's **Export and share changes → Download `diff.json`** button produces one. ### HTML report `--format html` produces a self-contained HTML file that can be opened in any browser with no install or server required. See [examples/](examples/) for sample reports you can open immediately. The report includes: -- **Header** with bill number, congress, and version numbers (e.g., "v1: reported-in-house → v2: engrossed-in-house") +- **Header** with the bill number and title, then one line per version ("Before: …" / "After: …", with the version number where the input has one; for an upload, the file name as given) and the Congress - **Sidebar** listing all changed sections with color-coded change type badges. Type in the filter box to narrow the list. Click any item to jump to that section. -- **Financial summary table** showing dollar amounts before and after, with change amounts and percentages. Click column headers to sort. Click a row to jump to that section's detail. Sections with floor amendment annotations show a warning badge. +- **Financial views** ([ADR 0023](docs/decisions/0023-financial-ledger-views.md)): **Financials – Version A** and **Version B** list every money-bearing section of one version with each dollar amount's type (appropriation, rescission, cap, earmark, …), a category key, sorting, and the section's text highlighted clause by clause; **Inferred Financial Comparison** lists the changes whose sections hold money, Version A's amount beside Version B's, the difference in money given out, and links to each version's section. Each exports a CSV. The types are read from the bill's wording by a versioned classifier and are not authoritative; each view says so. - **Change cards** for each modified, added, removed, or moved section. Modified sections show word-level inline diffs: additions highlighted in green, deletions in red strikethrough. Moved sections show both the old and new location, plus body text. - **Prev/next buttons** in the bottom right corner to step through changes one at a time. diff --git a/TESTING.md b/TESTING.md index b2427e13..b114b113 100644 --- a/TESTING.md +++ b/TESTING.md @@ -19,7 +19,7 @@ comparison needs no key and no internet connection. ## How accuracy is checked -Accuracy is checked in six ways. Each one answers a different question, and +Accuracy is checked in eight ways. Each one answers a different question, and each has limits worth being honest about. There is no single accuracy percentage that would be truthful across all of appropriations, so we describe what each layer does and does not establish. @@ -172,6 +172,45 @@ position, so it confirms a passage is present somewhere in the version, not that it appears in the right place. It also cannot cover draft bills at all, which have no official version to compare against; that is check 4's job. +### 7. Cross-checking *where* the PDF files each dollar amount + +Check 5 confirms every dollar amount in the official version turns up somewhere in +the PDF. This one asks whether it turns up under the right heading. For every bill we +have in both forms, each amount in the official version is lined up with the same +amount in the PDF reading, in document order, and the two locations (the headings the +amount sits under) are compared. Each amount lands in one of a few grades: same place; +same place under a slightly different label; the right place but with a parent heading +missing; the right heading under the wrong parent; or filed under a different heading +entirely. The first three are acceptable, since a heading left out is visible, while a +wrong one is not. The count in each grade is frozen per bill version, so a change that +files money under the wrong heading more often fails, and one that improves it has to +be locked in on purpose. + +**Limit:** the official version's own reader is the answer key, and it leaves out some +headings the print shows (a heading with no text of its own, such as an umbrella agency +over its accounts). Where the PDF keeps such a heading the two disagree even though the +PDF is right, so the "wrong parent" grade overstates PDF errors. Draft bills have no +official version, so this check cannot cover them. + +### 8. Checking the financial views against the research that defined them + +The report's financial views type every dollar amount (appropriation, rescission, cap, +earmark, …) with the rules the research notebook in `docs/research/financial-semantics/` +defined and proved on the official version of a bill. The product mostly reads PDFs, so +the check is that the same bill read from its PDF gives the notebook's rows: the same +clauses, types, amounts and review flags, in the same order. On H.R. 4366 as reported it +does, row for row; on the Senate's three-bill amendment the PDF gives 619 of the 622 rows, +with the same appropriation total, and the three rows it misses are sections the PDF +reader does not start (lettered numbers such as `SEC. 119A.`), which the report flags. The +rows are frozen under the version of the rules that produced them, so the rules cannot +change without their version number changing ([ADR 0023](docs/decisions/0023-financial-ledger-views.md)). +Each view also states how many dollar figures the version's text holds and whether all +of them are shown. + +**Limit:** this checks that the rules are applied the same way to a PDF as to the official +text, not that the rules are right. A type is the rules' reading of the wording, and the +report says so on every financial view. + ## Known soft spots We keep these in the open rather than papering over them: @@ -324,6 +363,97 @@ Note what the sentinel cannot see. Two corpus-invisible behaviours move zero of pairs and are bound only by synthetic fixtures in `tests/test_round1_stages.py`, so "the corpus is still green" is not evidence about them. +### The ledger-location pin, and when you may regenerate it + +`tests/test_pdf_ledger_location.py` measures where the PDF pipeline files each dollar amount, +against the XML twin of the same version ([ADR 0022](docs/decisions/0022-pdf-heading-convergence.md)). +Every amount in the XML ledger is aligned with the same amount in the PDF ledger and its two +locations are compared level by level, the whole path, not just the account and its parent: + +| tier | the PDF path, against the XML's | counted as | +|---|---|---| +| `T0` | identical | true hit | +| `T1` | the same levels in the same order, a label differs (a joined, tail or near-variant name) | tolerated | +| `T2` | the XML's ancestors in order, with some left out | tolerated | +| `T3` | an ancestor that is wrong, extra or out of order | not tolerated | +| `T4` | a different account or section | not tolerated | +| `MISS` | no PDF amount to pair with | not tolerated | + +`tests/test_ledger_location_scorer.py` pins each case on hand-built paths. The tier counts for +every committed dual-format version (enrolled excluded) are pinned in +`tests/data/ledger_location_baseline.json`. + +The pin is exact in both directions. More not-tolerated amounts (`T3`, `T4`, `MISS`) or fewer +true hits (`T0`) is a regression. An improvement also fails, so it gets locked in: + +```bash +UPDATE_LEDGER_BASELINE=1 uv run pytest tests/test_pdf_ledger_location.py +``` + +#### Two readings: the tier totals, and the per-amount check + +The **tier totals** are the formal result. They are what the pin holds, what a heading change is +optimized toward, and what an ADR reports. The **per-amount check** is informal: it follows each +amount from one parser to another and lists every amount whose tier got worse, with the XML +path and the path before and after. It is not pinned and is not a gate. It exists because a +total hides a regression whenever another amount in the same version improves: one amount +moving `T2 → T3` and another `T3 → T2` leaves every count as it was. Use it to find and explain +regressions; report the totals as the result. + +Both come from `tests/ledger_location.py`, which prints the totals for whichever parser is +importable, so the "before" is the base branch's `src/` on `PYTHONPATH`. `--save` writes every +amount's locations; `--against` compares a later run with them. The comparison grades both runs +with the scorer that is checked out, so a scorer change never passes for a parser change. +`--extra bills` adds your locally fetched versions to both readings (they are not pinned): + +```bash +git archive origin/develop src | tar -x -C /tmp/before +PYTHONPATH=/tmp/before/src uv run python -m tests.ledger_location --save /tmp/before.json # before +uv run python -m tests.ledger_location --against /tmp/before.json # after +``` + +For a change to PDF headings or breadcrumbs, put both in the pull request: the before and after +totals, and the check's count of amounts better and worse, with each group of worse amounts +explained or fixed. + +#### What the answer key gets wrong + +The answer key is DeltaTrack's own XML reader, which leaves heading-only elements (an agency +heading with no text of its own) out of the breadcrumb, so where the PDF keeps such a heading +it is graded "wrong parent" although it matches the page. Read `T3` as an upper bound on PDF +errors. The reference for headings is the raw XML file's heading tags, not the reader +([ADR 0022](docs/decisions/0022-pdf-heading-convergence.md)): a change to the reader that +restores them shows up here as fewer `T3`. The same holds for two headings the XML sets on one +level where the PDF nests one under the other, and for a file whose own tags are misplaced +(in division G of 117-hr-4502 the `TITLE I` element is empty and its accounts sit in an unnamed +title after it, while the print nests them correctly). `T1` is lenient by design: a label that is a +near-variant of the XML's, or ends with the same words, counts as the same place. A fragment +passes too: a PDF heading read as just `ADMINISTRATION` grades `T1` against `FEDERAL HIGHWAY +ADMINISTRATION`, and reading the full printed `ADMINISTRATIVE PROVISIONS—FEDERAL HIGHWAY +ADMINISTRATION` instead (a heading the reader drops) shows in the per-amount check as `T1 → T3`. The XML is read only by these tests, never by the product. + +### The financial rows pin, and when you may regenerate it + +`tests/test_financial_corpus.py` holds the ledger rows of H.R. 4366 (as reported, and the +Senate amendment) frozen in `tests/data/financial_rows/`, one clause per line, under the +classifier version that produced them (`deltatrack.financial.CLASSIFIER`, +[ADR 0023](docs/decisions/0023-financial-ledger-views.md)). The XML reading must give them +exactly; the PDF reading must match a pinned number of them with the same appropriation +total. A further test checks that the frozen rows are exactly what the research notebook +computes, for as long as the version is 1.0. + +A rule change that moves any row fails the pin. To lock in a deliberate change, first bump +`CLASSIFIER` and add its changelog line beside it, then regenerate: + +```bash +UPDATE_FINANCIAL_ROWS=1 uv run pytest tests/test_financial_corpus.py +``` + +Regeneration refuses to write different rows under an unchanged version, so it cannot be +used to bless a change silently. The fixture diff shows exactly which clauses moved; say in +the pull request why each should have. Add a fast test for the wording that motivated the +change to `tests/test_financial.py` first, failing before the rule change. + ### The rest of the slow suite runs in CI too A further CI step runs the remaining slow modules (`CI_SLOW_MODULES` in diff --git a/docs/bill-structure.md b/docs/bill-structure.md index ccec2743..c7abe549 100644 --- a/docs/bill-structure.md +++ b/docs/bill-structure.md @@ -218,12 +218,15 @@ Closing the gap (DeltaTrack#54 and beyond — applies to both pipelines): Re-checked against the corpus 2026-06-27. Glyph size yields **two** bands, not three: a body band (~14pt) and a single heading band (~11.2pt small-caps) that holds -*both* `appropriations-intermediate` and `-small`. Casing does **not** rescue the -split — an agency header that is title-case in the XML `header` +*both* `appropriations-intermediate` and `-small`. The *text* does not rescue the +split: an agency header that is title-case in the XML `header` (`Management directorate`) extracts **ALL-CAPS** in heading position -(`MANAGEMENT DIRECTORATE`); small-caps flattens to caps in PDF text, so it is -lexically indistinguishable from an all-caps account header. `Line` carries no -x-position, so indentation/centering is not a usable signal either. +(`MANAGEMENT DIRECTORATE`), because small capitals flatten to capitals in PDF text. +The *letters* do. GPO sets an agency in title case in small caps, each word's first +letter printed larger (11.2 pt over 14.0 pt initials), and an account in even small +caps. The line's median glyph size hides that; `LineGeom.initial_caps` reads it per +word ([ADR 0022](decisions/0022-pdf-heading-convergence.md)). A department heading is +set in capitals at body size throughout (`LineGeom.size_min`/`size_max`). What that leaves recoverable from PDF-only input, keyed on *what follows* a heading-band line: @@ -233,18 +236,41 @@ heading-band line: | Major (department), `appropriations-major` | body-size + uppercase, sits above the band | Yes, with a dedicated detector | | Intermediate **carry-over** (one agency over ≥2 accounts) | another heading-band line | **Yes** — today silently dropped | | Intermediate **grouping header** (`ADMINISTRATIVE PROVISIONS`) | a `SEC.` line | **Yes** — today mislabeled `account`; the `SEC.` is detectable | -| Intermediate **prose-leading** (agency → "For necessary expenses…") | appropriation prose | **No** — identical to an account by size, case, and position | +| Intermediate **prose-leading** (agency → "For necessary expenses…") | appropriation prose | **Partly**: same size and position as an account, but printed agency-style. The case pattern ends the previous agency's reach; the heading itself is still emitted as an account | | Account, `appropriations-small` | appropriation prose | Yes | The unrecoverable row is not an edge case: prose-leading intermediates are the **majority** of the agency level in many bills (H.R. 8774 Defense: 59 of 67; H.R. 5895 Energy-Water: 37 of 44). Only H.R. 8752 and S. 2625 in the working corpus have none, which is why H.R. 8752 alone can look like full agency recovery. So on -PDF-only input the agency level is recoverable only in its carry-over and -grouping-header forms; the prose-leading form is not separable from an account by any -signal currently extracted. On the XML side all three levels are present in the path -tuples (`display_path` / `match_path`), even though they are dropped as standalone -nodes. +PDF-only input the agency level is recovered in its carry-over and grouping-header +forms; the prose-leading form is recognized only by its case pattern, which is used for +scope, not to re-label the heading. On the XML side all three levels are present in +the path tuples (`display_path` / `match_path`), even though they are dropped as +standalone nodes. + +### Reading heading structure across lines (ADR 0022) + +The **ledger** of a bill is every dollar amount it contains, each with its location: +the breadcrumb of headings it sits under. It is the same idea for an appropriations +account and a reconciliation section, and it is how PDF heading quality is measured: +for a published bill, each amount's PDF location is compared with its location in the +XML twin ([ADR 0022](decisions/0022-pdf-heading-convergence.md)). Forms that only show +across lines: + +| Form | Example | How the print shows it | +|---|---|---| +| Wrapped name | `SALARIES AND EXPENSES, FOREIGN CLAIMS` / `SETTLEMENT COMMISSION` | letters alike and the upper line full: one heading | +| Stacked headings | `BUREAU OF PRISONS` / `BUILDINGS AND FACILITIES` | letters differ, or the upper line broke early: two headings | +| Hanging-indent heading | first line at the margin, the rest at the paragraph indent | one heading, whatever the letters | +| Department mid-title | `UNITED STATES SECRET SERVICE` after an account's prose | capitals at body size, centered | +| Umbrella heading | `ATOMIC ENERGY DEFENSE ACTIVITIES` over `NATIONAL NUCLEAR SECURITY ADMINISTRATION` | same style as the heading below it, nothing between: **not separable** (read as one) | +| Quoted heading | `‘‘CHAPTER 16—…` inside an amendment to another law | inside an open quotation: not a heading of this bill | + +The XML reader leaves heading-only elements (an umbrella agency, `FOOD AND DRUG +ADMINISTRATION` over its accounts, a topical general-provisions heading) out of +`display_path`, so where the PDF keeps such a heading the two disagree even though the +PDF matches the print. ## Why not USLM? diff --git a/docs/decisions/0006-canonical-diff-contract.md b/docs/decisions/0006-canonical-diff-contract.md index 85db0172..3c40c03a 100644 --- a/docs/decisions/0006-canonical-diff-contract.md +++ b/docs/decisions/0006-canonical-diff-contract.md @@ -124,6 +124,19 @@ model in #115 and the leveled tree in #175 first, so an amount can be attached t account and classified as appropriation, sub-allocation, ceiling or limitation before it is shown as a number in a Change column. +**Since 3.1: a per-side ledger, still no money on a `Change`** +([0023](0023-financial-ledger-views.md), Proposed). The document may carry a top-level +`financial` field: for each version separately, every dollar amount with its chain of +headings, its clause and a type read from the wording by a versioned classifier, with +ceilings and amounts inside amended law marked. It is an observation per side with an +inference attached, not a pairing: nothing in it says a figure in one version became a +figure in the other, and a `Change` still carries no money field. The report's own +`diff.json` download strips it; the API's and CLIs' canonical JSON carry it (0023, +Consequences, for the open question this leaves). Of #115's model, the first part is +there (each section split into typed clauses; its location, the breadcrumb, made +dependable on PDF by [0022](0022-pdf-heading-convergence.md)); pairing a sub-amount across +versions, which is what money on a `Change` would claim, is not. + ### Compatibility The serialized contract carries a required `schema_version` under an diff --git a/docs/decisions/0012-pdf-heading-levels.md b/docs/decisions/0012-pdf-heading-levels.md index 18c838e3..d0f8c955 100644 --- a/docs/decisions/0012-pdf-heading-levels.md +++ b/docs/decisions/0012-pdf-heading-levels.md @@ -48,18 +48,27 @@ discarded — so the probe added zero PDFium calls). The findings: (split line sums 250–315 pt, wrap sums 354–418 pt), robust to the small constants and to the one bill whose column measured 311 pt rather than 339. - **No geometric signal separates boundary 1.** A prose-leading agency is a single - body-size line followed by prose: there is no second heading line to test for - fullness, and x-centering and leading are null as above. + heading-band line followed by prose: there is no second heading line to test for + fullness, and x-centering and leading are null as above. The spike did not measure + the letters themselves: an agency prints in title case in small caps, an account in + even small caps ([0022](0022-pdf-heading-convergence.md)). ## Decision We will recover the PDF heading levels from **deterministic geometry and structural position only**, and accept the one boundary that no such signal separates. -- **Boundary 1 (prose-leading agency vs account): accept the gap.** A lone - body-size heading followed by prose is emitted as an account. We do not guess it is - an agency. The level is genuinely not present in the print as a distinguishable - signal, so inventing one would trade an honest gap for unauditable noise. +- **Boundary 1 (prose-leading agency vs account): accept the gap, narrowed.** A lone + heading followed by prose is emitted as an account; we do not guess it is an agency + from its print alone. Its case pattern ([0022](0022-pdf-heading-convergence.md)) ends + the reach of an earlier agency, and re-labels the heading only with one more fact: a + title-case heading whose text carries no dollar amount, followed by an account-style + heading, introduces accounts rather than holding money, and is an agency + (`BUREAU OF RECLAMATION`, `GREAT LAKES ST. LAWRENCE SEAWAY DEVELOPMENT CORPORATION`). + Measured on 118 PDFs: 183 amounts toward the XML, none away. Title case alone was + measured and rejected (265 amounts the wrong way against 8), because account names + print title case too. The dollar sign is read as a fact about the text, not as + appropriations wording ([0018](0018-text-triggers-are-financial-only.md)). - **Boundary 2 (stacked vs wrapped major): split on line-fullness.** The major detector splits a post-`TITLE` body-size run at each line-fullness hard break: `w_i + space + first_word_width(line_{i+1}) ≤ column_width − slack` means line *i* @@ -95,8 +104,8 @@ Alternatives considered: level fully separated (the 4/12 stacked residue from #105 is retired) and the carry-over agency level recovered where it is followed by another heading (#104). - Defense-class bills keep a known, documented agency gap: prose-leading agencies - surface as accounts. This is recorded as an accepted limitation, not a bug to chase - with a heuristic. Where the agency level matters for those bills, it comes from the + whose own text carries money surface as accounts. This is recorded as an accepted + limitation, not a bug to chase with a heuristic. Where the agency level matters for those bills, it comes from the XML pipeline (which the user already supplies for a published bill, [0010](0010-pdf-pipeline-pre-publication.md)). - The line-fullness rule carries one residual edge, absent from the FY2025 corpus: diff --git a/docs/decisions/0014-leveled-heading-tree-scope.md b/docs/decisions/0014-leveled-heading-tree-scope.md index 28da39e0..eda399d7 100644 --- a/docs/decisions/0014-leveled-heading-tree-scope.md +++ b/docs/decisions/0014-leveled-heading-tree-scope.md @@ -85,9 +85,9 @@ not an equality it would have to fudge: | bill (v1→v2) | XML | PDF | gap | attributed cause | |--------------|----:|----:|----:|------------------| -| 118-hr-8752 | 39 | 37 | −2 | clean; XML sees 2 bare subsections of added SEC.s the PDF folds into the section block (#188) | +| 118-hr-8752 | 39 | 38 | −1 | clean; XML sees 2 bare subsections of added SEC.s the PDF folds into the section block (#188); the PDF sees the added SPENDING REDUCTION ACCOUNT heading as its own block ([0022](0022-pdf-heading-convergence.md)) | | 118-hr-8774 | 31 | 33 | +2 | PDF over-segments a few blocks | -| 117-hr-4502 | 1415 | 1500 | +85 | PDF over-segments a large added block; XML +304 from subsection nodes (#188) | +| 117-hr-4502 | 1415 | 1456 | +41 | PDF over-segments a large added block; XML +304 from subsection nodes (#188) | | 115-hr-5895 | 300 | 336 | +36 | division-collapse + segmentation; XML +54 from subsection nodes (#188) | Snapshot 2026-06-29; the 117-hr-4502 and 115-hr-5895 PDF rows re-measured @@ -100,7 +100,10 @@ the #96 PDF recalibration), and the XML↔PDF gap narrows sharply (4502 +389→+ 5895 +90→+36) — much of what was attributed to PDF over-segmentation was subsection granularity the XML tree previously lacked. 118-hr-8752's exact parity gives way to a −2 by design: XML emits bare subsections the PDF cannot detect -(the all-subsections scope decision recorded on #188). The live numbers and the bands that gate them are in +(the all-subsections scope decision recorded on #188). Two PDF rows re-measured +2026-09-27 after the heading passes of [0022](0022-pdf-heading-convergence.md): 4502 +1500→1456 (wrapped account names now one block), 8752 37→38 (a heading the PDF +now reads). The live numbers and the bands that gate them are in `tests/test_pipeline_parity.py` (regenerate the table: `.venv/bin/python scripts/parity_table.py`). The Senate #89 residual `118-s-4795` recovers its heading hierarchy at a size-band ratio of 1.02 diff --git a/docs/decisions/0022-pdf-heading-convergence.md b/docs/decisions/0022-pdf-heading-convergence.md new file mode 100644 index 00000000..3b6b7d18 --- /dev/null +++ b/docs/decisions/0022-pdf-heading-convergence.md @@ -0,0 +1,293 @@ +# 22. Recover PDF heading structure with ordered, fail-closed passes over the reading order, measured by where each dollar amount lands against the XML twin + +- Status: Proposed +- Date: 2026-09-27 + +## Context + +The **ledger** of a bill is every dollar amount it contains, each with its location: the +breadcrumb of headings it sits under (`TITLE II > U.S. CUSTOMS AND BORDER PROTECTION > +OPERATIONS AND SUPPORT`). It is bill-type neutral (an appropriations account, a +reconciliation section) and is what the financial views ([#115](https://github.com/civictechdc/DeltaTrack/issues/115)) +read. On the PDF pipeline the location comes from the heading anchors in `pdf_anchors`, so +a misread heading files money under the wrong name. + +The anchor detectors judge each printed line largely on its own: its glyph-size band, its +casing, whether prose follows ([0012](0012-pdf-heading-levels.md), [0018](0018-text-triggers-are-financial-only.md)). +That misses anything only visible across lines ([#524](https://github.com/civictechdc/DeltaTrack/issues/524)): + +- a wrapped account name read as an agency fragment over an account fragment + (`SALARIES AND EXPENSES, FOREIGN CLAIMS` / `SETTLEMENT COMMISSION`); +- two stacked headings glued into one, or one glued onto the account below it; +- a heading-shaped line that is inside a quotation (a reconciliation bill quoting the law it + amends) or inside an unfinished sentence; +- a department-level heading in the middle of a title (`UNITED STATES SECRET SERVICE`), which + the major detector looks for only directly under `TITLE n`; +- an agency carried over to accounts that are not its own. + +**Measuring against the XML twin.** A published bill has both a PDF and an XML version of the +same text. The XML carries the heading levels as tags, so for every amount in the XML ledger we +can ask where the PDF ledger files the same amount. The two amount sequences are aligned in +document order and each matched amount is scored at account/section grain, comparing every level +of the two paths: + +| tier | the PDF path, against the XML's | counted as | +|---|---|---| +| T0 | identical | true hit | +| T1 | the same levels in the same order, a label differs (a joined, tail or near-variant name) | tolerated | +| T2 | true but shallower: the XML's ancestors in order, with some left out | tolerated | +| T3 | right heading, wrong parent: an ancestor that is wrong, extra or out of order | not tolerated | +| T4 | filed under a different heading | not tolerated | +| MISS | XML amount with no PDF counterpart | not tolerated | + +The tier totals are the result this record reports and later changes are held to. Totals can +hide a regression when another amount in the same version improves, so a change is also +checked amount by amount: each amount is followed from the parser before to the parser after, +both graded by the same scorer, and every amount whose tier got worse is listed and explained. +That check is a working tool, not a gate +([TESTING.md](../../TESTING.md#two-readings-the-tier-totals-and-the-per-amount-check)). + +T2 is tolerated for the reason [0018](0018-text-triggers-are-financial-only.md) gives: a +shallower breadcrumb that is true beats a deeper one that is invented. The XML is used only +by this test. Nothing reads it at runtime ([0005](0005-contained-two-version-tool.md), +[0010](0010-pdf-pipeline-pre-publication.md), [0011](0011-local-only-processing.md)). + +**The answer key is incomplete, and that is an assumption this record corrects.** Checks that +compare the PDF with its XML twin have taken DeltaTrack's XML reader (`bill_tree.normalize_bill`) +as the reference for headings. It is a sound reference for where money sits, not for every +heading above it: the reader leaves out a heading that carries no text of its own, while the raw +XML file tags it and the PDF prints it. In H.R. 4502 (117th Congress) the file tags +`Department of Health and Human Services`, `Food and Drug Administration` and `Salaries and +expenses` in turn, and the reader's breadcrumb keeps the first and last. The reference for +headings is therefore the raw XML file's heading tags; the reader is measured against it, +not the other way round. This record changes the PDF side only and leaves the reader as it is, +so a PDF heading the reader drops is graded "wrong parent" here although it matches the page. + +**A signal the earlier spike did not measure.** [0012](0012-pdf-heading-levels.md) found no +geometric signal between an agency and an account: same size band, same centering, same +leading. Both are small capitals, but GPO sets an agency in *title case* in small caps (each +word's first letter printed larger) and an account in *even* small caps. The line's median +glyph size, which is all `Line.glyph_size` kept, hides the difference. `pdf_text` now also +records, from the same glyph walk and with no new PDFium calls, `LineGeom.initial_caps` (the +case pattern) and `size_min`/`size_max` (a department heading is capitals at body size). The +case pattern is judged one word at a time, from the size of each letter. Sizes exist only in the +raw character data (each character the file draws, with its position and size), not in the +simplified page text PDFium also returns. The two differ on word spaces: the page text fills a +space in from the gap between words, while the raw data has only the spaces the file draws, and +in heading lines the file mostly draws none (65–92% of heading-line spaces are filled in, in two +sampled bills; about 11% in prose). So word breaks are found from the gaps between letters. Small +words (`and`, `of`, `the`, …) and words printed all in large capitals, such as acronyms, have no +case pattern of their own and are skipped. + +## Decision + +We will re-read the detected headings against the whole reading order in a fixed sequence of +passes (`parsers/pdf_heading_passes.py`, `converge_headings`), after detection and before +divisions are assigned. Each pass reads only what earlier passes produced, and each **fails +closed**: where its evidence is missing (no geometry, no case pattern), it leaves the detectors' +reading unchanged. + +1. **Stream.** One reading-order stream across page breaks, with page furniture removed. A + page break is never a heading boundary. +2. **Lines that cannot be headings.** Quoted text (a quote is open from an opening mark to the + next closing mark; an unquoted `SEC.`/`TITLE`/`DIVISION` line closes it), a line inside an + unfinished sentence, and the wrapped tail of a section or subsection title. Heading anchors + on such lines are dropped. Missing punctuation alone does not put a line inside a sentence: a + run of centered lines followed by a new, indented paragraph (or by nothing that is prose) is + set apart from the text around it and ends the sentence carried down to it. Without that, a + paragraph that ends without a period (`(Public Law 111–241)`) or an enacting clause that + `TITLE I` interrupts (`… assembled, That`) hides every heading printed after it. Heading-shaped + text inside prose runs the full column, or is followed by its sentence's continuation at the + left margin, and stays excluded. +3. **Department headings anywhere in a title.** A centered line set entirely at body size in + capitals is a department-level heading, not only directly under `TITLE n`. A line opening + with a structural token (`TITLE`, `SUBTITLE`, `CHAPTER`, `PART`, …), or directly under one, + is that level or its wrapped name, not a department. +4. **Heading runs.** Every run of heading lines between prose is re-segmented at each line + break. The first decisive signal wins: a line-break hyphen joins; a line ending in a small + word (`AND`, `OR`, `OF`, `FOR`, `TO`, `THE`, …) joins, since a heading cannot end there; + letters printed differently (title case over even) split; letters printed alike join unless + a veto applies. The vetoes: the lower line stands alone as a heading at least twice elsewhere + in the bill, or the next word would have fitted on the upper line (line fullness, + [0012](0012-pdf-heading-levels.md), measured against the widest centered heading rather than + the body column, with the lower line's first word ended at a space or a letter gap as for the + case pattern). A hanging-indent block (first line at the margin running full width, the rest + at the paragraph indent) is one heading. The department line directly under a bare `TITLE n` + is left as the major detector read it, and lines that run on, uncentered, from a numbered + level's own line (`TITLE VIII—COUNTERING THE MALIGN INFLU-` / `ENCE OF THE PEOPLE'S …`) + finish that line's name and are not headings. Each piece of a re-segmented run takes a level: + the last keeps the detectors' level, a piece printed like the last one (same case pattern and + size) takes the same level, and any other piece is an agency. GPO stacks a name over its own + account at one level (`FOOD AND DRUG ADMINISTRATION` / `SALARIES AND EXPENSES`); read as an + agency, the upper line would end the reach of the real agency above it. The run's last heading + is then judged whole, after joining: printed title case, with no dollar amount in the text + under it and an account-style heading next, it introduces accounts and is an agency + ([0012](0012-pdf-heading-levels.md), boundary 1). + +Scope, which heading an account inherits, is decided in `pdf_anchors._breadcrumb_core` from +the case pattern the passes record on each heading (`Anchor.caps`): a department ends the +agency above it; an agency carried past other accounts does not reach an account when it prints +like the heading right below it, when the account itself prints agency-style, or when an +agency-styled heading lies between them. An unknown case pattern never blocks. + +**No wording decides a line break.** Every signal above is format or grammar, as +[0018](0018-text-triggers-are-financial-only.md) requires. Where format cannot tell two stacked +headings from one wrapped name, the run stays joined: a missed split is accepted rather than +hierarchy inferred from what a line says (Known residuals, below). +`tests/test_pdf_heading_passes.py::TestNoWordingDecidesALineBreak` pins this on stacks whose +wording alone would have split them. + +Alternatives considered: + +- **Promote a joined title-case heading to agency on its print alone.** The detectors read + `GREAT LAKES ST. LAWRENCE SEAWAY DEVELOPMENT` as an agency over its wrapped tail `CORPORATION`; + keeping that first line's level for any title-case join was measured on 118 PDFs and moved 265 + amounts the wrong way against 8, because account names print title case too (`FAMILY HOUSING + OPERATION AND MAINTENANCE, ARMY`) and as agencies they took the accounts after them. The case + pattern ends an agency's reach reliably; making one also needs the text under the heading to + carry no money. +- **A repeat-words veto.** Split two lines printed alike when every word of the lower line + appears in the upper one, on the idea that a wrapped name continues with new words. Rejected + on measurement: over the same 118 PDFs it changed 9 headings and was wrong in all 9, because + names repeat words (`NATIONAL SECURITY COUNCIL AND HOMELAND` / `SECURITY COUNCIL`; `COUNCIL ON + ENVIRONMENTAL QUALITY AND OFFICE OF` / `ENVIRONMENTAL QUALITY`). +- **Named wording exceptions.** Split a stack when its lower line is exactly `SALARIES AND + EXPENSES`, or its upper line exactly one of the fifteen executive departments (5 U.S.C. 101). + Rejected in review: each entry encodes the answer to a known example rather than showing that + the print distinguishes the structures, and admitting them would amend 0018. Measured before + removal on 118 PDFs (the committed corpus plus 65 fetched versions of reconciliation, + continuing-resolution, supplemental, authorizing and holdout bills), comparing the heading + output with and without them: the departments rule changed nothing (54 firing sites, no + difference). The `SALARIES AND EXPENSES` rule changed 8 versions, which are 4 headings in 4 + bills, each an office name stacked over its own `SALARIES AND EXPENSES` (Office of Terrorism and + Financial Intelligence in 117-hr-4502 and 118-s-4928; Office of Congressional Accessibility + Services in 115-hr-5895 and 117-hr-4346): 8 of 9,129 organizing headings and 19 of 60,508 + amounts. Without the rule each reads as one heading; its amounts stay with their account, and + every version of a bill reads it the same way, so version diffs are unaffected. +- **Defer to the detectors' joins and only add splits.** Rejected: measured worse (56 more + not-tolerated headings, leaks 20 → 48 on the development corpus), because the detectors' + joins are where most glued stacks come from. +- **Attach the first heading after a bare `TITLE n` to the title's name.** Rejected: it + re-decides what the major detector already decided (#105), and title text is a key other + code relies on. +- **Label the PDF from the XML at runtime.** Rejected for the reasons in + [0012](0012-pdf-heading-levels.md): a network call and input automation the engine forbids. +- **A model to classify the ambiguous lines.** Rejected by [0008](0008-deterministic-engine.md). + +## Consequences + +Measured with the shipped parser before and after this change, both graded by the same +whole-path scorer, on every committed dual-format version whose XML carries appropriations +headings (33 versions, 13,422 amounts; `python -m tests.ledger_location` prints these totals +for whichever parser is importable), on an unseen holdout of four FY2024 House bills fetched +after the rules were fixed (Agriculture, Energy and Water, State and Foreign Operations, +Interior: 10 versions, 2,272 amounts), and on the fetched versions whose XML carries no +appropriations headings (reconciliation, authorizing and continuing-resolution text: 32 +versions, 11,736 amounts): + +| | corpus before | corpus after | holdout before | holdout after | +|---|---:|---:|---:|---:| +| same location (T0) | 58.3% | 78.7% | 29.2% | 53.8% | +| OK (T0–T2) | 66.7% | 91.8% | 45.1% | 83.0% | +| wrong parent (T3) | 4,373 | 1,055 | 1,202 | 375 | +| misfiled (T4) | 99 | 40 | 45 | 12 | + +Without appropriations headings: misfiled 368 → 4, OK 96.1% → 99.2%. No version's totals got +worse: none of the 42 committed versions and none of the 55 fetched ones. + +Amount by amount, across all 97 versions, 6,413 amounts are filed better and 105 worse. 103 +of the 105 are one case: +the PDF now reads the printed `ADMINISTRATIVE PROVISIONS—FEDERAL HIGHWAY ADMINISTRATION` (and +its FTA, FRA, FAA, NHTSA, GSA, SBA and NARA twins) where before it kept only the fragment +`ADMINISTRATION`, which T1 passes as a variant of the XML's `Federal Highway Administration`. +The XML file tags the full heading and the reader drops it, so the correct reading grades T3: +the answer-key gap below. The other two are the sec. 614 straggler listed under open issues. + +**The OK rate undercounts the PDF.** The answer key is DeltaTrack's existing XML reader, which +leaves heading-only elements out of the breadcrumb: in H.R. 4502 the XML file tags +`Department of Health and Human Services`, `Food and Drug Administration` and `Salaries and +expenses` in turn, the PDF reads all three, and the reader keeps only the first and last, so +the PDF is graded "wrong parent" where it matches the page. Of the 1,055 wrong-parent amounts +in the corpus, 905 have PDF ancestors that the XML path lacks and the XML file tags as headings +(334 of 375 on the holdout). That check only asks whether each heading is tagged somewhere in +the file, so it is an upper bound, and the reader is left as the answer key here: changing it +is a separate decision. + +- The same table is the yardstick for the XML side: a change to the reader counts as an + improvement when its breadcrumbs carry the file's heading tags, which shows here as the + corpus OK rate rising toward the roughly 98.5% the upper bound above allows. + +- Account names, measured the older way (is each PDF account heading a heading the XML twin + has, `scripts/heading_precision.py`): on the nine bills of + `tests/test_pdf_anchor_golden.py::TestCorpusAccountPrecision` the lowest recall and precision + rise from 0.744 / 0.750 to 0.967 / 0.967, and seven of the nine score 1.000 / 1.000. Its floors + are re-derived to 0.95 in the same change. On 118-hr-4820 every one of the fourteen wrapped + names listed on [#524](https://github.com/civictechdc/DeltaTrack/issues/524) is read as one + account, and its unmatched account headings fall from 17 to 0. #524's governing case, + `NATIONAL OCEANIC AND ATMOSPHERIC ADMINISTRATION` on 114-hr-2578 (not in the corpus, fetched + to check it), stays an agency over its own accounts in all four versions. + +- The per-version tiers are the **baseline** for later heading changes. + `tests/test_pdf_ledger_location.py` pins them on the committed corpus, exactly: a version + whose not-tolerated count rises or whose true hits fall fails, and an improvement fails until + it is locked in, so every later change reports its before and after from the same scorer. + `tests/test_pdf_heading_passes.py` pins each rule on synthetic lines, each proved by + removing the rule. +- The passes change the parser's output, so the parser revision changes + ([0019](0019-observation-identity.md)), and the canonical baselines and the round-1 pairing + sentinel are regenerated in the same change. This record's measurement is the independent + precision/recall evidence [0020](0020-matching-stages.md) asks for. +- **Known residuals.** Two same-styled headings stacked with no prose between them are glued + into one (`ATOMIC ENERGY DEFENSE ACTIVITIES` over `NATIONAL NUCLEAR SECURITY ADMINISTRATION`, + [#501](https://github.com/civictechdc/DeltaTrack/issues/501); `STATE AND LOCAL LAW + ENFORCEMENT ACTIVITIES` over `OFFICE ON VIOLENCE AGAINST WOMEN`). The XML marks the upper one + as a heading with no text of its own; the print gives no format signal for it. An office name + over its own `SALARIES AND EXPENSES`, printed alike, is glued the same way (`OFFICE OF + TERRORISM AND FINANCIAL INTELLIGENCE SALARIES AND EXPENSES`; see the rejected named + exceptions above). A department + name glued by the major detector itself (`OVERSEAS CONTINGENCY OPERATIONS DEPARTMENT OF + DEFENSE`) is left as detected. A wrapped account name that opens with the word `TITLE` + (`TITLE 17 INNOVATIVE TECHNOLOGY LOAN GUARANTEE` / `PROGRAM`, 115-hr-5895) is read as the + structural token and stays split. A name broken early for balance after a complete word + would still be split by the line-fullness veto; the one balanced break in the corpus ends in + `OF` and is joined. +- **A name over its own account can still be read one level too high.** A piece the passes + split out takes its level from the print, but where the detectors already made the upper line + an agency it stays one: `NORTH ATLANTIC TREATY ORGANIZATION` over `SECURITY INVESTMENT + PROGRAM`, which the XML sets as two headings on one level. +- **A title's name is displaced from breadcrumbs.** The heading under a bare `TITLE n` and the + major-level heading below it compete for one breadcrumb level, and the nearer wins: + `TITLE I › LEGISLATIVE BRANCH › HOUSE OF REPRESENTATIVES` reaches its accounts as + `TITLE I › HOUSE OF REPRESENTATIVES`. True but one level shallower, and true of develop too. + The follow-up is to keep the title's name as a level of its own. +- **Reconciliation bills stay shallower.** Their prints carry no subtitle or part level the + passes can read, so their amounts land T2 at best. Recovering those levels, and measuring + other bill vehicles (continuing resolutions, supplementals, authorizing bills with direct + appropriations), is follow-up work. +- The prose-leading agency gap in [0012](0012-pdf-heading-levels.md) narrows but stays: the + case pattern ends an agency's scope at an agency-styled account, and an agency-styled heading + that introduces accounts (no dollar amount under it, an account-style heading next) is now an + agency, but one whose own text carries money is still emitted as an account. + +**Open issues** found by the measurement and left for later changes: + +- **A department repeated as its own parent.** `DEPARTMENT OF DEFENSE › DEPARTMENT OF DEFENSE` + over `FAMILY HOUSING IMPROVEMENT FUND` and `MILITARY UNACCOMPANIED HOUSING IMPROVEMENT FUND`: + 16 amounts in 8 versions, graded T3. Not yet traced. +- **An account carried over a general provision.** In division J of 117-hr-3684 (v5), two + amounts in sec. 614 are filed under `ENVIRONMENTAL PROGRAMS AND MANAGEMENT`, where the XML + files them under the EPA's general provisions. Before this change they sat directly under the + title (T2); now they are T3. +- **T1 passes a fragment.** A label that ends with the same word as the XML's counts as the + same place, so `ADMINISTRATION` grades T1 against `Federal Highway Administration` (the 103 + amounts above). Tightening it regrades the parser before this change, not after, and is a + scorer change to measure on its own. +- **The answer key.** Restoring the heading-only elements the reader drops is the separate + decision above. A change to the answer key (#739) moves these grades too: under whole-path + T0 a heading the reader carries too far shows as T3 against a correct PDF path, so its + effect should be read amount by amount before the totals are re-pinned. + + diff --git a/docs/decisions/0023-financial-ledger-views.md b/docs/decisions/0023-financial-ledger-views.md new file mode 100644 index 00000000..109d2a5b --- /dev/null +++ b/docs/decisions/0023-financial-ledger-views.md @@ -0,0 +1,232 @@ +# 23. Type each version's dollar amounts in a versioned, per-side ledger, and show it in three report views without pairing money in the diff + +- Status: Proposed +- Date: 2026-09-28 + +## Context + +The research notebook in `docs/research/financial-semantics/` classifies the dollar amounts of a +bill: it splits each money-bearing section into clauses (`Provided, That`, `; and in addition,`, +`of which`) and types each clause with a small set of wording rules (appropriation, rescission, +restriction, cap, earmark, …; `classify_bill.py`, rationale in `classifier_notes.md`). Its report, +`financial_118_hr_4366.html`, shows one bill version as a table of 72 money-bearing sections with +a category key, a sort bar, a "review" badge where a clause holds more than one figure, and each +section's text highlighted clause by clause. It reads the XML, through `normalize_bill`. + +The product compares two versions, and mostly from PDF: a draft or a print often has no XML +([0010](0010-pdf-pipeline-pre-publication.md)). #115 asks for the notebook's view in the product, +for both versions, plus a comparison of the two. + +Three constraints shape how: + +- **The diff document carries no money** ([0006](0006-canonical-diff-contract.md)). Paired amounts + were removed from `Change` in 3.0 (#671) because the pipeline could not say what a figure is, and + because the export is read by machines (the report tells staffers to upload `diff.json` to an AI + assistant) that cannot see a caveat written elsewhere. 0006 names the preconditions for bringing + typed money back: an account model (#115) and the leveled tree (#175). +- **Wording may interpret money but never define structure** + ([0018](0018-text-triggers-are-financial-only.md)). The sections have to come from the tree the + report already builds; the classifier only reads inside them. +- **One renderer** ([0007](0007-single-renderer.md)), deterministic output + ([0008](0008-deterministic-engine.md)), no network and no XML at runtime for a PDF input + ([0005](0005-contained-two-version-tool.md), [0011](0011-local-only-processing.md)). + +PDF sections became dependable enough to carry this with [0022](0022-pdf-heading-convergence.md): +on the committed corpus 87.6% of amounts sit under the same headings as in the XML (from 62.3%). + +## Decision + +### A per-side ledger in the canonical document (schema 3.1) + +We will add an optional top-level `financial` field: for each version, its **ledger**, every +dollar amount in that version with where it sits and what it appears to be. + +```jsonc +"financial": { + "classifier": "1.0", + "v1": { "amounts_in_text": 141, "sections": [ /* LedgerSection */ ] }, + "v2": { "amounts_in_text": 206, "sections": [ … ] } +} +``` + +- A **section** is one money-bearing node of that version's tree: `path` (its chain of headings + with their levels), `span` (a character range into `full_text`), `clauses`, and `flags`. +- A **clause** has a `level` (primary or sub), a `type`, the clause `amount`, `needs_review`, and + every figure in it as `amounts[]` (`value`, `cap` for "not to exceed", `in_amended_law`). +- **No text is copied.** Every span points into the `full_text` the document already carries. +- **Built in the compare run**, where both producers finish (`formatters/canonical.py`), so the web + app and the CLIs get the same field. `deltatrack.financial` is the one module that reads the + wording, allowlisted under 0018; `formatters/financial_views.py` renders it. + +A `Change` still carries no money. The ledger is per side and unpaired: it states what each +version contains, not what changed. Where the report compares the two, it does so at render time +and stores nothing (below). + +Alternatives: a money field on `Change` (the claim 0006 removed); a separate JSON file (the report +is one self-contained file, 0011); computing the ledger in the browser (the classifier is Python, +and the CLIs would not get it). + +### Same values from the PDF as the notebook found in the XML + +The notebook's rules move into the package unchanged. What is new is how a PDF section's text is +read before the rules see it, reusing the PDF tools the parser already has: + +- the 7-character line-number gutter comes off, and the unnumbered running header + (`pdf_heading_passes._FURNITURE`) is skipped; +- a word the printer broke across a page seam (`speci-` / `fied`) is rejoined, by the same test as + `pdf_blocks._rejoin_cross_page_hyphens` (within a page `pdf_text._merge_print_lines` already did); +- a section is split into paragraphs where the XML has separate nodes: at a heading line after a + finished sentence (`pdf_blocks._is_strippable_heading_line`), and at `(a) ` after a finished + sentence outside a quotation. An all-caps line in the middle of a sentence is a continuation; +- GPO's quote marks are removed from the copy the rules read, only: the XML the rules were built on + has none, and they pushed one clause past a rule's 300-character window. They stay on the page. + +**Amended law.** An amount inside text this bill writes into another law (inside quote marks, +after an amendment lead-in such as "is amended to read as follows:", or just after +"striking"/"inserting") is marked `in_amended_law`. It is shown, listed separately, and never +counted as money given out. **Money given out** is an appropriation (positive) or a rescission +(negative) outside amended law; nothing else is summed. + +Result on H.R. 4366 (the parity bill): v1 gives the notebook's 108 clause rows exactly, from PDF +and from XML. The Senate amendment (v4) gives 622 of 622 from XML and 619 of 622 from PDF, with the +same appropriation total; the three missing rows are sections the PDF parser does not start +(`SEC. 109A.`, `SEC. 119A.`, `SEC. 119B.`, lettered section numbers), and the two rows that absorb +them are flagged on the page. + +### Misses are visible, never a silent zero + +- **Coverage:** each view states how many `$` figures the version's text holds and whether all are + shown (`amounts_in_text`). On the parity bill all are: 141 of 141 (v1), 206 of 206 (v2). +- **`may_hold_several_sections`:** a row whose text contains a further section heading the parser + did not start ("may hold more than one section"). +- **"review":** a clause with more than one non-ceiling figure, so its amount is ambiguous. +- **An information alert** on every view: the classifications are not authoritative; users must + audit and verify values against the bill text; they are subject to change as the application + improves; certain sections and amounts may have been missed and not shown at all; where to send + feedback (#congressional-tech on the Civic Tech DC Slack). It names the classifier version. + +### The classifier is versioned, and cannot change silently + +`deltatrack.financial.CLASSIFIER` (now `"1.0"`) is recorded in every document and shown in every +view. **Why a version, in a project without releases:** a report is often saved and passed on, and +its reader needs to know which rules typed the money in front of them. A rule change will move rows +(a clause that was "unknown" becomes an "earmark"), and anyone comparing two reports, or rows from +two runs, needs to see that the rules differ rather than conclude the bill did. Contributors here +cannot cut releases, so the version is a number in the code with a changelog comment beside it, and +the way to use an earlier version is to check out the commit before the change its changelog line +names. + +What keeps it honest is a test, not discipline: `tests/test_financial_corpus.py` holds the parity +bill's rows frozen under the version that produced them (`tests/data/financial_rows/`). If a rule +change moves any row, the test fails; regenerating the rows (`UPDATE_FINANCIAL_ROWS=1`) refuses to +write different rows under the same version. So the rules cannot move without the number moving. +Bump rules: the minor for a changed rule; the major for a type added, removed or renamed (which +also changes the schema's type list). While the version is 1.0, a further test checks the frozen +rows are exactly what the notebook itself computes from the XML; it is retired when 1.1 lands. + +### How the classifier improves over time + +The notebook's rules are a starting line, not a finished classifier. A change goes: + +1. Find the case: a row on a real bill with the wrong type, a wrong amount, or "unknown". The + exports make this cheap: each row carries its clause text. +2. Add a fast test with that wording in `tests/test_financial.py` that fails. +3. Change the rule, bump `CLASSIFIER`, add its changelog line (what changed and why). +4. Regenerate the frozen rows. The fixture diff is one clause per line, so the PR shows exactly + which rows moved; each moved row should be one the change meant to move. +5. Say in the PR which bills were checked beyond the parity bill. + +Known candidates, from building this and from `classifier_notes.md`: + +- **Earmark wording (1.1 candidate).** The earmark rule needs "specified in the table". The House + prints "…specified in the table under the heading … in the explanatory statement"; the Senate + prints "…specified in the report accompanying this Act". The same kind of provision is typed + "earmark" in one version and "unknown" in the other. +- **"For purposes of" openers** read as appropriations in authorization bills (NDAA sec. 1414). +- **Lettered section numbers** (`SEC. 119A.`) are not started by the PDF parser; today they are + flagged, not fixed. That is a parser change, not a classifier one. +- **The last section of a PDF** carries everything printed after it (short title, attestation, the + reported print's back cover) in all 27 line-numbered corpus versions checked; its text shows + cover lines and a renumbered last section is not recognised as moved. Also a parser change. + +### Three views in the report + +Tabs after Changes and Full bill: **Inferred Financial Comparison**, **Financials – Version A**, +**Financials – Version B**. + +- **Version A / B** are the notebook's report for one version: the table, the category key with + counts, the sort bar, the review badge, and a row that opens onto its clauses and the section's + text, highlighted by clause type. The text is read from the embedded document by the page (it + reproduces the same gutter, header and hyphen handling as the Python), so the report does not + carry it twice. "Export Inferred Financials" downloads that version's whole ledger as CSV, one row + per dollar amount, the heading chain spread into columns (division, title, department, agency, + account, …) so it sorts and pivots without nesting. +- **Inferred Financial Comparison** has one row per change the differ already reported whose + section holds money on either side: the differ's own pairing, no second matching engine. Each + side shows its section's type and amount; the difference counts money given out only ("$0 · text + changed" when the amount did not move, "not given-out money" when neither side gives any). A row + opens onto that change's word diff from the Changes view, and links to its section in Version A + and in Version B, where every clause is laid out. The column is headed "Text block change": the + badge is the differ's verdict on the text, not a claim about money. Amended-law amounts that + differ are listed underneath and not counted. "Export Comparison" writes the table as CSV, one row + per change, computed by the same Python the table uses (embedded as JSON), with the change's + words before and after on one line each. +- **The report's own `diff.json` download strips `financial`**, so what a staffer uploads to an AI + assistant stays the 0006 contract. + +Alternatives: two filtered notebook tables side by side (no diff, compare by eye); shaded column +bands (clear but wide); a clause-by-clause comparison, mocked up from real data and set aside: +denser than the one-line rows, and pairing clauses across versions is itself an inference that goes +wrong exactly where the classifier types one clause differently in the two versions. The links to +Version A / B show every clause without inferring a pairing. + +## Consequences + +- **It stands on 0022's sectioning.** 0006 waits for two things before typed money returns: where + an amount sits (the leveled tree, #175) and what it is (an account model, #115). 0022 (#734) is + the first, on PDF: the PDF's chain of headings now agrees with the XML's for 87.6% of amounts on + the committed corpus (92.7% within tolerance), and every ledger row takes its location, its + section boundaries and the comparison's pairing from that tree. Where a heading is misread, the + ledger files money under the wrong name and no classifier rule can put it right, so this lands + after #734 and 0022's corpus measure is also the measure of where ledger rows sit. +- **#115's model is partly here.** #115 asks to (1) decompose an account into its sub-amounts and + what each means, (2) track each sub-amount's change across versions, and (3) tell a per-account + change narrative. This ADR does a first cut of (1): each section split into clauses, each typed + by category from its wording (a category, not the purpose "$X for A" states). (2) and (3) are + not built, and the comparison deliberately does not pair clauses across versions (Decision). + Inter-account roll-up (#108, #147) stays parked. +- **The tension with 0006, stated.** 0006's condition for money on a `Change` is an amount + "attached to an account and classified as appropriation, sub-allocation, ceiling or + limitation". The ledger does that for each version on its own: attached to its tree node, + classified by clause (by inference, versioned). It does not pair a figure in one version with a + figure in the other, which is what a money column on a `Change` would claim and what #115's + (2) would have to earn. What keeps it inside 0006's line: a `Change` carries no money; the + ledger is per side, unpaired and versioned; every figure is kept, with ceilings and amended-law + amounts marked; the report says plainly that the types are inferred. What does not: the API's + `output=json` and the CLIs' + canonical JSON include `financial`, and a machine reading only that file sees classifier types + without the page's alert. Whether to keep it there, strip it from those outputs too, or add a + machine-readable caveat to the field is left to the maintainers. +- **Report size.** The ledger adds about 94 KB to the H.R. 4366 v1→v2 report, the comparison's + export data a little more; the text itself is never duplicated. +- **Changes to existing report behaviour** that came with the views: "Export and share changes" + (renamed) and the change navigator show only on Changes and Full bill; the find box is larger and + also searches rows that are collapsed (it fills and opens them); each tab keeps its own scroll + position; jump targets clear the sticky bar at its measured height (the fixed 64px cleared the + one-row bar, not the two-row one); the three financial tabs use the full window on a wide screen + while Changes and Full bill keep their 940px reading width; the PDF report's heading keeps the + whole bill title; upload labels keep the whole filename, extension included. +- **Accepted:** tabs add no browser history (Back leaves the report). Reports are often written + into a tab with no address of its own, where history entries do not behave the same across + browsers. +- **Not done, by decision:** totals and roll-up (a mechanical roll-up overcounted committee-report + title totals by 32–44%, [0014](0014-leveled-heading-tree-scope.md); semantic roll-up is #147); a note that + reconciliation locations are shallower (planned; there is no reliable bill-type signal to show it + on); "authorization" and "by reference" as kinds of money (the types exist; the flags wait for + bill types beyond appropriations and reconciliation). +- **Tests:** `tests/test_financial.py` (one rule per test on synthetic text; comparison rows; + amended-law amounts never in the difference; the comparison export), `tests/test_financial_corpus.py` + (PDF and XML against the frozen rows, the version gate, provenance at 1.0, coverage, the flags, + schema validity, the three views wired to the changes; in CI's corpus-gates step), and browser + tests for search in collapsed rows, the section links, per-tab scroll and jumps below the bar. + Every rule and the version gate were broken on purpose and the tests went red. diff --git a/docs/decisions/README.md b/docs/decisions/README.md index e2975bda..afce8b54 100644 --- a/docs/decisions/README.md +++ b/docs/decisions/README.md @@ -103,3 +103,5 @@ presents itself as the architecture in force. | [0019](0019-observation-identity.md) | Accepted | Identify a parsed observation by its source, its parser revision and its ordinal; never by its text | | [0020](0020-matching-stages.md) | Accepted | Separate retrieval, correspondence evidence, correspondence assignment and change classification | | [0021](0021-naming-authority-and-boundaries.md) | Accepted | Name things in the vocabulary an outside reader already speaks, scoped to the boundary being named | +| [0022](0022-pdf-heading-convergence.md) | Proposed | Recover PDF heading structure with ordered, fail-closed passes over the reading order, measured by where each dollar amount lands against the XML twin | +| [0023](0023-financial-ledger-views.md) | Proposed | Type each version's dollar amounts in a versioned, per-side ledger, and show it in three report views without pairing money in the diff | diff --git a/docs/research/financial-semantics/README.md b/docs/research/financial-semantics/README.md index b0cbe22c..6ec8da7c 100644 --- a/docs/research/financial-semantics/README.md +++ b/docs/research/financial-semantics/README.md @@ -2,6 +2,21 @@ Financial classifier and analysis tools for DeltaTrack bill data. +## Status: the classifier is in the product + +The rules in `classify_bill.py` now type the dollar amounts in the report's financial views: +they moved into `src/deltatrack/financial.py` unchanged, as classifier version 1.0 +([ADR 0023](../../decisions/0023-financial-ledger-views.md)). Changes to the rules are made +there, not here: they bump its version and regenerate the frozen rows the product is tested +against (`TESTING.md`, "The financial rows pin"). + +This directory stays as the provenance of version 1.0. `tests/test_financial_corpus.py` +imports `classify_bill.py` and checks that the product's rows on H.R. 4366 are exactly what it +computes; that test is retired when the product moves past 1.0, and this directory can then +be condensed under the [retention policy](../README.md). `classifier_notes.md` records known +weak spots (the "For purposes of" opener, the intentionally unknown categories) that are +candidates for later versions. + ## Files | File | Purpose | diff --git a/docs/source-signal-inventory.md b/docs/source-signal-inventory.md index 2abd5e75..e9c3b755 100644 --- a/docs/source-signal-inventory.md +++ b/docs/source-signal-inventory.md @@ -24,7 +24,11 @@ in-file signal read from the input the user already supplied. - **PDF** (`src/deltatrack/parsers/pdf_text.py`, `src/deltatrack/parsers/pdf_anchors.py`): codepoints; per-glyph char boxes → baseline clustering (visual lines), x-gaps (word spacing), line extent (justified column width, line-fullness split); text-matrix scale → glyph **size** - (body/heading/major size bands). That is the complete set of PDFium signals we read. + (body/heading/major size bands), and per line the **letter size range** and the + **case pattern** (whether each word's first letter prints larger than the rest: title + case in small caps, the agency style, vs even small caps, the account style; + `LineGeom.initial_caps`, [ADR 0022](decisions/0022-pdf-heading-convergence.md)). That + is the complete set of PDFium signals we read. - **XML** (`src/deltatrack/bill_tree.py`, `src/deltatrack/structure_tree.py`, `src/deltatrack/diff_bill.py`): the structural tag hierarchy + text, the `form` metadata block, and three attributes — `@id` (stored on `BillNode.element_id`, **not** used for matching), `@display-inline`, diff --git a/examples/hr4366_committee_vs_floor_xml_diff.html b/examples/hr4366_committee_vs_floor_xml_diff.html index fdd5b3cf..08e33512 100644 --- a/examples/hr4366_committee_vs_floor_xml_diff.html +++ b/examples/hr4366_committee_vs_floor_xml_diff.html @@ -19,6 +19,8 @@ --gold: #c9944e; --destructive: #c04040; --success: #3d9b6d; + --info: #e3f2f8; + --info-foreground: #0b4f63; --diff-add: #d3f0e2; --diff-add-foreground: #1a6647; --diff-remove: #f5ddd8; @@ -27,7 +29,22 @@ --diff-modified-foreground: #8a6320; --diff-moved: #eef0f8; --diff-moved-foreground: #2c2c5c; + --fin-transfer: #cffafe; + --fin-transfer-foreground: #155e75; + --fin-authorization: #e0f2fe; + --fin-authorization-foreground: #0c4a6e; + --fin-fee: #ffedd5; + --fin-fee-foreground: #9a3412; + --fin-directive: #e0e7ff; + --fin-directive-foreground: #3730a3; + --fin-earmark: #ede9fe; + --fin-earmark-foreground: #5b21b6; + --fin-availability: #d1fae5; + --fin-availability-foreground: #065f46; + --fin-sub-allocation: #dbeafe; + --fin-sub-allocation-foreground: #1e40af; --radius: 0.625rem; + --sticky-bar-height: 64px; --font-sans: ui-sans-serif, system-ui, -apple-system, 'Segoe UI', Roboto, Helvetica, Arial, sans-serif; --font-serif: ui-serif, Georgia, 'Times New Roman', serif; --font-mono: ui-monospace, 'SF Mono', Menlo, Consolas, monospace; @@ -88,11 +105,13 @@ .filter-empty[hidden] { display: none; } /* Main content */ -.main { margin-left: 280px; padding: 28px 36px; max-width: 940px; flex: 1; } +.main { margin-left: 280px; padding: 28px 36px; max-width: 940px; flex: 1; min-width: 0; } /* Header */ .report-header h1 { font-size: 24px; margin-bottom: 4px; } -.report-header .versions { color: var(--muted-foreground); font-size: 15px; margin-bottom: 16px; } +.report-header .versions { color: var(--muted-foreground); font-size: 15px; margin-bottom: 16px; line-height: 1.5; } +.versions__row { display: block; overflow-wrap: anywhere; } +.versions__row b { color: var(--foreground); font-weight: 600; } .summary-bar { display: flex; gap: 10px; margin-bottom: 24px; flex-wrap: wrap; } .summary-item { display: inline-flex; align-items: center; gap: 6px; padding: 4px 12px; border-radius: 999px; font-size: 13px; background: var(--secondary); } @@ -164,16 +183,25 @@ border-bottom: 1px solid var(--border); padding: 10px 0; margin-bottom: 16px; } .action-bar__left { display: flex; align-items: center; gap: 10px; flex-wrap: wrap; } .action-bar__group { display: flex; align-items: center; gap: 10px; flex-wrap: wrap; } -.find-bar { display: inline-flex; align-items: center; gap: 4px; } -.find-bar input { padding: 5px 10px; border: 1px solid var(--border); border-radius: var(--radius); - font: inherit; font-family: var(--font-sans); font-size: 13px; width: 180px; background: var(--card); } -.find-bar button { padding: 5px 9px; border: 1px solid var(--border); border-radius: var(--radius); - background: var(--card); cursor: pointer; font-family: var(--font-sans); font-size: 13px; } +.action-bar__left { flex: 1 1 auto; } +/* The page's one search: it reaches text that is collapsed, which the browser's own find + cannot, so it is sized and marked to be found first. */ +.find-bar { display: inline-flex; align-items: center; gap: 8px; flex: 1 1 380px; max-width: 640px; } +.find-field { position: relative; display: flex; flex: 1; min-width: 0; } +.find-icon { position: absolute; left: 14px; top: 50%; width: 22px; height: 22px; transform: translateY(-50%); + fill: none; stroke: var(--muted-foreground); stroke-width: 2.2; stroke-linecap: round; pointer-events: none; } +.find-bar input { width: 100%; padding: 10px 14px 10px 46px; border: 1px solid var(--border); + border-radius: var(--radius); font: inherit; font-family: var(--font-sans); font-size: 20px; + background: var(--card); color: var(--foreground); } +.find-bar input:focus { outline: 2px solid var(--primary); outline-offset: 1px; border-color: var(--primary); } +.find-bar button { padding: 8px 12px; border: 1px solid var(--border); border-radius: var(--radius); + background: var(--card); cursor: pointer; font-family: var(--font-sans); font-size: 15px; } .find-bar button:hover { background: var(--secondary); } .find-bar button[disabled] { opacity: 0.4; cursor: default; } .find-counter { font-variant-numeric: tabular-nums; font-size: 12px; color: var(--muted-foreground); min-width: 3.5em; text-align: center; } -mark.find-hit { background: var(--accent); color: inherit; border-radius: 2px; scroll-margin-top: 64px; } +mark.find-hit { background: var(--accent); color: inherit; border-radius: 2px; + scroll-margin-top: calc(var(--sticky-bar-height) + 8px); } mark.find-hit--current { background: var(--gold); color: #fff; } .nav-controls { display: inline-flex; align-items: center; gap: 4px; } .nav-controls button { padding: 6px 12px; border: 1px solid var(--border); border-radius: var(--radius); @@ -244,9 +272,13 @@ .prompt-copy:hover { background: var(--accent); } .prompt-text { line-height: 1.5; } -/* Nav targets clear the sticky action bar when scrolled to via Prev/Next */ +/* Nav targets clear the sticky action bar when scrolled to via Prev/Next or a link: + --sticky-bar-height is the bar's measured height (padForStickyBar), plus a small gap. A + fixed 64px cleared the one-row bar but not the two-row one. On the targets, not as the + page's scroll-padding: that would also count the bar's own controls as covered, so + clicking or tabbing to one would scroll the page. */ .change-card, .full-bill [id^="attr-"], .full-bill [id^="sec-"], .full-bill [id^="fb-off-"], -.removed-block { scroll-margin-top: 64px; } +.removed-block { scroll-margin-top: calc(var(--sticky-bar-height) + 8px); } /* Full-bill section TOC (sidebar variant) */ .sidebar-changes[hidden], .sidebar-toc[hidden] { display: none; } @@ -295,7 +327,25 @@ .find-bar { position: fixed; left: 0; right: 0; bottom: 46px; z-index: 35; justify-content: center; background: var(--card); border-top: 1px solid var(--border); padding: 8px 16px; } - .find-bar input { flex: 1; max-width: 320px; } + .find-bar { max-width: none; } + body.outside-changes { padding-bottom: 62px; } + body.outside-changes .find-bar { bottom: 0; padding-bottom: calc(8px + env(safe-area-inset-bottom)); } +} +/* Change navigation and the changes export serve Changes and Full bill only. */ +body.outside-changes .nav-controls, body.outside-changes #export-open { display: none; } + +/* Five view buttons get a row of their own when the financial views are present (ADR 0023). + The views' own styles live with them, in formatters/financial_views.py. */ +.action-bar--stacked { display: grid; grid-template-columns: 1fr auto; row-gap: 10px; align-items: center; } +.action-bar--stacked .action-bar__left { display: contents; } +.action-bar--stacked .view-toggle { grid-column: 1 / -1; justify-self: start; } +.action-bar--stacked .find-bar { grid-column: 1; } +.action-bar--stacked .action-bar__group { grid-column: 2; } +.fin-scroll { overflow-x: auto; } +@media (max-width: 820px) { + .action-bar--stacked { display: flex; flex-direction: column; align-items: stretch; } + .view-toggle { display: flex; flex-wrap: wrap; } + .view-toggle__btn { flex: 1 1 auto; } } /* Print */ @@ -304,6 +354,133 @@ .main { margin-left: 0; } .change-card { break-inside: avoid; } } +/* Financial views (ADR 0023): Version A / B, and the comparison */ +/* The three financial tabs use the full window on a wide screen: their tables are columns, + cramped at the report's 940px reading width. Changes and Full bill keep that width (their + cards are prose). `body[data-active-view]` is set by the report's view switch. */ +@media (min-width: 821px) { + body[data-active-view="fin-a"] .main, body[data-active-view="fin-b"] .main, + body[data-active-view="fin-compare"] .main { + max-width: none; padding-right: 10%; } +} +.fin-head { display: flex; align-items: baseline; justify-content: space-between; gap: 12px; flex-wrap: wrap; } +.view-fin-a h2, .view-fin-b h2, .view-fin-compare h2 { font-size: 20px; margin-bottom: 2px; } +.fin-version { color: var(--muted-foreground); font-size: 14px; margin-bottom: 12px; } +.fin-coverage, .fin-note, .fin-explain { font-size: 13px; color: var(--muted-foreground); margin-bottom: 10px; } +/* "About these figures": an information alert, left-accented, above each view's content. */ +.fin-alert { display: flex; gap: 12px; align-items: flex-start; margin: 4px 0 16px; padding: 12px 16px; + background: var(--info); color: var(--info-foreground); border: 1px solid var(--info); + border-left: 4px solid var(--info-foreground); border-radius: var(--radius); font-size: 14px; line-height: 1.55; } +.fin-alert__icon { flex: none; width: 20px; height: 20px; margin-top: 1px; fill: none; + stroke: var(--info-foreground); stroke-width: 2; stroke-linecap: round; } +.fin-alert__dot { fill: var(--info-foreground); } +.fin-alert a { color: inherit; font-weight: 600; text-decoration: underline; } +.fin-alert__version { display: block; margin-top: 4px; font-size: 12px; opacity: 0.85; } +.fin-coverage--gap { color: var(--destructive); font-weight: 600; } +.fin-empty { color: var(--muted-foreground); padding: 16px 2px; font-size: 14px; } +.view-fin-a h3, .view-fin-b h3 { font-size: 18px; margin: 18px 0 10px; } +.fin-key { border: 1px solid var(--border); border-radius: var(--radius); padding: 12px 14px; + background: var(--card); margin-bottom: 8px; } +.fin-key__title { font-weight: 600; font-size: 14px; margin-bottom: 8px; } +.fin-key__item { display: flex; align-items: baseline; gap: 8px; padding: 2px 0; } +.fin-key__count { color: var(--muted-foreground); font-size: 12px; min-width: 32px; display: inline-block; } +.fin-key__desc { color: var(--muted-foreground); font-size: 13px; } +.fin-type { display: inline-block; padding: 2px 7px; border-radius: 999px; font-size: 11px; font-weight: 600; + white-space: nowrap; background: var(--muted); color: var(--muted-foreground); } +.fin-type--appropriation { background: var(--diff-add); color: var(--diff-add-foreground); } +.fin-type--rescission, .fin-type--restriction { background: var(--diff-remove); color: var(--diff-remove-foreground); } +.fin-type--cap { background: var(--diff-modified); color: var(--diff-modified-foreground); } +.fin-type--transfer { background: var(--fin-transfer); color: var(--fin-transfer-foreground); } +.fin-type--authorization { background: var(--fin-authorization); color: var(--fin-authorization-foreground); } +.fin-type--fee { background: var(--fin-fee); color: var(--fin-fee-foreground); } +.fin-type--directive { background: var(--fin-directive); color: var(--fin-directive-foreground); } +.fin-type--earmark { background: var(--fin-earmark); color: var(--fin-earmark-foreground); } +.fin-type--availability { background: var(--fin-availability); color: var(--fin-availability-foreground); } +.fin-type--sub_allocation { background: var(--fin-sub-allocation); color: var(--fin-sub-allocation-foreground); } +.fin-sort { display: flex; gap: 8px; align-items: center; margin-bottom: 12px; font-size: 13px; } +.fin-sort span { color: var(--muted-foreground); } +.fin-sort__btn { padding: 4px 12px; border: 1px solid var(--border); border-radius: 999px; background: var(--card); + cursor: pointer; font: inherit; font-family: var(--font-sans); font-size: 13px; color: var(--foreground); } +.fin-sort__btn:hover { background: var(--secondary); } +.fin-sort__btn.is-active { background: var(--foreground); border-color: var(--foreground); color: var(--card); } +.fin-table { width: 100%; border-collapse: collapse; background: var(--card); border-radius: var(--radius); + box-shadow: var(--shadow-soft); overflow: hidden; font-size: 14px; table-layout: fixed; } +.fin-table th { background: var(--accent); text-align: left; padding: 9px 12px; border-bottom: 2px solid var(--border); + font-weight: 600; font-size: 13px; } +.fin-table td { padding: 8px 12px; border-bottom: 1px solid var(--border); vertical-align: middle; } +.fin-col-name { width: 72%; } .fin-col-type { width: 13%; } .fin-col-amount { width: 15%; text-align: right; } +.fin-amount { text-align: right; font-family: var(--font-mono); font-variant-numeric: tabular-nums; + white-space: nowrap; } +.fin-row, .fin-crow { cursor: pointer; } +.fin-row { background: var(--muted); font-weight: 600; } +.fin-row:hover td, .fin-crow:hover td { background: var(--secondary); } +.fin-row__name::before { content: "\25b6"; font-size: 10px; margin-right: 8px; color: var(--muted-foreground); + display: inline-block; transition: transform 0.15s; } +.fin-row.is-open .fin-row__name::before { transform: rotate(90deg); } +.fin-count { font-size: 11px; color: var(--muted-foreground); font-weight: 400; margin-left: 6px; } +.fin-review { background: var(--diff-modified); color: var(--diff-modified-foreground); font-size: 11px; + padding: 1px 6px; border-radius: 999px; margin-left: 6px; font-weight: 500; white-space: nowrap; } +/* The warning mark (`_WARNING_ICON`): drawn in the surrounding text's colour and size. */ +.fin-icon { display: inline-block; width: 1.15em; height: 1.15em; vertical-align: -0.2em; margin-right: 0.25em; + fill: none; stroke: currentColor; stroke-width: 2; stroke-linejoin: round; stroke-linecap: round; } +.fin-icon__dot { fill: currentColor; } +.fin-review[role="img"] .fin-icon { margin-right: 0; } +.fin-chip { font-size: 11px; font-weight: 600; border-radius: 4px; padding: 0 5px; margin-left: 6px; } +.fin-chip--law { background: var(--muted); color: var(--muted-foreground); border: 1px dashed var(--border); } +.fin-detail > td { padding: 0; background: var(--card); } +.fin-detail__inner { display: flex; } +.fin-detail__amounts { min-width: 220px; max-width: 260px; padding: 12px; border-right: 1px solid var(--border); + display: flex; flex-direction: column; gap: 10px; } +.fin-clause { display: flex; flex-direction: column; gap: 3px; align-items: flex-start; } +.fin-clause__amount { font-family: var(--font-mono); font-size: 15px; font-variant-numeric: tabular-nums; } +.fin-detail__text { padding: 12px 16px; font-size: 13px; line-height: 1.9; flex: 1; font-weight: 400; } +.fin-piece { border-radius: 3px; padding: 1px 3px; } +.fin-piece.fin-type--unknown { background: var(--muted); color: var(--foreground); } +.fin-proviso { color: var(--muted-foreground); } +.fin-card { background: var(--card); border: 1px solid var(--border); border-radius: var(--radius); overflow: hidden; + box-shadow: var(--shadow-soft); } +.fin-card .fin-table { box-shadow: none; border-radius: 0; table-layout: auto; } +.fin-summary { display: flex; gap: 18px; flex-wrap: wrap; padding: 10px 14px; border-bottom: 1px solid var(--border); + font-size: 13px; color: var(--muted-foreground); } +.fin-summary b { color: var(--foreground); font-variant-numeric: tabular-nums; } +.fin-table--compare td { vertical-align: top; } +.fin-table--compare th { background: var(--secondary); font-size: 12px; text-transform: uppercase; + letter-spacing: 0.04em; color: var(--muted-foreground); border-bottom: 1px solid var(--border); padding: 8px 10px; } +.fin-table--compare td { padding: 8px 10px; } +.fin-table--compare .fin-amount { font-family: var(--font-sans); } +.fin-stack__tag { white-space: nowrap; } +.fin-caret { color: var(--muted-foreground); width: 16px; display: inline-block; } +.fin-loc { font-weight: 600; } +.fin-loc small { display: block; font-weight: 400; color: var(--muted-foreground); font-size: 12px; } +/* Links from a comparison row to its sections in Version A / B. */ +.fin-jumps { display: flex; flex-wrap: wrap; gap: 4px 14px; margin-top: 6px; } +.fin-jump { background: none; border: 0; padding: 0; font: inherit; font-size: 12px; color: var(--primary); + text-decoration: underline; text-underline-offset: 2px; cursor: pointer; } +.fin-jump:hover { color: var(--foreground); } +.fin-jump--none { color: var(--muted-foreground); text-decoration: none; cursor: default; } +.fin-row.is-target > td { background: var(--accent); transition: background 0.6s ease; } +.fin-stack { display: grid; grid-template-columns: auto 1fr auto; gap: 2px 8px; align-items: center; } +.fin-ab { font-size: 11px; font-weight: 700; border-radius: 4px; padding: 0 5px; } +.fin-ab--a { background: var(--secondary); color: var(--primary); } +.fin-ab--b { background: var(--accent); color: var(--gold); } +.fin-stack__amt { text-align: right; font-variant-numeric: tabular-nums; white-space: nowrap; } +.fin-stack__amt--old { color: var(--muted-foreground); } +.fin-stack__amt--new { font-weight: 600; } +.fin-stack__none { color: var(--muted-foreground); font-style: italic; text-align: right; } +.fin-up { color: var(--diff-add-foreground); } .fin-down { color: var(--diff-remove-foreground); } +.fin-same { color: var(--muted-foreground); } +.fin-same small, .fin-up small, .fin-down small { display: block; font-size: 11px; } +.fin-cdetail > td { background: var(--background); padding: 12px 14px; } +.fin-subhead { font-size: 13px; font-weight: 600; color: var(--muted-foreground); padding: 14px 14px 6px; + border-top: 1px solid var(--border); } +.fin-table--plain { box-shadow: none; } +@media (max-width: 820px) { + .fin-table--compare { min-width: 640px; } + .fin-detail__inner { flex-direction: column; } + .fin-detail__amounts { max-width: none; border-right: 0; border-bottom: 1px solid var(--border); } + .fin-col-name { width: 55%; } .fin-col-type { width: 20%; } .fin-col-amount { width: 25%; } +} + @@ -324,17 +501,17 @@

H.R. 4366 — Making appropriations for military construction, the Department of Veterans Affairs, and related agencies for the fiscal year ending September 30, 2024, and for other purposes.

-
v1: reported-in-house → v2: engrossed-in-house · 118th Congress
+
Before (v1): reported-in-houseAfter (v2): engrossed-in-house118th Congress
modified 17added 7moved 1
-
+
-
- +
+
- +

Changes

@@ -602,31 +779,72 @@

TITLE IV > GENERAL PROVISIONS > sec. 425

-
+
- + \ No newline at end of file diff --git a/examples/hr4366_house_vs_senate_xml_diff.html b/examples/hr4366_house_vs_senate_xml_diff.html index 340ff42c..6cd1f918 100644 --- a/examples/hr4366_house_vs_senate_xml_diff.html +++ b/examples/hr4366_house_vs_senate_xml_diff.html @@ -19,6 +19,8 @@ --gold: #c9944e; --destructive: #c04040; --success: #3d9b6d; + --info: #e3f2f8; + --info-foreground: #0b4f63; --diff-add: #d3f0e2; --diff-add-foreground: #1a6647; --diff-remove: #f5ddd8; @@ -27,7 +29,22 @@ --diff-modified-foreground: #8a6320; --diff-moved: #eef0f8; --diff-moved-foreground: #2c2c5c; + --fin-transfer: #cffafe; + --fin-transfer-foreground: #155e75; + --fin-authorization: #e0f2fe; + --fin-authorization-foreground: #0c4a6e; + --fin-fee: #ffedd5; + --fin-fee-foreground: #9a3412; + --fin-directive: #e0e7ff; + --fin-directive-foreground: #3730a3; + --fin-earmark: #ede9fe; + --fin-earmark-foreground: #5b21b6; + --fin-availability: #d1fae5; + --fin-availability-foreground: #065f46; + --fin-sub-allocation: #dbeafe; + --fin-sub-allocation-foreground: #1e40af; --radius: 0.625rem; + --sticky-bar-height: 64px; --font-sans: ui-sans-serif, system-ui, -apple-system, 'Segoe UI', Roboto, Helvetica, Arial, sans-serif; --font-serif: ui-serif, Georgia, 'Times New Roman', serif; --font-mono: ui-monospace, 'SF Mono', Menlo, Consolas, monospace; @@ -88,11 +105,13 @@ .filter-empty[hidden] { display: none; } /* Main content */ -.main { margin-left: 280px; padding: 28px 36px; max-width: 940px; flex: 1; } +.main { margin-left: 280px; padding: 28px 36px; max-width: 940px; flex: 1; min-width: 0; } /* Header */ .report-header h1 { font-size: 24px; margin-bottom: 4px; } -.report-header .versions { color: var(--muted-foreground); font-size: 15px; margin-bottom: 16px; } +.report-header .versions { color: var(--muted-foreground); font-size: 15px; margin-bottom: 16px; line-height: 1.5; } +.versions__row { display: block; overflow-wrap: anywhere; } +.versions__row b { color: var(--foreground); font-weight: 600; } .summary-bar { display: flex; gap: 10px; margin-bottom: 24px; flex-wrap: wrap; } .summary-item { display: inline-flex; align-items: center; gap: 6px; padding: 4px 12px; border-radius: 999px; font-size: 13px; background: var(--secondary); } @@ -164,16 +183,25 @@ border-bottom: 1px solid var(--border); padding: 10px 0; margin-bottom: 16px; } .action-bar__left { display: flex; align-items: center; gap: 10px; flex-wrap: wrap; } .action-bar__group { display: flex; align-items: center; gap: 10px; flex-wrap: wrap; } -.find-bar { display: inline-flex; align-items: center; gap: 4px; } -.find-bar input { padding: 5px 10px; border: 1px solid var(--border); border-radius: var(--radius); - font: inherit; font-family: var(--font-sans); font-size: 13px; width: 180px; background: var(--card); } -.find-bar button { padding: 5px 9px; border: 1px solid var(--border); border-radius: var(--radius); - background: var(--card); cursor: pointer; font-family: var(--font-sans); font-size: 13px; } +.action-bar__left { flex: 1 1 auto; } +/* The page's one search: it reaches text that is collapsed, which the browser's own find + cannot, so it is sized and marked to be found first. */ +.find-bar { display: inline-flex; align-items: center; gap: 8px; flex: 1 1 380px; max-width: 640px; } +.find-field { position: relative; display: flex; flex: 1; min-width: 0; } +.find-icon { position: absolute; left: 14px; top: 50%; width: 22px; height: 22px; transform: translateY(-50%); + fill: none; stroke: var(--muted-foreground); stroke-width: 2.2; stroke-linecap: round; pointer-events: none; } +.find-bar input { width: 100%; padding: 10px 14px 10px 46px; border: 1px solid var(--border); + border-radius: var(--radius); font: inherit; font-family: var(--font-sans); font-size: 20px; + background: var(--card); color: var(--foreground); } +.find-bar input:focus { outline: 2px solid var(--primary); outline-offset: 1px; border-color: var(--primary); } +.find-bar button { padding: 8px 12px; border: 1px solid var(--border); border-radius: var(--radius); + background: var(--card); cursor: pointer; font-family: var(--font-sans); font-size: 15px; } .find-bar button:hover { background: var(--secondary); } .find-bar button[disabled] { opacity: 0.4; cursor: default; } .find-counter { font-variant-numeric: tabular-nums; font-size: 12px; color: var(--muted-foreground); min-width: 3.5em; text-align: center; } -mark.find-hit { background: var(--accent); color: inherit; border-radius: 2px; scroll-margin-top: 64px; } +mark.find-hit { background: var(--accent); color: inherit; border-radius: 2px; + scroll-margin-top: calc(var(--sticky-bar-height) + 8px); } mark.find-hit--current { background: var(--gold); color: #fff; } .nav-controls { display: inline-flex; align-items: center; gap: 4px; } .nav-controls button { padding: 6px 12px; border: 1px solid var(--border); border-radius: var(--radius); @@ -244,9 +272,13 @@ .prompt-copy:hover { background: var(--accent); } .prompt-text { line-height: 1.5; } -/* Nav targets clear the sticky action bar when scrolled to via Prev/Next */ +/* Nav targets clear the sticky action bar when scrolled to via Prev/Next or a link: + --sticky-bar-height is the bar's measured height (padForStickyBar), plus a small gap. A + fixed 64px cleared the one-row bar but not the two-row one. On the targets, not as the + page's scroll-padding: that would also count the bar's own controls as covered, so + clicking or tabbing to one would scroll the page. */ .change-card, .full-bill [id^="attr-"], .full-bill [id^="sec-"], .full-bill [id^="fb-off-"], -.removed-block { scroll-margin-top: 64px; } +.removed-block { scroll-margin-top: calc(var(--sticky-bar-height) + 8px); } /* Full-bill section TOC (sidebar variant) */ .sidebar-changes[hidden], .sidebar-toc[hidden] { display: none; } @@ -295,7 +327,25 @@ .find-bar { position: fixed; left: 0; right: 0; bottom: 46px; z-index: 35; justify-content: center; background: var(--card); border-top: 1px solid var(--border); padding: 8px 16px; } - .find-bar input { flex: 1; max-width: 320px; } + .find-bar { max-width: none; } + body.outside-changes { padding-bottom: 62px; } + body.outside-changes .find-bar { bottom: 0; padding-bottom: calc(8px + env(safe-area-inset-bottom)); } +} +/* Change navigation and the changes export serve Changes and Full bill only. */ +body.outside-changes .nav-controls, body.outside-changes #export-open { display: none; } + +/* Five view buttons get a row of their own when the financial views are present (ADR 0023). + The views' own styles live with them, in formatters/financial_views.py. */ +.action-bar--stacked { display: grid; grid-template-columns: 1fr auto; row-gap: 10px; align-items: center; } +.action-bar--stacked .action-bar__left { display: contents; } +.action-bar--stacked .view-toggle { grid-column: 1 / -1; justify-self: start; } +.action-bar--stacked .find-bar { grid-column: 1; } +.action-bar--stacked .action-bar__group { grid-column: 2; } +.fin-scroll { overflow-x: auto; } +@media (max-width: 820px) { + .action-bar--stacked { display: flex; flex-direction: column; align-items: stretch; } + .view-toggle { display: flex; flex-wrap: wrap; } + .view-toggle__btn { flex: 1 1 auto; } } /* Print */ @@ -304,6 +354,133 @@ .main { margin-left: 0; } .change-card { break-inside: avoid; } } +/* Financial views (ADR 0023): Version A / B, and the comparison */ +/* The three financial tabs use the full window on a wide screen: their tables are columns, + cramped at the report's 940px reading width. Changes and Full bill keep that width (their + cards are prose). `body[data-active-view]` is set by the report's view switch. */ +@media (min-width: 821px) { + body[data-active-view="fin-a"] .main, body[data-active-view="fin-b"] .main, + body[data-active-view="fin-compare"] .main { + max-width: none; padding-right: 10%; } +} +.fin-head { display: flex; align-items: baseline; justify-content: space-between; gap: 12px; flex-wrap: wrap; } +.view-fin-a h2, .view-fin-b h2, .view-fin-compare h2 { font-size: 20px; margin-bottom: 2px; } +.fin-version { color: var(--muted-foreground); font-size: 14px; margin-bottom: 12px; } +.fin-coverage, .fin-note, .fin-explain { font-size: 13px; color: var(--muted-foreground); margin-bottom: 10px; } +/* "About these figures": an information alert, left-accented, above each view's content. */ +.fin-alert { display: flex; gap: 12px; align-items: flex-start; margin: 4px 0 16px; padding: 12px 16px; + background: var(--info); color: var(--info-foreground); border: 1px solid var(--info); + border-left: 4px solid var(--info-foreground); border-radius: var(--radius); font-size: 14px; line-height: 1.55; } +.fin-alert__icon { flex: none; width: 20px; height: 20px; margin-top: 1px; fill: none; + stroke: var(--info-foreground); stroke-width: 2; stroke-linecap: round; } +.fin-alert__dot { fill: var(--info-foreground); } +.fin-alert a { color: inherit; font-weight: 600; text-decoration: underline; } +.fin-alert__version { display: block; margin-top: 4px; font-size: 12px; opacity: 0.85; } +.fin-coverage--gap { color: var(--destructive); font-weight: 600; } +.fin-empty { color: var(--muted-foreground); padding: 16px 2px; font-size: 14px; } +.view-fin-a h3, .view-fin-b h3 { font-size: 18px; margin: 18px 0 10px; } +.fin-key { border: 1px solid var(--border); border-radius: var(--radius); padding: 12px 14px; + background: var(--card); margin-bottom: 8px; } +.fin-key__title { font-weight: 600; font-size: 14px; margin-bottom: 8px; } +.fin-key__item { display: flex; align-items: baseline; gap: 8px; padding: 2px 0; } +.fin-key__count { color: var(--muted-foreground); font-size: 12px; min-width: 32px; display: inline-block; } +.fin-key__desc { color: var(--muted-foreground); font-size: 13px; } +.fin-type { display: inline-block; padding: 2px 7px; border-radius: 999px; font-size: 11px; font-weight: 600; + white-space: nowrap; background: var(--muted); color: var(--muted-foreground); } +.fin-type--appropriation { background: var(--diff-add); color: var(--diff-add-foreground); } +.fin-type--rescission, .fin-type--restriction { background: var(--diff-remove); color: var(--diff-remove-foreground); } +.fin-type--cap { background: var(--diff-modified); color: var(--diff-modified-foreground); } +.fin-type--transfer { background: var(--fin-transfer); color: var(--fin-transfer-foreground); } +.fin-type--authorization { background: var(--fin-authorization); color: var(--fin-authorization-foreground); } +.fin-type--fee { background: var(--fin-fee); color: var(--fin-fee-foreground); } +.fin-type--directive { background: var(--fin-directive); color: var(--fin-directive-foreground); } +.fin-type--earmark { background: var(--fin-earmark); color: var(--fin-earmark-foreground); } +.fin-type--availability { background: var(--fin-availability); color: var(--fin-availability-foreground); } +.fin-type--sub_allocation { background: var(--fin-sub-allocation); color: var(--fin-sub-allocation-foreground); } +.fin-sort { display: flex; gap: 8px; align-items: center; margin-bottom: 12px; font-size: 13px; } +.fin-sort span { color: var(--muted-foreground); } +.fin-sort__btn { padding: 4px 12px; border: 1px solid var(--border); border-radius: 999px; background: var(--card); + cursor: pointer; font: inherit; font-family: var(--font-sans); font-size: 13px; color: var(--foreground); } +.fin-sort__btn:hover { background: var(--secondary); } +.fin-sort__btn.is-active { background: var(--foreground); border-color: var(--foreground); color: var(--card); } +.fin-table { width: 100%; border-collapse: collapse; background: var(--card); border-radius: var(--radius); + box-shadow: var(--shadow-soft); overflow: hidden; font-size: 14px; table-layout: fixed; } +.fin-table th { background: var(--accent); text-align: left; padding: 9px 12px; border-bottom: 2px solid var(--border); + font-weight: 600; font-size: 13px; } +.fin-table td { padding: 8px 12px; border-bottom: 1px solid var(--border); vertical-align: middle; } +.fin-col-name { width: 72%; } .fin-col-type { width: 13%; } .fin-col-amount { width: 15%; text-align: right; } +.fin-amount { text-align: right; font-family: var(--font-mono); font-variant-numeric: tabular-nums; + white-space: nowrap; } +.fin-row, .fin-crow { cursor: pointer; } +.fin-row { background: var(--muted); font-weight: 600; } +.fin-row:hover td, .fin-crow:hover td { background: var(--secondary); } +.fin-row__name::before { content: "\25b6"; font-size: 10px; margin-right: 8px; color: var(--muted-foreground); + display: inline-block; transition: transform 0.15s; } +.fin-row.is-open .fin-row__name::before { transform: rotate(90deg); } +.fin-count { font-size: 11px; color: var(--muted-foreground); font-weight: 400; margin-left: 6px; } +.fin-review { background: var(--diff-modified); color: var(--diff-modified-foreground); font-size: 11px; + padding: 1px 6px; border-radius: 999px; margin-left: 6px; font-weight: 500; white-space: nowrap; } +/* The warning mark (`_WARNING_ICON`): drawn in the surrounding text's colour and size. */ +.fin-icon { display: inline-block; width: 1.15em; height: 1.15em; vertical-align: -0.2em; margin-right: 0.25em; + fill: none; stroke: currentColor; stroke-width: 2; stroke-linejoin: round; stroke-linecap: round; } +.fin-icon__dot { fill: currentColor; } +.fin-review[role="img"] .fin-icon { margin-right: 0; } +.fin-chip { font-size: 11px; font-weight: 600; border-radius: 4px; padding: 0 5px; margin-left: 6px; } +.fin-chip--law { background: var(--muted); color: var(--muted-foreground); border: 1px dashed var(--border); } +.fin-detail > td { padding: 0; background: var(--card); } +.fin-detail__inner { display: flex; } +.fin-detail__amounts { min-width: 220px; max-width: 260px; padding: 12px; border-right: 1px solid var(--border); + display: flex; flex-direction: column; gap: 10px; } +.fin-clause { display: flex; flex-direction: column; gap: 3px; align-items: flex-start; } +.fin-clause__amount { font-family: var(--font-mono); font-size: 15px; font-variant-numeric: tabular-nums; } +.fin-detail__text { padding: 12px 16px; font-size: 13px; line-height: 1.9; flex: 1; font-weight: 400; } +.fin-piece { border-radius: 3px; padding: 1px 3px; } +.fin-piece.fin-type--unknown { background: var(--muted); color: var(--foreground); } +.fin-proviso { color: var(--muted-foreground); } +.fin-card { background: var(--card); border: 1px solid var(--border); border-radius: var(--radius); overflow: hidden; + box-shadow: var(--shadow-soft); } +.fin-card .fin-table { box-shadow: none; border-radius: 0; table-layout: auto; } +.fin-summary { display: flex; gap: 18px; flex-wrap: wrap; padding: 10px 14px; border-bottom: 1px solid var(--border); + font-size: 13px; color: var(--muted-foreground); } +.fin-summary b { color: var(--foreground); font-variant-numeric: tabular-nums; } +.fin-table--compare td { vertical-align: top; } +.fin-table--compare th { background: var(--secondary); font-size: 12px; text-transform: uppercase; + letter-spacing: 0.04em; color: var(--muted-foreground); border-bottom: 1px solid var(--border); padding: 8px 10px; } +.fin-table--compare td { padding: 8px 10px; } +.fin-table--compare .fin-amount { font-family: var(--font-sans); } +.fin-stack__tag { white-space: nowrap; } +.fin-caret { color: var(--muted-foreground); width: 16px; display: inline-block; } +.fin-loc { font-weight: 600; } +.fin-loc small { display: block; font-weight: 400; color: var(--muted-foreground); font-size: 12px; } +/* Links from a comparison row to its sections in Version A / B. */ +.fin-jumps { display: flex; flex-wrap: wrap; gap: 4px 14px; margin-top: 6px; } +.fin-jump { background: none; border: 0; padding: 0; font: inherit; font-size: 12px; color: var(--primary); + text-decoration: underline; text-underline-offset: 2px; cursor: pointer; } +.fin-jump:hover { color: var(--foreground); } +.fin-jump--none { color: var(--muted-foreground); text-decoration: none; cursor: default; } +.fin-row.is-target > td { background: var(--accent); transition: background 0.6s ease; } +.fin-stack { display: grid; grid-template-columns: auto 1fr auto; gap: 2px 8px; align-items: center; } +.fin-ab { font-size: 11px; font-weight: 700; border-radius: 4px; padding: 0 5px; } +.fin-ab--a { background: var(--secondary); color: var(--primary); } +.fin-ab--b { background: var(--accent); color: var(--gold); } +.fin-stack__amt { text-align: right; font-variant-numeric: tabular-nums; white-space: nowrap; } +.fin-stack__amt--old { color: var(--muted-foreground); } +.fin-stack__amt--new { font-weight: 600; } +.fin-stack__none { color: var(--muted-foreground); font-style: italic; text-align: right; } +.fin-up { color: var(--diff-add-foreground); } .fin-down { color: var(--diff-remove-foreground); } +.fin-same { color: var(--muted-foreground); } +.fin-same small, .fin-up small, .fin-down small { display: block; font-size: 11px; } +.fin-cdetail > td { background: var(--background); padding: 12px 14px; } +.fin-subhead { font-size: 13px; font-weight: 600; color: var(--muted-foreground); padding: 14px 14px 6px; + border-top: 1px solid var(--border); } +.fin-table--plain { box-shadow: none; } +@media (max-width: 820px) { + .fin-table--compare { min-width: 640px; } + .fin-detail__inner { flex-direction: column; } + .fin-detail__amounts { max-width: none; border-right: 0; border-bottom: 1px solid var(--border); } + .fin-col-name { width: 55%; } .fin-col-type { width: 20%; } .fin-col-amount { width: 25%; } +} + @@ -324,17 +501,17 @@

H.R. 4366

-
v2: engrossed-in-house → v4: engrossed-amendment-senate · 118th Congress
+
Before (v2): engrossed-in-houseAfter (v4): engrossed-amendment-senate118th Congress
modified 48added 561removed 41moved 53
-
+
-
- +
+
- +

Changes

@@ -7835,31 +8012,72 @@

Division C: Transportation, Housing and Urban Development, and Related Agenc (4) Executive Order No. 14030, relating to Climate-Related Financial Risk. (5) Executive Order No. 14057, relating to Catalyzing Clean Energy Industries and Jobs Through Federal Sustainability. (6) Executive Order No. 14082, relating to Implementation of the Energy and Infrastructure Provisions of the Inflation Reduction Act of 2022. - (7) Executive Order No. 14096, relating to Revitalizing Our Nation's Commitment to Environmental Justice for All.

+ (7) Executive Order No. 14096, relating to Revitalizing Our Nation's Commitment to Environmental Justice for All.
- + \ No newline at end of file diff --git a/examples/hr8752_pdf_diff.html b/examples/hr8752_pdf_diff.html index 1ec03259..f224bfc1 100644 --- a/examples/hr8752_pdf_diff.html +++ b/examples/hr8752_pdf_diff.html @@ -19,6 +19,8 @@ --gold: #c9944e; --destructive: #c04040; --success: #3d9b6d; + --info: #e3f2f8; + --info-foreground: #0b4f63; --diff-add: #d3f0e2; --diff-add-foreground: #1a6647; --diff-remove: #f5ddd8; @@ -27,7 +29,22 @@ --diff-modified-foreground: #8a6320; --diff-moved: #eef0f8; --diff-moved-foreground: #2c2c5c; + --fin-transfer: #cffafe; + --fin-transfer-foreground: #155e75; + --fin-authorization: #e0f2fe; + --fin-authorization-foreground: #0c4a6e; + --fin-fee: #ffedd5; + --fin-fee-foreground: #9a3412; + --fin-directive: #e0e7ff; + --fin-directive-foreground: #3730a3; + --fin-earmark: #ede9fe; + --fin-earmark-foreground: #5b21b6; + --fin-availability: #d1fae5; + --fin-availability-foreground: #065f46; + --fin-sub-allocation: #dbeafe; + --fin-sub-allocation-foreground: #1e40af; --radius: 0.625rem; + --sticky-bar-height: 64px; --font-sans: ui-sans-serif, system-ui, -apple-system, 'Segoe UI', Roboto, Helvetica, Arial, sans-serif; --font-serif: ui-serif, Georgia, 'Times New Roman', serif; --font-mono: ui-monospace, 'SF Mono', Menlo, Consolas, monospace; @@ -88,11 +105,13 @@ .filter-empty[hidden] { display: none; } /* Main content */ -.main { margin-left: 280px; padding: 28px 36px; max-width: 940px; flex: 1; } +.main { margin-left: 280px; padding: 28px 36px; max-width: 940px; flex: 1; min-width: 0; } /* Header */ .report-header h1 { font-size: 24px; margin-bottom: 4px; } -.report-header .versions { color: var(--muted-foreground); font-size: 15px; margin-bottom: 16px; } +.report-header .versions { color: var(--muted-foreground); font-size: 15px; margin-bottom: 16px; line-height: 1.5; } +.versions__row { display: block; overflow-wrap: anywhere; } +.versions__row b { color: var(--foreground); font-weight: 600; } .summary-bar { display: flex; gap: 10px; margin-bottom: 24px; flex-wrap: wrap; } .summary-item { display: inline-flex; align-items: center; gap: 6px; padding: 4px 12px; border-radius: 999px; font-size: 13px; background: var(--secondary); } @@ -164,16 +183,25 @@ border-bottom: 1px solid var(--border); padding: 10px 0; margin-bottom: 16px; } .action-bar__left { display: flex; align-items: center; gap: 10px; flex-wrap: wrap; } .action-bar__group { display: flex; align-items: center; gap: 10px; flex-wrap: wrap; } -.find-bar { display: inline-flex; align-items: center; gap: 4px; } -.find-bar input { padding: 5px 10px; border: 1px solid var(--border); border-radius: var(--radius); - font: inherit; font-family: var(--font-sans); font-size: 13px; width: 180px; background: var(--card); } -.find-bar button { padding: 5px 9px; border: 1px solid var(--border); border-radius: var(--radius); - background: var(--card); cursor: pointer; font-family: var(--font-sans); font-size: 13px; } +.action-bar__left { flex: 1 1 auto; } +/* The page's one search: it reaches text that is collapsed, which the browser's own find + cannot, so it is sized and marked to be found first. */ +.find-bar { display: inline-flex; align-items: center; gap: 8px; flex: 1 1 380px; max-width: 640px; } +.find-field { position: relative; display: flex; flex: 1; min-width: 0; } +.find-icon { position: absolute; left: 14px; top: 50%; width: 22px; height: 22px; transform: translateY(-50%); + fill: none; stroke: var(--muted-foreground); stroke-width: 2.2; stroke-linecap: round; pointer-events: none; } +.find-bar input { width: 100%; padding: 10px 14px 10px 46px; border: 1px solid var(--border); + border-radius: var(--radius); font: inherit; font-family: var(--font-sans); font-size: 20px; + background: var(--card); color: var(--foreground); } +.find-bar input:focus { outline: 2px solid var(--primary); outline-offset: 1px; border-color: var(--primary); } +.find-bar button { padding: 8px 12px; border: 1px solid var(--border); border-radius: var(--radius); + background: var(--card); cursor: pointer; font-family: var(--font-sans); font-size: 15px; } .find-bar button:hover { background: var(--secondary); } .find-bar button[disabled] { opacity: 0.4; cursor: default; } .find-counter { font-variant-numeric: tabular-nums; font-size: 12px; color: var(--muted-foreground); min-width: 3.5em; text-align: center; } -mark.find-hit { background: var(--accent); color: inherit; border-radius: 2px; scroll-margin-top: 64px; } +mark.find-hit { background: var(--accent); color: inherit; border-radius: 2px; + scroll-margin-top: calc(var(--sticky-bar-height) + 8px); } mark.find-hit--current { background: var(--gold); color: #fff; } .nav-controls { display: inline-flex; align-items: center; gap: 4px; } .nav-controls button { padding: 6px 12px; border: 1px solid var(--border); border-radius: var(--radius); @@ -244,9 +272,13 @@ .prompt-copy:hover { background: var(--accent); } .prompt-text { line-height: 1.5; } -/* Nav targets clear the sticky action bar when scrolled to via Prev/Next */ +/* Nav targets clear the sticky action bar when scrolled to via Prev/Next or a link: + --sticky-bar-height is the bar's measured height (padForStickyBar), plus a small gap. A + fixed 64px cleared the one-row bar but not the two-row one. On the targets, not as the + page's scroll-padding: that would also count the bar's own controls as covered, so + clicking or tabbing to one would scroll the page. */ .change-card, .full-bill [id^="attr-"], .full-bill [id^="sec-"], .full-bill [id^="fb-off-"], -.removed-block { scroll-margin-top: 64px; } +.removed-block { scroll-margin-top: calc(var(--sticky-bar-height) + 8px); } /* Full-bill section TOC (sidebar variant) */ .sidebar-changes[hidden], .sidebar-toc[hidden] { display: none; } @@ -295,7 +327,25 @@ .find-bar { position: fixed; left: 0; right: 0; bottom: 46px; z-index: 35; justify-content: center; background: var(--card); border-top: 1px solid var(--border); padding: 8px 16px; } - .find-bar input { flex: 1; max-width: 320px; } + .find-bar { max-width: none; } + body.outside-changes { padding-bottom: 62px; } + body.outside-changes .find-bar { bottom: 0; padding-bottom: calc(8px + env(safe-area-inset-bottom)); } +} +/* Change navigation and the changes export serve Changes and Full bill only. */ +body.outside-changes .nav-controls, body.outside-changes #export-open { display: none; } + +/* Five view buttons get a row of their own when the financial views are present (ADR 0023). + The views' own styles live with them, in formatters/financial_views.py. */ +.action-bar--stacked { display: grid; grid-template-columns: 1fr auto; row-gap: 10px; align-items: center; } +.action-bar--stacked .action-bar__left { display: contents; } +.action-bar--stacked .view-toggle { grid-column: 1 / -1; justify-self: start; } +.action-bar--stacked .find-bar { grid-column: 1; } +.action-bar--stacked .action-bar__group { grid-column: 2; } +.fin-scroll { overflow-x: auto; } +@media (max-width: 820px) { + .action-bar--stacked { display: flex; flex-direction: column; align-items: stretch; } + .view-toggle { display: flex; flex-wrap: wrap; } + .view-toggle__btn { flex: 1 1 auto; } } /* Print */ @@ -304,6 +354,133 @@ .main { margin-left: 0; } .change-card { break-inside: avoid; } } +/* Financial views (ADR 0023): Version A / B, and the comparison */ +/* The three financial tabs use the full window on a wide screen: their tables are columns, + cramped at the report's 940px reading width. Changes and Full bill keep that width (their + cards are prose). `body[data-active-view]` is set by the report's view switch. */ +@media (min-width: 821px) { + body[data-active-view="fin-a"] .main, body[data-active-view="fin-b"] .main, + body[data-active-view="fin-compare"] .main { + max-width: none; padding-right: 10%; } +} +.fin-head { display: flex; align-items: baseline; justify-content: space-between; gap: 12px; flex-wrap: wrap; } +.view-fin-a h2, .view-fin-b h2, .view-fin-compare h2 { font-size: 20px; margin-bottom: 2px; } +.fin-version { color: var(--muted-foreground); font-size: 14px; margin-bottom: 12px; } +.fin-coverage, .fin-note, .fin-explain { font-size: 13px; color: var(--muted-foreground); margin-bottom: 10px; } +/* "About these figures": an information alert, left-accented, above each view's content. */ +.fin-alert { display: flex; gap: 12px; align-items: flex-start; margin: 4px 0 16px; padding: 12px 16px; + background: var(--info); color: var(--info-foreground); border: 1px solid var(--info); + border-left: 4px solid var(--info-foreground); border-radius: var(--radius); font-size: 14px; line-height: 1.55; } +.fin-alert__icon { flex: none; width: 20px; height: 20px; margin-top: 1px; fill: none; + stroke: var(--info-foreground); stroke-width: 2; stroke-linecap: round; } +.fin-alert__dot { fill: var(--info-foreground); } +.fin-alert a { color: inherit; font-weight: 600; text-decoration: underline; } +.fin-alert__version { display: block; margin-top: 4px; font-size: 12px; opacity: 0.85; } +.fin-coverage--gap { color: var(--destructive); font-weight: 600; } +.fin-empty { color: var(--muted-foreground); padding: 16px 2px; font-size: 14px; } +.view-fin-a h3, .view-fin-b h3 { font-size: 18px; margin: 18px 0 10px; } +.fin-key { border: 1px solid var(--border); border-radius: var(--radius); padding: 12px 14px; + background: var(--card); margin-bottom: 8px; } +.fin-key__title { font-weight: 600; font-size: 14px; margin-bottom: 8px; } +.fin-key__item { display: flex; align-items: baseline; gap: 8px; padding: 2px 0; } +.fin-key__count { color: var(--muted-foreground); font-size: 12px; min-width: 32px; display: inline-block; } +.fin-key__desc { color: var(--muted-foreground); font-size: 13px; } +.fin-type { display: inline-block; padding: 2px 7px; border-radius: 999px; font-size: 11px; font-weight: 600; + white-space: nowrap; background: var(--muted); color: var(--muted-foreground); } +.fin-type--appropriation { background: var(--diff-add); color: var(--diff-add-foreground); } +.fin-type--rescission, .fin-type--restriction { background: var(--diff-remove); color: var(--diff-remove-foreground); } +.fin-type--cap { background: var(--diff-modified); color: var(--diff-modified-foreground); } +.fin-type--transfer { background: var(--fin-transfer); color: var(--fin-transfer-foreground); } +.fin-type--authorization { background: var(--fin-authorization); color: var(--fin-authorization-foreground); } +.fin-type--fee { background: var(--fin-fee); color: var(--fin-fee-foreground); } +.fin-type--directive { background: var(--fin-directive); color: var(--fin-directive-foreground); } +.fin-type--earmark { background: var(--fin-earmark); color: var(--fin-earmark-foreground); } +.fin-type--availability { background: var(--fin-availability); color: var(--fin-availability-foreground); } +.fin-type--sub_allocation { background: var(--fin-sub-allocation); color: var(--fin-sub-allocation-foreground); } +.fin-sort { display: flex; gap: 8px; align-items: center; margin-bottom: 12px; font-size: 13px; } +.fin-sort span { color: var(--muted-foreground); } +.fin-sort__btn { padding: 4px 12px; border: 1px solid var(--border); border-radius: 999px; background: var(--card); + cursor: pointer; font: inherit; font-family: var(--font-sans); font-size: 13px; color: var(--foreground); } +.fin-sort__btn:hover { background: var(--secondary); } +.fin-sort__btn.is-active { background: var(--foreground); border-color: var(--foreground); color: var(--card); } +.fin-table { width: 100%; border-collapse: collapse; background: var(--card); border-radius: var(--radius); + box-shadow: var(--shadow-soft); overflow: hidden; font-size: 14px; table-layout: fixed; } +.fin-table th { background: var(--accent); text-align: left; padding: 9px 12px; border-bottom: 2px solid var(--border); + font-weight: 600; font-size: 13px; } +.fin-table td { padding: 8px 12px; border-bottom: 1px solid var(--border); vertical-align: middle; } +.fin-col-name { width: 72%; } .fin-col-type { width: 13%; } .fin-col-amount { width: 15%; text-align: right; } +.fin-amount { text-align: right; font-family: var(--font-mono); font-variant-numeric: tabular-nums; + white-space: nowrap; } +.fin-row, .fin-crow { cursor: pointer; } +.fin-row { background: var(--muted); font-weight: 600; } +.fin-row:hover td, .fin-crow:hover td { background: var(--secondary); } +.fin-row__name::before { content: "\25b6"; font-size: 10px; margin-right: 8px; color: var(--muted-foreground); + display: inline-block; transition: transform 0.15s; } +.fin-row.is-open .fin-row__name::before { transform: rotate(90deg); } +.fin-count { font-size: 11px; color: var(--muted-foreground); font-weight: 400; margin-left: 6px; } +.fin-review { background: var(--diff-modified); color: var(--diff-modified-foreground); font-size: 11px; + padding: 1px 6px; border-radius: 999px; margin-left: 6px; font-weight: 500; white-space: nowrap; } +/* The warning mark (`_WARNING_ICON`): drawn in the surrounding text's colour and size. */ +.fin-icon { display: inline-block; width: 1.15em; height: 1.15em; vertical-align: -0.2em; margin-right: 0.25em; + fill: none; stroke: currentColor; stroke-width: 2; stroke-linejoin: round; stroke-linecap: round; } +.fin-icon__dot { fill: currentColor; } +.fin-review[role="img"] .fin-icon { margin-right: 0; } +.fin-chip { font-size: 11px; font-weight: 600; border-radius: 4px; padding: 0 5px; margin-left: 6px; } +.fin-chip--law { background: var(--muted); color: var(--muted-foreground); border: 1px dashed var(--border); } +.fin-detail > td { padding: 0; background: var(--card); } +.fin-detail__inner { display: flex; } +.fin-detail__amounts { min-width: 220px; max-width: 260px; padding: 12px; border-right: 1px solid var(--border); + display: flex; flex-direction: column; gap: 10px; } +.fin-clause { display: flex; flex-direction: column; gap: 3px; align-items: flex-start; } +.fin-clause__amount { font-family: var(--font-mono); font-size: 15px; font-variant-numeric: tabular-nums; } +.fin-detail__text { padding: 12px 16px; font-size: 13px; line-height: 1.9; flex: 1; font-weight: 400; } +.fin-piece { border-radius: 3px; padding: 1px 3px; } +.fin-piece.fin-type--unknown { background: var(--muted); color: var(--foreground); } +.fin-proviso { color: var(--muted-foreground); } +.fin-card { background: var(--card); border: 1px solid var(--border); border-radius: var(--radius); overflow: hidden; + box-shadow: var(--shadow-soft); } +.fin-card .fin-table { box-shadow: none; border-radius: 0; table-layout: auto; } +.fin-summary { display: flex; gap: 18px; flex-wrap: wrap; padding: 10px 14px; border-bottom: 1px solid var(--border); + font-size: 13px; color: var(--muted-foreground); } +.fin-summary b { color: var(--foreground); font-variant-numeric: tabular-nums; } +.fin-table--compare td { vertical-align: top; } +.fin-table--compare th { background: var(--secondary); font-size: 12px; text-transform: uppercase; + letter-spacing: 0.04em; color: var(--muted-foreground); border-bottom: 1px solid var(--border); padding: 8px 10px; } +.fin-table--compare td { padding: 8px 10px; } +.fin-table--compare .fin-amount { font-family: var(--font-sans); } +.fin-stack__tag { white-space: nowrap; } +.fin-caret { color: var(--muted-foreground); width: 16px; display: inline-block; } +.fin-loc { font-weight: 600; } +.fin-loc small { display: block; font-weight: 400; color: var(--muted-foreground); font-size: 12px; } +/* Links from a comparison row to its sections in Version A / B. */ +.fin-jumps { display: flex; flex-wrap: wrap; gap: 4px 14px; margin-top: 6px; } +.fin-jump { background: none; border: 0; padding: 0; font: inherit; font-size: 12px; color: var(--primary); + text-decoration: underline; text-underline-offset: 2px; cursor: pointer; } +.fin-jump:hover { color: var(--foreground); } +.fin-jump--none { color: var(--muted-foreground); text-decoration: none; cursor: default; } +.fin-row.is-target > td { background: var(--accent); transition: background 0.6s ease; } +.fin-stack { display: grid; grid-template-columns: auto 1fr auto; gap: 2px 8px; align-items: center; } +.fin-ab { font-size: 11px; font-weight: 700; border-radius: 4px; padding: 0 5px; } +.fin-ab--a { background: var(--secondary); color: var(--primary); } +.fin-ab--b { background: var(--accent); color: var(--gold); } +.fin-stack__amt { text-align: right; font-variant-numeric: tabular-nums; white-space: nowrap; } +.fin-stack__amt--old { color: var(--muted-foreground); } +.fin-stack__amt--new { font-weight: 600; } +.fin-stack__none { color: var(--muted-foreground); font-style: italic; text-align: right; } +.fin-up { color: var(--diff-add-foreground); } .fin-down { color: var(--diff-remove-foreground); } +.fin-same { color: var(--muted-foreground); } +.fin-same small, .fin-up small, .fin-down small { display: block; font-size: 11px; } +.fin-cdetail > td { background: var(--background); padding: 12px 14px; } +.fin-subhead { font-size: 13px; font-weight: 600; color: var(--muted-foreground); padding: 14px 14px 6px; + border-top: 1px solid var(--border); } +.fin-table--plain { box-shadow: none; } +@media (max-width: 820px) { + .fin-table--compare { min-width: 640px; } + .fin-detail__inner { flex-direction: column; } + .fin-detail__amounts { max-width: none; border-right: 0; border-bottom: 1px solid var(--border); } + .fin-col-name { width: 55%; } .fin-col-type { width: 20%; } .fin-col-amount { width: 25%; } +} + @@ -317,24 +494,24 @@ - + - +

H.R. 8752 — Making appropriations for the Department of Homeland Security for the fiscal year ending September 30, 2025, and for other purposes.

-
v1: reported-in-house → v2: engrossed-in-house · 118th Congress
-
modified 18added 16removed 2moved 1
+
Before (v1): reported-in-houseAfter (v2): engrossed-in-house118th Congress
+
modified 18added 17removed 2moved 1
-
+
-
- +
+
- +

Changes

@@ -571,21 +748,33 @@

TITLE IV > RESEARCH, DEVELOPMENT, TRAINING, AND SERVICES > ADMINISTRAT
TITLE V
GENERAL PROVISIONS -
SEC. 552 +
SPENDING REDUCTION ACCOUNT
added -

TITLE V > GENERAL PROVISIONS > SEC. 552

+

TITLE V > GENERAL PROVISIONS > SPENDING REDUCTION ACCOUNT

+
+
— (new in v2)p.97 L13
+
SPENDING REDUCTION ACCOUNT
+
+
+
+
SPENDING REDUCTION ACCOUNT +
SEC. 552 +
+
+added +

TITLE V > SPENDING REDUCTION ACCOUNT > SEC. 552

— (new in v2)p.97 L14
SEC. 552. $0 (increased by $10,000,000).
SEC. 553 -
+
added -

TITLE V > GENERAL PROVISIONS > SEC. 553

+

TITLE V > SPENDING REDUCTION ACCOUNT > SEC. 553

— (new in v2)p.97 L15 – p.100 L6
SEC. 553. (a) None of the funds made available by @@ -652,10 +841,10 @@

TITLE V > GENERAL PROVISIONS > SEC. 553

SEC. 554 -
+
added -

TITLE V > GENERAL PROVISIONS > SEC. 554

+

TITLE V > SPENDING REDUCTION ACCOUNT > SEC. 554

— (new in v2)p.100 L7 – p.100 L12
SEC. 554. None of the funds made available by this @@ -666,10 +855,10 @@

TITLE V > GENERAL PROVISIONS > SEC. 554

SEC. 555 -
+
added -

TITLE V > GENERAL PROVISIONS > SEC. 555

+

TITLE V > SPENDING REDUCTION ACCOUNT > SEC. 555

— (new in v2)p.100 L13 – p.100 L16
SEC. 555. None of the funds appropriated or otherwise made available by this Act may be used to implement, @@ -678,10 +867,10 @@

TITLE V > GENERAL PROVISIONS > SEC. 555

SEC. 556 -
+
added -

TITLE V > GENERAL PROVISIONS > SEC. 556

+

TITLE V > SPENDING REDUCTION ACCOUNT > SEC. 556

— (new in v2)p.100 L17 – p.100 L20
SEC. 556. None of the funds made available in this @@ -691,10 +880,10 @@

TITLE V > GENERAL PROVISIONS > SEC. 556

SEC. 557 -
+
added -

TITLE V > GENERAL PROVISIONS > SEC. 557

+

TITLE V > SPENDING REDUCTION ACCOUNT > SEC. 557

— (new in v2)p.100 L21 – p.100 L22
SEC. 557. None of the funds made available by this @@ -702,10 +891,10 @@

TITLE V > GENERAL PROVISIONS > SEC. 557

SEC. 558 -
+
added -

TITLE V > GENERAL PROVISIONS > SEC. 558

+

TITLE V > SPENDING REDUCTION ACCOUNT > SEC. 558

— (new in v2)p.100 L23 – p.100 L25
SEC. 558. None of the funds made available by this @@ -714,10 +903,10 @@

TITLE V > GENERAL PROVISIONS > SEC. 558

SEC. 559 -
+
added -

TITLE V > GENERAL PROVISIONS > SEC. 559

+

TITLE V > SPENDING REDUCTION ACCOUNT > SEC. 559

— (new in v2)p.101 L1 – p.101 L2
SEC. 559. None of the funds made available by this @@ -725,10 +914,10 @@

TITLE V > GENERAL PROVISIONS > SEC. 559

SEC. 560 -
+
added -

TITLE V > GENERAL PROVISIONS > SEC. 560

+

TITLE V > SPENDING REDUCTION ACCOUNT > SEC. 560

— (new in v2)p.101 L4 – p.101 L5
SEC. 560. None of the funds made available under @@ -736,10 +925,10 @@

TITLE V > GENERAL PROVISIONS > SEC. 560

SEC. 561 -
+
added -

TITLE V > GENERAL PROVISIONS > SEC. 561

+

TITLE V > SPENDING REDUCTION ACCOUNT > SEC. 561

— (new in v2)p.101 L8 – p.101 L11
SEC. 561. None of the funds appropriated or otherwise made available by this Act may be made available @@ -748,10 +937,10 @@

TITLE V > GENERAL PROVISIONS > SEC. 561

SEC. 562 -
+
added -

TITLE V > GENERAL PROVISIONS > SEC. 562

+

TITLE V > SPENDING REDUCTION ACCOUNT > SEC. 562

— (new in v2)p.101 L12 – p.102 L9
SEC. 562. None of the funds made available by this @@ -771,10 +960,10 @@

TITLE V > GENERAL PROVISIONS > SEC. 562

SEC. 563 -
+
added -

TITLE V > GENERAL PROVISIONS > SEC. 563

+

TITLE V > SPENDING REDUCTION ACCOUNT > SEC. 563

— (new in v2)p.102 L10 – p.102 L12
SEC. 563. The salary of Nejwa Ali, U.S. Citizenship @@ -783,10 +972,10 @@

TITLE V > GENERAL PROVISIONS > SEC. 563

SEC. 564 -
+
added -

TITLE V > GENERAL PROVISIONS > SEC. 564

+

TITLE V > SPENDING REDUCTION ACCOUNT > SEC. 564

— (new in v2)p.102 L13 – p.102 L16
SEC. 564. None of the funds made available by this @@ -796,10 +985,10 @@

TITLE V > GENERAL PROVISIONS > SEC. 564

SEC. 565 -
+
added -

TITLE V > GENERAL PROVISIONS > SEC. 565

+

TITLE V > SPENDING REDUCTION ACCOUNT > SEC. 565

— (new in v2)p.102 L17 – p.102 L20
SEC. 565. None of the funds made available by this @@ -809,10 +998,10 @@

TITLE V > GENERAL PROVISIONS > SEC. 565

SEC. 566 -
+
added -

TITLE V > GENERAL PROVISIONS > SEC. 566

+

TITLE V > SPENDING REDUCTION ACCOUNT > SEC. 566

— (new in v2)p.102 L21 – p.102 L25
SEC. 566. None of the funds made available by this @@ -856,10 +1045,10 @@

TITLE V > GENERAL PROVISIONS > SEC. 552

Committed to the Committee of the Whole House on the State of the Union and ordered to be printed
-
+
added -

TITLE V > GENERAL PROVISIONS > SEC. 567

+

TITLE V > SPENDING REDUCTION ACCOUNT > SEC. 567

— (new in v2)p.103 L1 – p.104
SEC. 567. None of the funds appropriated or otherwise made available by this Act may be used to make payments to the Department of Veterans Affairs for medical @@ -876,36 +1065,77 @@

TITLE V > GENERAL PROVISIONS > SEC. 567

Homeland Security for the fiscal year ending
-

+ 16 labor is, or services are, of a temporary or seasonal nature.
- + \ No newline at end of file diff --git a/examples/hr8752_xml_diff.html b/examples/hr8752_xml_diff.html index efaff8e3..daf4fe58 100644 --- a/examples/hr8752_xml_diff.html +++ b/examples/hr8752_xml_diff.html @@ -19,6 +19,8 @@ --gold: #c9944e; --destructive: #c04040; --success: #3d9b6d; + --info: #e3f2f8; + --info-foreground: #0b4f63; --diff-add: #d3f0e2; --diff-add-foreground: #1a6647; --diff-remove: #f5ddd8; @@ -27,7 +29,22 @@ --diff-modified-foreground: #8a6320; --diff-moved: #eef0f8; --diff-moved-foreground: #2c2c5c; + --fin-transfer: #cffafe; + --fin-transfer-foreground: #155e75; + --fin-authorization: #e0f2fe; + --fin-authorization-foreground: #0c4a6e; + --fin-fee: #ffedd5; + --fin-fee-foreground: #9a3412; + --fin-directive: #e0e7ff; + --fin-directive-foreground: #3730a3; + --fin-earmark: #ede9fe; + --fin-earmark-foreground: #5b21b6; + --fin-availability: #d1fae5; + --fin-availability-foreground: #065f46; + --fin-sub-allocation: #dbeafe; + --fin-sub-allocation-foreground: #1e40af; --radius: 0.625rem; + --sticky-bar-height: 64px; --font-sans: ui-sans-serif, system-ui, -apple-system, 'Segoe UI', Roboto, Helvetica, Arial, sans-serif; --font-serif: ui-serif, Georgia, 'Times New Roman', serif; --font-mono: ui-monospace, 'SF Mono', Menlo, Consolas, monospace; @@ -88,11 +105,13 @@ .filter-empty[hidden] { display: none; } /* Main content */ -.main { margin-left: 280px; padding: 28px 36px; max-width: 940px; flex: 1; } +.main { margin-left: 280px; padding: 28px 36px; max-width: 940px; flex: 1; min-width: 0; } /* Header */ .report-header h1 { font-size: 24px; margin-bottom: 4px; } -.report-header .versions { color: var(--muted-foreground); font-size: 15px; margin-bottom: 16px; } +.report-header .versions { color: var(--muted-foreground); font-size: 15px; margin-bottom: 16px; line-height: 1.5; } +.versions__row { display: block; overflow-wrap: anywhere; } +.versions__row b { color: var(--foreground); font-weight: 600; } .summary-bar { display: flex; gap: 10px; margin-bottom: 24px; flex-wrap: wrap; } .summary-item { display: inline-flex; align-items: center; gap: 6px; padding: 4px 12px; border-radius: 999px; font-size: 13px; background: var(--secondary); } @@ -164,16 +183,25 @@ border-bottom: 1px solid var(--border); padding: 10px 0; margin-bottom: 16px; } .action-bar__left { display: flex; align-items: center; gap: 10px; flex-wrap: wrap; } .action-bar__group { display: flex; align-items: center; gap: 10px; flex-wrap: wrap; } -.find-bar { display: inline-flex; align-items: center; gap: 4px; } -.find-bar input { padding: 5px 10px; border: 1px solid var(--border); border-radius: var(--radius); - font: inherit; font-family: var(--font-sans); font-size: 13px; width: 180px; background: var(--card); } -.find-bar button { padding: 5px 9px; border: 1px solid var(--border); border-radius: var(--radius); - background: var(--card); cursor: pointer; font-family: var(--font-sans); font-size: 13px; } +.action-bar__left { flex: 1 1 auto; } +/* The page's one search: it reaches text that is collapsed, which the browser's own find + cannot, so it is sized and marked to be found first. */ +.find-bar { display: inline-flex; align-items: center; gap: 8px; flex: 1 1 380px; max-width: 640px; } +.find-field { position: relative; display: flex; flex: 1; min-width: 0; } +.find-icon { position: absolute; left: 14px; top: 50%; width: 22px; height: 22px; transform: translateY(-50%); + fill: none; stroke: var(--muted-foreground); stroke-width: 2.2; stroke-linecap: round; pointer-events: none; } +.find-bar input { width: 100%; padding: 10px 14px 10px 46px; border: 1px solid var(--border); + border-radius: var(--radius); font: inherit; font-family: var(--font-sans); font-size: 20px; + background: var(--card); color: var(--foreground); } +.find-bar input:focus { outline: 2px solid var(--primary); outline-offset: 1px; border-color: var(--primary); } +.find-bar button { padding: 8px 12px; border: 1px solid var(--border); border-radius: var(--radius); + background: var(--card); cursor: pointer; font-family: var(--font-sans); font-size: 15px; } .find-bar button:hover { background: var(--secondary); } .find-bar button[disabled] { opacity: 0.4; cursor: default; } .find-counter { font-variant-numeric: tabular-nums; font-size: 12px; color: var(--muted-foreground); min-width: 3.5em; text-align: center; } -mark.find-hit { background: var(--accent); color: inherit; border-radius: 2px; scroll-margin-top: 64px; } +mark.find-hit { background: var(--accent); color: inherit; border-radius: 2px; + scroll-margin-top: calc(var(--sticky-bar-height) + 8px); } mark.find-hit--current { background: var(--gold); color: #fff; } .nav-controls { display: inline-flex; align-items: center; gap: 4px; } .nav-controls button { padding: 6px 12px; border: 1px solid var(--border); border-radius: var(--radius); @@ -244,9 +272,13 @@ .prompt-copy:hover { background: var(--accent); } .prompt-text { line-height: 1.5; } -/* Nav targets clear the sticky action bar when scrolled to via Prev/Next */ +/* Nav targets clear the sticky action bar when scrolled to via Prev/Next or a link: + --sticky-bar-height is the bar's measured height (padForStickyBar), plus a small gap. A + fixed 64px cleared the one-row bar but not the two-row one. On the targets, not as the + page's scroll-padding: that would also count the bar's own controls as covered, so + clicking or tabbing to one would scroll the page. */ .change-card, .full-bill [id^="attr-"], .full-bill [id^="sec-"], .full-bill [id^="fb-off-"], -.removed-block { scroll-margin-top: 64px; } +.removed-block { scroll-margin-top: calc(var(--sticky-bar-height) + 8px); } /* Full-bill section TOC (sidebar variant) */ .sidebar-changes[hidden], .sidebar-toc[hidden] { display: none; } @@ -295,7 +327,25 @@ .find-bar { position: fixed; left: 0; right: 0; bottom: 46px; z-index: 35; justify-content: center; background: var(--card); border-top: 1px solid var(--border); padding: 8px 16px; } - .find-bar input { flex: 1; max-width: 320px; } + .find-bar { max-width: none; } + body.outside-changes { padding-bottom: 62px; } + body.outside-changes .find-bar { bottom: 0; padding-bottom: calc(8px + env(safe-area-inset-bottom)); } +} +/* Change navigation and the changes export serve Changes and Full bill only. */ +body.outside-changes .nav-controls, body.outside-changes #export-open { display: none; } + +/* Five view buttons get a row of their own when the financial views are present (ADR 0023). + The views' own styles live with them, in formatters/financial_views.py. */ +.action-bar--stacked { display: grid; grid-template-columns: 1fr auto; row-gap: 10px; align-items: center; } +.action-bar--stacked .action-bar__left { display: contents; } +.action-bar--stacked .view-toggle { grid-column: 1 / -1; justify-self: start; } +.action-bar--stacked .find-bar { grid-column: 1; } +.action-bar--stacked .action-bar__group { grid-column: 2; } +.fin-scroll { overflow-x: auto; } +@media (max-width: 820px) { + .action-bar--stacked { display: flex; flex-direction: column; align-items: stretch; } + .view-toggle { display: flex; flex-wrap: wrap; } + .view-toggle__btn { flex: 1 1 auto; } } /* Print */ @@ -304,6 +354,133 @@ .main { margin-left: 0; } .change-card { break-inside: avoid; } } +/* Financial views (ADR 0023): Version A / B, and the comparison */ +/* The three financial tabs use the full window on a wide screen: their tables are columns, + cramped at the report's 940px reading width. Changes and Full bill keep that width (their + cards are prose). `body[data-active-view]` is set by the report's view switch. */ +@media (min-width: 821px) { + body[data-active-view="fin-a"] .main, body[data-active-view="fin-b"] .main, + body[data-active-view="fin-compare"] .main { + max-width: none; padding-right: 10%; } +} +.fin-head { display: flex; align-items: baseline; justify-content: space-between; gap: 12px; flex-wrap: wrap; } +.view-fin-a h2, .view-fin-b h2, .view-fin-compare h2 { font-size: 20px; margin-bottom: 2px; } +.fin-version { color: var(--muted-foreground); font-size: 14px; margin-bottom: 12px; } +.fin-coverage, .fin-note, .fin-explain { font-size: 13px; color: var(--muted-foreground); margin-bottom: 10px; } +/* "About these figures": an information alert, left-accented, above each view's content. */ +.fin-alert { display: flex; gap: 12px; align-items: flex-start; margin: 4px 0 16px; padding: 12px 16px; + background: var(--info); color: var(--info-foreground); border: 1px solid var(--info); + border-left: 4px solid var(--info-foreground); border-radius: var(--radius); font-size: 14px; line-height: 1.55; } +.fin-alert__icon { flex: none; width: 20px; height: 20px; margin-top: 1px; fill: none; + stroke: var(--info-foreground); stroke-width: 2; stroke-linecap: round; } +.fin-alert__dot { fill: var(--info-foreground); } +.fin-alert a { color: inherit; font-weight: 600; text-decoration: underline; } +.fin-alert__version { display: block; margin-top: 4px; font-size: 12px; opacity: 0.85; } +.fin-coverage--gap { color: var(--destructive); font-weight: 600; } +.fin-empty { color: var(--muted-foreground); padding: 16px 2px; font-size: 14px; } +.view-fin-a h3, .view-fin-b h3 { font-size: 18px; margin: 18px 0 10px; } +.fin-key { border: 1px solid var(--border); border-radius: var(--radius); padding: 12px 14px; + background: var(--card); margin-bottom: 8px; } +.fin-key__title { font-weight: 600; font-size: 14px; margin-bottom: 8px; } +.fin-key__item { display: flex; align-items: baseline; gap: 8px; padding: 2px 0; } +.fin-key__count { color: var(--muted-foreground); font-size: 12px; min-width: 32px; display: inline-block; } +.fin-key__desc { color: var(--muted-foreground); font-size: 13px; } +.fin-type { display: inline-block; padding: 2px 7px; border-radius: 999px; font-size: 11px; font-weight: 600; + white-space: nowrap; background: var(--muted); color: var(--muted-foreground); } +.fin-type--appropriation { background: var(--diff-add); color: var(--diff-add-foreground); } +.fin-type--rescission, .fin-type--restriction { background: var(--diff-remove); color: var(--diff-remove-foreground); } +.fin-type--cap { background: var(--diff-modified); color: var(--diff-modified-foreground); } +.fin-type--transfer { background: var(--fin-transfer); color: var(--fin-transfer-foreground); } +.fin-type--authorization { background: var(--fin-authorization); color: var(--fin-authorization-foreground); } +.fin-type--fee { background: var(--fin-fee); color: var(--fin-fee-foreground); } +.fin-type--directive { background: var(--fin-directive); color: var(--fin-directive-foreground); } +.fin-type--earmark { background: var(--fin-earmark); color: var(--fin-earmark-foreground); } +.fin-type--availability { background: var(--fin-availability); color: var(--fin-availability-foreground); } +.fin-type--sub_allocation { background: var(--fin-sub-allocation); color: var(--fin-sub-allocation-foreground); } +.fin-sort { display: flex; gap: 8px; align-items: center; margin-bottom: 12px; font-size: 13px; } +.fin-sort span { color: var(--muted-foreground); } +.fin-sort__btn { padding: 4px 12px; border: 1px solid var(--border); border-radius: 999px; background: var(--card); + cursor: pointer; font: inherit; font-family: var(--font-sans); font-size: 13px; color: var(--foreground); } +.fin-sort__btn:hover { background: var(--secondary); } +.fin-sort__btn.is-active { background: var(--foreground); border-color: var(--foreground); color: var(--card); } +.fin-table { width: 100%; border-collapse: collapse; background: var(--card); border-radius: var(--radius); + box-shadow: var(--shadow-soft); overflow: hidden; font-size: 14px; table-layout: fixed; } +.fin-table th { background: var(--accent); text-align: left; padding: 9px 12px; border-bottom: 2px solid var(--border); + font-weight: 600; font-size: 13px; } +.fin-table td { padding: 8px 12px; border-bottom: 1px solid var(--border); vertical-align: middle; } +.fin-col-name { width: 72%; } .fin-col-type { width: 13%; } .fin-col-amount { width: 15%; text-align: right; } +.fin-amount { text-align: right; font-family: var(--font-mono); font-variant-numeric: tabular-nums; + white-space: nowrap; } +.fin-row, .fin-crow { cursor: pointer; } +.fin-row { background: var(--muted); font-weight: 600; } +.fin-row:hover td, .fin-crow:hover td { background: var(--secondary); } +.fin-row__name::before { content: "\25b6"; font-size: 10px; margin-right: 8px; color: var(--muted-foreground); + display: inline-block; transition: transform 0.15s; } +.fin-row.is-open .fin-row__name::before { transform: rotate(90deg); } +.fin-count { font-size: 11px; color: var(--muted-foreground); font-weight: 400; margin-left: 6px; } +.fin-review { background: var(--diff-modified); color: var(--diff-modified-foreground); font-size: 11px; + padding: 1px 6px; border-radius: 999px; margin-left: 6px; font-weight: 500; white-space: nowrap; } +/* The warning mark (`_WARNING_ICON`): drawn in the surrounding text's colour and size. */ +.fin-icon { display: inline-block; width: 1.15em; height: 1.15em; vertical-align: -0.2em; margin-right: 0.25em; + fill: none; stroke: currentColor; stroke-width: 2; stroke-linejoin: round; stroke-linecap: round; } +.fin-icon__dot { fill: currentColor; } +.fin-review[role="img"] .fin-icon { margin-right: 0; } +.fin-chip { font-size: 11px; font-weight: 600; border-radius: 4px; padding: 0 5px; margin-left: 6px; } +.fin-chip--law { background: var(--muted); color: var(--muted-foreground); border: 1px dashed var(--border); } +.fin-detail > td { padding: 0; background: var(--card); } +.fin-detail__inner { display: flex; } +.fin-detail__amounts { min-width: 220px; max-width: 260px; padding: 12px; border-right: 1px solid var(--border); + display: flex; flex-direction: column; gap: 10px; } +.fin-clause { display: flex; flex-direction: column; gap: 3px; align-items: flex-start; } +.fin-clause__amount { font-family: var(--font-mono); font-size: 15px; font-variant-numeric: tabular-nums; } +.fin-detail__text { padding: 12px 16px; font-size: 13px; line-height: 1.9; flex: 1; font-weight: 400; } +.fin-piece { border-radius: 3px; padding: 1px 3px; } +.fin-piece.fin-type--unknown { background: var(--muted); color: var(--foreground); } +.fin-proviso { color: var(--muted-foreground); } +.fin-card { background: var(--card); border: 1px solid var(--border); border-radius: var(--radius); overflow: hidden; + box-shadow: var(--shadow-soft); } +.fin-card .fin-table { box-shadow: none; border-radius: 0; table-layout: auto; } +.fin-summary { display: flex; gap: 18px; flex-wrap: wrap; padding: 10px 14px; border-bottom: 1px solid var(--border); + font-size: 13px; color: var(--muted-foreground); } +.fin-summary b { color: var(--foreground); font-variant-numeric: tabular-nums; } +.fin-table--compare td { vertical-align: top; } +.fin-table--compare th { background: var(--secondary); font-size: 12px; text-transform: uppercase; + letter-spacing: 0.04em; color: var(--muted-foreground); border-bottom: 1px solid var(--border); padding: 8px 10px; } +.fin-table--compare td { padding: 8px 10px; } +.fin-table--compare .fin-amount { font-family: var(--font-sans); } +.fin-stack__tag { white-space: nowrap; } +.fin-caret { color: var(--muted-foreground); width: 16px; display: inline-block; } +.fin-loc { font-weight: 600; } +.fin-loc small { display: block; font-weight: 400; color: var(--muted-foreground); font-size: 12px; } +/* Links from a comparison row to its sections in Version A / B. */ +.fin-jumps { display: flex; flex-wrap: wrap; gap: 4px 14px; margin-top: 6px; } +.fin-jump { background: none; border: 0; padding: 0; font: inherit; font-size: 12px; color: var(--primary); + text-decoration: underline; text-underline-offset: 2px; cursor: pointer; } +.fin-jump:hover { color: var(--foreground); } +.fin-jump--none { color: var(--muted-foreground); text-decoration: none; cursor: default; } +.fin-row.is-target > td { background: var(--accent); transition: background 0.6s ease; } +.fin-stack { display: grid; grid-template-columns: auto 1fr auto; gap: 2px 8px; align-items: center; } +.fin-ab { font-size: 11px; font-weight: 700; border-radius: 4px; padding: 0 5px; } +.fin-ab--a { background: var(--secondary); color: var(--primary); } +.fin-ab--b { background: var(--accent); color: var(--gold); } +.fin-stack__amt { text-align: right; font-variant-numeric: tabular-nums; white-space: nowrap; } +.fin-stack__amt--old { color: var(--muted-foreground); } +.fin-stack__amt--new { font-weight: 600; } +.fin-stack__none { color: var(--muted-foreground); font-style: italic; text-align: right; } +.fin-up { color: var(--diff-add-foreground); } .fin-down { color: var(--diff-remove-foreground); } +.fin-same { color: var(--muted-foreground); } +.fin-same small, .fin-up small, .fin-down small { display: block; font-size: 11px; } +.fin-cdetail > td { background: var(--background); padding: 12px 14px; } +.fin-subhead { font-size: 13px; font-weight: 600; color: var(--muted-foreground); padding: 14px 14px 6px; + border-top: 1px solid var(--border); } +.fin-table--plain { box-shadow: none; } +@media (max-width: 820px) { + .fin-table--compare { min-width: 640px; } + .fin-detail__inner { flex-direction: column; } + .fin-detail__amounts { max-width: none; border-right: 0; border-bottom: 1px solid var(--border); } + .fin-col-name { width: 55%; } .fin-col-type { width: 20%; } .fin-col-amount { width: 25%; } +} + @@ -324,17 +501,17 @@

H.R. 8752 — Making appropriations for the Department of Homeland Security for the fiscal year ending September 30, 2025, and for other purposes.

-
v1: reported-in-house → v2: engrossed-in-house · 118th Congress
+
Before (v1): reported-in-houseAfter (v2): engrossed-in-house118th Congress
modified 18added 18removed 2moved 1
-
+
-
- +
+
- +

Changes

@@ -784,31 +961,72 @@

TITLE V > GENERAL PROVISIONS > SPENDING REDUCTION ACCOUNT > sec. 55

-
+
- + \ No newline at end of file diff --git a/schema/canonical-diff.md b/schema/canonical-diff.md index 2bb653f9..3a97801e 100644 --- a/schema/canonical-diff.md +++ b/schema/canonical-diff.md @@ -1,4 +1,4 @@ -# Canonical Diff JSON — v3.0 +# Canonical Diff JSON — v3.1 This document specifies the canonical JSON shape produced when comparing two versions of a bill. It is the public contract between the diff engine and any @@ -8,10 +8,19 @@ XML inputs and a diff produced from PDF inputs share this shape. ## Versioning -Top-level field: `schema_version: "3.0"`. +Top-level field: `schema_version: "3.1"`. ## Changelog +- **3.1** — Added optional top-level `financial: { classifier, v1, v2 } | null`: each + version's **ledger**, every dollar amount in it with its chain of headings, its clause and + a type read from the wording ([ADR 0023](../docs/decisions/0023-financial-ledger-views.md)). + Per side and unpaired, like `tree`: it states what each version contains, never that a + figure in one became a figure in the other, and a change object still carries no money + (the 3.0 rule below stands). `classifier` names the version of the rules that typed it, so + a saved document says which rules produced its types. Requires `full_text` and `tree` + present (every span indexes into `full_text`). Additive, backward compatible. The HTML + report's own `diff.json` download strips the field. - **3.0** — **Breaking:** removed `amount_entries` from each change object and from its `required` list (#671). No field replaces it: a change object now carries no money at all. The field paired a dollar figure on one side with a figure on the @@ -103,7 +112,7 @@ Top-level field: `schema_version: "3.0"`. ```jsonc { - "schema_version": "3.0", + "schema_version": "3.1", "generator": { "name": "deltatrack", "version": "0.x" }, "bill": { "type": "HR", "number": 4366, "congress": 118 }, "versions": { @@ -111,6 +120,7 @@ Top-level field: `schema_version: "3.0"`. "v2": { "label": "Public Law", "version_number": 4, "source": "xml" } }, "summary": { "added": 12, "removed": 8, "modified": 47, "moved": 3 }, + "financial": { "classifier": "1.0", "v1": { … }, "v2": { … } }, // optional, v3.1+ "full_text": { // optional, v1.1+ "v1": "TITLE I—…\n\nSECTION 101. …", "v2": "TITLE I—…\n\nSECTION 101. …" @@ -163,6 +173,50 @@ derivable from this tree, and since #462 it is the renderer's only source for the navigation: the separate flat `sections` jump-list and the builder that read it were removed. +### `financial` (optional, v3.1+) + +Top-level object: each version's ledger ([ADR 0023](../docs/decisions/0023-financial-ledger-views.md)). +`null` (or absent) when there is no full text and tree to build it from. **Co-presence:** a +non-null `financial` requires non-null `full_text` and `tree`. + +| Field | Type | Notes | +|---|---|---| +| `classifier` | string | Version of the rules that typed the clauses (`deltatrack.financial.CLASSIFIER`). A rule change that moves any ledger row changes it. | +| `v1`, `v2` | Ledger | One per side, built independently. | + +A **Ledger**: + +| Field | Type | Notes | +|---|---|---| +| `amounts_in_text` | int | How many `$` figures that side's `full_text` holds. A consumer can compare it with the figures under `sections` to see whether any were not placed. | +| `sections` | LedgerSection[] | The money-bearing nodes of that side's `tree`, in document order. | + +A **LedgerSection** (one money-bearing tree node): + +| Field | Type | Notes | +|---|---|---| +| `path` | `[label, level][]` | The node's chain of headings with their `tree` levels, outermost first. The location of the text, not an account. | +| `span` | `[start, end]` | Character range into `full_text[side]`. | +| `clauses` | LedgerClause[] | The section split at `Provided, That`, `; and in addition,` and `of which`; only clauses holding a dollar figure. | +| `pieces` | `[start, end, kind][]` | Ranges of the section's text by clause type (`kind` a type, or `"proviso"` for the `Provided, That` wording itself), for highlighting. | +| `flags` | string[] | `"may_hold_several_sections"`: the text holds a further section heading the parser did not start a node for, so the section may carry more than one. | + +A **LedgerClause**: + +| Field | Type | Notes | +|---|---|---| +| `level` | `"primary"` \| `"sub"` | The section's opening clause, or a clause split off it. | +| `type` | enum | `appropriation`, `transfer`, `rescission`, `authorization`, `fee`, `restriction`, `directive`, `cap`, `earmark`, `availability`, `sub_allocation`, `unknown` (a figure no rule recognised). An inference from wording. | +| `amount` | number \| null | The clause's amount: its first figure that is not a "not to exceed" ceiling. | +| `in_amended_law` | bool | The clause amount sits inside text this bill writes into another law, so it is not money this bill gives out. | +| `needs_review` | bool | The clause holds more than one non-ceiling figure, so `amount` is ambiguous. | +| `span` | `[start, end]` | The clause's range in `full_text[side]`. | +| `amounts` | `{ value, cap, in_amended_law }[]` | Every figure in the clause, in order. | + +No text is duplicated: every range points into `full_text`. **Money given out** (what a +comparison may add up) is an appropriation, positive, or a rescission, negative, with +`in_amended_law` false; nothing else is summed. + ### `bill` | Field | Type | Notes | @@ -320,7 +374,9 @@ ceiling, or a commitment limitation. Presenting them as bare Old/New/Change colu is what this break exists to stop. Re-adding a typed money field is planned: it needs the account-level model in #115 -and the leveled tree in #175 first. +and the leveled tree in #175 first. Since v3.1 the per-side, unpaired `financial` ledger +types each figure where it sits ([`financial`](#financial-optional-v31)); that is not this +field, and a change object still carries none. ### `full_text_span` (optional, v1.2+) diff --git a/schema/canonical-diff.schema.json b/schema/canonical-diff.schema.json index d3e5d697..424bbb38 100644 --- a/schema/canonical-diff.schema.json +++ b/schema/canonical-diff.schema.json @@ -42,6 +42,22 @@ ], "description": "Optional (v1.3+) per-side leveled structure tree (#108). Each side is an ordered list of root nodes. Requires full_text present (node spans index into it). A leveled TOC is derivable from the tree, and since #462 the renderer builds the navigation from it alone." }, + "financial": { + "anyOf": [ + { "type": "null" }, + { + "type": "object", + "required": ["classifier", "v1", "v2"], + "additionalProperties": false, + "properties": { + "classifier": { "type": "string" }, + "v1": { "$ref": "#/$defs/Ledger" }, + "v2": { "$ref": "#/$defs/Ledger" } + } + } + ], + "description": "Optional (v3.1+) per-side ledger (ADR 0023): every dollar amount in each version, typed by clause and located in the structure tree. Unpaired across sides, like tree; a change object still carries no money. Requires full_text and tree present (spans index into full_text). The report's diff.json download strips it." + }, "generator": { "type": "object", "additionalProperties": false, @@ -79,6 +95,77 @@ } }, "$defs": { + "Ledger": { + "type": "object", + "required": ["amounts_in_text", "sections"], + "additionalProperties": false, + "properties": { + "amounts_in_text": { + "type": "integer", + "minimum": 0, + "description": "Every dollar amount in this side's full_text; the sections' amounts account for all of them when nothing was dropped." + }, + "sections": { "type": "array", "items": { "$ref": "#/$defs/LedgerSection" } } + } + }, + "LedgerSection": { + "type": "object", + "required": ["path", "span", "clauses", "pieces", "flags"], + "additionalProperties": false, + "properties": { + "path": { + "type": "array", + "items": { "type": "array", "prefixItems": [{ "type": "string" }, { "type": "string" }], "minItems": 2, "maxItems": 2 }, + "description": "[label, level] pairs from the tree root to the node holding this prose." + }, + "span": { "$ref": "#/$defs/LedgerSpan" }, + "clauses": { "type": "array", "minItems": 1, "items": { "$ref": "#/$defs/LedgerClause" } }, + "pieces": { + "type": "array", + "items": { "type": "array", "minItems": 3, "maxItems": 3 }, + "description": "[start, end, kind] highlight pieces over full_text: the prose split at each `Provided, That` (kind `proviso`), every other piece typed." + }, + "flags": { "type": "array", "items": { "enum": ["may_hold_several_sections"] } } + } + }, + "LedgerClause": { + "type": "object", + "required": ["level", "type", "amount", "in_amended_law", "needs_review", "span", "amounts"], + "additionalProperties": false, + "properties": { + "level": { "enum": ["primary", "sub"] }, + "type": { + "enum": ["appropriation", "transfer", "rescission", "authorization", "fee", "restriction", "directive", "cap", "earmark", "availability", "sub_allocation", "unknown"] + }, + "amount": { + "anyOf": [{ "type": "null" }, { "type": "number", "minimum": 0 }], + "description": "The clause's first dollar figure outside a `not to exceed` ceiling, else its first." + }, + "in_amended_law": { "type": "boolean", "description": "That amount sits inside text this bill writes into another law." }, + "needs_review": { "type": "boolean", "description": "More than one non-ceiling dollar figure, so `amount` is ambiguous." }, + "span": { "$ref": "#/$defs/LedgerSpan" }, + "amounts": { + "type": "array", + "items": { + "type": "object", + "required": ["value", "cap", "in_amended_law"], + "additionalProperties": false, + "properties": { + "value": { "type": "number", "minimum": 0 }, + "cap": { "type": "boolean" }, + "in_amended_law": { "type": "boolean" } + } + } + } + } + }, + "LedgerSpan": { + "type": "array", + "items": { "type": "integer", "minimum": 0 }, + "minItems": 2, + "maxItems": 2, + "description": "[start, end) character offsets into full_text for the same side." + }, "TreeNode": { "type": "object", "required": ["label", "level", "own_amounts", "full_text_span", "children"], diff --git a/scripts/heading_precision.py b/scripts/heading_precision.py index 95bfc96e..9b8ad8e5 100644 --- a/scripts/heading_precision.py +++ b/scripts/heading_precision.py @@ -88,7 +88,13 @@ def measure(pdf_path: Path, xml_path: Path) -> dict: # Vocabulary recall/precision: are the distinct account names recovered, and are # PDF account names real XML headings (catches spurious/false accounts)? vocab_hit = pdf_account_names & xml_leaf_vocab - vocab_recall = len(vocab_hit) / len(xml_leaf_vocab) if xml_leaf_vocab else None + # The XML side stands in intermediate (agency-level) headings for accounts until #54, so a + # PDF agency with an intermediate heading's name is recovered too: recall credits it, since + # reading BUREAU OF RECLAMATION as the agency over its accounts is not a missed account. + # Precision stays over account anchors only. + pdf_agency_names = {normalize_header(a.text) for a in anchors if a.kind == "agency"} + recovered = vocab_hit | (pdf_agency_names & unique["appropriations-intermediate"]) + vocab_recall = len(recovered) / len(xml_leaf_vocab) if xml_leaf_vocab else None vocab_precision = len(vocab_hit) / len(pdf_account_names) if pdf_account_names else None numbered = [ln for pg in pages for ln in pg.lines if ln.line_number is not None] diff --git a/src/deltatrack/compare/pdf.py b/src/deltatrack/compare/pdf.py index aaf22afc..843979b1 100644 --- a/src/deltatrack/compare/pdf.py +++ b/src/deltatrack/compare/pdf.py @@ -210,9 +210,9 @@ def _derive_bill_title(canonical: dict) -> str: r"\bA BILL\b\s+(.+?\bpurposes\.)", head, re.IGNORECASE ) if m2: + # Whole, as the XML path's `bill_title` gives it: the heading wraps, so a cut + # only hid the end of the title (the fiscal year, on an appropriations bill). title = re.sub(r"\s+", " ", m2.group(1)).strip() - if len(title) > 140: - title = title[:137].rstrip() + "…" if designator and title: return f"{designator} — {title}" diff --git a/src/deltatrack/financial.py b/src/deltatrack/financial.py new file mode 100644 index 00000000..b36289e5 --- /dev/null +++ b/src/deltatrack/financial.py @@ -0,0 +1,424 @@ +"""The bill ledger: every dollar amount in one bill version, typed and located (ADR 0023). + +A ledger section is one money-bearing node of a version's structure tree (the same +sections the report's drill-down shows), split into clauses, each typed by a small +regex classifier (appropriation, rescission, restriction, cap, …). Every amount in a +clause is kept, and flagged when it sits inside an amendment to another law ("changes +existing law"), which is text this bill writes into the U.S. Code, not money it gives out. + +The classifier and the row model are the research notebook's, moved here as they were so +the report can show what the notebook showed (`docs/research/financial-semantics/`, +`02_financial_report.ipynb`); `tests/test_financial_corpus.py` pins that parity. Improving the +classifier is later work, and the parity pin is where it starts. + +This module is the financial layer ADR 0018 permits to read appropriations wording: it +interprets money and never decides structure (the sections come from the tree). + +Output is JSON-ready and holds no text: every span is a character range into the +canonical document's ``full_text`` for that side, so the report derives what it shows from +the text it already embeds. +""" + +from __future__ import annotations + +import re + +from deltatrack.parsers.pdf_blocks import _is_strippable_heading_line +from deltatrack.parsers.pdf_heading_passes import _FURNITURE + +# ---------- Classifier (from docs/research/financial-semantics/classify_bill.py) ----------- + +# Node-opening patterns +_RESTRICT = re.compile(r"^\s*(?:\([a-z0-9]+\)\s*)?None of the funds", re.IGNORECASE) +_RESTRICT_NOTWITHSTANDING = re.compile(r"^\s*Notwithstanding\b.{0,200}\bnone of the funds\b", re.IGNORECASE | re.DOTALL) +_TRANSFER = re.compile(r"^\s*Of (?:the )?amounts.{0,300}\btransferr?ed?\b", re.IGNORECASE | re.DOTALL) +_APPROP = re.compile(r"^\s*(?:\([a-z0-9]+\)\s*)?For\b", re.IGNORECASE | re.DOTALL) +_RESCISSION = re.compile(r"(?:is|are) hereby rescinded", re.IGNORECASE) +_DIRECTIVE = re.compile(r"^\s*The\s+\w[\w\s]+(?:shall|may not)\b", re.IGNORECASE) +_REPROGRAM = re.compile(r"^\s*no project may be (?:increased|decreased)", re.IGNORECASE) +_DELAYED_APPROP = re.compile(r"^\s*\$[\d,]+.{0,50}\bshall become available\b", re.IGNORECASE | re.DOTALL) +_APPROP_ALT = re.compile(r"there (?:is|are)(?: hereby)? appropriated", re.IGNORECASE) +_AUTHORIZATION = re.compile(r"\bauthorized to be appropriated\b", re.IGNORECASE) +_FEE = re.compile(r"fee in the amount of\s+\$|impose a fee|\bpays a fee of\s+\$|\ba fee of\s+\$", re.IGNORECASE) + +# Sub-clause patterns +_PROVIDED = re.compile(r"\bProvided(?:\s+further)?,?\s+That\b", re.IGNORECASE) +_EARMARK = re.compile( + r"of the amount.{0,50}under this heading.{0,100}specified in the table", re.IGNORECASE | re.DOTALL +) +_AVAILABILITY = re.compile(r"of the amount.{0,100}shall remain available until", re.IGNORECASE | re.DOTALL) +_SUB_ALLOC = re.compile(r"^\s*,?\s*\$[\d,]+\s+shall\s+be\s+(?:for|available)", re.IGNORECASE) +_CAP = re.compile(r"not (?:more than|to exceed)\s+\$[\d,]+", re.IGNORECASE) +_OF_WHICH_AVAIL = re.compile(r"^\s*of which.{0,80}\bshall remain available\b", re.IGNORECASE | re.DOTALL) +_OF_WHICH_ALLOC = re.compile(r"^\s*of which\b", re.IGNORECASE) + +# Splitting / extraction +_OF_WHICH = re.compile(r",?\s*\bof which\b", re.IGNORECASE) +_IN_ADDITION = re.compile(r";\s*and,?\s*in addition,", re.IGNORECASE) +_CAP_AMOUNT = re.compile(r"not (?:more than|to exceed)\s+\$[\d,]+(?:\.\d+)?", re.IGNORECASE) +_DOLLAR = re.compile(r"\$([\d,]+(?:\.\d+)?)") + +#: Every clause type, in the order the report's key lists them. +TYPES = ( + "appropriation", + "transfer", + "rescission", + "authorization", + "fee", + "restriction", + "directive", + "cap", + "earmark", + "availability", + "sub_allocation", + "unknown", +) + + +def classify_text(text: str) -> str | None: + """The clause type of ``text``; None for empty text. First matching rule wins.""" + if not text: + return None + if _RESTRICT.match(text) or _RESTRICT_NOTWITHSTANDING.match(text): + return "restriction" + if _TRANSFER.match(text): + return "transfer" + if _APPROP.match(text): + return "rescission" if _RESCISSION.search(text) else "appropriation" + if _RESCISSION.search(text): + return "rescission" + if _DIRECTIVE.match(text): + return "directive" + if _REPROGRAM.match(text): + return "cap" + if _DELAYED_APPROP.match(text): + return "appropriation" + if _APPROP_ALT.search(text): + return "rescission" if _RESCISSION.search(text) else "appropriation" + if _AUTHORIZATION.search(text): + return "authorization" + if _FEE.search(text): + return "fee" + if _EARMARK.search(text): + return "earmark" + if _AVAILABILITY.search(text): + return "availability" + if _SUB_ALLOC.match(text): + return "sub_allocation" + if _OF_WHICH_AVAIL.match(text): + return "availability" + if _OF_WHICH_ALLOC.match(text): + return "sub_allocation" + if _CAP.search(text): + return "cap" + return "unknown" + + +# ---------- Section text ------------------------------------------------------------------ + +_GUTTER = 7 # the PDF full_text line prefix: `{line:>5} ` +_SEC_PREFIX = re.compile(r"^SEC\.\s*\d+[A-Z]?\.\s*") +# GPO's quote marks, which the PDF prints and the XML text the classifier was built on +# does not carry. Left in, they lengthen a clause past a rule's reach (`Of the amounts … +# ''Medical Services'' … transferred` is a transfer). A single apostrophe is kept. +_QUOTE_MARKS = re.compile(r"‘‘|’’|''|``|“|”") + + +def _is_soft_hyphen_break(upper: list[str], lower: str) -> bool: + """A word the printer broke across two lines (`speci-` / `fied`): one word. + + The same test as ``pdf_blocks._rejoin_cross_page_hyphens`` (an alphanumeric before + the hyphen, a lowercase continuation), which rejoins these for the differ's blocks. + Within a page ``pdf_text._merge_print_lines`` has already rejoined them in the + canonical text; a break at a page seam reaches it intact, so it is rejoined here. + """ + return len(upper) >= 2 and upper[-1] == "-" and upper[-2].isalnum() and lower[:1].islower() + + +# A subsection opening a printed line after a sentence ends starts its own paragraph: +# the XML reader makes each such subsection its own node, and the PDF block holds them all. +_SUBSECTION_START = re.compile(r"\([a-z]\)\s") +_SENTENCE_END = (".", ":", ";", "—") + + +def _prose_blocks(full_text: str, start: int, end: int, *, guttered: bool) -> list[tuple[str, list[int]]]: + """The prose of one tree node, one block per printed paragraph, each as a single line + plus every character's offset in ``full_text`` (so a clause or highlight maps back). + + PDF text (``guttered``) goes through the PDF pipeline's own cleanup, reused rather than + restated: the line-number gutter comes off; page furniture (the unnumbered running + header, ``pdf_heading_passes._FURNITURE``) is skipped; heading lines are the ones + ``pdf_blocks._is_strippable_heading_line`` strips from a diff block (the account name, + ``(INCLUDING TRANSFER OF FUNDS)``, never an all-caps line carrying a dollar amount). A + heading line after prose closes the paragraph (``FEDERAL-AID HIGHWAYS`` prints its + limitation, then ``(LIQUIDATION OF CONTRACT AUTHORIZATION)`` and its payment), and so + does a subsection opening a line after a sentence end, outside a quotation. + + XML text is one node per subsection already, so it is only joined into one line. + """ + blocks: list[tuple[str, list[int]]] = [] + chars: list[str] = [] + offsets: list[int] = [] + quote_open = False + + def close() -> None: + nonlocal chars, offsets + if chars: + blocks.append(("".join(chars), offsets)) + chars, offsets = [], [] + + pos = start + for raw in full_text[start:end].split("\n"): + line_start = pos + pos += len(raw) + 1 + content_at = _GUTTER if guttered else 0 + content = raw[content_at:].rstrip() + if not content.strip(): + continue + if guttered: + if not raw[:content_at].strip() and _FURNITURE.match(content): + continue + ended = "".join(chars[-2:]).rstrip().endswith(_SENTENCE_END) + if _is_strippable_heading_line(content) and (not chars or ended): + # A heading only where one can start: opening the node, or after a sentence + # ends. Mid-sentence, an all-caps line is a continuation (`account (69 X`). + close() + continue + if chars and ended and not quote_open and _SUBSECTION_START.match(content.lstrip()): + close() + lead = len(content) - len(content.lstrip()) + body = content[lead:] + if _is_soft_hyphen_break(chars, body): + chars.pop() + offsets.pop() + elif chars: + chars.append(" ") + offsets.append(line_start + content_at + lead) + for k, ch in enumerate(body): + chars.append(ch) + offsets.append(line_start + content_at + lead + k) + for mark in re.findall(r"‘‘|’’|“|”", body): + quote_open = mark in ("‘‘", "“") + close() + return blocks + + +# Text this bill writes into another law. PDF prints it inside GPO's quote marks; the XML +# serializer drops them, so an amendment's lead-in (`… is amended to read as follows:`) +# stands in for the opening mark there, running to the end of the section. +_AMENDMENT_LEAD_IN = re.compile(r"\b(?:as follows|the following)\s*:", re.IGNORECASE) +_STRIKE_INSERT = re.compile(r"\b(?:striking|inserting)\s+[\"'‘“]*\s*$", re.IGNORECASE) + + +def _amended_law_ranges(text: str) -> list[tuple[int, int]]: + """Character ranges of ``text`` that quote another law (see ``_AMENDMENT_LEAD_IN``).""" + ranges: list[tuple[int, int]] = [] + opened = None + for m in re.finditer(r"‘‘|’’|“|”", text): + mark = m.group(0) + if mark in ("‘‘", "“"): + opened = m.start() if opened is None else opened + elif opened is not None: + ranges.append((opened, m.end())) + opened = None + if opened is not None: + ranges.append((opened, len(text))) + lead_in = _AMENDMENT_LEAD_IN.search(text) + if lead_in is not None: + ranges.append((lead_in.end(), len(text))) + return ranges + + +def _in_amended_law(text: str, at: int, ranges: list[tuple[int, int]]) -> bool: + if any(s <= at < e for s, e in ranges): + return True + return bool(_STRIKE_INSERT.search(text[max(0, at - 40) : at])) + + +# ---------- Clauses ---------------------------------------------------------------------- + + +def _split_clauses(text: str) -> list[tuple[str, str, int, int]]: + """``(clause text, level, start, end)``, split exactly as the notebook split them. + + On ``Provided, That`` first, then ``; and in addition,``, then ``of which``. The clause + text is the notebook's (an ``of which`` clause gets the words back as a prefix, and it is + stripped); ``start``/``end`` locate the clause's own characters in ``text``. + """ + out: list[tuple[str, str, int, int]] = [] + + def pieces(pattern: re.Pattern, s: int, e: int): + cursor = s + for m in pattern.finditer(text, s, e): + yield cursor, m.start() + cursor = m.end() + yield cursor, e + + for i, (ps, pe) in enumerate(pieces(_PROVIDED, 0, len(text))): + level = "primary" if i == 0 else "sub" + for j, (as_, ae) in enumerate(pieces(_IN_ADDITION, ps, pe)): + for k, (os_, oe) in enumerate(pieces(_OF_WHICH, as_, ae)): + raw = text[os_:oe] + lead = len(raw) - len(raw.lstrip()) + body = raw.strip() + clause = ("of which " if k > 0 else "") + body + out.append( + (clause.strip(), level if (j == 0 and k == 0) else "sub", os_ + lead, os_ + lead + len(body)) + ) + return out + + +def _amount_value(digits: str) -> int | float: + value = float(digits.replace(",", "")) + return int(value) if value.is_integer() else value + + +def _clause_json(text: str, clause: str, level: str, start: int, end: int, ranges, offsets, section_type): + caps = [(m.start(), m.end()) for m in _CAP_AMOUNT.finditer(text, start, end)] + amounts = [] + first_plain = first_any = None + for m in _DOLLAR.finditer(text, start, end): + is_cap = any(s <= m.start() < e for s, e in caps) + entry = { + "value": _amount_value(m.group(1)), + "cap": is_cap, + "in_amended_law": _in_amended_law(text, m.start(), ranges), + } + amounts.append(entry) + first_any = entry if first_any is None else first_any + if not is_cap and first_plain is None: + first_plain = entry + primary = first_plain or first_any + return { + "level": level, + "type": section_type if level == "primary" else classify_text(_classifiable(clause)), + "amount": primary["value"] if primary else None, + "in_amended_law": bool(primary and primary["in_amended_law"]), + "needs_review": sum(1 for a in amounts if not a["cap"]) > 1, + "span": [offsets[start], offsets[end - 1] + 1], + "amounts": amounts, + } + + +def _classifiable(text: str) -> str: + """What the classifier reads: no leading `SEC. N.` enumerator, no quote marks.""" + return _SEC_PREFIX.sub("", _QUOTE_MARKS.sub("", text)) + + +def _highlight_pieces(text: str, offsets: list[int], section_type: str) -> list[list]: + """``[start, end, kind]`` over ``full_text``: the section text split at each + ``Provided, That`` (kind ``proviso``), every other piece typed as the notebook's + highlight typed it (the first by the whole section's type).""" + out = [] + cursor, first = 0, True + marks = list(_PROVIDED.finditer(text)) + [None] + for m in marks: + stop = m.start() if m else len(text) + if stop > cursor: + kind = section_type if first else (classify_text(_classifiable(text[cursor:stop])) or "unknown") + out.append([offsets[cursor], offsets[stop - 1] + 1, kind]) + first = False + if m: + out.append([offsets[m.start()], offsets[m.end() - 1] + 1, "proviso"]) + cursor = m.end() + return out + + +# ---------- The ledger ------------------------------------------------------------------- + + +def _walk(nodes: list[dict], path: tuple = ()): + for node in nodes: + label = (node.get("label") or "").strip() + here = path + ([label, node.get("level") or ""],) if label else path + yield node, here + yield from _walk(node.get("children") or [], here) + + +def section_ledger(full_text: str, tree: list[dict], *, guttered: bool) -> list[dict]: + """The ledger of one bill version: its money-bearing tree nodes, in document order.""" + sections = [] + for node, path in _walk(tree): + span = node.get("full_text_span") + if not span or span["end"] <= span["start"] or not node.get("label"): + continue + for text, offsets in _prose_blocks(full_text, span["start"], span["end"], guttered=guttered): + if not _DOLLAR.search(text): + continue + ranges = _amended_law_ranges(text) + section_type = classify_text(_classifiable(text)) or "unknown" + clauses = [ + _clause_json(text, clause, level, s, e, ranges, offsets, section_type) + for clause, level, s, e in _split_clauses(text) + if e > s and _DOLLAR.search(clause) + ] + if not clauses: + continue + sections.append( + { + "path": [list(p) for p in path], + "span": [offsets[0], offsets[-1] + 1], + "clauses": clauses, + "pieces": _highlight_pieces(text, offsets, section_type), + "flags": ["may_hold_several_sections"] if _INNER_SECTION.search(text) else [], + } + ) + return sections + + +# A section heading after a sentence end, inside one node's prose: the parser did not +# start a section there (lettered numbers such as `SEC. 119A.` are the case seen), so +# this row may carry more than one section's money. Shown to the reader, never hidden. +_INNER_SECTION = re.compile(r"[.:;—]\s+SEC\.\s*\d+[A-Z]?\.\s") + + +def version_ledger(full_text: str, tree: list[dict], *, guttered: bool) -> dict: + """One side's ledger plus the count that proves nothing was dropped: every dollar + amount the version's text holds, against the amounts its sections show.""" + return { + "amounts_in_text": len(_DOLLAR.findall(full_text)), + "sections": section_ledger(full_text, tree, guttered=guttered), + } + + +def financial_for(full_text: dict | None, tree: dict | None, source: str) -> dict | None: + """The canonical ``financial`` field (schema 3.1), or None without text and tree.""" + if not full_text or not tree: + return None + guttered = source != "xml" + return { + "classifier": CLASSIFIER, + "v1": version_ledger(full_text["v1"], tree["v1"], guttered=guttered), + "v2": version_ledger(full_text["v2"], tree["v2"], guttered=guttered), + } + + +#: The classifier's version. Every document records it (`financial.classifier`) and every +#: financial view shows it, so a reader can tell which rules typed the money in front of them. +#: +#: Bump it whenever a rule change moves any ledger row: the minor for a changed rule, the +#: major for a type added, removed or renamed (that also changes the schema's type enum). +#: `tests/test_financial_corpus.py` holds the frozen rows of the parity bill under the version that +#: produced them, and refuses different rows under the same version, so the rules cannot +#: move without this number moving. There are no releases to pin: to use an earlier +#: version, check out the commit before the change its line below names. +#: +#: - 1.0: the research notebook's rules (`docs/research/financial-semantics/classify_bill.py`), +#: moved into the package unchanged; its PDF reading and paragraph split added (ADR 0023). +CLASSIFIER = "1.0" + + +def given_out(clause: dict) -> int | float | None: + """Money a clause gives out (+) or takes back (−), or None when it does neither. + + Only an appropriation or a rescission counts, and never an amount inside an + amendment to another law: that is text written into the U.S. Code, not funding. + """ + amount = clause.get("amount") + if amount is None or clause.get("in_amended_law"): + return None + if clause.get("type") == "appropriation": + return amount + if clause.get("type") == "rescission": + return -amount + return None diff --git a/src/deltatrack/formatters/canonical.py b/src/deltatrack/formatters/canonical.py index 26ce7ae2..4c7a0925 100644 --- a/src/deltatrack/formatters/canonical.py +++ b/src/deltatrack/formatters/canonical.py @@ -24,11 +24,12 @@ from deltatrack.amounts import extract_amounts from deltatrack.diff_pdf import PdfDiff, PdfHunk +from deltatrack.financial import financial_for from deltatrack.formatters.view_model import ChangeView, DiffView from deltatrack.parsers.pdf_anchors import Anchor, breadcrumb_for from deltatrack.structure_tree import TreeNode, build_pdf_tree -SCHEMA_VERSION = "3.0" +SCHEMA_VERSION = "3.1" GENERATOR_NAME = "deltatrack" @@ -163,6 +164,7 @@ def xml_diff_to_canonical( """ diffed = [c for c in (diff_dict.get("changes") or []) if c.get("change_type") != "unchanged"] normalized_full_text = _normalize_full_text(full_text) + normalized_tree = _normalize_tree(tree, normalized_full_text) search_state: dict = {} return { "schema_version": SCHEMA_VERSION, @@ -186,7 +188,8 @@ def xml_diff_to_canonical( }, "summary": dict(diff_dict.get("summary") or {}), "full_text": normalized_full_text, - "tree": _normalize_tree(tree, normalized_full_text), + "tree": normalized_tree, + "financial": financial_for(normalized_full_text, normalized_tree, "xml"), "changes": [ _xml_change_to_canonical(c, i, normalized_full_text, full_text_spans, search_state) for i, c in enumerate(diffed) @@ -441,6 +444,7 @@ def pdf_diff_to_canonical( "summary": dict(diff.summary), "full_text": normalized_full_text, "tree": _normalize_tree(tree, normalized_full_text), + "financial": financial_for(normalized_full_text, tree, "pdf"), "changes": [ _pdf_hunk_to_canonical(h, i, diff.v1_anchors, diff.v2_anchors, line_offsets_v1, line_offsets_v2) for i, h in enumerate(diff.hunks) diff --git a/src/deltatrack/formatters/diff_html.py b/src/deltatrack/formatters/diff_html.py index 0ea68b21..745d2b7e 100644 --- a/src/deltatrack/formatters/diff_html.py +++ b/src/deltatrack/formatters/diff_html.py @@ -19,6 +19,13 @@ from deltatrack.formatters._text import word_diff from deltatrack.formatters.canonical import view_from_canonical +from deltatrack.formatters.financial_views import ( + FINANCIAL_CSS, + FINANCIAL_JS, + financial_toggle_buttons, + financial_views_html, + has_financial, +) from deltatrack.formatters.view_model import ChangeView, DiffView from deltatrack.palette import root_block @@ -380,32 +387,28 @@ def _build_sidebar( def _versions_html(view: DiffView) -> str: - """Render the versions line. - - Canonical form: "v1: {label} → v2: {label} · {congress}th Congress". - The "vN: " prefix is dropped when both version numbers are None — PDF - inputs don't carry a version index, and "v1: Reported" is misleading - when no such index exists. + """Render the versions lines: one per version, then the Congress. + + Canonical form, one row each: + Before (v1): {label} + After (v2): {label} + {congress}th Congress + The label is shown as given, which for an upload is its filename: a stage name + ("Engrossed in House") is known only when the file keeps GPO's own name, so none + is inferred. The "(vN)" is dropped for a version without a number (PDF inputs + carry no version index, and "v1" would be misleading when none exists). The + Congress row is omitted when unknown, not left as "th Congress". """ - if view.v1_version_number is not None or view.v2_version_number is not None: - v1 = ( - f"v{view.v1_version_number}: {escape(view.v1_label)}" - if view.v1_version_number is not None - else escape(view.v1_label) - ) - v2 = ( - f"v{view.v2_version_number}: {escape(view.v2_label)}" - if view.v2_version_number is not None - else escape(view.v2_label) - ) - else: - v1 = escape(view.v1_label) - v2 = escape(view.v2_label) - line = f"{v1} → {v2}" + + def row(tag: str, number: int | None, label: str) -> str: + suffix = f" (v{number})" if number is not None else "" + return f'{tag}{suffix}: {escape(label)}' + + rows = [row("Before", view.v1_version_number, view.v1_label), row("After", view.v2_version_number, view.v2_label)] congress = str(view.congress).strip() - if congress: # omit the suffix entirely when unknown, not "· th Congress" - line += f" · {escape(congress)}th Congress" - return line + if congress: + rows.append(f'{escape(congress)}th Congress') + return "".join(rows) def _summary_bar_html(summary: dict[str, int]) -> str: @@ -488,12 +491,13 @@ def _view_toggle_html(canonical: dict | None) -> str: """Changes/Full segmented control. Empty when there's no full text to show.""" if not _has_full_bill(canonical): return "" + financial = financial_toggle_buttons() if has_financial(canonical) else "" return ( '
' '' '' + f' aria-selected="false">Full bill{financial}' "
" ) @@ -755,7 +759,10 @@ def _views_html( if not _has_full_bill(canonical): return changes_inner full_bill = _full_bill_html(display_canonical or canonical) - return f'
{changes_inner}
' + return ( + f'
{changes_inner}
' + f"{financial_views_html(canonical)}" + ) # Ready-made questions a staffer can paste into an LLM alongside the diff.json, @@ -789,7 +796,7 @@ def _export_button_html(canonical: dict | None) -> str: pipeline that supplies it — XML and PDF alike, not PDF-only.""" if not _has_full_bill(canonical): return "" - return '' + return '' def _nav_controls_html(canonical: dict | None) -> str: @@ -818,7 +825,11 @@ def _find_bar_html(canonical: dict | None) -> str: return "" return ( ' -
+
{_view_toggle_html(canonical)} {_find_bar_html(canonical)} @@ -952,6 +965,7 @@ def format_diff_html( {data_script} """ @@ -1027,11 +1041,13 @@ def format_diff_html( .filter-empty[hidden] { display: none; } /* Main content */ -.main { margin-left: 280px; padding: 28px 36px; max-width: 940px; flex: 1; } +.main { margin-left: 280px; padding: 28px 36px; max-width: 940px; flex: 1; min-width: 0; } /* Header */ .report-header h1 { font-size: 24px; margin-bottom: 4px; } -.report-header .versions { color: var(--muted-foreground); font-size: 15px; margin-bottom: 16px; } +.report-header .versions { color: var(--muted-foreground); font-size: 15px; margin-bottom: 16px; line-height: 1.5; } +.versions__row { display: block; overflow-wrap: anywhere; } +.versions__row b { color: var(--foreground); font-weight: 600; } .summary-bar { display: flex; gap: 10px; margin-bottom: 24px; flex-wrap: wrap; } .summary-item { display: inline-flex; align-items: center; gap: 6px; padding: 4px 12px; border-radius: 999px; font-size: 13px; background: var(--secondary); } @@ -1103,16 +1119,25 @@ def format_diff_html( border-bottom: 1px solid var(--border); padding: 10px 0; margin-bottom: 16px; } .action-bar__left { display: flex; align-items: center; gap: 10px; flex-wrap: wrap; } .action-bar__group { display: flex; align-items: center; gap: 10px; flex-wrap: wrap; } -.find-bar { display: inline-flex; align-items: center; gap: 4px; } -.find-bar input { padding: 5px 10px; border: 1px solid var(--border); border-radius: var(--radius); - font: inherit; font-family: var(--font-sans); font-size: 13px; width: 180px; background: var(--card); } -.find-bar button { padding: 5px 9px; border: 1px solid var(--border); border-radius: var(--radius); - background: var(--card); cursor: pointer; font-family: var(--font-sans); font-size: 13px; } +.action-bar__left { flex: 1 1 auto; } +/* The page's one search: it reaches text that is collapsed, which the browser's own find + cannot, so it is sized and marked to be found first. */ +.find-bar { display: inline-flex; align-items: center; gap: 8px; flex: 1 1 380px; max-width: 640px; } +.find-field { position: relative; display: flex; flex: 1; min-width: 0; } +.find-icon { position: absolute; left: 14px; top: 50%; width: 22px; height: 22px; transform: translateY(-50%); + fill: none; stroke: var(--muted-foreground); stroke-width: 2.2; stroke-linecap: round; pointer-events: none; } +.find-bar input { width: 100%; padding: 10px 14px 10px 46px; border: 1px solid var(--border); + border-radius: var(--radius); font: inherit; font-family: var(--font-sans); font-size: 20px; + background: var(--card); color: var(--foreground); } +.find-bar input:focus { outline: 2px solid var(--primary); outline-offset: 1px; border-color: var(--primary); } +.find-bar button { padding: 8px 12px; border: 1px solid var(--border); border-radius: var(--radius); + background: var(--card); cursor: pointer; font-family: var(--font-sans); font-size: 15px; } .find-bar button:hover { background: var(--secondary); } .find-bar button[disabled] { opacity: 0.4; cursor: default; } .find-counter { font-variant-numeric: tabular-nums; font-size: 12px; color: var(--muted-foreground); min-width: 3.5em; text-align: center; } -mark.find-hit { background: var(--accent); color: inherit; border-radius: 2px; scroll-margin-top: 64px; } +mark.find-hit { background: var(--accent); color: inherit; border-radius: 2px; + scroll-margin-top: calc(var(--sticky-bar-height) + 8px); } mark.find-hit--current { background: var(--gold); color: #fff; } .nav-controls { display: inline-flex; align-items: center; gap: 4px; } .nav-controls button { padding: 6px 12px; border: 1px solid var(--border); border-radius: var(--radius); @@ -1183,9 +1208,13 @@ def format_diff_html( .prompt-copy:hover { background: var(--accent); } .prompt-text { line-height: 1.5; } -/* Nav targets clear the sticky action bar when scrolled to via Prev/Next */ +/* Nav targets clear the sticky action bar when scrolled to via Prev/Next or a link: + --sticky-bar-height is the bar's measured height (padForStickyBar), plus a small gap. A + fixed 64px cleared the one-row bar but not the two-row one. On the targets, not as the + page's scroll-padding: that would also count the bar's own controls as covered, so + clicking or tabbing to one would scroll the page. */ .change-card, .full-bill [id^="attr-"], .full-bill [id^="sec-"], .full-bill [id^="fb-off-"], -.removed-block { scroll-margin-top: 64px; } +.removed-block { scroll-margin-top: calc(var(--sticky-bar-height) + 8px); } /* Full-bill section TOC (sidebar variant) */ .sidebar-changes[hidden], .sidebar-toc[hidden] { display: none; } @@ -1234,7 +1263,25 @@ def format_diff_html( .find-bar { position: fixed; left: 0; right: 0; bottom: 46px; z-index: 35; justify-content: center; background: var(--card); border-top: 1px solid var(--border); padding: 8px 16px; } - .find-bar input { flex: 1; max-width: 320px; } + .find-bar { max-width: none; } + body.outside-changes { padding-bottom: 62px; } + body.outside-changes .find-bar { bottom: 0; padding-bottom: calc(8px + env(safe-area-inset-bottom)); } +} +/* Change navigation and the changes export serve Changes and Full bill only. */ +body.outside-changes .nav-controls, body.outside-changes #export-open { display: none; } + +/* Five view buttons get a row of their own when the financial views are present (ADR 0023). + The views' own styles live with them, in formatters/financial_views.py. */ +.action-bar--stacked { display: grid; grid-template-columns: 1fr auto; row-gap: 10px; align-items: center; } +.action-bar--stacked .action-bar__left { display: contents; } +.action-bar--stacked .view-toggle { grid-column: 1 / -1; justify-self: start; } +.action-bar--stacked .find-bar { grid-column: 1; } +.action-bar--stacked .action-bar__group { grid-column: 2; } +.fin-scroll { overflow-x: auto; } +@media (max-width: 820px) { + .action-bar--stacked { display: flex; flex-direction: column; align-items: stretch; } + .view-toggle { display: flex; flex-wrap: wrap; } + .view-toggle__btn { flex: 1 1 auto; } } /* Print */ @@ -1244,6 +1291,7 @@ def format_diff_html( .change-card { break-inside: avoid; } } """ + + FINANCIAL_CSS ) @@ -1253,20 +1301,61 @@ def format_diff_html( var toggleBtns = document.querySelectorAll('.view-toggle__btn'); var sidebarChanges = document.querySelector('.sidebar-changes'); var sidebarToc = document.querySelector('.sidebar-toc'); + // Jump targets land below the sticky action bar, not under it: their scroll-margin-top is + // --sticky-bar-height plus a gap, and that is the bar's measured height, which varies (the + // view switcher takes its own row when the financial views are present, the bar wraps on + // narrower windows). Where the bar does not stick (phones) nothing covers the top: 0. + var stickyBar = document.querySelector('.action-bar'); + function padForStickyBar() { + var sticky = stickyBar && getComputedStyle(stickyBar).position === 'sticky'; + document.documentElement.style.setProperty('--sticky-bar-height', sticky ? stickyBar.offsetHeight + 'px' : '0px'); + } + padForStickyBar(); + window.addEventListener('resize', padForStickyBar); + if (stickyBar && window.ResizeObserver) new ResizeObserver(padForStickyBar).observe(stickyBar); + // Each view keeps its own scroll position. The views share one page, so without this a + // view comes back at wherever the last one left the page: neither where the reader was + // in it nor its top. Restored synchronously, inside the switch, so a caller that scrolls + // afterwards still wins: the browser's jump to a sidebar #change-N link, a search's first + // match, a financial view's section link. Tabs add no browser history (Back leaves the + // report): the report is often written into a tab with no address of its own, where + // history entries do not behave the same across browsers. + var viewScroll = {}; + var shownView = 'changes'; + document.body.dataset.activeView = shownView; function showView(name) { + if (name !== shownView) viewScroll[shownView] = window.scrollY; + // Which view is showing, for styles: `body[data-active-view]`, and `outside-changes` for the + // controls that serve only Changes and Full bill (the change navigator, the changes + // export). Set here, on every switch, so no path into a view can leave them stale. + document.body.dataset.activeView = name; + document.body.classList.toggle('outside-changes', name !== 'changes' && name !== 'full'); toggleBtns.forEach(function(b) { var on = b.dataset.view === name; b.classList.toggle('is-active', on); b.setAttribute('aria-selected', on ? 'true' : 'false'); }); + var target = null; document.querySelectorAll('.view').forEach(function(el) { el.hidden = !el.classList.contains('view-' + name); + if (!el.hidden) target = el; }); // Swap the sidebar variant (only when a TOC variant was rendered). if (sidebarToc) { sidebarToc.hidden = name !== 'full'; if (sidebarChanges) sidebarChanges.hidden = name === 'full'; } + if (name === shownView || !target) return; + shownView = name; + if (viewScroll[name] !== undefined) { + window.scrollTo(0, viewScroll[name]); + } else { + // First visit: the view's own top, just under the sticky bar, when the page is + // already past it; otherwise leave the page where it is (the header stays in sight). + var bar = document.querySelector('.action-bar'); + var top = target.getBoundingClientRect().top + window.scrollY - (bar ? bar.offsetHeight : 0) - 8; + if (window.scrollY > top) window.scrollTo(0, Math.max(0, top)); + } } toggleBtns.forEach(function(b) { b.addEventListener('click', function() { showView(b.dataset.view); }); @@ -1278,6 +1367,10 @@ def format_diff_html( for (var d = el && el.parentElement; d; d = d.parentElement) { if (d.tagName === 'DETAILS') d.open = true; } + // A collapsed row that find indexes while hidden (`data-find-reveal`) is opened by + // the view that owns it, which knows what opening means there. + var row = el && el.closest && el.closest('[data-find-reveal][hidden]'); + if (row) row.dispatchEvent(new CustomEvent('find:reveal', {bubbles: true})); } // Change-list anchors (#change-N) live in the changes view; jump back to it // first. TOC links (.sidebar-toc a) just scroll within the full-bill view. @@ -1312,7 +1405,11 @@ def format_diff_html( var dlJson = document.getElementById('dl-json'); if (dlJson) dlJson.addEventListener('click', function() { var raw = document.getElementById('diff-data').textContent; - downloadBlob('diff.json', JSON.stringify(JSON.parse(raw), null, 2), 'application/json'); + // The changes export stays what it was: the typed ledger feeds the financial views + // and their own CSV export, not the document handed to an assistant (ADR 0023). + var doc = JSON.parse(raw); + delete doc.financial; + downloadBlob('diff.json', JSON.stringify(doc, null, 2), 'application/json'); }); var dlHtml = document.getElementById('dl-html'); if (dlHtml) dlHtml.addEventListener('click', function() { @@ -1406,6 +1503,8 @@ def format_diff_html( .filter(function(c) { return c.offsetParent !== null; }); } function refreshNav() { + // Outside Changes and Full bill the navigator is hidden (`outside-changes`, set by + // showView), rather than show a disabled 0 / 0 beside the find counter. var n = navTargets().length; if (current >= n) current = n - 1; if (counter) counter.textContent = (current + 1) + ' / ' + n; @@ -1502,9 +1601,8 @@ def format_diff_html( var findHits = []; var findIdx = -1; function activeView() { - var full = document.querySelector('.view-full'); - if (full && !full.hidden) return full; - return document.querySelector('.view-changes') || document.body; + var shown = [].slice.call(document.querySelectorAll('.view')).filter(function(v) { return !v.hidden; }); + return shown[0] || document.querySelector('.view-changes') || document.body; } function clearFind() { var parents = []; @@ -1607,8 +1705,10 @@ def format_diff_html( if (seg.gutter || !seg.block) continue; if (seg.block !== visBlock) { // one layout read per block, not per node visBlock = seg.block; + // Collapsed content marked `data-find-reveal` is searched too: a hit opens it. visible = !seg.block.classList.contains('fb-page') - && (seg.block.offsetParent !== null || seg.block.tagName === 'BODY'); + && (seg.block.offsetParent !== null || seg.block.tagName === 'BODY' + || seg.block.closest('[data-find-reveal]') !== null); } if (!visible) continue; var b = seg.block; @@ -1671,7 +1771,11 @@ def format_diff_html( // ("House of Representa-"). That hyphen is gone from the searchable text, // so drop it rather than return nothing for a phrase the reader copied. if (/[A-Za-z0-9]-$/.test(q) && q.length > 3) q = q.slice(0, -1); - var idx = buildFindIndex(activeView()); + // A view that renders collapsed text lazily fills it in now (synchronously), so find + // sees text no one has opened yet. + var root = activeView(); + root.dispatchEvent(new CustomEvent('find:prepare', {bubbles: true})); + var idx = buildFindIndex(root); var ql = q.toLowerCase(); var hits = [], ranges = [], at = 0; while ((at = idx.lower.indexOf(ql, at)) !== -1) { diff --git a/src/deltatrack/formatters/financial_views.py b/src/deltatrack/formatters/financial_views.py new file mode 100644 index 00000000..af06d03e --- /dev/null +++ b/src/deltatrack/formatters/financial_views.py @@ -0,0 +1,823 @@ +"""The report's three financial views, rendered from the canonical ``financial`` field (ADR 0023). + +- **Financials – Version A / B**: one version's ledger as the research notebook's financial + report showed it (`docs/research/financial-semantics/financial_118_hr_4366.html`): one row + per money-bearing section with its type and amount, a key of the types, a sort bar, a + "review" badge where a clause holds more than one figure, and a row that opens onto + the section's clauses and its text, highlighted clause by clause. +- **Inferred Financial Comparison**: one row per change the differ already reported whose + section holds money on either side, Version A's amount over Version B's, and the + difference in money given out. A row opens onto that change's card from the Changes view, + so the comparison has no matching of its own. + +Money given out means an appropriation or a rescission outside an amendment to another law +(``financial.given_out``); everything else is shown and never summed. +""" + +from __future__ import annotations + +import json +from html import escape + +from deltatrack.financial import TYPES, given_out + +__all__ = ["FINANCIAL_CSS", "FINANCIAL_JS", "financial_toggle_buttons", "financial_views_html", "has_financial"] + +_TYPE_DESCRIPTIONS = { + "appropriation": "Funds directly allocated to an agency or program", + "transfer": "Funds moved between accounts", + "rescission": "Previously appropriated funds clawed back", + "restriction": "Prohibition on how funds may be used", + "authorization": "Congressional authorization to request an appropriation; funds not yet available for obligation", + "cap": "Upper limit on spending for a specific purpose", + "earmark": "Funds designated for a recipient listed in a table", + "availability": "Time extension specifying when funds may be spent", + "sub_allocation": "Portion of funds reserved for a specific use", + "directive": "Mandatory action required of an agency", + "fee": "Charge imposed on individuals or entities", + "unknown": "Could not be automatically classified", +} + +_SIDES = {"v1": "A", "v2": "B"} + + +def has_financial(canonical: dict | None) -> bool: + """The views render only when the document carries a ledger for both sides.""" + fin = (canonical or {}).get("financial") or {} + return bool(fin.get("v1") and fin.get("v2")) + + +def _money(value) -> str: + if value is None: + return "—" + return f"${value:,.0f}" if float(value).is_integer() else f"${value:,.2f}" + + +def _signed(value) -> str: + if value == 0: + return "$0" + return ("+" if value > 0 else "−") + _money(abs(value)) + + +def _type_badge(kind: str) -> str: + kind = kind if kind in _TYPE_DESCRIPTIONS else "unknown" + return f'{escape(kind.replace("_", " "))}' + + +def _label(section: dict) -> str: + """The row's name, as the notebook named it: the last two breadcrumb levels.""" + return " > ".join(label for label, _level in section["path"][-2:]) + + +def _about_figures_html(canonical: dict) -> str: + """The information alert at the top of each financial view: what these figures are and + are not, and which classifier typed them. The version is the one the document records, + so a saved report names the rules that produced it, not whichever rules run now.""" + version = escape(str((canonical.get("financial") or {}).get("classifier") or "unknown")) + return ( + '
' + '' + "
These classifications are not authoritative. DeltaTrack assigns each " + "dollar amount a type (appropriation, rescission, cap, and so on) from the wording of its " + "clause. Audit and verify every value against the bill text before you rely on it. The " + "classifications are subject to change as the application is improved. Certain sections " + "and amounts may have been missed and not shown at all. Send feedback, " + "suggestions, screenshots or issues to the #congressional-tech channel of the " + '' + "Civic Tech DC Slack." + f'Pattern classifier version {version}.
' + "
" + ) + + +def _version_label(canonical: dict, side: str) -> str: + return (canonical.get("versions") or {}).get(side, {}).get("label") or "" + + +# ---------- Version A / B -------------------------------------------------------------------- + + +#: The views' warning mark: an inline SVG triangle in the text's own colour, not the +#: U+26A0 glyph, which fonts draw inconsistently (some as a colour emoji) and screen +#: readers announce as "warning sign" mid-sentence. Decorative (`aria-hidden`): every use +#: carries its meaning in words beside it, or in `_REVIEW_MARK`'s label. +_WARNING_ICON = ( + '' +) +#: The mark alone, where a row has no room for the word (the comparison's A / B lines). +_REVIEW_MARK = ( + f'{_WARNING_ICON}' +) + + +def _coverage_html(ledger: dict) -> str: + """Every figure in the version's text is on this page, or the page says how many are not.""" + total = ledger.get("amounts_in_text", 0) + shown = sum(len(c["amounts"]) for s in ledger["sections"] for c in s["clauses"]) + if shown >= total: + return ( + f'

All {total:,} dollar figures found in this version\'s text are shown below.

' + ) + missing = total - shown + return ( + f'

{_WARNING_ICON}' + f"{missing:,} of {total:,} dollar amounts in this version could not be placed in a section " + "and are not shown here. They are in the Full bill view.

" + ) + + +def _amended_note_html(ledger: dict) -> str: + n = sum(1 for s in ledger["sections"] for c in s["clauses"] for a in c["amounts"] if a["in_amended_law"]) + if not n: + return "" + return ( + f'

{n:,} amounts sit inside text this bill writes into other laws. ' + 'They are marked in amended law and are never ' + "counted as money given out.

" + ) + + +def _legend_html(ledger: dict) -> str: + counts: dict[str, int] = {} + for section in ledger["sections"]: + for clause in section["clauses"]: + counts[clause["type"]] = counts.get(clause["type"], 0) + 1 + items = [] + for kind in TYPES: + n = counts.get(kind, 0) + count = f'({n})' if n else '' + items.append( + f'
{_type_badge(kind)}{count}' + f'{escape(_TYPE_DESCRIPTIONS[kind])}
' + ) + return f'
Category key
{"".join(items)}
' + + +def _rows_html(side: str, ledger: dict) -> str: + rows = [] + for gid, section in enumerate(ledger["sections"]): + clauses = section["clauses"] + first = clauses[0] + review = any(c["needs_review"] for c in clauses) + badges = "" + if len(clauses) > 1: + badges += f'({len(clauses) - 1})' + if review: + badges += f'{_WARNING_ICON}review' + if "may_hold_several_sections" in section["flags"]: + badges += ( + '' + f"{_WARNING_ICON}may hold more than one section" + ) + if first["in_amended_law"]: + badges += 'in amended law' + amount = first["amount"] if first["amount"] is not None else 0 + rows.append( + f'' + f'{escape(_label(section))}{badges}' + f"{_type_badge(first['type'])}" + f'{_money(first["amount"])}' + ) + items = "".join( + f'
{_type_badge(c["type"])}' + f'{_money(c["amount"])}' + + ('in amended law' if c["in_amended_law"] else "") + + "
" + for c in clauses + ) + pieces = escape(json.dumps(section["pieces"], separators=(",", ":"))) + rows.append( + f'' + '
' + f'
{items}
' + f'
' + f"
" + ) + return "".join(rows) + + +def _version_view_html(canonical: dict, side: str) -> str: + ledger = canonical["financial"][side] + letter = _SIDES[side] + label = _version_label(canonical, side) + table = ( + f'
' + '' + '' + f"{_rows_html(side, ledger)}
AccountTypeAmount
" + if ledger["sections"] + else '

No dollar amounts found in this version.

' + ) + return ( + f'" + ) + + +# ---------- Comparison ----------------------------------------------------------------------- + + +def _overlapping_groups(sections: list[dict], span: dict | None) -> list[int]: + """Positions of the ledger sections a change's span overlaps.""" + if not span: + return [] + return [i for i, s in enumerate(sections) if s["span"][0] < span["end"] and span["start"] < s["span"][1]] + + +def _side_lines(sections: list[dict]) -> list[str]: + return [ + _type_badge(s["clauses"][0]["type"]) + + (f" {_REVIEW_MARK}" if any(c["needs_review"] for c in s["clauses"]) else "") + for s in sections + ] + + +def _given(sections: list[dict]): + values = [given_out(s["clauses"][0]) for s in sections] + values = [v for v in values if v is not None] + return sum(values) if values else None + + +def comparison_rows(canonical: dict) -> list[dict]: + """The comparison's rows: each reported change whose section holds money on either side. + + ``change_index`` is the change's position in ``canonical["changes"]``, which is also its + card's ``id="change-{index}"`` in the Changes view. ``a_groups`` / ``b_groups`` are the + sections' positions in that version's ledger, which are their rows' ``data-group`` in + the Version A / B views (the comparison's links open them there). + """ + fin = canonical["financial"] + rows = [] + for index, change in enumerate(canonical.get("changes") or []): + span = change.get("full_text_span") or {} + a_groups = _overlapping_groups(fin["v1"]["sections"], span.get("v1")) + b_groups = _overlapping_groups(fin["v2"]["sections"], span.get("v2")) + if a_groups or b_groups: + rows.append( + { + "change_index": index, + "change": change, + "a": [fin["v1"]["sections"][g] for g in a_groups], + "b": [fin["v2"]["sections"][g] for g in b_groups], + "a_groups": a_groups, + "b_groups": b_groups, + } + ) + return rows + + +def _jump_links_html(row: dict) -> str: + """Links from a comparison row to its sections in Version A and Version B, where every + clause, tag and amount is laid out; the comparison itself stays one line per side.""" + links = [] + for side, letter, groups in (("v1", "A", row["a_groups"]), ("v2", "B", row["b_groups"])): + if not groups: + links.append(f'not in Version {letter}') + continue + for n, group in enumerate(groups): + text = f"Version {letter} section" + (f" {n + 1}" if len(groups) > 1 else "") + links.append( + f'' + ) + return f'{"".join(links)}' + + +def _location_html(change: dict) -> str: + path = (change.get("path") or {}).get("v2") or (change.get("path") or {}).get("v1") or [] + if not path: + return '(location not resolved)' + parents = " › ".join(escape(p) for p in path[:-1]) + return f'{escape(path[-1])}{parents}' + + +def _stack_html(row: dict) -> str: + cells = [] + for letter, sections, cls in (("A", row["a"], "old"), ("B", row["b"], "new")): + if not sections: + cells.append( + f'{letter}' + f'not in {letter}' + ) + continue + for tag, section in zip(_side_lines(sections), sections): + cells.append( + f'{letter}' + f'{tag}' + f'{_money(section["clauses"][0]["amount"])}' + ) + return f'
{"".join(cells)}
' + + +def _difference(row: dict) -> tuple[float | None, str]: + """(B minus A in money given out, or None when neither side gives any out; its note).""" + a, b = _given(row["a"]), _given(row["b"]) + if a is None and b is None: + return None, "not given-out money" + delta = (b or 0) - (a or 0) + return delta, "text changed" if delta == 0 else "" + + +def _difference_html(row: dict) -> tuple[str, float]: + delta, note = _difference(row) + if delta is None: + return f'—{note}', 0 + if delta == 0: + return f'$0{note}', 0 + cls = "fin-up" if delta > 0 else "fin-down" + return f'{_signed(delta)}', delta + + +def _amended_figures(sections: list[dict]) -> list: + return [a["value"] for s in sections for c in s["clauses"] for a in c["amounts"] if a["in_amended_law"]] + + +def comparison_export_rows(canonical: dict) -> list[dict]: + """The comparison table as data, one entry per row, for its CSV export. + + Built from the same `comparison_rows` and `_difference` the table renders, so the export + cannot disagree with the page. Carries no bill text: the page adds each change's words + from the embedded document (`changes[change_index].text`). + """ + out = [] + for row in comparison_rows(canonical): + change = row["change"] + path = (change.get("path") or {}).get("v2") or (change.get("path") or {}).get("v1") or [] + delta, note = _difference(row) + entry = { + "change_index": row["change_index"], + "change_type": change["change_type"], + "location": " > ".join(path), + "difference": delta, + "difference_note": note, + } + for key, sections in (("a", row["a"]), ("b", row["b"])): + entry[key] = { + "sections": [_label(s) for s in sections], + "types": [s["clauses"][0]["type"] for s in sections], + "amounts": [s["clauses"][0]["amount"] for s in sections], + "given_out": _given(sections), + "needs_review": any(c["needs_review"] for s in sections for c in s["clauses"]), + "flags": sorted({f for s in sections for f in s["flags"]}), + "amended_law_amounts": _amended_figures(sections), + } + out.append(entry) + return out + + +def _amended_rows_html(rows: list[dict]) -> str: + items = [] + for row in rows: + + def figures(sections): + return ", ".join(_money(v) for v in _amended_figures(sections)) + + a, b = figures(row["a"]), figures(row["b"]) + if (a or b) and a != b: + items.append( + f"{_location_html(row['change'])}" + f'' + f"{escape(row['change']['change_type'])}" + f"A: {a or '—'} → B: {b or '—'}" + ) + if not items: + return "" + return ( + '
Amounts in laws this bill amends (not counted above)
' + f'{"".join(items)}
' + ) + + +def _comparison_view_html(canonical: dict) -> str: + rows = comparison_rows(canonical) + body, counts, net = [], {}, 0 + for row in rows: + ct = row["change"]["change_type"] + counts[ct] = counts.get(ct, 0) + 1 + diff_html, delta = _difference_html(row) + net += delta + i = row["change_index"] + body.append( + f'' + f'▸{_location_html(row["change"])}{_jump_links_html(row)}' + f'{escape(ct)}' + f"{_stack_html(row)}" + f'{diff_html}' + f'' + ) + summary = [f"Sections with money changes {len(rows)}"] + summary += [f"{escape(k.capitalize())} {v}" for k, v in counts.items()] + net_cls = "fin-up" if net > 0 else ("fin-down" if net < 0 else "fin-same") + summary.append(f'Net change in money given out {_signed(net)}') + a_label, b_label = _version_label(canonical, "v1"), _version_label(canonical, "v2") + table = ( + '
' + "" + '' + f"{''.join(body)}
Location in the billText block changeVersion A → Version BDifference
" + if rows + else '

No change touches a section that holds money.

' + ) + # `' + '
' + "

Inferred Financial Comparison

" + '' + "
" + f'' + f'
Version A: {escape(a_label)} → Version B: {escape(b_label)}
' + f"{_about_figures_html(canonical)}" + '

Every change in the Changes view whose section holds money, with the same ' + "change words. The difference counts money given out or taken back only. Open a row for that " + "change's word diff.

" + f'
{"".join(summary)}
{table}' + f"{_amended_rows_html(rows)}
" + ) + + +def financial_views_html(canonical: dict) -> str: + """All three views, each a hidden ``.view`` the view toggle reveals.""" + if not has_financial(canonical): + return "" + return _version_view_html(canonical, "v1") + _version_view_html(canonical, "v2") + _comparison_view_html(canonical) + + +def financial_toggle_buttons() -> str: + """The three view-toggle buttons, after Changes / Full bill. The comparison comes first: + like Changes and Full bill it is about the pair, where Version A / B are one version each.""" + return "".join( + f'' + for view, label in ( + ("fin-compare", "Inferred Financial Comparison"), + ("fin-a", "Financials – Version A"), + ("fin-b", "Financials – Version B"), + ) + ) + + +# The views' styles and script, which `diff_html.format_diff_html` includes in every report. +# They live here with the money vocabulary they are written in (type names such as +# `rescission`), so the renderer module itself stays outside the financial layer (ADR 0018). +FINANCIAL_CSS = """\ +/* Financial views (ADR 0023): Version A / B, and the comparison */ +/* The three financial tabs use the full window on a wide screen: their tables are columns, + cramped at the report's 940px reading width. Changes and Full bill keep that width (their + cards are prose). `body[data-active-view]` is set by the report's view switch. */ +@media (min-width: 821px) { + body[data-active-view="fin-a"] .main, body[data-active-view="fin-b"] .main, + body[data-active-view="fin-compare"] .main { + max-width: none; padding-right: 10%; } +} +.fin-head { display: flex; align-items: baseline; justify-content: space-between; gap: 12px; flex-wrap: wrap; } +.view-fin-a h2, .view-fin-b h2, .view-fin-compare h2 { font-size: 20px; margin-bottom: 2px; } +.fin-version { color: var(--muted-foreground); font-size: 14px; margin-bottom: 12px; } +.fin-coverage, .fin-note, .fin-explain { font-size: 13px; color: var(--muted-foreground); margin-bottom: 10px; } +/* "About these figures": an information alert, left-accented, above each view's content. */ +.fin-alert { display: flex; gap: 12px; align-items: flex-start; margin: 4px 0 16px; padding: 12px 16px; + background: var(--info); color: var(--info-foreground); border: 1px solid var(--info); + border-left: 4px solid var(--info-foreground); border-radius: var(--radius); font-size: 14px; line-height: 1.55; } +.fin-alert__icon { flex: none; width: 20px; height: 20px; margin-top: 1px; fill: none; + stroke: var(--info-foreground); stroke-width: 2; stroke-linecap: round; } +.fin-alert__dot { fill: var(--info-foreground); } +.fin-alert a { color: inherit; font-weight: 600; text-decoration: underline; } +.fin-alert__version { display: block; margin-top: 4px; font-size: 12px; opacity: 0.85; } +.fin-coverage--gap { color: var(--destructive); font-weight: 600; } +.fin-empty { color: var(--muted-foreground); padding: 16px 2px; font-size: 14px; } +.view-fin-a h3, .view-fin-b h3 { font-size: 18px; margin: 18px 0 10px; } +.fin-key { border: 1px solid var(--border); border-radius: var(--radius); padding: 12px 14px; + background: var(--card); margin-bottom: 8px; } +.fin-key__title { font-weight: 600; font-size: 14px; margin-bottom: 8px; } +.fin-key__item { display: flex; align-items: baseline; gap: 8px; padding: 2px 0; } +.fin-key__count { color: var(--muted-foreground); font-size: 12px; min-width: 32px; display: inline-block; } +.fin-key__desc { color: var(--muted-foreground); font-size: 13px; } +.fin-type { display: inline-block; padding: 2px 7px; border-radius: 999px; font-size: 11px; font-weight: 600; + white-space: nowrap; background: var(--muted); color: var(--muted-foreground); } +.fin-type--appropriation { background: var(--diff-add); color: var(--diff-add-foreground); } +.fin-type--rescission, .fin-type--restriction { background: var(--diff-remove); color: var(--diff-remove-foreground); } +.fin-type--cap { background: var(--diff-modified); color: var(--diff-modified-foreground); } +.fin-type--transfer { background: var(--fin-transfer); color: var(--fin-transfer-foreground); } +.fin-type--authorization { background: var(--fin-authorization); color: var(--fin-authorization-foreground); } +.fin-type--fee { background: var(--fin-fee); color: var(--fin-fee-foreground); } +.fin-type--directive { background: var(--fin-directive); color: var(--fin-directive-foreground); } +.fin-type--earmark { background: var(--fin-earmark); color: var(--fin-earmark-foreground); } +.fin-type--availability { background: var(--fin-availability); color: var(--fin-availability-foreground); } +.fin-type--sub_allocation { background: var(--fin-sub-allocation); color: var(--fin-sub-allocation-foreground); } +.fin-sort { display: flex; gap: 8px; align-items: center; margin-bottom: 12px; font-size: 13px; } +.fin-sort span { color: var(--muted-foreground); } +.fin-sort__btn { padding: 4px 12px; border: 1px solid var(--border); border-radius: 999px; background: var(--card); + cursor: pointer; font: inherit; font-family: var(--font-sans); font-size: 13px; color: var(--foreground); } +.fin-sort__btn:hover { background: var(--secondary); } +.fin-sort__btn.is-active { background: var(--foreground); border-color: var(--foreground); color: var(--card); } +.fin-table { width: 100%; border-collapse: collapse; background: var(--card); border-radius: var(--radius); + box-shadow: var(--shadow-soft); overflow: hidden; font-size: 14px; table-layout: fixed; } +.fin-table th { background: var(--accent); text-align: left; padding: 9px 12px; border-bottom: 2px solid var(--border); + font-weight: 600; font-size: 13px; } +.fin-table td { padding: 8px 12px; border-bottom: 1px solid var(--border); vertical-align: middle; } +.fin-col-name { width: 72%; } .fin-col-type { width: 13%; } .fin-col-amount { width: 15%; text-align: right; } +.fin-amount { text-align: right; font-family: var(--font-mono); font-variant-numeric: tabular-nums; + white-space: nowrap; } +.fin-row, .fin-crow { cursor: pointer; } +.fin-row { background: var(--muted); font-weight: 600; } +.fin-row:hover td, .fin-crow:hover td { background: var(--secondary); } +.fin-row__name::before { content: "\\25b6"; font-size: 10px; margin-right: 8px; color: var(--muted-foreground); + display: inline-block; transition: transform 0.15s; } +.fin-row.is-open .fin-row__name::before { transform: rotate(90deg); } +.fin-count { font-size: 11px; color: var(--muted-foreground); font-weight: 400; margin-left: 6px; } +.fin-review { background: var(--diff-modified); color: var(--diff-modified-foreground); font-size: 11px; + padding: 1px 6px; border-radius: 999px; margin-left: 6px; font-weight: 500; white-space: nowrap; } +/* The warning mark (`_WARNING_ICON`): drawn in the surrounding text's colour and size. */ +.fin-icon { display: inline-block; width: 1.15em; height: 1.15em; vertical-align: -0.2em; margin-right: 0.25em; + fill: none; stroke: currentColor; stroke-width: 2; stroke-linejoin: round; stroke-linecap: round; } +.fin-icon__dot { fill: currentColor; } +.fin-review[role="img"] .fin-icon { margin-right: 0; } +.fin-chip { font-size: 11px; font-weight: 600; border-radius: 4px; padding: 0 5px; margin-left: 6px; } +.fin-chip--law { background: var(--muted); color: var(--muted-foreground); border: 1px dashed var(--border); } +.fin-detail > td { padding: 0; background: var(--card); } +.fin-detail__inner { display: flex; } +.fin-detail__amounts { min-width: 220px; max-width: 260px; padding: 12px; border-right: 1px solid var(--border); + display: flex; flex-direction: column; gap: 10px; } +.fin-clause { display: flex; flex-direction: column; gap: 3px; align-items: flex-start; } +.fin-clause__amount { font-family: var(--font-mono); font-size: 15px; font-variant-numeric: tabular-nums; } +.fin-detail__text { padding: 12px 16px; font-size: 13px; line-height: 1.9; flex: 1; font-weight: 400; } +.fin-piece { border-radius: 3px; padding: 1px 3px; } +.fin-piece.fin-type--unknown { background: var(--muted); color: var(--foreground); } +.fin-proviso { color: var(--muted-foreground); } +.fin-card { background: var(--card); border: 1px solid var(--border); border-radius: var(--radius); overflow: hidden; + box-shadow: var(--shadow-soft); } +.fin-card .fin-table { box-shadow: none; border-radius: 0; table-layout: auto; } +.fin-summary { display: flex; gap: 18px; flex-wrap: wrap; padding: 10px 14px; border-bottom: 1px solid var(--border); + font-size: 13px; color: var(--muted-foreground); } +.fin-summary b { color: var(--foreground); font-variant-numeric: tabular-nums; } +.fin-table--compare td { vertical-align: top; } +.fin-table--compare th { background: var(--secondary); font-size: 12px; text-transform: uppercase; + letter-spacing: 0.04em; color: var(--muted-foreground); border-bottom: 1px solid var(--border); padding: 8px 10px; } +.fin-table--compare td { padding: 8px 10px; } +.fin-table--compare .fin-amount { font-family: var(--font-sans); } +.fin-stack__tag { white-space: nowrap; } +.fin-caret { color: var(--muted-foreground); width: 16px; display: inline-block; } +.fin-loc { font-weight: 600; } +.fin-loc small { display: block; font-weight: 400; color: var(--muted-foreground); font-size: 12px; } +/* Links from a comparison row to its sections in Version A / B. */ +.fin-jumps { display: flex; flex-wrap: wrap; gap: 4px 14px; margin-top: 6px; } +.fin-jump { background: none; border: 0; padding: 0; font: inherit; font-size: 12px; color: var(--primary); + text-decoration: underline; text-underline-offset: 2px; cursor: pointer; } +.fin-jump:hover { color: var(--foreground); } +.fin-jump--none { color: var(--muted-foreground); text-decoration: none; cursor: default; } +.fin-row.is-target > td { background: var(--accent); transition: background 0.6s ease; } +.fin-stack { display: grid; grid-template-columns: auto 1fr auto; gap: 2px 8px; align-items: center; } +.fin-ab { font-size: 11px; font-weight: 700; border-radius: 4px; padding: 0 5px; } +.fin-ab--a { background: var(--secondary); color: var(--primary); } +.fin-ab--b { background: var(--accent); color: var(--gold); } +.fin-stack__amt { text-align: right; font-variant-numeric: tabular-nums; white-space: nowrap; } +.fin-stack__amt--old { color: var(--muted-foreground); } +.fin-stack__amt--new { font-weight: 600; } +.fin-stack__none { color: var(--muted-foreground); font-style: italic; text-align: right; } +.fin-up { color: var(--diff-add-foreground); } .fin-down { color: var(--diff-remove-foreground); } +.fin-same { color: var(--muted-foreground); } +.fin-same small, .fin-up small, .fin-down small { display: block; font-size: 11px; } +.fin-cdetail > td { background: var(--background); padding: 12px 14px; } +.fin-subhead { font-size: 13px; font-weight: 600; color: var(--muted-foreground); padding: 14px 14px 6px; + border-top: 1px solid var(--border); } +.fin-table--plain { box-shadow: none; } +@media (max-width: 820px) { + .fin-table--compare { min-width: 640px; } + .fin-detail__inner { flex-direction: column; } + .fin-detail__amounts { max-width: none; border-right: 0; border-bottom: 1px solid var(--border); } + .fin-col-name { width: 55%; } .fin-col-type { width: 20%; } .fin-col-amount { width: 25%; } +} + +""" + +FINANCIAL_JS = """\ +document.addEventListener('DOMContentLoaded', function() { + // Financial views (ADR 0023). Their text is read from the embedded document's full_text, + // sliced by the ledger's offsets, so no bill text is duplicated into the page. + var finDoc = null; + function finData() { + if (!finDoc) finDoc = JSON.parse(document.getElementById('diff-data').textContent); + return finDoc; + } + // The prose between two full_text offsets as one line, read the way + // `financial._prose_blocks` reads it: the PDF line-number gutter off, the unnumbered + // running header skipped, lines joined with a space, and a word the printer broke + // across lines (`speci-` / `fied`) rejoined. + function finText(side, s, e) { + var doc = finData(), ft = doc.full_text[side]; + var guttered = ((doc.versions || {})[side] || {}).source !== 'xml'; + var out = '', pos = ft.lastIndexOf('\\n', s - 1) + 1; + while (pos < e) { + var nl = ft.indexOf('\\n', pos); + if (nl < 0) nl = ft.length; + var cs = guttered ? pos + 7 : pos; + var a = Math.max(cs, s), b = Math.min(nl, e); + var furniture = guttered && !ft.slice(pos, Math.min(cs, nl)).trim() + && /^\\s*[\\u2020\\u2021\\u2022]/.test(ft.slice(cs, nl)); + var body = b > a ? ft.slice(a, b).replace(/\\s+$/, '').replace(/^\\s+/, '') : ''; + if (body && !furniture) { + if (out.length >= 2 && out.slice(-1) === '-' && /[A-Za-z0-9]/.test(out.charAt(out.length - 2)) + && /^[a-z]/.test(body)) { + out = out.slice(0, -1) + body; + } else { + out = out ? out + ' ' + body : body; + } + } + pos = nl + 1; + } + return out; + } + function finEsc(t) { + return t.replace(/[&<>"]/g, function(c) { return {'&': '&', '<': '<', '>': '>', '"': '"'}[c]; }); + } + function finFillText(box) { + if (box.dataset.done) return; + var side = box.dataset.side, pieces = JSON.parse(box.dataset.pieces), html = ''; + pieces.forEach(function(p) { + var t = finEsc(finText(side, p[0], p[1])); + html += p[2] === 'proviso' + ? '' + t + ' ' + : '' + t + ' '; + }); + box.innerHTML = html; + box.dataset.done = '1'; + } + // Comparison: a row opens onto that change's card from the Changes view, cloned, so the + // word diff shown is the Changes view's own. + function finFillChange(detail) { + var cell = detail.firstElementChild; + if (cell.childElementCount) return; + var card = document.getElementById(detail.dataset.change); + if (!card) return; + [].slice.call(card.children).forEach(function(part) { + if (!part.classList.contains('change-header')) cell.appendChild(part.cloneNode(true)); + }); + } + function finFill(detail) { + if (detail.classList.contains('fin-cdetail')) finFillChange(detail); + else finFillText(detail.querySelector('.fin-detail__text')); + } + // A detail row always follows its summary row (the sort moves them as a pair). + function finToggle(detail) { + var row = detail.previousElementSibling, open = detail.hidden; + detail.hidden = !open; + row.classList.toggle('is-open', open); + row.setAttribute('aria-expanded', open ? 'true' : 'false'); + var caret = row.querySelector('.fin-caret'); + if (caret) caret.textContent = open ? '\\u25be' : '\\u25b8'; + if (open) finFill(detail); + } + document.querySelectorAll('.fin-table').forEach(function(table) { + table.addEventListener('click', function(e) { + var row = e.target.closest('tr.fin-row, tr.fin-crow'); + if (row) finToggle(row.nextElementSibling); + }); + table.addEventListener('keydown', function(e) { + if (e.key !== 'Enter' && e.key !== ' ') return; + var row = e.target.closest('tr.fin-row, tr.fin-crow'); + if (row && row === e.target) { e.preventDefault(); finToggle(row.nextElementSibling); } + }); + }); + // The page's search (`find:prepare` / `find:reveal` in the report script) reads the + // collapsed rows too: their text is filled in before it indexes, and a hit opens its row. + document.addEventListener('find:prepare', function(e) { + if (!e.target.querySelectorAll) return; + e.target.querySelectorAll('tr.fin-detail, tr.fin-cdetail').forEach(finFill); + }); + document.addEventListener('find:reveal', function(e) { + var detail = e.target; + if (detail.matches && detail.matches('tr.fin-detail, tr.fin-cdetail') && detail.hidden) finToggle(detail); + }); + // A comparison row's "Version A section" / "Version B section": switch to that view (through + // its own tab, so everything a tab change does happens), open the section's row, show it. + document.querySelectorAll('.fin-jump[data-group]').forEach(function(link) { + link.addEventListener('click', function(e) { + e.stopPropagation(); // the link sits in a comparison row, which opens on click + var view = link.dataset.side === 'v1' ? 'fin-a' : 'fin-b'; + [].slice.call(document.querySelectorAll('.view-toggle__btn')) + .filter(function(b) { return b.dataset.view === view; }) + .forEach(function(b) { b.click(); }); + var row = document.querySelector('.view-' + view + ' tr.fin-row[data-group="' + link.dataset.group + '"]'); + if (!row) return; + if (row.nextElementSibling.hidden) finToggle(row.nextElementSibling); + row.scrollIntoView({behavior: 'smooth', block: 'center'}); + row.classList.add('is-target'); + setTimeout(function() { row.classList.remove('is-target'); }, 1600); + row.focus({preventScroll: true}); + }); + }); + // Sort: Default (bill order) / Amount, largest first / Needs review first. Each row + // travels with its detail row. + document.querySelectorAll('.fin-sort').forEach(function(bar) { + var table = bar.parentElement.querySelector('.fin-table'); + if (!table || !table.tBodies.length) return; + bar.addEventListener('click', function(e) { + var btn = e.target.closest('.fin-sort__btn'); + if (!btn) return; + var key = btn.dataset.sort, body = table.tBodies[0]; + var groups = [].slice.call(body.querySelectorAll('tr.fin-row')).map(function(r) { + return [r, body.querySelector('tr.fin-detail[data-group="' + r.dataset.group + '"]')]; + }); + groups.sort(function(x, y) { + if (key === 'amount') return parseFloat(y[0].dataset.amount) - parseFloat(x[0].dataset.amount); + if (key === 'review') return (y[0].dataset.review - x[0].dataset.review) + || (x[0].dataset.group - y[0].dataset.group); + return x[0].dataset.group - y[0].dataset.group; + }); + groups.forEach(function(g) { body.appendChild(g[0]); body.appendChild(g[1]); }); + bar.querySelectorAll('.fin-sort__btn').forEach(function(b) { b.classList.toggle('is-active', b === btn); }); + }); + }); + // Export Inferred Financials: that version's whole ledger, one row per dollar amount, + // the breadcrumb spread into repeated columns so it sorts and pivots without nesting. + var FIN_LEVEL_COLUMN = {division: 'division', title: 'title', major: 'department', agency: 'agency', + account: 'account', grouping: 'grouping', section: 'section', subsection: 'subsection'}; + function csvCell(v) { + var t = v === null || v === undefined ? '' : String(v); + return /[",\\r\\n]/.test(t) ? '"' + t.replace(/"/g, '""') + '"' : t; + } + document.querySelectorAll('.fin-export').forEach(function(btn) { + btn.addEventListener('click', function() { + var doc = finData(), side = btn.dataset.side, letter = side === 'v1' ? 'A' : 'B'; + var cols = ['version', 'version_label', 'location', 'division', 'title', 'department', 'agency', + 'account', 'grouping', 'section', 'subsection', 'section_no', 'clause_no', 'clause_level', + 'clause_type', 'amount', 'is_clause_amount', 'not_to_exceed', 'in_amended_law', + 'needs_review', 'flags', 'clause_text']; + var lines = [cols.join(',')]; + doc.financial[side].sections.forEach(function(sec, si) { + var levels = {}; + sec.path.forEach(function(p) { var c = FIN_LEVEL_COLUMN[p[1]]; if (c) levels[c] = p[0]; }); + var location = sec.path.map(function(p) { return p[0]; }).join(' > '); + sec.clauses.forEach(function(cl, ci) { + var text = finText(side, cl.span[0], cl.span[1]); + var clauseAmountTaken = false; + cl.amounts.forEach(function(a) { + var isClause = !clauseAmountTaken && a.value === cl.amount; + if (isClause) clauseAmountTaken = true; + lines.push([letter, doc.versions[side].label, location, levels.division, levels.title, + levels.department, levels.agency, levels.account, levels.grouping, levels.section, + levels.subsection, si + 1, ci + 1, cl.level, cl.type, a.value, isClause, a.cap, + a.in_amended_law, cl.needs_review, sec.flags.join(' '), text].map(csvCell).join(',')); + }); + }); + }); + finDownload(lines, ['version', letter]); + }); + }); + // The comparison's CSV: one row per comparison row, as the table shows it. The rows come + // from Python (`comparison_export_rows`), so the difference is the table's own; the + // change's words come from the embedded document. + function finDownload(lines, suffix) { + var bill = finData().bill || {}; + // A PDF upload knows its congress but not its bill number: leave out what is empty. + var parts = [bill.congress, bill.type, bill.number].filter(function(x) { return x; }); + var name = ['financials'].concat(parts, suffix).join('-') + '.csv'; + var url = URL.createObjectURL(new Blob(['\\ufeff' + lines.join('\\r\\n')], {type: 'text/csv'})); + var link = document.createElement('a'); + link.href = url; link.download = name; + document.body.appendChild(link); link.click(); link.remove(); + setTimeout(function() { URL.revokeObjectURL(url); }, 1000); + } + var compareBtn = document.querySelector('.fin-export-compare'); + if (compareBtn) compareBtn.addEventListener('click', function() { + var doc = finData(), rows = JSON.parse(document.getElementById('fin-compare-data').textContent); + var cols = ['change_index', 'change_type', 'location', 'version_a_label', 'version_b_label', + 'a_sections', 'a_types', 'a_amounts', 'a_given_out', 'b_sections', 'b_types', 'b_amounts', + 'b_given_out', 'difference', 'difference_note', 'a_needs_review', 'b_needs_review', 'flags', + 'a_amended_law_amounts', 'b_amended_law_amounts', 'text_before', 'text_after']; + var lines = [cols.join(',')]; + function joined(list) { return list.join(' | '); } + // One line per change: before and after side by side, each as one line of words. + function oneLine(t) { return t ? String(t).replace(/\\s+/g, ' ').trim() : ''; } + rows.forEach(function(r) { + var text = (doc.changes[r.change_index] || {}).text || {}; + var flags = r.a.flags.concat(r.b.flags.filter(function(f) { return r.a.flags.indexOf(f) < 0; })); + lines.push([r.change_index, r.change_type, r.location, doc.versions.v1.label, doc.versions.v2.label, + joined(r.a.sections), joined(r.a.types), joined(r.a.amounts), r.a.given_out, + joined(r.b.sections), joined(r.b.types), joined(r.b.amounts), r.b.given_out, + r.difference, r.difference_note, r.a.needs_review, r.b.needs_review, flags.join(' '), + joined(r.a.amended_law_amounts), joined(r.b.amended_law_amounts), oneLine(text.old), oneLine(text.new)] + .map(csvCell).join(',')); + }); + finDownload(lines, ['comparison']); + }); +}); +""" diff --git a/src/deltatrack/palette.py b/src/deltatrack/palette.py index ed4a053b..0c9d6b54 100644 --- a/src/deltatrack/palette.py +++ b/src/deltatrack/palette.py @@ -44,6 +44,9 @@ # Status "--destructive": "#c04040", "--success": "#3d9b6d", + # Information notices (the financial views' "About these figures" alert, ADR 0023) + "--info": "#e3f2f8", + "--info-foreground": "#0b4f63", # Diff states, one background/foreground pair each "--diff-add": "#d3f0e2", "--diff-add-foreground": "#1a6647", @@ -53,8 +56,29 @@ "--diff-modified-foreground": "#8a6320", "--diff-moved": "#eef0f8", "--diff-moved-foreground": "#2c2c5c", + # Money types in the financial views (ADR 0023), for the types no diff state already + # colours: appropriation takes --diff-add, rescission and restriction --diff-remove, cap + # --diff-modified, unknown --muted. Values are the research notebook's report's. + "--fin-transfer": "#cffafe", + "--fin-transfer-foreground": "#155e75", + "--fin-authorization": "#e0f2fe", + "--fin-authorization-foreground": "#0c4a6e", + "--fin-fee": "#ffedd5", + "--fin-fee-foreground": "#9a3412", + "--fin-directive": "#e0e7ff", + "--fin-directive-foreground": "#3730a3", + "--fin-earmark": "#ede9fe", + "--fin-earmark-foreground": "#5b21b6", + "--fin-availability": "#d1fae5", + "--fin-availability-foreground": "#065f46", + "--fin-sub-allocation": "#dbeafe", + "--fin-sub-allocation-foreground": "#1e40af", # Geometry "--radius": "0.625rem", + # How far a jump target clears the report's sticky action bar. 64px is the one-row + # bar; the report's script sets the bar's measured height (it is two rows with the + # financial views, wraps on narrow windows, and is not sticky on phones). + "--sticky-bar-height": "64px", # Typography. System stacks, never a webfont: a report fetches nothing when opened. "--font-sans": ("ui-sans-serif, system-ui, -apple-system, 'Segoe UI', Roboto, Helvetica, Arial, sans-serif"), "--font-serif": "ui-serif, Georgia, 'Times New Roman', serif", diff --git a/src/deltatrack/parsers/pdf_anchors.py b/src/deltatrack/parsers/pdf_anchors.py index cf97a124..aa366cba 100644 --- a/src/deltatrack/parsers/pdf_anchors.py +++ b/src/deltatrack/parsers/pdf_anchors.py @@ -16,6 +16,7 @@ from dataclasses import dataclass, replace from typing import Literal +from deltatrack.parsers.pdf_heading_passes import converge_headings from deltatrack.parsers.pdf_text import Page, parse_lines, strip_page_chrome AnchorKind = Literal["title", "section", "account", "grouping", "agency", "major", "subsection", "preamble"] @@ -39,6 +40,10 @@ class Anchor: # so a bill that gains/loses division wrappers across versions still aligns). # Empty on single-division bills, leaving all existing behavior unchanged. division: str = "" + # Case pattern of the heading's first printed line (ADR 0022): True for title case in small + # caps (agency style), False for even letters, None when unknown or not a heading anchor. Set + # by ``pdf_heading_passes.converge_headings``; read by the scope rules in ``_breadcrumb_core``. + caps: bool | None = None @dataclass(frozen=True) @@ -876,6 +881,8 @@ def extract_anchors(pages: list[Page]) -> list[Anchor]: # appropriations-specific English phrase (#114, ADR 0018). anchors.sort(key=lambda a: (a.page_number, a.line_number)) + # Re-read the detected headings against the whole reading order (ADR 0022). + anchors = converge_headings(pages, anchors) return _assign_divisions(anchors, _flatten(pages)) @@ -895,6 +902,33 @@ def breadcrumb_for(anchor: Anchor, all_anchors: tuple[Anchor, ...] | list[Anchor return (anchor.division, *core) if anchor.division else core +def _agency_reaches( + account: Anchor, + agency: Anchor, + j: int, + idx: int, + all_anchors: tuple[Anchor, ...] | list[Anchor], + agency_styled_between: bool, +) -> bool: + """Whether an agency heading at ``j`` is the parent of the account at ``idx`` (ADR 0022). + + Directly above the account it always is. Carried over past other accounts, it is not when + the case pattern shows it is not a real container: it prints like the heading right below it + (the top of a wrap, or a same-level stack), the account itself prints agency-style, or an + agency-styled heading lies between them (an agency the detectors read as an account, such as + one followed by prose). Unknown case pattern (``caps is None``) never blocks: fail closed to + the carry-over reading. + """ + if j == idx - 1: + return True + below = all_anchors[j + 1] + if below.caps is not None and below.caps == agency.caps: + return False + if agency.caps and (account.caps is True or agency_styled_between): + return False + return True + + def _breadcrumb_core(anchor: Anchor, all_anchors: tuple[Anchor, ...] | list[Anchor]) -> tuple[str, ...]: """Walk back through `all_anchors` from `anchor` to assemble a parent chain. @@ -921,10 +955,14 @@ def _breadcrumb_core(anchor: Anchor, all_anchors: tuple[Anchor, ...] | list[Anch For a MAJOR / department anchor (DeltaTrack#105): the body-size heading under a TITLE deepens the chain to `("TITLE I", "DEPARTMENTAL MANAGEMENT", "MANAGEMENT - DIRECTORATE", "OPERATIONS AND SUPPORT")`. One major scopes the whole title, so it - is captured INDEPENDENTLY of the agency `agency_blocked` gate (a title-level - account after a grouping boundary still carries the major) and threaded leftmost, - just inside the TITLE. + DIRECTORATE", "OPERATIONS AND SUPPORT")`. The nearest major above an anchor scopes + it; a title can hold several, since department headings are also read mid-title + (ADR 0022). It is captured INDEPENDENTLY of the agency `agency_blocked` gate (a + title-level account after a grouping boundary still carries the major) and threaded + leftmost, just inside the TITLE. A major also ends the scope of any agency above it. + + An agency carried past other accounts reaches an account only when its case pattern + (`Anchor.caps`) allows it (`_agency_reaches`, ADR 0022). Breadcrumb DEPTH is detection-path dependent: major/agency/grouping parents exist only on the size path, so a low-coverage/no-band bill has no account level at all @@ -969,6 +1007,7 @@ def chain(title: tuple[str, ...]) -> tuple[str, ...]: parents = ((major,) if major else ()) + ((agency,) if agency else ()) + ((grouping,) if grouping else ()) return title + parents + (anchor.text,) + agency_styled_between = False # an agency-styled heading lies between this account and the agency for j in range(idx - 1, -1, -1): prev = all_anchors[j] if prev.kind == "title": @@ -981,10 +1020,17 @@ def chain(title: tuple[str, ...]) -> tuple[str, ...]: grouping = prev.text if anchor.kind == "account" and not agency_blocked: if prev.kind == "agency": - agency = prev.text agency_blocked = True # nearest agency only - elif prev.kind in ("grouping", "section"): - agency_blocked = True # agency scope ended before this account + if not _agency_reaches(anchor, prev, j, idx, all_anchors, agency_styled_between): + continue + agency = prev.text + elif prev.kind in ("grouping", "section", "major"): + # A grouping or section ends the agency's scope; so does a department heading, + # which now also appears mid-title (ADR 0022): an agency above it belongs to an + # earlier department. + agency_blocked = True + elif prev.kind == "account" and prev.caps is True: + agency_styled_between = True parents = chain(()) if len(parents) > 1: return parents diff --git a/src/deltatrack/parsers/pdf_heading_passes.py b/src/deltatrack/parsers/pdf_heading_passes.py new file mode 100644 index 00000000..802f3d55 --- /dev/null +++ b/src/deltatrack/parsers/pdf_heading_passes.py @@ -0,0 +1,591 @@ +"""Ordered cleanup passes that recover PDF heading structure for the bill ledger (ADR 0022). + +The anchor detectors in ``pdf_anchors`` judge each printed line largely on its own, so a +heading that wraps, a heading glued to its neighbour, a quoted heading name inside a sentence, +or a department heading in the middle of a title is misread. ``converge_headings`` re-reads the +detected headings against the whole reading order and returns the corrected anchor list. It runs +after detection and before divisions are assigned, and every pass fails closed: where its +evidence is missing it leaves the parser's reading unchanged. + +Order of operations (ADR 0022). Each pass reads only what earlier passes produced: + +1. **Stream**: one reading-order stream of lines across page breaks, with page furniture removed + (unnumbered running lines opening with a bullet or dagger). A page break is never a heading + boundary. +2. **Lines that cannot be headings**: quoted text (tracked across lines until the quote closes), + lines inside an unfinished sentence (a centered run followed by a new paragraph never is, and + ends the sentence carried down to it), and subsection-title continuations. Heading anchors on + such lines are dropped. +3. **Department-level headings anywhere in a title**: a centered line set entirely at body size + in capitals is a department-level heading, not only directly under ``TITLE n``. +4. **Heading runs**: every run of heading lines between body text is re-segmented line break by + line break. First decisive signal wins: a line-break hyphen joins; a line ending in a small + word (``AND``, ``OF``, ``FOR``, …) joins; an unknown case pattern keeps the detectors' reading; + letters printed differently split; letters printed alike join unless a veto applies (the lower + line stands alone as a heading at least twice elsewhere, or line fullness says the break was + deliberate). A hanging-indent block is then one heading whatever those decisions were. The + department line directly under a bare ``TITLE n`` is left as the major detector read it, and + uncentered lines running on from a numbered level's own line finish its name. A piece printed + like the run's last heading takes the same level; any other non-last piece is an agency. + +Scope (which heading an account inherits) is decided afterwards in +``pdf_anchors._breadcrumb_core`` from the ``Anchor.caps`` this module records. + +Structure is read from format alone: glyph size and case pattern, position, punctuation, and the +universal legislative tokens (ADR 0012, ADR 0018). No heading wording decides a line break; where +format cannot tell two stacked headings from one wrapped name, the run stays joined (ADR 0022, +"Known residuals"). +""" + +from __future__ import annotations + +import re +from collections import Counter +from dataclasses import replace +from typing import TYPE_CHECKING + +from deltatrack.parsers.pdf_text import TITLE_CASE_SMALL_WORDS + +if TYPE_CHECKING: + from deltatrack.parsers.pdf_anchors import Anchor + from deltatrack.parsers.pdf_text import Line, Page + +# Anchor kinds these passes may drop, join, split or add. Titles, sections, subsections and the +# front-matter preamble come from universal tokens and are left as detected. +_HEADING_KINDS = frozenset({"account", "agency", "major", "grouping"}) +_CONTAINER_KINDS = frozenset({"agency", "major", "grouping"}) + +# Page furniture PDFium floats into the text layer: an unnumbered line opening with a bullet or +# dagger ("† HR 5895 EAS"). Inside the reading-order stream it would sit between the last body +# line of one page and the first heading of the next. +_FURNITURE = re.compile(r"^\s*[†‡•]") +_TITLE_BARE = re.compile(r"^\s*TITLE\s+[IVXLC]+\s*$") +# Universal legislative tokens: a line opening with one is its own structural level (or the +# wrapped name of one), never a department-level heading. +_STRUCTURAL_TOKEN = re.compile(r"^\s*(TITLE|SUBTITLE|CHAPTER|SUBCHAPTER|PART|SUBPART|DIVISION)\b") +_LEADING_TOKEN = re.compile(r"^\s*(SEC\.|TITLE\b|DIVISION\b)") +# GPO opens a quoted block with ‘‘, opens every further quoted paragraph with ‘‘ again, and +# closes once at the end with ’’: a quote is open from its last opening mark to the next closing +# mark, not while openings outnumber closings. +_QUOTE_OPEN = ("‘‘", "“") +_QUOTE_MARK = re.compile("‘‘|’’|“|”") +_SENTENCE_END = (".", ":", ";", ".—", ".–", "—", "’’", "''", "”") + +# Geometry tolerances, points. +_CENTER_TOL = 3.0 # a centered line's midpoint sits within this of the column's +_INSET_MIN = 10.0 # a centered heading is inset at least this far on both sides +_PARA_INDENT_MIN = 14.0 # half the ~28 pt first-line indent of a GPO paragraph; continuations start at 0 +_HANG_TOL = 3.0 # a hanging continuation starts within this of the paragraph indent +_SPACE_WIDTH = 4.0 # one inter-word space, for the "next word would have fit" test +_BODY_SIZE_TOL = 0.3 # a department-style line's letters all sit within this of body size +_MAX_ROUNDS = 5 # the text vetoes are re-read until no decision changes + + +def _norm(text: str) -> str: + return re.sub(r"[^a-z0-9']+", " ", text.replace("’", "'").replace("‘", "'").lower()).strip() + + +def _canon(text: str) -> str: + return re.sub(r"\s+", " ", text.strip().upper().rstrip(",;")) + + +def _is_body(text: str) -> bool: + return bool(re.search(r"[a-z]", text)) + + +def _is_paren(text: str) -> bool: + t = text.strip() + return t.startswith("(") and not _is_body(t) + + +def _is_prose_tail(text: str) -> bool: + """The last line of a paragraph can carry no lowercase at all: ``2028.``, ``$324,386,000.``.""" + t = text.strip() + return bool(re.match(r"^[\s\d$,.;:()]+$", t)) or (t.endswith((".", ";", ":")) and not t.endswith(".—")) + + +def _is_heading(text: str) -> bool: + t = text.strip() + return ( + len(t) > 3 + and not _is_body(t) + and not t.startswith("(") + and "$" not in t + and not _LEADING_TOKEN.match(t) + and not _is_prose_tail(t) + ) + + +def _starts_quote(text: str) -> bool: + return text.strip().startswith(_QUOTE_OPEN) + + +def _is_boundary_before(text: str) -> bool: + return _is_body(text) or _is_prose_tail(text) or bool(re.match(r"^\s*(TITLE|DIVISION)\b", text.strip())) + + +def _join(parts: list[str]) -> str: + out = "" + for part in parts: + p = part.strip() + if out.endswith("-") and not out.endswith("—"): + out = out[:-1] + p + else: + out = f"{out} {p}" if out else p + return out + + +class _Stream: + """The reading-order line stream and the per-line facts every pass reads.""" + + def __init__(self, pages: list[Page]): + self.lines: list[tuple[int, Line]] = [ + (pg.page_number, ln) + for pg in pages + for ln in pg.lines + if ln.text.strip() and not (ln.line_number is None and _FURNITURE.match(ln.text)) + ] + self.index: dict[tuple[int, int | None], int] = {} + for k, (p, ln) in enumerate(self.lines): + self.index.setdefault((p, ln.line_number), k) + self.edges: dict[int, tuple[float, float]] = {} + self.body_size: dict[int, float] = {} + for pg in pages: + body = [ln for ln in pg.lines if ln.geom is not None and _is_body(ln.text)] + if body: + self.edges[pg.page_number] = ( + min(ln.geom.content_left for ln in body), + max(ln.geom.content_right for ln in body), + ) + sizes = Counter(ln.glyph_size for ln in pg.lines if ln.glyph_size is not None and _is_body(ln.text)) + if sizes: + self.body_size[pg.page_number] = sizes.most_common(1)[0][0] + self.in_quote = self._quote_state() + self.mid_sentence = self._mid_sentence() + + def _quote_state(self) -> set[int]: + """Lines that are quoted text: they open a quote, or a quote opened above is still open. + An unquoted SEC./TITLE/DIVISION line closes any open quote, so a closing mark lost in + the extracted text cannot silence the rest of the bill.""" + quoted: set[int] = set() + is_open = False + for k, (_, ln) in enumerate(self.lines): + t = ln.text.strip() + if _LEADING_TOKEN.match(t): + is_open = False + if is_open or _starts_quote(t): + quoted.add(k) + for mark in _QUOTE_MARK.findall(t): + is_open = mark in _QUOTE_OPEN + return quoted + + def is_centered(self, k: int) -> bool: + """The line is inset on both sides and its midpoint sits on the column's.""" + page, line = self.lines[k] + g = line.geom + if g is None or page not in self.edges: + return False + left, right = self.edges[page] + return ( + g.content_left - left >= _INSET_MIN + and right - g.content_right >= _INSET_MIN + and abs((g.content_left + g.content_right) / 2 - (left + right) / 2) <= _CENTER_TOL + ) + + def set_apart(self, k: int) -> bool: + """The line is centered, and the run of centered lines it belongs to is not followed by the + continuation of a sentence: what comes next is a new, indented paragraph, a line that is not + prose, or nothing. A continuation line starts at the column's left edge.""" + if not self.is_centered(k): + return False + j = k + 1 + while j < len(self.lines) and self.is_centered(j): + j += 1 + if j == len(self.lines): + return True + page, line = self.lines[j] + if not _is_body(line.text) or line.geom is None or page not in self.edges: + return True + return line.geom.content_left - self.edges[page][0] >= _PARA_INDENT_MIN + + def _mid_sentence(self) -> set[int]: + """Lines that continue an unfinished sentence: the line above is body text (or itself in a + sentence) and does not end one. A heading never starts mid-sentence. + + A line set apart from the prose (``set_apart``) is never inside a sentence and ends any + sentence carried down to it, so an unpunctuated paragraph end (``(Public Law 111–241)``) or + an enacting clause interrupted by ``TITLE I`` does not hide the headings printed after it. + Heading-shaped text inside prose is either full-width or followed by the sentence's own + continuation, so it stays inside.""" + inside: set[int] = set() + for k in range(1, len(self.lines)): + if self.set_apart(k): + continue + prev = self.lines[k - 1][1].text.rstrip() + if (_is_body(prev) or (k - 1) in inside) and (prev.endswith("-") or not prev.endswith(_SENTENCE_END)): + inside.add(k) + return inside + + def can_be_heading(self, k: int) -> bool: + line = self.lines[k][1] + return ( + line.line_number is not None # an anchor needs a printed line number + and k not in self.mid_sentence + and k not in self.in_quote + and _is_heading(line.text) + ) + + def boundary_before(self, k: int) -> int | None: + """Index of the line that closes off a heading run starting at ``k`` from above, skipping + parenthetical qualifier lines, or None.""" + b = k - 1 + while b >= 0 and _is_paren(self.lines[b][1].text): + b -= 1 + return b if b >= 0 and _is_boundary_before(self.lines[b][1].text) else None + + def body_after(self, k: int) -> bool: + a = k + 1 + while a < len(self.lines) and _is_paren(self.lines[a][1].text): + a += 1 + return a < len(self.lines) and _is_body(self.lines[a][1].text) + + +def _heading_measure(stream: _Stream) -> dict[int, float]: + """Width a centered heading can fill, per page: the body column less the smallest inset any + centered heading line in this document keeps. Fullness is judged against this width, not the + body width, or a wrapped name looks like a deliberate break.""" + insets = [] + for p, ln in stream.lines: + g = ln.geom + if g is None or p not in stream.edges or not _is_heading(ln.text): + continue + left, right = stream.edges[p] + li, ri = g.content_left - left, right - g.content_right + if li >= _INSET_MIN and ri >= _INSET_MIN and abs(li - ri) <= _CENTER_TOL: + insets.append(min(li, ri)) + inset = min(insets) if insets else 0.0 + return {p: (right - left) - 2 * inset for p, (left, right) in stream.edges.items()} + + +def _lone_headings(stream: _Stream) -> Counter: + """How often each multi-word line stands alone as a heading between body lines. One lone + appearance is weak evidence (it can be a wrap tail whose top line was lost); two is kept.""" + lone: Counter = Counter() + for k in range(1, len(stream.lines) - 1): + t = stream.lines[k][1].text + if ( + stream.can_be_heading(k) + and _is_boundary_before(stream.lines[k - 1][1].text) + and _is_body(stream.lines[k + 1][1].text) + ): + lone[_norm(t)] += 1 + return Counter({name: n for name, n in lone.items() if n >= 2 and len(name.split()) > 1}) + + +def _introduces_accounts(stream: _Stream, end: int) -> bool: + """The text under the heading run ending at ``end`` carries no dollar amount, and the next + heading prints even (account style): the heading introduces the accounts below it, as an + agency's authority paragraph does, instead of holding money itself as an account does.""" + j = end + 1 + while j < len(stream.lines) and not stream.can_be_heading(j): + if "$" in stream.lines[j][1].text: + return False + j += 1 + if j == len(stream.lines): + return False + g = stream.lines[j][1].geom + return g is not None and g.initial_caps is False + + +def _printed_alike(a: Line, b: Line) -> bool: + """Two heading lines set in the same case pattern and glyph size.""" + return ( + a.geom is not None + and b.geom is not None + and a.geom.initial_caps is not None + and a.geom.initial_caps == b.geom.initial_caps + and a.glyph_size is not None + and b.glyph_size is not None + and abs(a.glyph_size - b.glyph_size) <= _BODY_SIZE_TOL + ) + + +def _decided_by_grammar(upper: str) -> bool | None: + """True join, None no opinion. A heading line that ends in a small word ("…DECONTAMINATION + AND", "GRANTS FOR CONSTRUCTION OF") always continues onto the next line: grammar, not + appropriations vocabulary (ADR 0018).""" + words = _canon(upper).split() + if words and words[-1] in TITLE_CASE_SMALL_WORDS: + return True + return None + + +def _segment( + run: list[tuple[int, Line]], anchor_starts: set[tuple[int, int | None]], lone: Counter, measure: dict[int, float] +) -> list[list[int]]: + """Decide every line break inside one heading run; return the groups of line indexes.""" + n = len(run) + caps = [ln.geom.initial_caps if ln.geom is not None else None for _, ln in run] + decisions: list[bool | None] = [None] * (n - 1) + for _ in range(_MAX_ROUNDS): + previous = list(decisions) + for b in range(n - 1): + (p_up, up_line), (p_lo, lo_line) = run[b], run[b + 1] + up, lo = up_line.text.strip(), lo_line.text.strip() + grammar = _decided_by_grammar(up) + if up.endswith("-") and not up.endswith("—"): + join = True + elif grammar is not None: + join = grammar + elif caps[b] is None or caps[b + 1] is None: + # Fail closed: without the case pattern, keep the detectors' split. Why not defer + # to the detectors' joins everywhere: their joins are where most glued stacks come + # from, and measured worse against the XML twin (ADR 0022). + join = (p_lo, lo_line.line_number) not in anchor_starts + elif caps[b] != caps[b + 1]: + join = False + else: + join = True + if lone[_norm(lo)]: + join = False # a heading in its own right elsewhere in this bill + elif up_line.geom is not None and lo_line.geom is not None and p_up in measure: + g1, g2 = up_line.geom, lo_line.geom + width = ( + (g1.content_right - g1.content_left) + _SPACE_WIDTH + (g2.first_word_right - g2.content_left) + ) + if width <= measure[p_up]: + join = False # the next word would have fitted: the break was deliberate + decisions[b] = join + if decisions == previous: + break + groups, current = [], [0] + for b, join in enumerate(decisions): + if join: + current.append(b + 1) + else: + groups.append(current) + current = [b + 1] + groups.append(current) + return groups + + +def _merge_hanging( + run: list[tuple[int, Line]], groups: list[list[int]], stream: _Stream, para_left: float | None +) -> list[list[int]]: + """A hanging-indent block is one heading: a line starting at the left margin and running full + width, then lines starting exactly where the following paragraph starts. Overrides the + decisions inside the block.""" + if para_left is None: + return groups + owner = {x: gi for gi, g in enumerate(groups) for x in g} + changed = False + k = 0 + while k < len(run) - 1: + page, line = run[k] + g0, edges = line.geom, stream.edges.get(page) + if g0 is None or edges is None: + k += 1 + continue + left, right = edges + if abs(g0.content_left - left) <= _HANG_TOL and abs(g0.content_right - right) <= 4 * _HANG_TOL: + m = k + 1 + while m < len(run) and run[m][0] == page and run[m][1].geom is not None: + if abs(run[m][1].geom.content_left - para_left) > _HANG_TOL: + break + m += 1 + if m - k >= 2: + for x in range(k + 1, m): + changed |= owner[x] != owner[k] + owner[x] = owner[k] + k = m + continue + k += 1 + if not changed: + return groups + merged, current = [], [0] + for x in range(1, len(run)): + if owner[x] == owner[x - 1]: + current.append(x) + else: + merged.append(current) + current = [x] + merged.append(current) + return merged + + +def _is_department_style(stream: _Stream, k: int) -> bool: + """A centered line whose letters are all at body size (capitals, not small caps).""" + page, line = stream.lines[k] + g = line.geom + if g is None or page not in stream.body_size or g.size_min is None or not stream.is_centered(k): + return False + body = stream.body_size[page] + return abs(g.size_min - body) <= _BODY_SIZE_TOL and abs(g.size_max - body) <= _BODY_SIZE_TOL + + +def converge_headings(pages: list[Page], anchors: list[Anchor]) -> list[Anchor]: + """Return ``anchors`` with heading structure recovered from the reading order (ADR 0022). + + ``anchors`` are the detectors' output in document order. Title, section, subsection and + preamble anchors are kept as they are; account, agency, major and grouping anchors may be + dropped, joined, split or added. Every heading anchor gets ``caps`` set from its first printed + line, for the scope rules in ``pdf_anchors._breadcrumb_core``. + """ + if not anchors: + return anchors + stream = _Stream(pages) + if not stream.lines: + return anchors + + def pos(a: Anchor) -> int | None: + return stream.index.get((a.page_number, a.line_number)) + + # Pass 2: heading anchors on lines that cannot be headings. + kept: list[Anchor] = [] + for i, a in enumerate(anchors): + if a.kind in _HEADING_KINDS: + k = pos(a) + t = a.text.strip() + prev = kept[-1] if kept else None + if t.endswith((".—", ".–")) or ( + prev is not None + and prev.kind in ("subsection", "section") + and _norm(prev.text).endswith(_norm(t.rstrip(".—– "))) + ): + continue # the continuation of a wrapped section or subsection title + if k is not None and (k in stream.in_quote or k in stream.mid_sentence): + continue + kept.append(a) + anchors = kept + if not anchors: + return anchors + + # Pass 3: department-level headings anywhere in a title. + taken = {(a.page_number, a.line_number) for a in anchors} + added = [] + for k, (page, line) in enumerate(stream.lines): + if (page, line.line_number) in taken or not stream.can_be_heading(k) or not _is_department_style(stream, k): + continue + b = stream.boundary_before(k) + if b is None or _TITLE_BARE.match(stream.lines[b][1].text): + continue # directly under TITLE n is the major detector's own case + if _STRUCTURAL_TOKEN.match(line.text) or _STRUCTURAL_TOKEN.match(stream.lines[b][1].text): + continue + added.append((k, _anchor_like(anchors, page, line.line_number, "major", line.text.strip()))) + if added: + anchors = _merge_in_order(anchors, added, pos) + + # Pass 4: re-segment every heading run between body text. + measure = _heading_measure(stream) + lone = _lone_headings(stream) + by_key = {(a.page_number, a.line_number): i for i, a in enumerate(anchors)} + replace_at: dict[int, list[Anchor]] = {} + k = 0 + while k < len(stream.lines): + if not stream.can_be_heading(k): + k += 1 + continue + end = k + while end + 1 < len(stream.lines) and stream.can_be_heading(end + 1): + end += 1 + start, k = k, end + 1 + run = stream.lines[start : end + 1] + b = stream.boundary_before(start) + if b is None or not stream.body_after(end): + continue + above = stream.lines[b][1].text.strip() + if _STRUCTURAL_TOKEN.match(above) and not _TITLE_BARE.match(above): + # "TITLE VIII—COUNTERING THE MALIGN INFLU-" / "ENCE OF … OF" / "CHINA": lines that run on + # from a numbered level's own line, and are not centered, finish that line's name. + lead = 0 + while lead < len(run) and not stream.is_centered(start + lead): + lead += 1 + run, start = run[lead:], start + lead + if not run: + continue + idx = sorted(by_key[(p, ln.line_number)] for p, ln in run if (p, ln.line_number) in by_key) + if not idx or any(anchors[i].kind not in _HEADING_KINDS for i in idx) or any(i in replace_at for i in idx): + continue + if _TITLE_BARE.match(stream.lines[b][1].text) and anchors[idx[0]].kind == "major": + # The line(s) right under TITLE n are the parser's department heading (the major + # detector's own case, pdf_anchors._major_anchors_by_size). Keep it as detected and + # re-segment only the heading lines after it. Why not attach it to the title's name + # (`TITLE IV—`): that re-decides what the major detector decided (#105) and + # changes title text other code keys on (ADR 0022). + covered = _lines_covered(run, anchors[idx[0]].text) + run = run[covered:] + idx = idx[1:] + if not run or not idx: + continue + starts = {(anchors[i].page_number, anchors[i].line_number): anchors[i] for i in idx} + groups = _segment(run, set(starts), lone, measure) + a_ = end + 1 + while a_ < len(stream.lines) and _is_paren(stream.lines[a_][1].text): + a_ += 1 + para = stream.lines[a_][1].geom if a_ < len(stream.lines) else None + groups = _merge_hanging(run, groups, stream, para.content_left if para is not None else None) + last_kind = anchors[idx[-1]].kind + rebuilt = [] + for gi, g in enumerate(groups): + page, first = run[g[0]] + original = starts.get((page, first.line_number)) + if gi == len(groups) - 1: + kind = last_kind + elif original is not None and original.kind in _CONTAINER_KINDS: + kind = original.kind + elif _printed_alike(first, run[groups[-1][0]][1]): + # Printed like the run's last heading (same case pattern and size): the same level. + # GPO stacks a name over its own account (FOOD AND DRUG ADMINISTRATION / SALARIES + # AND EXPENSES) as two same-level headings; read as an agency, the upper one would + # end the reach of the real agency above it. + kind = last_kind + else: + kind = "agency" + rebuilt.append(_anchor_like(anchors, page, first.line_number, kind, _join([run[x][1].text for x in g]))) + # The run's last heading is judged whole, after joining: printed agency style (title case) + # and introducing accounts rather than holding money, it is an agency (ADR 0012, boundary 1). + last, lead = rebuilt[-1], run[groups[-1][0]][1] + if last.kind == "account" and lead.geom is not None and lead.geom.initial_caps is True: + if _introduces_accounts(stream, end): + rebuilt[-1] = replace(last, kind="agency") + replace_at[idx[0]] = rebuilt + for i in idx[1:]: + replace_at[i] = [] + + out: list[Anchor] = [] + for i, a in enumerate(anchors): + out.extend(replace_at.get(i, [a])) + + # Record each heading anchor's case pattern for the scope rules. + caps_at = {(p, ln.line_number): (ln.geom.initial_caps if ln.geom is not None else None) for p, ln in stream.lines} + return [ + replace(a, caps=caps_at.get((a.page_number, a.line_number))) if a.kind in _HEADING_KINDS else a for a in out + ] + + +def _lines_covered(run: list[tuple[int, Line]], text: str) -> int: + """How many leading lines of ``run`` make up ``text`` (a heading the detector joined).""" + target = _norm(text) + for n in range(1, len(run) + 1): + if _norm(_join([ln.text for _, ln in run[:n]])) == target: + return n + return 1 + + +def _anchor_like(anchors: list[Anchor], page: int, line_number: int | None, kind: str, text: str) -> Anchor: + """A new anchor of the same type as the detectors emit (avoids importing Anchor at runtime, + which would make this module and pdf_anchors import each other).""" + return replace(anchors[0], page_number=page, line_number=line_number, kind=kind, text=text, division="", caps=None) + + +def _merge_in_order(anchors: list[Anchor], added: list[tuple[int, Anchor]], pos) -> list[Anchor]: + """Insert ``added`` (stream index, anchor) into ``anchors`` by reading order.""" + out, j = [], 0 + for a in anchors: + ka = pos(a) + while j < len(added) and ka is not None and added[j][0] < ka: + out.append(added[j][1]) + j += 1 + out.append(a) + out.extend(a for _, a in added[j:]) + return out diff --git a/src/deltatrack/parsers/pdf_text.py b/src/deltatrack/parsers/pdf_text.py index f85e3d43..df2f6fab 100644 --- a/src/deltatrack/parsers/pdf_text.py +++ b/src/deltatrack/parsers/pdf_text.py @@ -89,6 +89,17 @@ class LineGeom: content_left: float content_right: float first_word_right: float + # Per-letter typography of the line (ADR 0022), read from the same content glyphs, so + # still no extra PDFium calls. `initial_caps` is True when most content words print + # their first letter larger than the rest (a title-cased heading in small caps, the + # agency style), False when the letters are even (an all-small-caps account heading, + # or full capitals), None when the line has no word to judge or is prose (has lowercase + # letters). `size_min`/`size_max` bound the letter sizes, so a line set entirely at body + # size in capitals (the department style) is recognizable. The line MEDIAN in + # `Line.glyph_size` hides both signals, which is why they are carried separately. + initial_caps: bool | None = None + size_min: float | None = None + size_max: float | None = None @dataclass(frozen=True) @@ -367,7 +378,9 @@ def _first_word_right(content_glyphs: list[tuple[float, float, float, int, float `(bottom, left, right, cp, size)` tuples in left-to-right x order. A word boundary is the first SPACE GLYPH (cp == 32) or, as a fallback, an x-gap - wider than SPACE_FACTOR × size. The space-glyph test is load-bearing: PDFium + wider than `_WORD_GAP` × size (the case pattern's word gap; a heading line usually has no + space glyph, and the text layer's wider `_SPACE_FACTOR` misses 0.5% of its word gaps, which + then runs the first word into the second). The space-glyph test is load-bearing: PDFium emits real space glyphs that sit IN the inter-word gap, so the gap between the last glyph of word one and the first of word two is bridged — a gap-only test never fires and silently returns the whole line as one word (this bit the #106 @@ -380,13 +393,79 @@ def _first_word_right(content_glyphs: list[tuple[float, float, float, int, float if first_word_right is None: continue # leading space before any word; skip it break - if prev_right is not None and left - prev_right > _SPACE_FACTOR * size: + if prev_right is not None and left - prev_right > _WORD_GAP * size: break # fallback: wide x-gap with no emitted space glyph first_word_right = right prev_right = right return first_word_right +# Words a title-cased heading leaves in small letters ("AGENCY FOR HEALTHCARE RESEARCH AND +# QUALITY" prints "AND" without a large initial). Typographic convention, not appropriations +# vocabulary: they are skipped when judging whether a line's words start with a large capital. +TITLE_CASE_SMALL_WORDS = frozenset({"AND", "OF", "THE", "FOR", "TO", "IN", "ON", "AT", "BY", "OR", "A", "AN"}) +# A first letter this much larger (points) than the smallest of the rest counts as a large +# initial. Small caps print at about 0.8 of the capital size (11.2 vs 14.0 in working prints). +_INITIAL_CAP_MARGIN = 0.5 +# A horizontal gap wider than this share of the glyph size separates two words. This walk reads the +# raw character data, which has only the spaces the file draws; the simplified page text fills a +# space in from each gap, but those generated spaces carry a size of 1.0 that does not clear +# _SIZE_FLOOR, and heading lines mostly draw none, so a space alone cannot find the word breaks. +# Measured on every page of the committed corpus (29,687 all-capital lines): 99.996% of the gaps +# inside a word are under 0.2 of the size, and 99.94% of the gaps where PDFium's text puts a space +# are over it. +_WORD_GAP = 0.2 + + +def _initial_caps(content_glyphs: list[tuple[float, float, float, int, float]]) -> bool | None: + """True when most content words print a first letter larger than the rest (title case in + small caps), False when their letters are even, None when no word of 2+ letters remains + after skipping `TITLE_CASE_SMALL_WORDS`. Glyphs are x-ordered (bottom, left, right, cp, size). + + Words are split at a space glyph or at a gap wider than `_WORD_GAP` of the glyph size. A word + printed entirely in large capitals (an acronym such as ``FDA``, on a line that also carries + small caps) has no case pattern of its own and is left out of the vote, like a small word. + + A line carrying a lowercase letter is prose, not a heading in small caps (small caps reach the + text layer as capitals), so it is not judged: None. Most lines are prose, and the check stops + at the first lowercase letter, which keeps this off the extraction's hot path.""" + if any(chr(glyph[3]).islower() for glyph in content_glyphs): + return None + words: list[list[tuple[str, float]]] = [] + current: list[tuple[str, float]] = [] + prev = None + for glyph in content_glyphs: + ch = chr(glyph[3]) + if ch.isspace(): + if current: + words.append(current) + current, prev = [], None + continue + if prev is not None and current and glyph[1] - prev[2] > _WORD_GAP * max(glyph[4], prev[4]): + words.append(current) + current = [] + if ch.isalpha(): + current.append((ch, glyph[4])) + prev = glyph + if current: + words.append(current) + letters = [s for w in words for _, s in w] + if not letters: + return None + small = min(letters) + judged = [ + w + for w in words + if len(w) > 1 + and "".join(c for c, _ in w).upper() not in TITLE_CASE_SMALL_WORDS + and not min(s for _, s in w) > small + _INITIAL_CAP_MARGIN + ] + if not judged: + return None + large = sum(1 for w in judged if w[0][1] > min(s for _, s in w[1:]) + _INITIAL_CAP_MARGIN) + return large / len(judged) >= 0.5 + + def _page_glyph_sizes(textpage, page_text: str) -> dict[int, tuple[float, LineGeom]]: """Map GPO margin line number → `(glyph size, horizontal extent)` for one page. @@ -486,7 +565,15 @@ def _page_glyph_sizes(textpage, page_text: str) -> dict[int, tuple[float, LineGe # finds it ⇒ never None here (it only returns None on no content glyphs at all). first_word_right = _first_word_right(content_glyphs) assert first_word_right is not None - geom = LineGeom(content_left, content_right, first_word_right) + letters = [c[4] for c in content_glyphs if chr(c[3]).isalpha()] + geom = LineGeom( + content_left, + content_right, + first_word_right, + initial_caps=_initial_caps(content_glyphs), + size_min=round(min(letters), 1) if letters else None, + size_max=round(max(letters), 1) if letters else None, + ) if line_number in sizes or line_number in ambiguous: ambiguous.add(line_number) sizes.pop(line_number, None) diff --git a/tests/data/canonical_baseline.json b/tests/data/canonical_baseline.json index e3e4043f..c4ca8045 100644 --- a/tests/data/canonical_baseline.json +++ b/tests/data/canonical_baseline.json @@ -1,8 +1,8 @@ { "113-hr-3547/1_introduced-in-house->2_engrossed-in-house": { - "bytes": 2999, + "bytes": 3114, "changes": 1, - "sha256": "dc6061cdbef8209e39f1cfce20881714ab1094eac136cce2a15e9dff763b67b4", + "sha256": "c4edbca4f6dc750242b231925f5fe9003c07ccca7f4020827354db2418532864", "summary": { "added": 0, "modified": 1, @@ -12,9 +12,9 @@ } }, "113-hr-3547/2_engrossed-in-house->3_received-in-senate": { - "bytes": 2658, + "bytes": 2773, "changes": 0, - "sha256": "4c0b8993fb1d06102b2a74c7d29c782372250bed8ab55535d05c08d073511bbf", + "sha256": "e1fadf3ae87d309bb2843e2fcf17224fab8b8e098cbf1dc339c3f6a7b055d569", "summary": { "added": 0, "modified": 0, @@ -24,9 +24,9 @@ } }, "113-hr-3547/3_received-in-senate->4_engrossed-amendment-senate": { - "bytes": 3650, + "bytes": 3765, "changes": 5, - "sha256": "dc30bd639254a05f2269c083fc9a656e3748b4b7c9c9ccf16f7a779c9b0f6f11", + "sha256": "76d9bdaa6cd04a4bcdac7a3e57b78240731c9e5cddbf007a1422d9e20d5a5f80", "summary": { "added": 0, "modified": 0, @@ -36,9 +36,9 @@ } }, "113-hr-3547/4_engrossed-amendment-senate->5_engrossed-amendment-house": { - "bytes": 5355122, + "bytes": 6331321, "changes": 2811, - "sha256": "37dc77993746fc80d8035b580bae055d900f248d0a8e145ae0b8108bf1b308e5", + "sha256": "6d1ca00a8aff9b65ff4b1437078f5e0f3d90ca0dd40d3ee1c74341dabaf5fa65", "summary": { "added": 2810, "modified": 0, @@ -48,9 +48,9 @@ } }, "113-hr-3547/5_engrossed-amendment-house->6_enrolled-bill": { - "bytes": 4763664, + "bytes": 6715950, "changes": 8, - "sha256": "e0197da6e3c3685da2bf6152a1c126f5365f85fb3d8090c9384a5a971529dbdd", + "sha256": "9fecaf1f19b0151295e2bc2583e488c3b7d9badeb959fd9dc9d04810bd93c9b0", "summary": { "added": 3, "modified": 5, @@ -60,9 +60,9 @@ } }, "113-hr-83/6_engrossed-amendment-house->7_enrolled-bill": { - "bytes": 5083801, + "bytes": 6972218, "changes": 9, - "sha256": "cb1af5947c2c6b2aa450400f9e065abbef56db89fc64d5716462695387ca921c", + "sha256": "5cc56776172c48ad13bf2e8eab7c72b382f73ded5a79e1767c5099465f0a551f", "summary": { "added": 3, "modified": 6, @@ -72,9 +72,9 @@ } }, "114-hr-2029/1_reported-in-house->3_referred-in-senate": { - "bytes": 249998, + "bytes": 330502, "changes": 23, - "sha256": "e4893da02b5ef7dfa7f6f8790676c33ca85de190c034b89a719e66ec8fdae206", + "sha256": "f0bd3d17f0829d074c3339db2d2b5c2cea0fe7e2ecca762d59cf03962988ba9d", "summary": { "added": 15, "modified": 8, @@ -84,9 +84,9 @@ } }, "114-hr-2029/3_referred-in-senate->4_reported-in-senate": { - "bytes": 466240, + "bytes": 589930, "changes": 154, - "sha256": "581aaec0c9c9efe42ef9b8fee85e71a7fd0cdd0f06955e52e9592d8da7067a05", + "sha256": "4801190d6a381abe79dd5fb1958062151aa188db82a384ef2e82af6d87e08f90", "summary": { "added": 150, "modified": 4, @@ -96,9 +96,9 @@ } }, "114-hr-2029/4_reported-in-senate->5_engrossed-amendment-senate": { - "bytes": 522125, + "bytes": 647925, "changes": 207, - "sha256": "9e4a0f8d0427b8d55e8ae4cadc730cd36a324e6bb1e18838e293a821df58836a", + "sha256": "deb952abbbb6cff5f189670b25ff7ce398c40b6b053d8d6267515552765eb67c", "summary": { "added": 35, "modified": 4, @@ -108,9 +108,9 @@ } }, "114-hr-2029/5_engrossed-amendment-senate->6_engrossed-amendment-house": { - "bytes": 6685416, + "bytes": 7740831, "changes": 3285, - "sha256": "93de9fd2271175e6393cd98e8eef83a812005290b2ff02da1bb6d118f01b0f34", + "sha256": "fff771487a60d83ba7d8473c905fbe77c36a08b45d0dd10cc0f7198f099dddcb", "summary": { "added": 3152, "modified": 39, @@ -120,9 +120,9 @@ } }, "114-hr-2029/6_engrossed-amendment-house->7_enrolled-bill": { - "bytes": 6660421, + "bytes": 8693480, "changes": 543, - "sha256": "6d2fbddee9c61f9400269897134d7177ae93057eddfc1e1e1d61c42b1a62a6c4", + "sha256": "44e11b71b857aa2e13d2055fdd7d3909c8a661200f3a671b3677d916dda868cf", "summary": { "added": 470, "modified": 73, @@ -132,9 +132,9 @@ } }, "115-hr-5895/1_reported-in-house->2_engrossed-in-house": { - "bytes": 690452, + "bytes": 861552, "changes": 302, - "sha256": "3092d722a670540b9b546c5c8953ebd8fd04541f6615cfb8d1503969e92e32a8", + "sha256": "53f13ffc2586f3bb6bb1c7c65af4837a9234fe7193d6b1418d6487a4d46aaf4c", "summary": { "added": 281, "modified": 20, @@ -144,9 +144,9 @@ } }, "115-hr-5895/2_engrossed-in-house->4_engrossed-amendment-senate": { - "bytes": 1165063, + "bytes": 1438680, "changes": 444, - "sha256": "9d8ebbb6ccbc5b1743143c4fd8abe3333dcfba13640b72b038b1c570dadfbbc2", + "sha256": "f80d2372bd169df69360fca7b8cef7c5e22293bae7aa7026d22fcea06ce3c2d2", "summary": { "added": 165, "modified": 96, @@ -156,9 +156,9 @@ } }, "115-hr-5895/4_engrossed-amendment-senate->5_enrolled-bill": { - "bytes": 1034798, + "bytes": 1327290, "changes": 304, - "sha256": "5bd4604d51dd677218ad7894ccb4b35e1631bcb3a55c2e58708001a7e7698c30", + "sha256": "acea73e434188592eeba1bf2f593e833cd2785e0fb6d1257957239926b4fcfba", "summary": { "added": 68, "modified": 96, @@ -168,9 +168,9 @@ } }, "117-hr-2471/1_introduced-in-house->6_enrolled-bill": { - "bytes": 8448235, + "bytes": 9571464, "changes": 4015, - "sha256": "c0d784540c70d9c16d31376acd8ac978790a5b84675de2bfcbbeb1e6c8af4e04", + "sha256": "3553d3d7bd0b87597b26fec76b2437541008969a43475e08721f84d4bf94aea1", "summary": { "added": 3997, "modified": 1, @@ -180,9 +180,9 @@ } }, "117-hr-4502/1_reported-in-house->2_engrossed-in-house": { - "bytes": 3626529, + "bytes": 4432426, "changes": 1415, - "sha256": "2fbdb9da81b8df9eac6a573fe4e9ec2a23013245e378e4ba7ba59fcc63973cfc", + "sha256": "c04f740898fd51f89fdf99b5401f2e0d86325c6dee3320e75fec5c430f7b1191", "summary": { "added": 1366, "modified": 48, @@ -192,9 +192,9 @@ } }, "118-hr-2882/1_introduced-in-house->4_engrossed-amendment-senate": { - "bytes": 6757, + "bytes": 6872, "changes": 6, - "sha256": "41811ed7c0bb75e4dcb537055163a115beb1156712a03d2d63d4556145c8f833", + "sha256": "e245e589fef8e145b96ce02830b98eddabd304a5cf0832b3c4d7c068311a4aa3", "summary": { "added": 2, "modified": 0, @@ -204,9 +204,9 @@ } }, "118-hr-2882/4_engrossed-amendment-senate->5_engrossed-amendment-house": { - "bytes": 3401926, + "bytes": 3965242, "changes": 1842, - "sha256": "4cdc5a3362da50305dcede57e6ded3506dd2cc73d4aace6e67e629428d137f36", + "sha256": "781bd093bf19afc205703b2b530bf0541a4343332b66e8acd03f5b4a8e19dd98", "summary": { "added": 1838, "modified": 1, @@ -216,9 +216,9 @@ } }, "118-hr-4366/1_reported-in-house->2_engrossed-in-house": { - "bytes": 326090, + "bytes": 413205, "changes": 25, - "sha256": "b812e00b4effb4fa8a5e0c50d7932292ded6508c8a7473b37798bbd9017d85db", + "sha256": "6f0d24430fd29f44c831a4a1ca796778eb0c69fffec2e63c7af70c71d08fc05d", "summary": { "added": 7, "modified": 17, @@ -228,9 +228,9 @@ } }, "118-hr-4366/2_engrossed-in-house->3_placed-on-calendar-senate": { - "bytes": 288433, + "bytes": 379012, "changes": 0, - "sha256": "cfbcd1f79c544ea5a37e0c71a2fed0b83c9c1bba79a735c2b243920b6e566637", + "sha256": "4dc8b19ed7236c071e3e4b57db93dc6fe8c10cb3701309b08e28d55f8d582444", "summary": { "added": 0, "modified": 0, @@ -240,9 +240,9 @@ } }, "118-hr-4366/3_placed-on-calendar-senate->4_engrossed-amendment-senate": { - "bytes": 1766251, + "bytes": 2084296, "changes": 703, - "sha256": "d5fe8c601f2b88e5433ef532516ccd0f5e33a68b05373dba0ec2d45860675a06", + "sha256": "551aae0427fa0b7c12d869ad7d364c143b92ae393008a637bc1abe764a8dc32d", "summary": { "added": 561, "modified": 48, @@ -252,9 +252,9 @@ } }, "118-hr-4366/4_engrossed-amendment-senate->5_engrossed-amendment-house": { - "bytes": 4174842, + "bytes": 4989046, "changes": 1199, - "sha256": "9bbae928be77fb2b5d0039b9d7a2add4462feb76de92b402d68f91592f3117e2", + "sha256": "2c17d47a7f99827808ec7a2abce12d997752db2b123aa140f6f1e6a2c2a8e742", "summary": { "added": 823, "modified": 187, @@ -264,9 +264,9 @@ } }, "118-hr-4366/5_engrossed-amendment-house->6_enrolled-bill": { - "bytes": 2981872, + "bytes": 4064784, "changes": 3, - "sha256": "89571991f6fca063b77958e2e00a182d058f5b9934f275521767408865ca933a", + "sha256": "6503c08c31a20ad8e6782e5948376fec5dc06fb79c7afa9ba8c0801e3a9d2edd", "summary": { "added": 3, "modified": 0, @@ -276,9 +276,9 @@ } }, "118-hr-8752/1_reported-in-house->2_engrossed-in-house": { - "bytes": 381625, + "bytes": 466004, "changes": 39, - "sha256": "1c4d2f720a7b32b39bbfe046ca06a55b61499a92c0c02ebcc5c21d0bddadcd6a", + "sha256": "385723f97eb545fe460abcabf315f160e3fc4dc97b5d6b8dcb8e40681f8310cc", "summary": { "added": 18, "modified": 18, @@ -288,9 +288,9 @@ } }, "118-hr-8774/1_reported-in-house->2_engrossed-in-house": { - "bytes": 527725, + "bytes": 664840, "changes": 31, - "sha256": "25119e72b08cd6e0ba1f5927b767a3c5e6c66f2f1a160768b0609686c427e715", + "sha256": "250472f79a6c625eb51e0f60c406e29c7f6fbe1267d6852ab371279011da6071", "summary": { "added": 7, "modified": 21, @@ -300,9 +300,9 @@ } }, "118-hr-9468/1_introduced-in-house->4_enrolled-bill": { - "bytes": 15778, + "bytes": 17527, "changes": 2, - "sha256": "1062758a01c2a717fbcbe803d69d377bf7b7ea2fc695a13e48a2feaa3b6f7702", + "sha256": "3a58f276667ea5f31eea2a9b08fe85d58a06f2893ded68a5f5cb5a70ef9cc236", "summary": { "added": 1, "modified": 0, @@ -312,9 +312,9 @@ } }, "119-hr-1/1_reported-in-house->2_engrossed-in-house": { - "bytes": 3967064, + "bytes": 4235309, "changes": 497, - "sha256": "16d93c61446dbb6a7f4141ad9646b7e0be84f640c64e89b3a33074d15ae61359", + "sha256": "841ed4d5af35d1a62744dfc575897b8d36c106bccd7b8f3b25c6aae2872764d1", "summary": { "added": 50, "modified": 148, diff --git a/tests/data/financial_rows/118-hr-4366-1_reported-in-house.json b/tests/data/financial_rows/118-hr-4366-1_reported-in-house.json new file mode 100644 index 00000000..92344399 --- /dev/null +++ b/tests/data/financial_rows/118-hr-4366-1_reported-in-house.json @@ -0,0 +1,254 @@ +{"bill": "118-hr-4366", "version": "1_reported-in-house", "classifier": "1.0", "sections": [ + [ + ["primary", "appropriation", 1517455000, false], + ["sub", "cap", 345775000, false], + ["sub", "earmark", 162900000, false] + ], + [ + ["primary", "appropriation", 4477961000, false], + ["sub", "cap", 602625000, false], + ["sub", "earmark", 52683000, false] + ], + [ + ["primary", "appropriation", 2439614000, false], + ["sub", "cap", 450614000, false], + ["sub", "earmark", 30300000, false] + ], + [ + ["primary", "appropriation", 2651047000, false], + ["sub", "cap", 304045000, false] + ], + [ + ["primary", "appropriation", 369261000, false], + ["sub", "cap", 42361000, false], + ["sub", "earmark", 19075000, false] + ], + [ + ["primary", "appropriation", 189322000, false], + ["sub", "cap", 36200000, false], + ["sub", "earmark", 600000, false] + ], + [ + ["primary", "appropriation", 117076000, false], + ["sub", "cap", 23389000, false] + ], + [ + ["primary", "appropriation", 61291000, false], + ["sub", "cap", 6495000, false] + ], + [ + ["primary", "appropriation", 330572000, false], + ["sub", "cap", 12146000, false], + ["sub", "earmark", 29000000, false] + ], + [ + ["primary", "appropriation", 293434000, false] + ], + [ + ["primary", "appropriation", 539174000, false] + ], + [ + ["primary", "appropriation", 304895000, false] + ], + [ + ["primary", "appropriation", 395485000, false] + ], + [ + ["primary", "appropriation", 277142000, false] + ], + [ + ["primary", "appropriation", 373854000, false] + ], + [ + ["primary", "appropriation", 237097000, false] + ], + [ + ["primary", "appropriation", 324386000, false] + ], + [ + ["primary", "appropriation", 50785000, false] + ], + [ + ["primary", "appropriation", 6611000, false] + ], + [ + ["primary", "appropriation", 496000, false] + ], + [ + ["primary", "restriction", 25000, false] + ], + [ + ["primary", "restriction", 25000, false] + ], + [ + ["primary", "restriction", 500000, false] + ], + [ + ["primary", "restriction", 1000000, false] + ], + [ + ["primary", "directive", 100000, false] + ], + [ + ["sub", "cap", 15000, false] + ], + [ + ["primary", "appropriation", 47700000, true] + ], + [ + ["primary", "appropriation", 30000000, true] + ], + [ + ["primary", "appropriation", 252000000, false] + ], + [ + ["primary", "appropriation", 25000000, true] + ], + [ + ["primary", "appropriation", 65000000, true] + ], + [ + ["primary", "appropriation", 15000000, true] + ], + [ + ["primary", "appropriation", 122210000, true] + ], + [ + ["primary", "appropriation", 50000000, false] + ], + [ + ["primary", "appropriation", 4655879000, false], + ["sub", "appropriation", 181390281000, false], + ["sub", "cap", 22109000, false] + ], + [ + ["primary", "appropriation", 11523134000, false] + ], + [ + ["primary", "appropriation", 12701000, false], + ["sub", "appropriation", 135119422, false] + ], + [ + ["sub", "cap", 316742419, false] + ], + [ + ["primary", "appropriation", 78337, false], + ["sub", "cap", 460698, false] + ], + [ + ["primary", "appropriation", 2718546, false] + ], + [ + ["primary", "appropriation", 3899000000, false] + ], + [ + ["primary", "appropriation", 71000000000, false], + ["sub", "availability", 2000000000, false] + ], + [ + ["primary", "appropriation", 20382000000, false], + ["sub", "availability", 2000000000, false] + ], + [ + ["primary", "appropriation", 11800000000, false], + ["sub", "availability", 350000000, false] + ], + [ + ["primary", "appropriation", 9400000000, false], + ["sub", "availability", 500000000, false] + ], + [ + ["primary", "appropriation", 938000000, false] + ], + [ + ["primary", "appropriation", 480000000, false] + ], + [ + ["primary", "appropriation", 475000000, false] + ], + [ + ["primary", "appropriation", 287000000, false] + ], + [ + ["primary", "appropriation", 6401000000, false], + ["sub", "sub_allocation", 1606977000, false], + ["sub", "sub_allocation", 4668373000, false], + ["sub", "sub_allocation", 125650000, false], + ["sub", "cap", 3000000, false] + ], + [ + ["primary", "appropriation", 1863000000, false] + ], + [ + ["primary", "appropriation", 296000000, false] + ], + [ + ["primary", "appropriation", 881000000, false], + ["sub", "availability", 373096000, false], + ["sub", "availability", 507904000, false], + ["sub", "sub_allocation", 110000000, false] + ], + [ + ["primary", "appropriation", 680000000, false], + ["sub", "availability", 612000000, false], + ["sub", "availability", 68000000, false] + ], + [ + ["primary", "appropriation", 164000000, false] + ], + [ + ["primary", "appropriation", 60000000, false] + ], + [ + ["primary", "cap", 6960000, true] + ], + [ + ["primary", "transfer", 430532000, false] + ], + [ + ["primary", "transfer", 456547000, false] + ], + [ + ["primary", "transfer", 15000000, false] + ], + [ + ["primary", "directive", 5000000, false] + ], + [ + ["primary", "directive", 1000000, false] + ], + [ + ["primary", "directive", 7000000, false] + ], + [ + ["primary", "unknown", 1279096000, false] + ], + [ + ["primary", "restriction", 223166.67, false] + ], + [ + ["primary", "rescission", 4933113000, true] + ], + [ + ["primary", "appropriation", 158630000, false] + ], + [ + ["primary", "appropriation", 47200000, false], + ["sub", "sub_allocation", 3385000, false] + ], + [ + ["primary", "appropriation", 100267000, false], + ["sub", "availability", 15000000, false] + ], + [ + ["primary", "appropriation", 88600000, false] + ], + [ + ["primary", "appropriation", 77000000, false], + ["sub", "availability", 8940000, false], + ["sub", "unknown", 25000000, false] + ], + [ + ["primary", "unknown", 0, false] + ] +]} diff --git a/tests/data/financial_rows/118-hr-4366-4_engrossed-amendment-senate.json b/tests/data/financial_rows/118-hr-4366-4_engrossed-amendment-senate.json new file mode 100644 index 00000000..cf1d5ee6 --- /dev/null +++ b/tests/data/financial_rows/118-hr-4366-4_engrossed-amendment-senate.json @@ -0,0 +1,1218 @@ +{"bill": "118-hr-4366", "version": "4_engrossed-amendment-senate", "classifier": "1.0", "sections": [ + [ + ["primary", "appropriation", 1876875000, false], + ["sub", "cap", 349245000, false], + ["sub", "unknown", 376320000, false] + ], + [ + ["primary", "appropriation", 6046309000, false], + ["sub", "cap", 708822000, false], + ["sub", "unknown", 282880000, false] + ], + [ + ["primary", "appropriation", 2802924000, false], + ["sub", "cap", 562074000, false], + ["sub", "unknown", 197610000, false] + ], + [ + ["primary", "appropriation", 3132782000, false], + ["sub", "cap", 318545000, false], + ["sub", "unknown", 36100000, false] + ], + [ + ["primary", "appropriation", 598572000, false], + ["sub", "cap", 71146000, false], + ["sub", "unknown", 251386000, false] + ], + [ + ["primary", "appropriation", 304426000, false], + ["sub", "cap", 67854000, false], + ["sub", "unknown", 125704000, false] + ], + [ + ["primary", "appropriation", 151076000, false], + ["sub", "cap", 27389000, false], + ["sub", "unknown", 44000000, false] + ], + [ + ["primary", "appropriation", 51291000, false], + ["sub", "cap", 6495000, false] + ], + [ + ["primary", "appropriation", 309572000, false], + ["sub", "cap", 14646000, false], + ["sub", "unknown", 18000000, false] + ], + [ + ["primary", "appropriation", 293434000, false] + ], + [ + ["primary", "appropriation", 439174000, false] + ], + [ + ["primary", "appropriation", 304895000, false] + ], + [ + ["primary", "appropriation", 385485000, false] + ], + [ + ["primary", "appropriation", 277142000, false] + ], + [ + ["primary", "appropriation", 363854000, false] + ], + [ + ["primary", "appropriation", 237097000, false] + ], + [ + ["primary", "appropriation", 314386000, false] + ], + [ + ["primary", "appropriation", 50785000, false] + ], + [ + ["primary", "appropriation", 6611000, false] + ], + [ + ["primary", "appropriation", 496000, false] + ], + [ + ["primary", "restriction", 25000, false] + ], + [ + ["primary", "restriction", 25000, false] + ], + [ + ["primary", "restriction", 500000, false] + ], + [ + ["primary", "restriction", 1000000, false] + ], + [ + ["primary", "directive", 100000, false] + ], + [ + ["sub", "cap", 35000, false] + ], + [ + ["primary", "appropriation", 38514000, true] + ], + [ + ["primary", "appropriation", 15000000, true] + ], + [ + ["primary", "appropriation", 150000000, false] + ], + [ + ["primary", "appropriation", 15000000, true] + ], + [ + ["primary", "appropriation", 83000000, false] + ], + [ + ["primary", "appropriation", 20000000, false] + ], + [ + ["primary", "unknown", 18800000, false] + ], + [ + ["primary", "appropriation", 48300000, true] + ], + [ + ["primary", "appropriation", 4655879000, false], + ["sub", "appropriation", 181390281000, false], + ["sub", "cap", 22109000, false] + ], + [ + ["primary", "appropriation", 11523134000, false] + ], + [ + ["primary", "appropriation", 12701000, false], + ["sub", "appropriation", 135119422, false] + ], + [ + ["sub", "cap", 316742419, false] + ], + [ + ["primary", "appropriation", 78337, false], + ["sub", "cap", 460698, false] + ], + [ + ["primary", "appropriation", 2718546, false] + ], + [ + ["primary", "appropriation", 3899000000, false] + ], + [ + ["primary", "rescission", 71000000000, false], + ["sub", "availability", 2000000000, false], + ["sub", "rescission", 74004000000, true] + ], + [ + ["primary", "rescission", 20382000000, false], + ["sub", "availability", 2000000000, false], + ["sub", "rescission", 33000000000, true] + ], + [ + ["primary", "appropriation", 11800000000, false], + ["sub", "availability", 350000000, false] + ], + [ + ["primary", "appropriation", 1000000000, false], + ["sub", "appropriation", 9400000000, false], + ["sub", "availability", 500000000, false] + ], + [ + ["primary", "appropriation", 938000000, false] + ], + [ + ["primary", "appropriation", 480000000, false] + ], + [ + ["primary", "appropriation", 475000000, false] + ], + [ + ["primary", "appropriation", 287000000, false] + ], + [ + ["primary", "appropriation", 6401000000, false], + ["sub", "sub_allocation", 1606977000, false], + ["sub", "sub_allocation", 4668373000, false], + ["sub", "availability", 75288000, false], + ["sub", "sub_allocation", 125650000, false], + ["sub", "cap", 3000000, false] + ], + [ + ["primary", "appropriation", 1334142000, false] + ], + [ + ["primary", "appropriation", 296000000, false] + ], + [ + ["primary", "appropriation", 881000000, false], + ["sub", "availability", 373096000, false], + ["sub", "availability", 507904000, false], + ["sub", "sub_allocation", 110000000, false] + ], + [ + ["primary", "appropriation", 680000000, false], + ["sub", "availability", 612000000, false], + ["sub", "availability", 68000000, false] + ], + [ + ["primary", "appropriation", 164000000, false] + ], + [ + ["primary", "appropriation", 60000000, false] + ], + [ + ["primary", "cap", 6960000, true] + ], + [ + ["primary", "transfer", 430532000, false] + ], + [ + ["primary", "transfer", 456547000, false] + ], + [ + ["primary", "transfer", 15000000, false] + ], + [ + ["primary", "directive", 5000000, false] + ], + [ + ["primary", "directive", 1000000, false] + ], + [ + ["primary", "directive", 7000000, false] + ], + [ + ["primary", "unknown", 1279096000, false], + ["sub", "sub_allocation", 10000000, false] + ], + [ + ["primary", "unknown", 600000000, false] + ], + [ + ["primary", "rescission", 439750000, false] + ], + [ + ["primary", "rescission", 1000000000, true] + ], + [ + ["primary", "unknown", 43500000, false] + ], + [ + ["primary", "unknown", 5180336000, false] + ], + [ + ["primary", "appropriation", 158630000, false] + ], + [ + ["primary", "appropriation", 47200000, false], + ["sub", "sub_allocation", 3000000, false] + ], + [ + ["primary", "appropriation", 99880000, false], + ["sub", "availability", 15000000, false] + ], + [ + ["primary", "appropriation", 88600000, false] + ], + [ + ["primary", "appropriation", 77000000, false], + ["sub", "availability", 8940000, false], + ["sub", "unknown", 25000000, false] + ], + [ + ["primary", "appropriation", 65067000, false], + ["sub", "sub_allocation", 7432000, false], + ["sub", "sub_allocation", 1000000, false], + ["sub", "sub_allocation", 1500000, false], + ["sub", "sub_allocation", 26716000, false], + ["sub", "cap", 4609000, false], + ["sub", "cap", 22000, false] + ], + [ + ["primary", "appropriation", 30181000, false], + ["sub", "sub_allocation", 10000000, false], + ["sub", "unknown", 2000000, false], + ["sub", "unknown", 500000, false] + ], + [ + ["primary", "appropriation", 16703000, false] + ], + [ + ["primary", "appropriation", 14967000, false] + ], + [ + ["primary", "appropriation", 92284000, false], + ["sub", "sub_allocation", 77428000, false] + ], + [ + ["primary", "appropriation", 7367000, false] + ], + [ + ["primary", "appropriation", 1466000, false] + ], + [ + ["primary", "appropriation", 37595000, false] + ], + [ + ["primary", "appropriation", 36081000, false] + ], + [ + ["primary", "appropriation", 6586000, false] + ], + [ + ["primary", "appropriation", 21800000, false] + ], + [ + ["primary", "appropriation", 111561000, false] + ], + [ + ["primary", "appropriation", 60537000, false] + ], + [ + ["primary", "appropriation", 5556000, false] + ], + [ + ["primary", "appropriation", 2384000, false], + ["sub", "unknown", 1000000, false] + ], + [ + ["primary", "appropriation", 92183000, false] + ], + [ + ["primary", "appropriation", 200563000, false], + ["sub", "sub_allocation", 55900000, false] + ], + [ + ["primary", "appropriation", 1792879000, false], + ["sub", "cap", 500000, true] + ], + [ + ["primary", "appropriation", 88869000, false], + ["sub", "sub_allocation", 47464000, false] + ], + [ + ["primary", "appropriation", 1084600000, false], + ["sub", "unknown", 1000000, false] + ], + [ + ["primary", "appropriation", 11880000, false] + ], + [ + ["primary", "appropriation", 567410000, false], + ["sub", "unknown", 1000000, false] + ], + [ + ["primary", "appropriation", 41500000, false] + ], + [ + ["primary", "appropriation", 1617000, false] + ], + [ + ["primary", "appropriation", 30000, true], + ["sub", "sub_allocation", 14276000, false], + ["sub", "sub_allocation", 6000000, false], + ["sub", "sub_allocation", 750000, false], + ["sub", "sub_allocation", 514000, false], + ["sub", "sub_allocation", 15450000, false], + ["sub", "sub_allocation", 39183000, false], + ["sub", "sub_allocation", 4096000, false], + ["sub", "sub_allocation", 64930000, false], + ["sub", "sub_allocation", 4251000, false], + ["sub", "sub_allocation", 217904000, false], + ["sub", "sub_allocation", 8500000, false], + ["sub", "sub_allocation", 13986000, false], + ["sub", "sub_allocation", 21567000, false], + ["sub", "sub_allocation", 48067000, false], + ["sub", "sub_allocation", 62562000, false], + ["sub", "sub_allocation", 6500000, false], + ["sub", "sub_allocation", 1500000, false], + ["sub", "sub_allocation", 2500000, false], + ["sub", "unknown", 1000000, false], + ["sub", "unknown", 4990000, false], + ["sub", "sub_allocation", 24527000, false] + ], + [ + ["primary", "appropriation", 3175000, false] + ], + [ + ["primary", "appropriation", 229891000, false], + ["sub", "sub_allocation", 7500000, false], + ["sub", "sub_allocation", 1000000, false], + ["sub", "unknown", 15000000, false], + ["sub", "sub_allocation", 14000000, false] + ], + [ + ["primary", "cap", 62596000, false] + ], + [ + ["primary", "cap", 21501000, false] + ], + [ + ["primary", "appropriation", 1235000, false] + ], + [ + ["primary", "cap", 55000000, false] + ], + [ + ["primary", "appropriation", 1117000, false] + ], + [ + ["primary", "appropriation", 1205009000, false], + ["sub", "unknown", 1000000, false] + ], + [ + ["primary", "appropriation", 1727000, false] + ], + [ + ["primary", "appropriation", 249684000, false], + ["sub", "sub_allocation", 1000000, false], + ["sub", "unknown", 60228000, false] + ], + [ + ["primary", "appropriation", 1215307000, false], + ["sub", "sub_allocation", 15000000, false], + ["sub", "unknown", 25000, false], + ["sub", "unknown", 696594000, false] + ], + [ + ["primary", "appropriation", 7000000, false] + ], + [ + ["primary", "appropriation", 7500000, false] + ], + [ + ["primary", "appropriation", 4000000, false] + ], + [ + ["primary", "appropriation", 3500000000, true], + ["sub", "directive", 3507000, true], + ["sub", "unknown", 321621000, false] + ], + [ + ["primary", "appropriation", 66870000, false], + ["sub", "unknown", 1000000, false], + ["sub", "cap", 1000, false] + ], + [ + ["primary", "appropriation", 922151000, false], + ["sub", "unknown", 250000, false], + ["sub", "unknown", 8500000, false] + ], + [ + ["primary", "appropriation", 90405000, false], + ["sub", "sub_allocation", 20405000, false], + ["sub", "unknown", 5000000, false], + ["sub", "unknown", 20000000, false] + ], + [ + ["primary", "unknown", 2000000, false] + ], + [ + ["sub", "unknown", 5000000, false] + ], + [ + ["primary", "appropriation", 15000000, false] + ], + [ + ["primary", "appropriation", 1620000, false] + ], + [ + ["primary", "appropriation", 351087000, false], + ["sub", "unknown", 3000000, false], + ["sub", "unknown", 100000000, false] + ], + [ + ["primary", "appropriation", 850000000, true], + ["sub", "unknown", 5000000, false], + ["sub", "directive", 7500, true], + ["sub", "directive", 18703000, true] + ], + [ + ["primary", "appropriation", 1608000000, false] + ], + [ + ["primary", "appropriation", 48000000, false], + ["sub", "cap", 1000000, false] + ], + [ + ["primary", "appropriation", 32000000, false] + ], + [ + ["primary", "appropriation", 48000000, false] + ], + [ + ["primary", "appropriation", 2800000000, true], + ["sub", "sub_allocation", 205134000, false], + ["sub", "unknown", 6000000, false], + ["sub", "unknown", 10000000, false] + ], + [ + ["primary", "appropriation", 77728000, false], + ["sub", "cap", 12000000, false], + ["sub", "unknown", 4000000, true] + ], + [ + ["primary", "appropriation", 18889000, true], + ["sub", "sub_allocation", 573000, false], + ["sub", "sub_allocation", 1147000, false], + ["sub", "unknown", 4468000, false] + ], + [ + ["primary", "appropriation", 75000000, true] + ], + [ + ["primary", "appropriation", 28300000, false], + ["sub", "sub_allocation", 3500000, false], + ["sub", "cap", 3000000, false], + ["sub", "sub_allocation", 16000000, false], + ["sub", "sub_allocation", 3000000, false] + ], + [ + ["primary", "appropriation", 20000000, true] + ], + [ + ["primary", "appropriation", 50000000, false] + ], + [ + ["primary", "appropriation", 1000000, false] + ], + [ + ["primary", "appropriation", 880000000, true], + ["sub", "sub_allocation", 1000000, false], + ["sub", "cap", 20000000, false], + ["sub", "sub_allocation", 10000000, false], + ["sub", "cap", 10000000, false], + ["sub", "unknown", 70000000, false], + ["sub", "cap", 37500000, false], + ["sub", "sub_allocation", 8500000, false], + ["sub", "sub_allocation", 800000, false], + ["sub", "cap", 25000000, false], + ["sub", "cap", 4000000, false], + ["sub", "cap", 8085000, false], + ["sub", "unknown", 10000000, false] + ], + [ + ["primary", "unknown", 2167000000, true], + ["sub", "unknown", 2000000000, true], + ["sub", "directive", 33270000, false] + ], + [ + ["primary", "appropriation", 62721000, false], + ["sub", "sub_allocation", 2721000, false], + ["sub", "unknown", 3000000, false], + ["sub", "unknown", 98000000, false], + ["sub", "appropriation", 35000000, false] + ], + [ + ["primary", "appropriation", 1376000, false] + ], + [ + ["primary", "appropriation", 32032897000, false], + ["sub", "unknown", 20162000, false], + ["sub", "unknown", 21876000, false], + ["sub", "unknown", 10000000, false], + ["sub", "unknown", 500000, false], + ["sub", "unknown", 20000000, true], + ["sub", "unknown", 2000000, false] + ], + [ + ["primary", "appropriation", 6300000000, false], + ["sub", "unknown", 90000000, true] + ], + [ + ["primary", "appropriation", 122141239000, false], + ["sub", "sub_allocation", 3000000000, false], + ["sub", "unknown", 998000, false], + ["sub", "unknown", 5000000, false], + ["sub", "unknown", 3000000, false] + ], + [ + ["primary", "appropriation", 501070000, false] + ], + [ + ["primary", "appropriation", 184348000, false], + ["sub", "unknown", 2000000, false] + ], + [ + ["primary", "appropriation", 932000, false] + ], + [ + ["primary", "appropriation", 4922000, false] + ], + [ + ["primary", "appropriation", 237330000, false], + ["sub", "unknown", 2000000, false] + ], + [ + ["primary", "appropriation", 1800000000, false] + ], + [ + ["primary", "appropriation", 248331000, false], + ["sub", "availability", 24800000, false] + ], + [ + ["primary", "appropriation", 6063000, false] + ], + [ + ["primary", "appropriation", 6625030000, false], + ["sub", "unknown", 1336525000, true], + ["sub", "unknown", 1198263000, false], + ["sub", "sub_allocation", 15000000, true], + ["sub", "sub_allocation", 10000000, true], + ["sub", "sub_allocation", 56411000, true], + ["sub", "cap", 25000, false], + ["sub", "unknown", 2000000, false], + ["sub", "transfer", 1500000, false] + ], + [ + ["primary", "appropriation", 12788000, false] + ], + [ + ["primary", "appropriation", 50000000, false] + ], + [ + ["primary", "cap", 94300000, false] + ], + [ + ["sub", "restriction", 25000, false], + ["sub", "unknown", 250000, false] + ], + [ + ["primary", "cap", 20000000, false] + ], + [ + ["primary", "cap", 2900000, false] + ], + [ + ["primary", "restriction", 1573666000, true], + ["sub", "unknown", 350000000, false] + ], + [ + ["primary", "restriction", 500000, false] + ], + [ + ["primary", "restriction", 500000, false] + ], + [ + ["primary", "unknown", 9156000, false] + ], + [ + ["primary", "unknown", 3000000, false] + ], + [ + ["primary", "unknown", 8000000, false] + ], + [ + ["primary", "appropriation", 3000000, false] + ], + [ + ["primary", "appropriation", 2000000, false] + ], + [ + ["primary", "unknown", 30000000, false] + ], + [ + ["primary", "rescission", 307526000, false] + ], + [ + ["primary", "appropriation", 2000000, false] + ], + [ + ["primary", "appropriation", 2000000, false] + ], + [ + ["primary", "appropriation", 500000, false] + ], + [ + ["primary", "appropriation", 9000000, false] + ], + [ + ["primary", "appropriation", 3000000, false] + ], + [ + ["primary", "unknown", 700000, false] + ], + [ + ["primary", "rescission", 20000000, false] + ], + [ + ["primary", "appropriation", 3000000, false] + ], + [ + ["primary", "unknown", 3000000, false] + ], + [ + ["primary", "appropriation", 1000000, false] + ], + [ + ["primary", "unknown", 6000000, false] + ], + [ + ["primary", "appropriation", 1000000, false] + ], + [ + ["primary", "unknown", 1000000, false] + ], + [ + ["primary", "appropriation", 2000000, false] + ], + [ + ["primary", "appropriation", 191295000, false], + ["sub", "unknown", 3770000, true], + ["sub", "sub_allocation", 2000000, false], + ["sub", "unknown", 22724000, true], + ["sub", "cap", 70000, false], + ["sub", "unknown", 2500000, false] + ], + [ + ["primary", "appropriation", 51358000, false], + ["sub", "availability", 35745000, false], + ["sub", "unknown", 14750000, false], + ["sub", "unknown", 8000000, false] + ], + [ + ["primary", "appropriation", 800000000, false], + ["sub", "unknown", 20000000, false], + ["sub", "unknown", 45000000, false] + ], + [ + ["primary", "appropriation", 9558000, false] + ], + [ + ["primary", "appropriation", 25000000, false] + ], + [ + ["primary", "appropriation", 5000000, false] + ], + [ + ["primary", "appropriation", 49000000, false] + ], + [ + ["primary", "appropriation", 18228000, false] + ], + [ + ["primary", "appropriation", 24069000, false], + ["sub", "unknown", 5436000, false], + ["sub", "unknown", 3443000, false] + ], + [ + ["primary", "appropriation", 522165000, false] + ], + [ + ["primary", "appropriation", 5330000, false] + ], + [ + ["primary", "unknown", 348554000, false] + ], + [ + ["sub", "unknown", 1000000, false] + ], + [ + ["primary", "unknown", 75000000, false] + ], + [ + ["primary", "unknown", 8973000, false] + ], + [ + ["primary", "appropriation", 12740627000, false], + ["sub", "sub_allocation", 12103596000, false], + ["sub", "unknown", 1745532000, true], + ["sub", "directive", 100000, false], + ["sub", "directive", 100000, false], + ["sub", "unknown", 194000000, false] + ], + [ + ["primary", "appropriation", 3429000000, false], + ["sub", "availability", 635000000, true], + ["sub", "unknown", 15000000, false], + ["sub", "unknown", 469000000, false] + ], + [ + ["primary", "appropriation", 260000000, false] + ], + [ + ["primary", "appropriation", 3350000000, false], + ["sub", "unknown", 3350000000, false], + ["sub", "unknown", 157475000, true] + ], + [ + ["primary", "appropriation", 500728000, false], + ["sub", "unknown", 200728000, false], + ["sub", "unknown", 300000000, false], + ["sub", "sub_allocation", 25000000, true] + ], + [ + ["primary", "unknown", 1590528.89, true] + ], + [ + ["primary", "unknown", 29350000, false] + ], + [ + ["primary", "cap", 483551671, false], + ["sub", "unknown", 3248000, false] + ], + [ + ["primary", "unknown", 60095782888, false] + ], + [ + ["primary", "appropriation", 60792659888, false] + ], + [ + ["primary", "appropriation", 2046738000, false], + ["sub", "unknown", 701738000, false], + ["sub", "unknown", 100000000, false], + ["sub", "directive", 10000000, true], + ["sub", "unknown", 5000000, false], + ["sub", "directive", 15000000, false], + ["sub", "directive", 12000000, false], + ["sub", "appropriation", 45000000, false], + ["sub", "unknown", 3000000, true], + ["sub", "unknown", 6000000, false], + ["sub", "unknown", 60000000, true] + ], + [ + ["primary", "unknown", 639000000, true] + ], + [ + ["primary", "unknown", 33437074.13, true] + ], + [ + ["primary", "unknown", 20000000, false], + ["sub", "unknown", 30000000, true], + ["sub", "sub_allocation", 125000000, true] + ], + [ + ["primary", "appropriation", 346000000, false], + ["sub", "unknown", 435000000, false], + ["sub", "sub_allocation", 60000000, false], + ["sub", "unknown", 14073000, true] + ], + [ + ["primary", "appropriation", 516300000, false], + ["sub", "unknown", 516300000, false], + ["sub", "unknown", 406500000, true] + ], + [ + ["primary", "appropriation", 222000000, false] + ], + [ + ["primary", "appropriation", 201200000, false], + ["sub", "restriction", 201200000, false], + ["sub", "unknown", 194000000, true], + ["sub", "unknown", 201200000, true] + ], + [ + ["primary", "appropriation", 813300800, false], + ["sub", "restriction", 813300800, false], + ["sub", "unknown", 378400000, true], + ["sub", "cap", 500000, false] + ], + [ + ["primary", "unknown", 130000, false] + ], + [ + ["primary", "appropriation", 267799000, false], + ["sub", "availability", 25000000, false] + ], + [ + ["primary", "appropriation", 59000000, false], + ["sub", "unknown", 3000000, false] + ], + [ + ["primary", "appropriation", 100000000, false] + ], + [ + ["primary", "appropriation", 572861000, false], + ["sub", "unknown", 72861000, false], + ["sub", "unknown", 5000000, false] + ], + [ + ["primary", "unknown", 1141442000, false], + ["sub", "unknown", 5000000, false] + ], + [ + ["primary", "unknown", 1313033000, false], + ["sub", "unknown", 3000000, false], + ["sub", "unknown", 66000000, false] + ], + [ + ["primary", "unknown", 126348, true] + ], + [ + ["primary", "restriction", 35000, false] + ], + [ + ["primary", "appropriation", 13990000000, false], + ["sub", "unknown", 13990000000, false] + ], + [ + ["primary", "appropriation", 268261000, false], + ["sub", "unknown", 80000000, true], + ["sub", "directive", 750000, true], + ["sub", "unknown", 5000000, true], + ["sub", "unknown", 23014000, false], + ["sub", "unknown", 10000000, false], + ["sub", "unknown", 5000000, false] + ], + [ + ["primary", "appropriation", 7500000, false] + ], + [ + ["primary", "appropriation", 2450000000, false], + ["sub", "unknown", 1910000000, true] + ], + [ + ["primary", "appropriation", 150000000, false] + ], + [ + ["primary", "unknown", 581046, false] + ], + [ + ["primary", "unknown", 4009637, false] + ], + [ + ["primary", "unknown", 4072214, false] + ], + [ + ["primary", "unknown", 1975409, false] + ], + [ + ["primary", "unknown", 2148414, false] + ], + [ + ["primary", "unknown", 45187599, false] + ], + [ + ["primary", "unknown", 40536306, false] + ], + [ + ["primary", "unknown", 81863444, false] + ], + [ + ["primary", "appropriation", 40288000, false], + ["sub", "unknown", 16300000, false] + ], + [ + ["primary", "appropriation", 318000000, false] + ], + [ + ["primary", "appropriation", 10000000, false] + ], + [ + ["primary", "appropriation", 120000000, false] + ], + [ + ["primary", "appropriation", 283546000, false], + ["sub", "unknown", 103500000, true] + ], + [ + ["primary", "appropriation", 131000000, false], + ["sub", "unknown", 22000000, true], + ["sub", "sub_allocation", 8900000, false], + ["sub", "unknown", 2400000, true] + ], + [ + ["primary", "unknown", 20000000, false] + ], + [ + ["primary", "appropriation", 6021000, false] + ], + [ + ["primary", "appropriation", 103020000, false], + ["sub", "availability", 100000000, false], + ["sub", "cap", 3020000, false] + ], + [ + ["primary", "unknown", 213000000, false], + ["sub", "unknown", 188000000, false], + ["sub", "appropriation", 1000000, false] + ], + [ + ["primary", "appropriation", 31681000, false], + ["sub", "availability", 4500000, false] + ], + [ + ["primary", "appropriation", 74556000, false], + ["sub", "availability", 12070000, false], + ["sub", "sub_allocation", 1000000, false], + ["sub", "unknown", 800000, false] + ], + [ + ["primary", "appropriation", 226228000, false], + ["sub", "sub_allocation", 30000000, false], + ["sub", "sub_allocation", 188828000, false], + ["sub", "sub_allocation", 400000, false], + ["sub", "sub_allocation", 7000000, false], + ["sub", "unknown", 1058000, false] + ], + [ + ["primary", "appropriation", 46825000, false] + ], + [ + ["primary", "appropriation", 116452000, false] + ], + [ + ["primary", "appropriation", 19400000, false], + ["sub", "cap", 25000, false] + ], + [ + ["primary", "appropriation", 698200000, false], + ["sub", "unknown", 95200000, true], + ["sub", "sub_allocation", 21700000, true] + ], + [ + ["primary", "appropriation", 1114100000, false], + ["sub", "unknown", 288500000, true] + ], + [ + ["primary", "appropriation", 27737961000, true], + ["sub", "unknown", 27765512000, false], + ["sub", "unknown", 200000000, false], + ["sub", "unknown", 5289210000, true], + ["sub", "unknown", 5000000, false], + ["sub", "unknown", 2781449000, false], + ["sub", "sub_allocation", 30000000, false], + ["sub", "unknown", 2751449000, false], + ["sub", "unknown", 686000000, false], + ["sub", "unknown", 10000000, false], + ["sub", "unknown", 7500000, false], + ["sub", "unknown", 30000000, false], + ["sub", "unknown", 10000000, true], + ["sub", "unknown", 5000000, true] + ], + [ + ["primary", "appropriation", 8875000000, false], + ["sub", "unknown", 5530000000, true], + ["sub", "unknown", 3200000000, false], + ["sub", "directive", 30000000, false], + ["sub", "unknown", 10000000, false], + ["sub", "unknown", 65000000, false], + ["sub", "unknown", 25000000, false], + ["sub", "appropriation", 15000000, false] + ], + [ + ["primary", "appropriation", 50000000, false] + ], + [ + ["primary", "appropriation", 150000000, false], + ["sub", "unknown", 75000000, false], + ["sub", "cap", 10000000, false] + ], + [ + ["primary", "appropriation", 198000000, false], + ["sub", "unknown", 140500000, true] + ], + [ + ["primary", "appropriation", 1081625000, false], + ["sub", "unknown", 848625000, false], + ["sub", "directive", 150000000, false], + ["sub", "unknown", 10000000, false], + ["sub", "unknown", 1000000, false], + ["sub", "sub_allocation", 75000000, false], + ["sub", "sub_allocation", 5000000, false], + ["sub", "unknown", 7000000, false], + ["sub", "unknown", 2000000, false] + ], + [ + ["primary", "appropriation", 905700, false], + ["sub", "sub_allocation", 1400000000, false] + ], + [ + ["primary", "appropriation", 22300000, false], + ["sub", "unknown", 1000000, false] + ], + [ + ["sub", "sub_allocation", 28000000, false] + ], + [ + ["primary", "appropriation", 505000000, false] + ], + [ + ["primary", "appropriation", 4491483000, false], + ["sub", "unknown", 3300000000, false], + ["sub", "unknown", 100000000, false], + ["sub", "unknown", 30000000, false], + ["sub", "directive", 1061483000, false] + ], + [ + ["sub", "sub_allocation", 400000000, false] + ], + [ + ["primary", "appropriation", 1500000000, false] + ], + [ + ["primary", "appropriation", 61500000, false], + ["sub", "unknown", 13500000, true], + ["sub", "sub_allocation", 5000000, false], + ["sub", "appropriation", 6000000, false] + ], + [ + ["primary", "appropriation", 3908000000, false], + ["sub", "unknown", 290000000, false], + ["sub", "directive", 3401000000, false], + ["sub", "unknown", 25000000, false], + ["sub", "unknown", 50000, false], + ["sub", "unknown", 52000000, false], + ["sub", "unknown", 10000000, false], + ["sub", "unknown", 107000000, false], + ["sub", "unknown", 25000000, false], + ["sub", "unknown", 10000000, false], + ["sub", "unknown", 100000000, false], + ["sub", "unknown", 35000000, false] + ], + [ + ["primary", "appropriation", 15390924000, true], + ["sub", "cap", 448000000, false], + ["sub", "unknown", 32924000, false], + ["sub", "unknown", 5081790000, false] + ], + [ + ["primary", "appropriation", 1075000000, false], + ["sub", "unknown", 120000000, false], + ["sub", "unknown", 6000000, false] + ], + [ + ["primary", "appropriation", 360000000, false] + ], + [ + ["primary", "appropriation", 57500000, true] + ], + [ + ["primary", "appropriation", 14000000, false], + ["sub", "sub_allocation", 14000000, false] + ], + [ + ["primary", "unknown", 400000000000, false], + ["sub", "unknown", 1000000, false], + ["sub", "appropriation", 150000000, false], + ["sub", "unknown", 200000000000, true] + ], + [ + ["primary", "unknown", 35000000000, false], + ["sub", "unknown", 1000000, false] + ], + [ + ["primary", "unknown", 900000000000, false], + ["sub", "unknown", 54000000, false], + ["sub", "unknown", 155000000000, true] + ], + [ + ["primary", "appropriation", 125400000, false], + ["sub", "unknown", 20000000, false] + ], + [ + ["primary", "appropriation", 86355000, false], + ["sub", "unknown", 1355000, false] + ], + [ + ["primary", "appropriation", 350000000, false], + ["sub", "unknown", 245000000, false], + ["sub", "sub_allocation", 105000000, true], + ["sub", "sub_allocation", 5000000, true], + ["sub", "unknown", 10000000, true] + ], + [ + ["primary", "appropriation", 374750000, false], + ["sub", "sub_allocation", 23950000, false] + ], + [ + ["primary", "appropriation", 152924000, false] + ], + [ + ["primary", "unknown", 5000000, false], + ["sub", "unknown", 5000000, false], + ["sub", "directive", 5000000, false] + ], + [ + ["primary", "unknown", 500000, false], + ["sub", "directive", 5000000, false], + ["sub", "unknown", 500000, false] + ], + [ + ["primary", "rescission", 65000000, false] + ], + [ + ["primary", "rescission", 49400000, false] + ], + [ + ["primary", "rescission", 20000000, false] + ], + [ + ["primary", "transfer", 3500000, false] + ], + [ + ["primary", "appropriation", 9955000, false] + ], + [ + ["primary", "appropriation", 43720000, false], + ["sub", "availability", 2000000, false], + ["sub", "cap", 3500, false] + ], + [ + ["primary", "appropriation", 29240000, false] + ], + [ + ["primary", "appropriation", 134300000, false], + ["sub", "sub_allocation", 2000, false] + ], + [ + ["primary", "appropriation", 168000000, false], + ["sub", "unknown", 2000000, false] + ], + [ + ["primary", "appropriation", 47452000, false], + ["sub", "cap", 1250000, false], + ["sub", "directive", 46202000, false] + ], + [ + ["primary", "appropriation", 4300000, false] + ], + [ + ["primary", "unknown", 5000000, true] + ] +]} diff --git a/tests/data/ledger_location_baseline.json b/tests/data/ledger_location_baseline.json new file mode 100644 index 00000000..a24f7183 --- /dev/null +++ b/tests/data/ledger_location_baseline.json @@ -0,0 +1,338 @@ +{ + "113-hr-3547/1_introduced-in-house": { + "T0": 0, + "T1": 0, + "T2": 0, + "T3": 0, + "T4": 0, + "MISS": 0 + }, + "113-hr-3547/2_engrossed-in-house": { + "T0": 0, + "T1": 0, + "T2": 0, + "T3": 0, + "T4": 0, + "MISS": 0 + }, + "113-hr-3547/3_received-in-senate": { + "T0": 0, + "T1": 0, + "T2": 0, + "T3": 0, + "T4": 0, + "MISS": 0 + }, + "113-hr-3547/4_engrossed-amendment-senate": { + "T0": 0, + "T1": 0, + "T2": 0, + "T3": 0, + "T4": 0, + "MISS": 0 + }, + "114-hr-2029/1_reported-in-house": { + "T0": 122, + "T1": 0, + "T2": 0, + "T3": 5, + "T4": 0, + "MISS": 0 + }, + "114-hr-2029/3_referred-in-senate": { + "T0": 122, + "T1": 0, + "T2": 0, + "T3": 10, + "T4": 0, + "MISS": 0 + }, + "114-hr-2029/4_reported-in-senate": { + "T0": 247, + "T1": 0, + "T2": 0, + "T3": 11, + "T4": 0, + "MISS": 0 + }, + "115-hr-5895/1_reported-in-house": { + "T0": 31, + "T1": 0, + "T2": 87, + "T3": 11, + "T4": 0, + "MISS": 0 + }, + "115-hr-5895/2_engrossed-in-house": { + "T0": 147, + "T1": 0, + "T2": 199, + "T3": 30, + "T4": 0, + "MISS": 0 + }, + "115-hr-5895/4_engrossed-amendment-senate": { + "T0": 145, + "T1": 0, + "T2": 252, + "T3": 44, + "T4": 3, + "MISS": 0 + }, + "117-hr-2471/1_introduced-in-house": { + "T0": 2, + "T1": 0, + "T2": 0, + "T3": 0, + "T4": 0, + "MISS": 0 + }, + "117-hr-4432/1_reported-in-house": { + "T0": 217, + "T1": 0, + "T2": 0, + "T3": 0, + "T4": 0, + "MISS": 0 + }, + "117-hr-4502/1_reported-in-house": { + "T0": 428, + "T1": 15, + "T2": 0, + "T3": 0, + "T4": 1, + "MISS": 0 + }, + "117-hr-4502/2_engrossed-in-house": { + "T0": 1400, + "T1": 22, + "T2": 229, + "T3": 301, + "T4": 8, + "MISS": 0 + }, + "118-hr-2882/1_introduced-in-house": { + "T0": 0, + "T1": 0, + "T2": 0, + "T3": 0, + "T4": 0, + "MISS": 0 + }, + "118-hr-2882/4_engrossed-amendment-senate": { + "T0": 0, + "T1": 0, + "T2": 0, + "T3": 0, + "T4": 0, + "MISS": 0 + }, + "118-hr-2882/5_engrossed-amendment-house": { + "T0": 1244, + "T1": 22, + "T2": 253, + "T3": 215, + "T4": 2, + "MISS": 0 + }, + "118-hr-4366/1_reported-in-house": { + "T0": 137, + "T1": 0, + "T2": 0, + "T3": 4, + "T4": 0, + "MISS": 0 + }, + "118-hr-4366/2_engrossed-in-house": { + "T0": 137, + "T1": 0, + "T2": 0, + "T3": 4, + "T4": 0, + "MISS": 0 + }, + "118-hr-4366/3_placed-on-calendar-senate": { + "T0": 137, + "T1": 0, + "T2": 0, + "T3": 4, + "T4": 0, + "MISS": 0 + }, + "118-hr-4366/4_engrossed-amendment-senate": { + "T0": 774, + "T1": 0, + "T2": 2, + "T3": 57, + "T4": 0, + "MISS": 0 + }, + "118-hr-4366/5_engrossed-amendment-house": { + "T0": 1346, + "T1": 36, + "T2": 235, + "T3": 106, + "T4": 11, + "MISS": 0 + }, + "118-hr-4820/1_reported-in-house": { + "T0": 273, + "T1": 1, + "T2": 0, + "T3": 14, + "T4": 0, + "MISS": 0 + }, + "118-hr-8282/1_introduced-in-house": { + "T0": 0, + "T1": 0, + "T2": 0, + "T3": 0, + "T4": 0, + "MISS": 0 + }, + "118-hr-8752/1_reported-in-house": { + "T0": 151, + "T1": 0, + "T2": 1, + "T3": 0, + "T4": 0, + "MISS": 0 + }, + "118-hr-8752/2_engrossed-in-house": { + "T0": 151, + "T1": 0, + "T2": 2, + "T3": 0, + "T4": 0, + "MISS": 0 + }, + "118-hr-8774/1_reported-in-house": { + "T0": 229, + "T1": 0, + "T2": 0, + "T3": 2, + "T4": 0, + "MISS": 0 + }, + "118-hr-8774/2_engrossed-in-house": { + "T0": 230, + "T1": 0, + "T2": 0, + "T3": 1, + "T4": 0, + "MISS": 0 + }, + "118-hr-9468/1_introduced-in-house": { + "T0": 2, + "T1": 0, + "T2": 0, + "T3": 0, + "T4": 0, + "MISS": 0 + }, + "118-s-2321/1_reported-in-senate": { + "T0": 321, + "T1": 34, + "T2": 0, + "T3": 2, + "T4": 2, + "MISS": 0 + }, + "118-s-2625/1_reported-in-senate": { + "T0": 174, + "T1": 0, + "T2": 0, + "T3": 0, + "T4": 0, + "MISS": 0 + }, + "118-s-4677/1_reported-in-senate": { + "T0": 123, + "T1": 0, + "T2": 0, + "T3": 3, + "T4": 0, + "MISS": 0 + }, + "118-s-4690/1_reported-in-senate": { + "T0": 292, + "T1": 0, + "T2": 2, + "T3": 13, + "T4": 0, + "MISS": 0 + }, + "118-s-4795/1_reported-in-senate": { + "T0": 323, + "T1": 35, + "T2": 0, + "T3": 0, + "T4": 2, + "MISS": 0 + }, + "118-s-4796/1_reported-in-senate": { + "T0": 341, + "T1": 1, + "T2": 0, + "T3": 20, + "T4": 0, + "MISS": 0 + }, + "118-s-4797/1_reported-in-senate": { + "T0": 97, + "T1": 0, + "T2": 69, + "T3": 153, + "T4": 2, + "MISS": 0 + }, + "118-s-4802/1_reported-in-senate": { + "T0": 193, + "T1": 0, + "T2": 77, + "T3": 12, + "T4": 9, + "MISS": 0 + }, + "118-s-4921/1_reported-in-senate": { + "T0": 269, + "T1": 0, + "T2": 0, + "T3": 0, + "T4": 0, + "MISS": 0 + }, + "118-s-4927/1_reported-in-senate": { + "T0": 32, + "T1": 0, + "T2": 135, + "T3": 12, + "T4": 0, + "MISS": 0 + }, + "118-s-4928/1_reported-in-senate": { + "T0": 291, + "T1": 4, + "T2": 29, + "T3": 21, + "T4": 0, + "MISS": 0 + }, + "118-s-4942/1_reported-in-senate": { + "T0": 437, + "T1": 22, + "T2": 0, + "T3": 0, + "T4": 0, + "MISS": 0 + }, + "119-hr-1/1_reported-in-house": { + "T0": 0, + "T1": 0, + "T2": 783, + "T3": 0, + "T4": 0, + "MISS": 0 + } +} diff --git a/tests/data/pdf/anchors_golden/118-s-4795.json b/tests/data/pdf/anchors_golden/118-s-4795.json index ca5d6ca8..ec46aea8 100644 --- a/tests/data/pdf/anchors_golden/118-s-4795.json +++ b/tests/data/pdf/anchors_golden/118-s-4795.json @@ -389,17 +389,11 @@ 35, 5 ], - [ - "agency", - "SALARIES AND EXPENSES, FOREIGN CLAIMS", - 36, - 11 - ], [ "account", - "SETTLEMENT COMMISSION", + "SALARIES AND EXPENSES, FOREIGN CLAIMS SETTLEMENT COMMISSION", 36, - 12 + 11 ], [ "account", @@ -407,17 +401,11 @@ 36, 17 ], - [ - "agency", - "SALARIES AND EXPENSES, COMMUNITY RELATIONS", - 37, - 9 - ], [ "account", - "SERVICE", + "SALARIES AND EXPENSES, COMMUNITY RELATIONS SERVICE", 37, - 10 + 9 ], [ "account", @@ -540,22 +528,22 @@ 6 ], [ - "agency", - "LIMITATION ON ADMINISTRATIVE EXPENSES, FEDERAL", + "account", + "LIMITATION ON ADMINISTRATIVE EXPENSES, FEDERAL PRISON INDUSTRIES, INCORPORATED", 47, 15 ], [ - "account", - "PRISON INDUSTRIES, INCORPORATED", - 47, - 16 + "agency", + "STATE AND LOCAL LAW ENFORCEMENT ACTIVITIES OFFICE ON VIOLENCE AGAINST WOMEN", + 48, + 6 ], [ "account", - "PROSECUTION PROGRAMS", + "VIOLENCE AGAINST WOMEN PREVENTION AND PROSECUTION PROGRAMS", 48, - 9 + 8 ], [ "agency", @@ -799,9 +787,9 @@ ], [ "account", - "MATHEMATICS ENGAGEMENT", + "SCIENCE, TECHNOLOGY, ENGINEERING, AND MATHEMATICS ENGAGEMENT", 92, - 4 + 3 ], [ "account", @@ -811,9 +799,9 @@ ], [ "account", - "RESTORATION", + "CONSTRUCTION AND ENVIRONMENTAL COMPLIANCE AND RESTORATION", 94, - 6 + 5 ], [ "account", @@ -839,17 +827,11 @@ 99, 7 ], - [ - "agency", - "MAJOR RESEARCH EQUIPMENT AND FACILITIES", - 100, - 8 - ], [ "account", - "CONSTRUCTION", + "MAJOR RESEARCH EQUIPMENT AND FACILITIES CONSTRUCTION", 100, - 9 + 8 ], [ "account", @@ -941,17 +923,11 @@ 106, 17 ], - [ - "agency", - "ADMINISTRATIVE PROVISION\u2014LEGAL SERVICES", - 107, - 15 - ], [ "account", - "CORPORATION", + "ADMINISTRATIVE PROVISION\u2014LEGAL SERVICES CORPORATION", 107, - 16 + 15 ], [ "agency", diff --git a/tests/data/pdf/anchors_golden/118-s-4795.pre-agency-anchors.json b/tests/data/pdf/anchors_golden/118-s-4795.pre-agency-anchors.json index 7d0be351..2736f6c1 100644 --- a/tests/data/pdf/anchors_golden/118-s-4795.pre-agency-anchors.json +++ b/tests/data/pdf/anchors_golden/118-s-4795.pre-agency-anchors.json @@ -295,9 +295,9 @@ ], [ "account", - "SETTLEMENT COMMISSION", + "SALARIES AND EXPENSES, FOREIGN CLAIMS SETTLEMENT COMMISSION", 36, - 12 + 11 ], [ "account", @@ -307,9 +307,9 @@ ], [ "account", - "SERVICE", + "SALARIES AND EXPENSES, COMMUNITY RELATIONS SERVICE", 37, - 10 + 9 ], [ "account", @@ -391,15 +391,15 @@ ], [ "account", - "PRISON INDUSTRIES, INCORPORATED", + "LIMITATION ON ADMINISTRATIVE EXPENSES, FEDERAL PRISON INDUSTRIES, INCORPORATED", 47, - 16 + 15 ], [ "account", - "PROSECUTION PROGRAMS", + "VIOLENCE AGAINST WOMEN PREVENTION AND PROSECUTION PROGRAMS", 48, - 9 + 8 ], [ "account", @@ -619,9 +619,9 @@ ], [ "account", - "MATHEMATICS ENGAGEMENT", + "SCIENCE, TECHNOLOGY, ENGINEERING, AND MATHEMATICS ENGAGEMENT", 92, - 4 + 3 ], [ "account", @@ -631,9 +631,9 @@ ], [ "account", - "RESTORATION", + "CONSTRUCTION AND ENVIRONMENTAL COMPLIANCE AND RESTORATION", 94, - 6 + 5 ], [ "account", @@ -655,9 +655,9 @@ ], [ "account", - "CONSTRUCTION", + "MAJOR RESEARCH EQUIPMENT AND FACILITIES CONSTRUCTION", 100, - 9 + 8 ], [ "account", @@ -721,9 +721,9 @@ ], [ "account", - "CORPORATION", + "ADMINISTRATIVE PROVISION\u2014LEGAL SERVICES CORPORATION", 107, - 16 + 15 ], [ "account", diff --git a/tests/data/pdf/anchors_golden/major_vocab.json b/tests/data/pdf/anchors_golden/major_vocab.json index 5896e6b2..261150df 100644 --- a/tests/data/pdf/anchors_golden/major_vocab.json +++ b/tests/data/pdf/anchors_golden/major_vocab.json @@ -1,10 +1,12 @@ { "agriculture": [ "AGRICULTURAL PROGRAMS", + "CORPORATIONS", "DOMESTIC FOOD PROGRAMS", "FARM PRODUCTION AND CONSERVATION PROGRAMS", "FOREIGN ASSISTANCE AND RELATED PROGRAMS", "GENERAL PROVISIONS", + "INDEPENDENT AGENCIES", "RELATED AGENCIES AND FOOD AND DRUG ADMINISTRATION", "RURAL DEVELOPMENT PROGRAMS" ], @@ -26,13 +28,19 @@ "REVOLVING AND MANAGEMENT FUNDS" ], "energy-water": [ + "ATOMIC ENERGY DEFENSE ACTIVITIES NATIONAL NUCLEAR SECURITY ADMINISTRATION", "CORPS OF ENGINEERS\u2014CIVIL", "DEPARTMENT OF ENERGY", "DEPARTMENT OF THE ARMY", "DEPARTMENT OF THE INTERIOR", "ENERGY PROGRAMS", + "ENVIRONMENTAL AND OTHER DEFENSE ACTIVITIES", "GENERAL PROVISIONS", - "INDEPENDENT AGENCIES" + "GENERAL PROVISIONS\u2014CORPS OF ENGINEERS\u2014CIVIL", + "GENERAL PROVISIONS\u2014DEPARTMENT OF ENERGY", + "GENERAL PROVISIONS\u2014INDEPENDENT AGENCIES", + "INDEPENDENT AGENCIES", + "POWER MARKETING ADMINISTRATIONS" ], "financial-services": [ "ADDITIONAL GENERAL PROVISIONS", @@ -54,9 +62,11 @@ ], "interior": [ "DEPARTMENT OF AGRICULTURE", + "DEPARTMENT OF HEALTH AND HUMAN SERVICES", "DEPARTMENT OF THE INTERIOR", "ENVIRONMENTAL PROTECTION AGENCY", "GENERAL PROVISIONS", + "OTHER RELATED AGENCIES", "RELATED AGENCIES" ], "labor-hhs": [ @@ -67,9 +77,18 @@ "RELATED AGENCIES" ], "legislative-branch": [ + "ARCHITECT OF THE CAPITOL", + "CAPITOL POLICE", + "CONGRESSIONAL BUDGET OFFICE", "GENERAL PROVISIONS", + "GOVERNMENT ACCOUNTABILITY OFFICE", + "GOVERNMENT PUBLISHING OFFICE", "HOUSE OF REPRESENTATIVES", - "LEGISLATIVE BRANCH" + "JOHN C. STENNIS CENTER FOR PUBLIC SERVICE TRAINING AND DEVELOPMENT", + "JOINT ITEMS", + "LEGISLATIVE BRANCH", + "LIBRARY OF CONGRESS", + "OFFICE OF CONGRESSIONAL WORKPLACE RIGHTS" ], "milcon-va": [ "DEPARTMENT OF DEFENSE", @@ -85,6 +104,9 @@ "GENERAL PROVISIONS", "INTERNATIONAL SECURITY ASSISTANCE", "MULTILATERAL ASSISTANCE", + "OTHER COMMISSIONS", + "RELATED AGENCY", + "RELATED PROGRAMS", "UNITED STATES AGENCY FOR INTERNATIONAL DEVELOPMENT" ], "transportation-hud": [ diff --git a/tests/data/pdf_canonical_baseline.json b/tests/data/pdf_canonical_baseline.json index 36586c6b..b73a6040 100644 --- a/tests/data/pdf_canonical_baseline.json +++ b/tests/data/pdf_canonical_baseline.json @@ -1,27 +1,27 @@ { "113-hr-3547/1_introduced-in-house->2_engrossed-in-house": { - "bytes": 5313, + "bytes": 5428, "changes": 2, "declined": false, - "sha256": "5ae9864b6b24e6a6f0186456ca90d6eb57fb669a0c28180d930ad9ba0a5eb82f", + "sha256": "e350da24c4b6d37f8d6ce839dda095439dc8bcf4d210b1047f8d81ac0166935e", "summary": { "modified": 2 } }, "113-hr-3547/2_engrossed-in-house->3_received-in-senate": { - "bytes": 4903, + "bytes": 5018, "changes": 2, "declined": false, - "sha256": "ed0901625befb4a8fd8709d48e8a3750073c96c0a32e97f8bb84909b05a13abf", + "sha256": "d8789f10c08047cc1e26476be671507f42fe71cf1ecc6a211fa5e59d81c8dc79", "summary": { "modified": 2 } }, "113-hr-3547/3_received-in-senate->4_engrossed-amendment-senate": { - "bytes": 5984, + "bytes": 6099, "changes": 5, "declined": false, - "sha256": "06c79e44e75172f3c600971a8f504ef6b3eeda8cd248e737120ffa2707df2ac4", + "sha256": "b8a2be8b3372c68445731aff2a427829ce62c1ac0c06d01a1244cf7e34e4ffe5", "summary": { "added": 2, "removed": 3 @@ -32,22 +32,22 @@ "message": "This document has no printed line numbers, so DeltaTrack can't diff it accurately yet \u2014 enrolled bills and public-law prints are typeset this way. Use the XML version of the bill instead." }, "114-hr-2029/1_reported-in-house->3_referred-in-senate": { - "bytes": 288238, + "bytes": 368566, "changes": 26, "declined": false, - "sha256": "10d326513397ba7d939cfc5192c319df8d8376ce606ea7b7957acab1c9b5ac8e", + "sha256": "48a58477faa98af62e43224d766ce03d5fa2ce35b0a117aa7c37bc3dce062ee5", "summary": { "added": 15, "modified": 11 } }, "114-hr-2029/3_referred-in-senate->4_reported-in-senate": { - "bytes": 524861, - "changes": 163, + "bytes": 645567, + "changes": 157, "declined": false, - "sha256": "969bc44f8867f36cd716538cfb7f39610b5200672b14b4b4e6ccedb0e33b2264", + "sha256": "1e2cc9454bc189140fd34d54e2fb4da1877852901199bf9cf27677ba00f8975b", "summary": { - "added": 158, + "added": 152, "modified": 5 } }, @@ -56,35 +56,35 @@ "message": "This document has no printed line numbers, so DeltaTrack can't diff it accurately yet \u2014 enrolled bills and public-law prints are typeset this way. Use the XML version of the bill instead." }, "115-hr-5895/1_reported-in-house->2_engrossed-in-house": { - "bytes": 775561, - "changes": 336, + "bytes": 947193, + "changes": 335, "declined": false, - "sha256": "f4448734cd6ffbb4221f78c11787573e0cf72dff12cdbc1210d7baef66a1713a", + "sha256": "5a5fca818eb0b3199d82422e65c860594b744c9391a469d7c4f178228bc3375b", "summary": { - "added": 317, + "added": 316, "modified": 18, "removed": 1 } }, "115-hr-5895/2_engrossed-in-house->3_placed-on-calendar-senate": { - "bytes": 637672, + "bytes": 900430, "changes": 2, "declined": false, - "sha256": "35f2524afe122bce39eeb1d242ef81bf84ab5853c462af2be225c09ef9ca52ef", + "sha256": "c9b4581f8d6bb9478a45cf0c8c63415b4d85eb2932bd8fa710663403893ee905", "summary": { "modified": 2 } }, "115-hr-5895/3_placed-on-calendar-senate->4_engrossed-amendment-senate": { - "bytes": 1442222, - "changes": 467, + "bytes": 1716868, + "changes": 458, "declined": false, - "sha256": "35d82cf81a170f4cbc6fedf58b607f71ddd072f872022db617054825a1737f1d", + "sha256": "44a40dca54f55dd8f008fe081ea187cc8678b29c0bc625158ac5a38496d14e87", "summary": { "added": 120, - "modified": 194, - "moved": 60, - "removed": 93 + "modified": 205, + "moved": 45, + "removed": 88 } }, "115-hr-5895/4_engrossed-amendment-senate->5_enrolled-bill": { @@ -96,20 +96,20 @@ "message": "This document has no printed line numbers, so DeltaTrack can't diff it accurately yet \u2014 enrolled bills and public-law prints are typeset this way. Use the XML version of the bill instead." }, "117-hr-4502/1_reported-in-house->2_engrossed-in-house": { - "bytes": 4009849, - "changes": 1500, + "bytes": 4802864, + "changes": 1451, "declined": false, - "sha256": "76fa92d4364915d409c8fda147f27cc14b093aaa295c3b4a9f03760dc6003dd7", + "sha256": "4291d25f5239f28d725c8073a8741840c707717c1f068f1930674bfa9329c943", "summary": { - "added": 1451, + "added": 1402, "modified": 49 } }, "118-hr-2882/1_introduced-in-house->4_engrossed-amendment-senate": { - "bytes": 10151, + "bytes": 10266, "changes": 5, "declined": false, - "sha256": "41c1bed3c950b16792999feab2b6d4af6f32adc075841c216d039fead37da6e2", + "sha256": "4595ce60de7ed3081d6a18ed2a99ce71d8362ea752980f3709ba84f4e5f85455", "summary": { "added": 3, "moved": 1, @@ -117,21 +117,21 @@ } }, "118-hr-2882/4_engrossed-amendment-senate->5_engrossed-amendment-house": { - "bytes": 3733327, - "changes": 1794, + "bytes": 4274598, + "changes": 1741, "declined": false, - "sha256": "36a1871fdfeba427cd21dcd24a881769c21e4a45721d2f17c5ace8353e9efdc9", + "sha256": "62ead366decf4265ad2b5726fbbb2df22fc9d25c17eaa253da58dbe1a83d1abb", "summary": { - "added": 1789, + "added": 1736, "modified": 2, "moved": 3 } }, "118-hr-4366/1_reported-in-house->2_engrossed-in-house": { - "bytes": 363539, + "bytes": 450022, "changes": 26, "declined": false, - "sha256": "a2268cec4866ea70a35faaa578a5453ea42be5063a787603878ffb7cd2463d87", + "sha256": "56bea01f893d113df81435fb1d504918bb475f73323f4dee5f585a762aaa3c0f", "summary": { "added": 8, "modified": 17, @@ -139,36 +139,36 @@ } }, "118-hr-4366/2_engrossed-in-house->3_placed-on-calendar-senate": { - "bytes": 324704, + "bytes": 414598, "changes": 2, "declined": false, - "sha256": "b99cd3de3901da47d6ac9923263857f0d0ada79246ffde190eb11940fbc95507", + "sha256": "e35bfb27f004a0034570b0fe00df0dbf0eab15133db90186de0b1ac16606a7a7", "summary": { "modified": 2 } }, "118-hr-4366/3_placed-on-calendar-senate->4_engrossed-amendment-senate": { - "bytes": 2037974, - "changes": 720, + "bytes": 2346348, + "changes": 689, "declined": false, - "sha256": "1e800a39d226173a5550d00f9435c79dfd9c8ada8614d5af6921869765a0081b", + "sha256": "728af428193fd379065512c3eae5fd76bea1482fa0a9a33554c3d6fa59e212cc", "summary": { - "added": 558, - "modified": 99, - "moved": 36, - "removed": 27 + "added": 529, + "modified": 101, + "moved": 33, + "removed": 26 } }, "118-hr-4366/4_engrossed-amendment-senate->5_engrossed-amendment-house": { - "bytes": 4874588, - "changes": 1283, + "bytes": 5675701, + "changes": 1245, "declined": false, - "sha256": "63670038fc80720df7a3281088b0580c1e2a190c09ead2174083b96e43458888", + "sha256": "537d026f88d0d5abd28aea21a375e51db790f46e4d06324ef96f825480fc5d43", "summary": { - "added": 869, + "added": 836, "modified": 277, - "moved": 64, - "removed": 73 + "moved": 61, + "removed": 71 } }, "118-hr-4366/5_engrossed-amendment-house->6_enrolled-bill": { @@ -176,22 +176,22 @@ "message": "This document has no printed line numbers, so DeltaTrack can't diff it accurately yet \u2014 enrolled bills and public-law prints are typeset this way. Use the XML version of the bill instead." }, "118-hr-8752/1_reported-in-house->2_engrossed-in-house": { - "bytes": 390800, - "changes": 37, + "bytes": 475100, + "changes": 38, "declined": false, - "sha256": "c00b7c107d1f5c967a94aa72b6dba4a5391cd60e5f5393a2baf34255856e1f2b", + "sha256": "c16e1f2f3a7699efd1dcd7a7a2f54c4b855b49ef3aa8ecc494ad869a22fe1ea8", "summary": { - "added": 16, + "added": 17, "modified": 18, "moved": 1, "removed": 2 } }, "118-hr-8774/1_reported-in-house->2_engrossed-in-house": { - "bytes": 566404, + "bytes": 699847, "changes": 33, "declined": false, - "sha256": "50f922e02e9bd51b22f41a48a3b7b30c9a06b04b1a7d23c389adfd4f0ecf5e29", + "sha256": "81549d1b4efbd6664d500fc3428fb4e98a1866112a49afc781717efb60e68deb", "summary": { "added": 8, "modified": 22, diff --git a/tests/data/round1_pairing_sentinel.json b/tests/data/round1_pairing_sentinel.json index 43c1e96d..ea9ae33b 100644 --- a/tests/data/round1_pairing_sentinel.json +++ b/tests/data/round1_pairing_sentinel.json @@ -2,163 +2,163 @@ "113-hr-3547/1_introduced-in-house->2_engrossed-in-house": { "new_source_sha256": "5eb6140f808c97d153ee06dd14d82528f8db2b4eada2345954c3783b80490c86", "old_source_sha256": "3e1b7e6e240c6e149b62307931d5628ac57108dd3ad4aee7b3a7ad41c47ac7c9", - "parser_revision": "5831bfb6b12f253bd758cd6a78750cb3e48b5e5663830dfe7f313de1d44c76e2", + "parser_revision": "a9bc09de9a48c5d99982913f269a1ff7013855d6fcb13da92d6fdc03695b6467", "stream_sha256": "640b45a014ad672a1da0d6bdea22ada4b9c2beaad3ee6d8a85ead5e7745fe602" }, "113-hr-3547/2_engrossed-in-house->3_received-in-senate": { "new_source_sha256": "b249d2bc25fa25e7f2f7165c71921a75cd072cacf460edfeb7d6f559e2673c36", "old_source_sha256": "5eb6140f808c97d153ee06dd14d82528f8db2b4eada2345954c3783b80490c86", - "parser_revision": "5831bfb6b12f253bd758cd6a78750cb3e48b5e5663830dfe7f313de1d44c76e2", + "parser_revision": "a9bc09de9a48c5d99982913f269a1ff7013855d6fcb13da92d6fdc03695b6467", "stream_sha256": "640b45a014ad672a1da0d6bdea22ada4b9c2beaad3ee6d8a85ead5e7745fe602" }, "113-hr-3547/3_received-in-senate->4_engrossed-amendment-senate": { "new_source_sha256": "760e4a0022cb1f59a17a2ac63d1892103ba46b944957b33c8315026502e76597", "old_source_sha256": "b249d2bc25fa25e7f2f7165c71921a75cd072cacf460edfeb7d6f559e2673c36", - "parser_revision": "5831bfb6b12f253bd758cd6a78750cb3e48b5e5663830dfe7f313de1d44c76e2", + "parser_revision": "a9bc09de9a48c5d99982913f269a1ff7013855d6fcb13da92d6fdc03695b6467", "stream_sha256": "55c5bb17e6428af4ddcb201b65ee8d831c9268875ae4e40f953ecd6fe453928a" }, "113-hr-3547/4_engrossed-amendment-senate->5_engrossed-amendment-house": { "new_source_sha256": "38154238c82849192779a0d780c5365e09f6ffddb305d723c7a1221dedd708e0", "old_source_sha256": "760e4a0022cb1f59a17a2ac63d1892103ba46b944957b33c8315026502e76597", - "parser_revision": "5831bfb6b12f253bd758cd6a78750cb3e48b5e5663830dfe7f313de1d44c76e2", + "parser_revision": "a9bc09de9a48c5d99982913f269a1ff7013855d6fcb13da92d6fdc03695b6467", "stream_sha256": "61cf9436f4daa241cc5d3c11ac376b19854441445538841e7c7f412013313e26" }, "113-hr-3547/5_engrossed-amendment-house->6_enrolled-bill": { "new_source_sha256": "4a87c798a18f3895d4fe45ba3eafd4c25de210f92a2a0152f4be77f21c0e0bfa", "old_source_sha256": "38154238c82849192779a0d780c5365e09f6ffddb305d723c7a1221dedd708e0", - "parser_revision": "5831bfb6b12f253bd758cd6a78750cb3e48b5e5663830dfe7f313de1d44c76e2", + "parser_revision": "a9bc09de9a48c5d99982913f269a1ff7013855d6fcb13da92d6fdc03695b6467", "stream_sha256": "99f620f1be9d48bc06e5191b010bc26f128568b7053e304c40985721c5db74ad" }, "113-hr-83/6_engrossed-amendment-house->7_enrolled-bill": { "new_source_sha256": "cf255ffe292764b73f0986e8d0d70cb9f679123b70f1462b23e074478f04faec", "old_source_sha256": "8d0bb7cea0c45e565c072498a9c61a1711e76f5dcabed4fed46c4f7d5fdacb83", - "parser_revision": "5831bfb6b12f253bd758cd6a78750cb3e48b5e5663830dfe7f313de1d44c76e2", + "parser_revision": "a9bc09de9a48c5d99982913f269a1ff7013855d6fcb13da92d6fdc03695b6467", "stream_sha256": "ba4a59b15f2351d19695ace0e3b9b1be12edca9d042364c7258156f83eb2cf12" }, "114-hr-2029/1_reported-in-house->3_referred-in-senate": { "new_source_sha256": "c5ebc5bdc835b44c93e7ec758f316cbae23c1fee47e4beb72187ce89bcfe906f", "old_source_sha256": "e5064e773c0b4d1931348e8e84500aca14f286f017b78251ac1cb095902d1310", - "parser_revision": "5831bfb6b12f253bd758cd6a78750cb3e48b5e5663830dfe7f313de1d44c76e2", + "parser_revision": "a9bc09de9a48c5d99982913f269a1ff7013855d6fcb13da92d6fdc03695b6467", "stream_sha256": "cc3cd9655ed64beb5dd4d99a8df4e6c669b72a0fe76032e3704dad8f1fa7152c" }, "114-hr-2029/3_referred-in-senate->4_reported-in-senate": { "new_source_sha256": "68e98a6b0b204cfec7b0d79fdd8e42184fd72cc2872ce93facf6907c1bcbb1a8", "old_source_sha256": "c5ebc5bdc835b44c93e7ec758f316cbae23c1fee47e4beb72187ce89bcfe906f", - "parser_revision": "5831bfb6b12f253bd758cd6a78750cb3e48b5e5663830dfe7f313de1d44c76e2", + "parser_revision": "a9bc09de9a48c5d99982913f269a1ff7013855d6fcb13da92d6fdc03695b6467", "stream_sha256": "5869d9693605782e4846c41b84bf139535ad25b5a0ef850492ebe6cddcb72925" }, "114-hr-2029/4_reported-in-senate->5_engrossed-amendment-senate": { "new_source_sha256": "4a10a81918ce2a61d68c81b114e9238e087cc1811af18193bee7cc85f527bf7d", "old_source_sha256": "68e98a6b0b204cfec7b0d79fdd8e42184fd72cc2872ce93facf6907c1bcbb1a8", - "parser_revision": "5831bfb6b12f253bd758cd6a78750cb3e48b5e5663830dfe7f313de1d44c76e2", + "parser_revision": "a9bc09de9a48c5d99982913f269a1ff7013855d6fcb13da92d6fdc03695b6467", "stream_sha256": "9c1bd850ea02ea75f0ed6de067372c68b8392d59910031fa179ac9c5a1bcb7ab" }, "114-hr-2029/5_engrossed-amendment-senate->6_engrossed-amendment-house": { "new_source_sha256": "c7df6115e0840a1e88a8a2b368de6fd59282a22c13ad17038ce14fcf5852afb5", "old_source_sha256": "4a10a81918ce2a61d68c81b114e9238e087cc1811af18193bee7cc85f527bf7d", - "parser_revision": "5831bfb6b12f253bd758cd6a78750cb3e48b5e5663830dfe7f313de1d44c76e2", + "parser_revision": "a9bc09de9a48c5d99982913f269a1ff7013855d6fcb13da92d6fdc03695b6467", "stream_sha256": "dcf6b1e6d188034efa7e9c2c9f51f476abaf6d823e13c917b95ba8630b7c546e" }, "114-hr-2029/6_engrossed-amendment-house->7_enrolled-bill": { "new_source_sha256": "3828daa3066abead31805cc2a618b9be26226feea42aa44a9af0bcf0460eae06", "old_source_sha256": "c7df6115e0840a1e88a8a2b368de6fd59282a22c13ad17038ce14fcf5852afb5", - "parser_revision": "5831bfb6b12f253bd758cd6a78750cb3e48b5e5663830dfe7f313de1d44c76e2", + "parser_revision": "a9bc09de9a48c5d99982913f269a1ff7013855d6fcb13da92d6fdc03695b6467", "stream_sha256": "94ffdef9de0c03e8c81e535987535db6fd0afa2fa3192cfe9521200291a6494b" }, "115-hr-5895/1_reported-in-house->2_engrossed-in-house": { "new_source_sha256": "e896e2ea0d472790613eebf708fdd032acb48ccf55719a76911225b15ac92c2f", "old_source_sha256": "cfef7e5da3830c7b3a8fa00e63535ddcbce1b9057e6d0678b8d9e3c0ed06f07b", - "parser_revision": "5831bfb6b12f253bd758cd6a78750cb3e48b5e5663830dfe7f313de1d44c76e2", + "parser_revision": "a9bc09de9a48c5d99982913f269a1ff7013855d6fcb13da92d6fdc03695b6467", "stream_sha256": "7058646042b7d87c662866c1e09b0fbdc7d3db4822c8d9f77359e4e505415111" }, "115-hr-5895/2_engrossed-in-house->4_engrossed-amendment-senate": { "new_source_sha256": "1ffb81c413c007f999df117c9509d976c76cbe0c55f63d43dad920d7fe5bf772", "old_source_sha256": "e896e2ea0d472790613eebf708fdd032acb48ccf55719a76911225b15ac92c2f", - "parser_revision": "5831bfb6b12f253bd758cd6a78750cb3e48b5e5663830dfe7f313de1d44c76e2", + "parser_revision": "a9bc09de9a48c5d99982913f269a1ff7013855d6fcb13da92d6fdc03695b6467", "stream_sha256": "e23b25327464fa47f2ebfdee9c84bd852cc6715b90b628951f1b353c58bbcff6" }, "115-hr-5895/4_engrossed-amendment-senate->5_enrolled-bill": { "new_source_sha256": "4be06372badb910c815956bdf89f900e6de77279457c0d1559605a820125d358", "old_source_sha256": "1ffb81c413c007f999df117c9509d976c76cbe0c55f63d43dad920d7fe5bf772", - "parser_revision": "5831bfb6b12f253bd758cd6a78750cb3e48b5e5663830dfe7f313de1d44c76e2", + "parser_revision": "a9bc09de9a48c5d99982913f269a1ff7013855d6fcb13da92d6fdc03695b6467", "stream_sha256": "bb04458743fec9b59143599a03f588e6238c07bc4edd88229f49b301fc446c00" }, "117-hr-2471/1_introduced-in-house->6_enrolled-bill": { "new_source_sha256": "cab7cbd0c02c7c90f384204c85cfbf7d6c2f2c46738536420a5b5ed4a9379288", "old_source_sha256": "4c8619f8e000fa9c32d1a53f0c3dd0a8d37cc88986b1d246e59dd359b50867b3", - "parser_revision": "5831bfb6b12f253bd758cd6a78750cb3e48b5e5663830dfe7f313de1d44c76e2", + "parser_revision": "a9bc09de9a48c5d99982913f269a1ff7013855d6fcb13da92d6fdc03695b6467", "stream_sha256": "67d6b01315690bf5d788c9040bcf8a5629753af6ebbf3dd36cdc1252aac35ca6" }, "117-hr-4502/1_reported-in-house->2_engrossed-in-house": { "new_source_sha256": "70954e5373aa831c1ced2eb7f3f7bc59c84affeadf9061c151a0f0ce77bd7da6", "old_source_sha256": "1cc2b5cbc13691c4b052ffc50a3219ec1f78371d69c3801b9b72be63af84d385", - "parser_revision": "5831bfb6b12f253bd758cd6a78750cb3e48b5e5663830dfe7f313de1d44c76e2", + "parser_revision": "a9bc09de9a48c5d99982913f269a1ff7013855d6fcb13da92d6fdc03695b6467", "stream_sha256": "8363a2b10965578f3d2a00a438855fd73d9153851d2255eb18676f27ac1c7e1d" }, "118-hr-2882/1_introduced-in-house->4_engrossed-amendment-senate": { "new_source_sha256": "cc362514dd8b52e0c19d098efd465291999bb2d21b221c99988fc50a61c6dbdb", "old_source_sha256": "4efacf3f5d8ad72dc679f13ec50c80240d2353a03bba06c3e495c81d049d3016", - "parser_revision": "5831bfb6b12f253bd758cd6a78750cb3e48b5e5663830dfe7f313de1d44c76e2", + "parser_revision": "a9bc09de9a48c5d99982913f269a1ff7013855d6fcb13da92d6fdc03695b6467", "stream_sha256": "94d32b465806eec4ca16535773f661c55bcde962ac1420c41a5851390df23b5f" }, "118-hr-2882/4_engrossed-amendment-senate->5_engrossed-amendment-house": { "new_source_sha256": "9076f2dc075915e1d7219c6f9ae7238544f035eab5633aca787de8f6906b0ac2", "old_source_sha256": "cc362514dd8b52e0c19d098efd465291999bb2d21b221c99988fc50a61c6dbdb", - "parser_revision": "5831bfb6b12f253bd758cd6a78750cb3e48b5e5663830dfe7f313de1d44c76e2", + "parser_revision": "a9bc09de9a48c5d99982913f269a1ff7013855d6fcb13da92d6fdc03695b6467", "stream_sha256": "251f0dc774d6571e43f9b6ca50310ec342edcfc40c23c79a5b9fc27520c50735" }, "118-hr-4366/1_reported-in-house->2_engrossed-in-house": { "new_source_sha256": "66206ecd5984f3e89480c38ff3c7e4938b43e8df6504e9c9251257c9cb074334", "old_source_sha256": "37ef6e4705d6d0ff0153d793b34e2e78f0e49c272e229868265761896d419124", - "parser_revision": "5831bfb6b12f253bd758cd6a78750cb3e48b5e5663830dfe7f313de1d44c76e2", + "parser_revision": "a9bc09de9a48c5d99982913f269a1ff7013855d6fcb13da92d6fdc03695b6467", "stream_sha256": "f79bfc2523ad1de7138efe30c167f3833e77842baaf7040ee7fb439ac4340376" }, "118-hr-4366/2_engrossed-in-house->3_placed-on-calendar-senate": { "new_source_sha256": "ea66f6b12f8e8c522e349da8b3a09be987ddaf4b4eade7518d6ae867b4dee496", "old_source_sha256": "66206ecd5984f3e89480c38ff3c7e4938b43e8df6504e9c9251257c9cb074334", - "parser_revision": "5831bfb6b12f253bd758cd6a78750cb3e48b5e5663830dfe7f313de1d44c76e2", + "parser_revision": "a9bc09de9a48c5d99982913f269a1ff7013855d6fcb13da92d6fdc03695b6467", "stream_sha256": "1dad92903695209366aba84ecd4837e0bd003ba2a63b616adfd2a13a00a7e5c1" }, "118-hr-4366/3_placed-on-calendar-senate->4_engrossed-amendment-senate": { "new_source_sha256": "7f5136031fbdc527798988b2527c9f2125a6c5f8901b44f58fd68c2a336ac80a", "old_source_sha256": "ea66f6b12f8e8c522e349da8b3a09be987ddaf4b4eade7518d6ae867b4dee496", - "parser_revision": "5831bfb6b12f253bd758cd6a78750cb3e48b5e5663830dfe7f313de1d44c76e2", + "parser_revision": "a9bc09de9a48c5d99982913f269a1ff7013855d6fcb13da92d6fdc03695b6467", "stream_sha256": "28249bda889cde983fa23105eb2de049a9634db1ef6567d06d0c6409ec615769" }, "118-hr-4366/4_engrossed-amendment-senate->5_engrossed-amendment-house": { "new_source_sha256": "9b470b0e42d91b2f580f8b9314a2fb88a062df3654665aa6401200ee49f6af24", "old_source_sha256": "7f5136031fbdc527798988b2527c9f2125a6c5f8901b44f58fd68c2a336ac80a", - "parser_revision": "5831bfb6b12f253bd758cd6a78750cb3e48b5e5663830dfe7f313de1d44c76e2", + "parser_revision": "a9bc09de9a48c5d99982913f269a1ff7013855d6fcb13da92d6fdc03695b6467", "stream_sha256": "2829a83aa2590a759cd91b550a5be99aa0dd23b8357393dce3cc20f2c82ac471" }, "118-hr-4366/5_engrossed-amendment-house->6_enrolled-bill": { "new_source_sha256": "63ae3f506d9e06145b3713736e8c3122e90fe1261abcc8a67b19ab726e14cdb9", "old_source_sha256": "9b470b0e42d91b2f580f8b9314a2fb88a062df3654665aa6401200ee49f6af24", - "parser_revision": "5831bfb6b12f253bd758cd6a78750cb3e48b5e5663830dfe7f313de1d44c76e2", + "parser_revision": "a9bc09de9a48c5d99982913f269a1ff7013855d6fcb13da92d6fdc03695b6467", "stream_sha256": "a9d61973ece0b5ee006c06d1e0e83fdde7d8ef674b33ee2d6011080e54bfe259" }, "118-hr-8752/1_reported-in-house->2_engrossed-in-house": { "new_source_sha256": "2664f5b619cde7f6806acd1a470bfe64a451a185120b3ad712bb9e93761ae837", "old_source_sha256": "e4bc6b90eeff87d05eaa5ba61792c8a3b24aebcb86e5d6c98bb03de839be536d", - "parser_revision": "5831bfb6b12f253bd758cd6a78750cb3e48b5e5663830dfe7f313de1d44c76e2", + "parser_revision": "a9bc09de9a48c5d99982913f269a1ff7013855d6fcb13da92d6fdc03695b6467", "stream_sha256": "fc88a42676147c3e41e434564a758003c5ba3b285ba93bcc26a7af4fe240320d" }, "118-hr-8774/1_reported-in-house->2_engrossed-in-house": { "new_source_sha256": "55c3181f80962f367fb9fd3c9bc2c2cd26d97d14c0d7d668c8cdea98acbdca12", "old_source_sha256": "aa5a70c19cc0b59441e906f6b549ae75abd849b75610a2b20e932a8a102ba2e1", - "parser_revision": "5831bfb6b12f253bd758cd6a78750cb3e48b5e5663830dfe7f313de1d44c76e2", + "parser_revision": "a9bc09de9a48c5d99982913f269a1ff7013855d6fcb13da92d6fdc03695b6467", "stream_sha256": "f09e16387a498ab36cf0f0eec1cdc2945a3052ef974542a975e9ce18c258a0a6" }, "118-hr-9468/1_introduced-in-house->4_enrolled-bill": { "new_source_sha256": "ae8e8a24fca374451976b1025ee35d578971ebb7b460b8106c1bdd9a0bdd771a", "old_source_sha256": "9f80b4d65bf0e82e07c4111f78d786e9be07f6dd6f027ce796427dfc37b4fbb6", - "parser_revision": "5831bfb6b12f253bd758cd6a78750cb3e48b5e5663830dfe7f313de1d44c76e2", + "parser_revision": "a9bc09de9a48c5d99982913f269a1ff7013855d6fcb13da92d6fdc03695b6467", "stream_sha256": "63fe11211b0b9fa0cf3ee584cdec705faeeaf5d4bf79c05cc944c6a4f3808030" }, "119-hr-1/1_reported-in-house->2_engrossed-in-house": { "new_source_sha256": "71822471fd717b67495ae467e82923db8d375e10f42f7299459baddde092431a", "old_source_sha256": "c837c7f62418fda8c22d19a11a40c12f99261e144634509b5e0c042242630858", - "parser_revision": "5831bfb6b12f253bd758cd6a78750cb3e48b5e5663830dfe7f313de1d44c76e2", + "parser_revision": "a9bc09de9a48c5d99982913f269a1ff7013855d6fcb13da92d6fdc03695b6467", "stream_sha256": "a5c3da51c69ccc671f17117aeb24d26fcfba8d7905f09eeb74cfd9e0e26763d1" } } diff --git a/tests/ledger_location.py b/tests/ledger_location.py new file mode 100644 index 00000000..8165b76c --- /dev/null +++ b/tests/ledger_location.py @@ -0,0 +1,293 @@ +"""Ledger location: where the PDF files each dollar amount, scored against the XML twin (ADR 0022). + +The ledger of a bill version is every dollar amount it contains, in document order, each with +its location. On the XML side the location is the node's ``display_path``; on the PDF side it is +the breadcrumb of the anchor block the amount sits in. The two amount sequences are aligned in +document order on their values, and each matched amount is scored by comparing the two +locations, cut after the section (sub-section depth is not heading structure). Every level of +the path is compared, not just the account and its parent: + +- ``T0`` the same path. +- ``T1`` the same levels in the same order, a label differs: a joined, tail or near-variant name. +- ``T2`` true but shallower: the PDF's ancestors are the XML's, in order, with some left out. +- ``T3`` right heading, wrong parent: an ancestor that is wrong, extra or out of order. +- ``T4`` filed under a different heading. +- ``MISS`` an XML amount the alignment could not place in the PDF sequence. + +Two readings come out of this module, and they answer different questions: + +- **The tier totals** (``score``) are the formal result. They are pinned per version by + ``tests/test_pdf_ledger_location.py``, and a heading change is judged and optimized by them. +- **The per-amount check** (``--save`` / ``--against``) is informal. It follows each amount from + one parser to another and lists the ones whose tier got worse, which a total hides when other + amounts improve. It is how a regression is found and explained; it is not pinned and is not a + gate. Both runs are graded by this module's ``tier``, so a scorer change cannot pass for a + parser change. + +The XML is read here, at test time, and nowhere at runtime (ADR 0005, 0010, 0011). +""" + +from __future__ import annotations + +import difflib +import re +from collections import Counter +from pathlib import Path + +from deltatrack.amounts import extract_amounts +from deltatrack.bill_tree import amount_text, normalize_bill, normalize_header +from deltatrack.parsers.pdf_anchors import breadcrumb_for, extract_anchors +from deltatrack.parsers.pdf_blocks import _flatten, _group_into_blocks +from tests.pdf_corpus import cached_pages, dual_format_versions + +TIERS = ("T0", "T1", "T2", "T3", "T4", "MISS") +NOT_TOLERATED = ("T3", "T4", "MISS") + +# A structural level printed with its own number, possibly followed by its name on the same +# element ("TITLE I—DEPARTMENT OF DEFENSE", "Division A: …"). +_LEVEL = re.compile(r"^(title|division|subtitle|part|chapter|subchapter)\s+([ivxlcdm]+|\d+|[a-z])\b") +_SECTION = re.compile(r"^sec(?:tion)?\.?\s*([0-9]+[a-z]?)") +_NEAR_VARIANT = 0.9 + + +def location(path: tuple[str, ...] | list[str]) -> tuple[str, ...]: + """Comparable location keys for a path, cut after the section. + + A level element that carries its own name yields two keys, the level and the name, because + the PDF prints the name as its own line under the level. + """ + out: list[str] = [] + for element in path: + s = " ".join(str(element).lower().split()) + level = _LEVEL.match(s) + if level: + out.append(f"{level.group(1)} {level.group(2)}") + name = normalize_header(s[level.end() :].lstrip(" —–-:.")) + if name: + out.append(name) + continue + section = _SECTION.match(s) + if section: + out.append(f"sec {section.group(1)}") + break + key = normalize_header(s) + if key: + out.append(key) + return tuple(out) + + +def _variant(a: str | None, b: str | None) -> bool: + if not a or not b: + return False + return ( + a.endswith(" " + b) + or b.endswith(" " + a) + or a.startswith(b + " ") + or b.startswith(a + " ") + or difflib.SequenceMatcher(None, a, b).ratio() >= _NEAR_VARIANT + ) + + +def _same(a: str, b: str) -> bool: + return a == b or _variant(a, b) + + +def tier(xml_loc: tuple[str, ...], pdf_loc: tuple[str, ...]) -> str: + """The tier of one amount filed at ``pdf_loc`` where the XML files it at ``xml_loc``. + + Every level is compared, not just the account and its parent: a wrong, extra or reordered + ancestor anywhere above the account is a wrong parent (T3). + """ + if xml_loc == pdf_loc: + return "T0" + if not xml_loc or not pdf_loc or not _same(xml_loc[-1], pdf_loc[-1]): + return "T4" + x_up, p_up = xml_loc[:-1], pdf_loc[:-1] + if len(x_up) == len(p_up) and all(_same(x, p) for x, p in zip(x_up, p_up, strict=True)): + return "T1" + j = 0 + for p in p_up: # the PDF's ancestors must be the XML's, in order, with some left out + while j < len(x_up) and x_up[j] != p: + j += 1 + if j == len(x_up): + return "T3" + j += 1 + return "T2" + + +def xml_ledger(xml_path: Path) -> list[tuple[int, tuple[str, ...]]]: + return [ + (value, location(node.display_path)) + for node in normalize_bill(xml_path).nodes + for value in extract_amounts(amount_text(node)) + ] + + +def pdf_ledger(pdf_path: Path) -> list[tuple[int, tuple[str, ...]]]: + pages = cached_pages(pdf_path) + anchors = extract_anchors(pages) + out = [] + for block in _group_into_blocks(_flatten(pages), anchors): + loc = location(breadcrumb_for(block.anchor, anchors)) if block.anchor is not None else () + out.extend((value, loc) for value in extract_amounts(block.text)) + return out + + +Located = tuple[int, tuple[str, ...], tuple[str, ...] | None] + + +def aligned(xml_path: Path, pdf_path: Path) -> list[Located]: + """Every XML amount in document order: its value, its XML location, and the location of the + PDF amount the alignment pairs it with (``None`` where it pairs with none, a ``MISS``).""" + xml, pdf = xml_ledger(xml_path), pdf_ledger(pdf_path) + out: list[Located] = [(value, loc, None) for value, loc in xml] + matcher = difflib.SequenceMatcher(None, [v for v, _ in xml], [v for v, _ in pdf], autojunk=False) + for block in matcher.get_matching_blocks(): + for k in range(block.size): + value, loc = xml[block.a + k] + out[block.a + k] = (value, loc, pdf[block.b + k][1]) + return out + + +def _graded(xml_loc: tuple[str, ...], pdf_loc: tuple[str, ...] | None) -> str: + return "MISS" if pdf_loc is None else tier(xml_loc, pdf_loc) + + +def score(xml_path: Path, pdf_path: Path) -> dict[str, int]: + """Tier counts for one bill version (every tier present, zero when empty).""" + tally = Counter({t: 0 for t in TIERS}) + tally.update(_graded(x, p) for _, x, p in aligned(xml_path, pdf_path)) + return dict(tally) + + +_RANK = {t: i for i, t in enumerate(TIERS)} + + +def compare_amounts( + before: dict[str, list[Located]], after: dict[str, list[Located]] +) -> tuple[Counter[str], dict[tuple[str, ...], list[int]], list[str]]: + """Amount by amount, how ``after`` files each amount against ``before`` (the informal check). + + Both are ``aligned`` output per version. Returns the count of moved amounts by direction + (``better``, ``worse``, ``same tier``), the worse amounts grouped by version and the three + locations (XML, before, after) with their values, and the versions that could not be compared + because their XML side differs (the check follows one XML ledger through two parsers). + """ + moved: Counter[str] = Counter() + worse: dict[tuple[str, ...], list[int]] = {} + skipped = [] + for key in sorted(before.keys() & after.keys()): + b, a = before[key], after[key] + if [(v, tuple(x)) for v, x, _ in b] != [(v, tuple(x)) for v, x, _ in a]: + skipped.append(key) + continue + for (value, x, pb), (_, _, pa) in zip(b, a, strict=True): + x, pb, pa = tuple(x), None if pb is None else tuple(pb), None if pa is None else tuple(pa) + if pb == pa: + continue + tb, ta = _graded(x, pb), _graded(x, pa) + direction = "worse" if _RANK[ta] > _RANK[tb] else "better" if _RANK[ta] < _RANK[tb] else "same tier" + moved[direction] += 1 + if direction == "worse": + where = ( + key, + f"{tb}->{ta}", + " > ".join(x), + " > ".join(pb or ("(none)",)), + " > ".join(pa or ("(none)",)), + ) + worse.setdefault(where, []).append(value) + return moved, worse, skipped + + +def committed_versions() -> dict[str, tuple[Path, Path]]: + """``bill/version`` -> (xml, pdf) for every committed dual-format version the PDF pipeline + accepts. Enrolled versions are left out: their unnumbered layout is declined.""" + return {f"{bill}/{xml.stem}": (xml, pdf) for bill, xml, pdf in dual_format_versions() if "enrolled" not in xml.stem} + + +def _versions_under(root: Path) -> dict[str, tuple[Path, Path]]: + """``bill/version`` -> (xml, pdf) for the dual-format, non-enrolled versions under ``root`` + (one directory per bill, as ``fetch_bills.py download`` writes them).""" + return { + f"{xml.parent.name}/{xml.stem}": (xml, xml.with_suffix(".pdf")) + for xml in sorted(root.glob("*/*.xml")) + if xml.with_suffix(".pdf").exists() and "enrolled" not in xml.stem + } + + +def _report(argv: list[str] | None = None) -> None: + """Tier totals for whichever ``deltatrack`` is importable, against the pinned baseline, and + optionally the per-amount check against an earlier run. + + The same scorer gives a change's before and after (ADR 0022): export the parser to compare + (``git archive origin/develop src | tar -x -C /tmp/before``), run with + ``PYTHONPATH=/tmp/before/src`` and ``--save before.json``, then without it and with + ``--against before.json``. Totals are split by whether the XML carries appropriations + headings, since a reconciliation bill has no account level to agree on. + """ + import argparse + import json + + import deltatrack + from tests.corpus_paths import DATA_DIR + + parser = argparse.ArgumentParser(prog="python -m tests.ledger_location") + parser.add_argument("--save", type=Path, help="write every amount's locations here, for a later --against") + parser.add_argument("--against", type=Path, help="list the amounts filed worse than in this --save file") + parser.add_argument( + "--extra", type=Path, action="append", default=[], help="also score the versions under this bills/ directory" + ) + parser.add_argument("--show", type=int, default=20, help="how many groups of worse amounts to print") + args = parser.parse_args(argv) + + versions = {} + for root in args.extra: + versions.update(_versions_under(root)) + versions.update(committed_versions()) + pinned = json.loads((DATA_DIR / "ledger_location_baseline.json").read_text(encoding="utf-8")) + groups: dict[str, list[dict[str, int]]] = {} + ledgers: dict[str, list[Located]] = {} + worse, better = [], [] + for key, (xml, pdf) in sorted(versions.items()): + ledgers[key] = aligned(xml, pdf) + tally = Counter({t: 0 for t in TIERS}) + tally.update(_graded(x, p) for _, x, p in ledgers[key]) + live = dict(tally) + appropriations = any(n.tag.startswith("appropriations-") for n in normalize_bill(xml).nodes) + groups.setdefault("appropriations" if appropriations else "other", []).append(live) + was = pinned.get(key) + if was: + live_bad, was_bad = sum(live[t] for t in NOT_TOLERATED), sum(was[t] for t in NOT_TOLERATED) + if live_bad > was_bad or live["T0"] < was["T0"]: + worse.append(key) + if live_bad < was_bad or live["T0"] > was["T0"]: + better.append(key) + print(f"deltatrack: {Path(deltatrack.__file__).parent}") + for name, tallies in sorted(groups.items()): + totals = sum((Counter(t) for t in tallies), Counter()) + n = sum(totals.values()) + ok = totals["T0"] + totals["T1"] + totals["T2"] + print(f"{name}: {len(tallies)} versions, {n} amounts, OK (T0-T2) {ok / n:.1%}") + for t in TIERS: + print(f" {t:<4} {totals[t]:>6} {totals[t] / n:6.1%}") + # Run on the base branch, "better" must be empty for "no version got worse" to hold. + print(f"versions worse than the pinned baseline: {len(worse)} {worse}") + print(f"versions better than the pinned baseline: {len(better)} {better}") + + if args.save: + args.save.write_text(json.dumps(ledgers), encoding="utf-8") + print(f"saved {sum(map(len, ledgers.values()))} amounts in {len(ledgers)} versions to {args.save}") + if args.against: + before = json.loads(args.against.read_text(encoding="utf-8")) + moved, worse_amounts, skipped = compare_amounts(before, ledgers) + print(f"per amount against {args.against}: {dict(moved)}") + if skipped: + print(f" not compared, XML side differs: {skipped}") + for (key, change, x, b, a), values in sorted(worse_amounts.items(), key=lambda kv: -len(kv[1]))[: args.show]: + print(f" {len(values):>4} amounts {key} {change}") + print(f" XML {x}\n before {b}\n after {a}") + + +if __name__ == "__main__": + _report() diff --git a/tests/test_financial.py b/tests/test_financial.py new file mode 100644 index 00000000..c5c0ff5b --- /dev/null +++ b/tests/test_financial.py @@ -0,0 +1,232 @@ +"""The ledger's rules, one per test, on synthetic text (ADR 0023). + +`deltatrack.financial` reads each money-bearing section of a version's structure tree, +splits it into clauses, types each clause and keeps every amount. These tests pin the +reading of the text (what a PDF block looks like once its print furniture is gone) and the +one safety rule the views depend on: an amount inside an amendment to another law is never +money given out. The corpus-level parity with the research notebook is in +`tests/test_financial_corpus.py`. +""" + +from __future__ import annotations + +from deltatrack.financial import ( + _amended_law_ranges, + _in_amended_law, + _prose_blocks, + classify_text, + financial_for, + given_out, + section_ledger, +) +from deltatrack.formatters.financial_views import comparison_rows + + +def pdf_text(*lines: tuple[int | None, str]) -> str: + """PDF canonical full_text: each line behind a 7-char line-number gutter.""" + return "\n".join((f"{n:>5}" if n is not None else " " * 5) + " " + text for n, text in lines) + + +def blocks(text: str, *, guttered: bool = True) -> list[str]: + return [b for b, _offsets in _prose_blocks(text, 0, len(text), guttered=guttered)] + + +def node(text: str, label: str = "OPERATIONS AND SUPPORT", level: str = "account") -> list[dict]: + return [{"label": label, "level": level, "full_text_span": {"start": 0, "end": len(text)}, "children": []}] + + +class TestReadingThePdfBlock: + def test_the_gutter_and_the_opening_headings_come_off(self): + text = pdf_text( + (10, "MILITARY CONSTRUCTION, ARMY"), + (11, "(INCLUDING TRANSFER OF FUNDS)"), + (12, "For acquisition, construction, installation,"), + (13, "$1,517,455,000, to remain available."), + ) + assert blocks(text) == ["For acquisition, construction, installation, $1,517,455,000, to remain available."] + + def test_every_character_maps_back_to_the_embedded_text(self): + # The views slice the embedded text by these offsets instead of carrying a copy. + text = pdf_text((1, "HEADING"), (2, "For necessary"), (3, "expenses, $5,000.")) + [(clean, offsets)] = _prose_blocks(text, 0, len(text), guttered=True) + assert len(offsets) == len(clean) + assert [text[o] for o, ch in zip(offsets, clean) if ch != " "] == [ch for ch in clean if ch != " "] + assert offsets == sorted(offsets) + + def test_the_running_header_is_skipped(self): + # `† HR 4366 EAS`, unnumbered, sits between two lines of one sentence. + text = pdf_text( + (23, "SEC. 111. None of the funds may be obligated unless"), + (None, "† HR 4366 EAS"), + (1, "such contracts are awarded to United States firms, $500,000."), + ) + assert blocks(text) == [ + "SEC. 111. None of the funds may be obligated unless such contracts are awarded to" + " United States firms, $500,000." + ] + + def test_a_word_broken_across_lines_is_rejoined(self): + # Within a page `pdf_text` already rejoins; this is the break at a page seam. + text = pdf_text((22, "amounts, speci-"), (1, "fied in the table, $52,683,000.")) + assert blocks(text) == ["amounts, specified in the table, $52,683,000."] + + def test_a_real_compound_keeps_its_hyphen(self): + text = pdf_text((1, "the Child-"), (2, "Rescue program, $5,000.")) + assert blocks(text) == ["the Child- Rescue program, $5,000."] + + def test_a_heading_after_a_sentence_starts_a_new_paragraph(self): + # FEDERAL-AID HIGHWAYS prints its limitation, then its payment under a qualifier. + text = pdf_text( + (9, "Funds shall not exceed $60,095,782,888 for fiscal year 2024."), + (17, "(LIQUIDATION OF CONTRACT AUTHORIZATION)"), + (19, "For the payment of obligations, $60,792,659,888 shall be derived."), + ) + assert blocks(text) == [ + "Funds shall not exceed $60,095,782,888 for fiscal year 2024.", + "For the payment of obligations, $60,792,659,888 shall be derived.", + ] + + def test_an_all_caps_line_mid_sentence_is_a_continuation(self): + text = pdf_text( + (1, "(7) $68,438.40 from funds available in the account"), + (2, "(69 X 0641);"), + (3, "(8) $133,231.12 from funds available."), + ) + assert blocks(text) == [ + "(7) $68,438.40 from funds available in the account (69 X 0641); (8) $133,231.12 from funds available." + ] + + def test_a_subsection_after_a_sentence_starts_a_new_paragraph(self): + text = pdf_text( + (1, "SEC. 5. (a) None of the funds, $500,000, may be used."), + (2, "(b) None of the funds, $500,000, may be used."), + ) + assert blocks(text) == [ + "SEC. 5. (a) None of the funds, $500,000, may be used.", + "(b) None of the funds, $500,000, may be used.", + ] + + def test_a_subsection_inside_a_quotation_does_not(self): + text = pdf_text( + (1, "is amended to read: ‘‘(a) The Secretary shall pay $5."), (2, "(b) The Secretary shall pay $6.’’.") + ) + assert len(blocks(text)) == 1 + + def test_xml_text_is_joined_and_not_split(self): + # The XML reader gives each subsection its own node already. + text = "For expenses, $5,000:\n (b) and $6,000." + assert blocks(text, guttered=False) == ["For expenses, $5,000: (b) and $6,000."] + + +class TestClassifying: + def test_quote_marks_do_not_push_a_clause_past_a_rule(self): + # The XML the rules were built on carries no quote marks here; the PDF prints GPO's. + lead = "Of the amounts appropriated to the Department for " + names = ", ".join(f"''Account {i}''" for i in range(20)) + text = pdf_text((1, lead + names + ", up to $430,532,000 may be transferred to the Fund.")) + [section] = section_ledger(text, node(text, "SEC. 219", "section"), guttered=True) + assert section["clauses"][0]["type"] == "transfer" + assert classify_text(lead + names.replace("''", "") + ", up to $430,532,000 may be transferred") == "transfer" + + def test_the_section_enumerator_is_not_read(self): + text = pdf_text((1, "SEC. 101. None of the funds made available, $5,000, may be used.")) + [section] = section_ledger(text, node(text, "SEC. 101", "section"), guttered=True) + assert section["clauses"][0]["type"] == "restriction" + + def test_a_ceiling_is_not_the_clause_amount(self): + text = pdf_text((1, "For expenses, not to exceed $2,000, $9,000.")) + [section] = section_ledger(text, node(text), guttered=True) + clause = section["clauses"][0] + assert clause["amount"] == 9000 and [a["cap"] for a in clause["amounts"]] == [True, False] + + +class TestAmendedLaw: + def test_a_quoted_amount_is_in_amended_law(self): + text = "Section 5 is amended by striking ‘‘$1,500’’ and inserting ‘‘$2,000’’." + ranges = _amended_law_ranges(text) + assert all(_in_amended_law(text, text.index(v), ranges) for v in ("$1,500", "$2,000")) + + def test_text_after_an_amendment_lead_in_is_in_amended_law(self): + # The XML serializer drops the quote marks; the lead-in stands in for them. + text = "Section 3(u) is amended to read as follows: (u) the plan costs $12,000." + assert _in_amended_law(text, text.index("$12,000"), _amended_law_ranges(text)) + + def test_an_amount_the_bill_gives_out_is_not(self): + text = "For necessary expenses, $5,000, to remain available." + assert not _in_amended_law(text, text.index("$5,000"), _amended_law_ranges(text)) + + def test_an_amended_law_amount_is_never_money_given_out(self): + assert given_out({"type": "appropriation", "amount": 5, "in_amended_law": True}) is None + assert given_out({"type": "appropriation", "amount": 5, "in_amended_law": False}) == 5 + assert given_out({"type": "rescission", "amount": 5, "in_amended_law": False}) == -5 + assert given_out({"type": "cap", "amount": 5, "in_amended_law": False}) is None + + +class TestFlags: + def test_a_further_section_inside_the_prose_is_flagged(self): + # The parser did not start `SEC. 119A.`, so its money rides in SEC. 119's row. + text = pdf_text( + (1, "SEC. 119. None of the funds, $5, may be used."), (2, "SEC. 119A. None of the funds, $6, may be used.") + ) + [section] = section_ledger(text, node(text, "SEC. 119", "section"), guttered=True) + assert section["flags"] == ["may_hold_several_sections"] + + def test_the_section_opening_its_own_block_is_not(self): + text = pdf_text((1, "SEC. 119. None of the funds, $5, may be used.")) + [section] = section_ledger(text, node(text, "SEC. 119", "section"), guttered=True) + assert section["flags"] == [] + + +class TestComparison: + def _canonical(self, a_text: str, b_text: str) -> dict: + full_text = {"v1": a_text, "v2": b_text} + tree = {"v1": node(a_text, "SEC. 5", "section"), "v2": node(b_text, "SEC. 5", "section")} + return { + "full_text": full_text, + "financial": financial_for(full_text, tree, "xml"), + "changes": [ + { + "change_type": "modified", + "path": {"v1": ["SEC. 5"], "v2": ["SEC. 5"]}, + "full_text_span": {"v1": {"start": 0, "end": len(a_text)}, "v2": {"start": 0, "end": len(b_text)}}, + } + ], + } + + def test_a_row_is_the_change_whose_section_holds_money(self): + canonical = self._canonical("For expenses, $5,000.", "For expenses, $7,000.") + [row] = comparison_rows(canonical) + assert row["change_index"] == 0 and len(row["a"]) == len(row["b"]) == 1 + + def test_an_amended_law_amount_never_enters_the_difference(self): + # Both versions quote a figure in the law they amend; only the quoted figure moves. + from deltatrack.formatters.financial_views import _difference_html + + canonical = self._canonical( + "Section 5 of the Act is amended to read as follows: (a) The limit is $1,500.", + "Section 5 of the Act is amended to read as follows: (a) The limit is $2,000.", + ) + [row] = comparison_rows(canonical) + html, delta = _difference_html(row) + assert delta == 0 and "not given-out money" in html + + def test_the_comparison_export_is_the_tables_own_rows(self): + from deltatrack.formatters.financial_views import _difference_html, comparison_export_rows + + canonical = self._canonical("For expenses, $5,000.", "For expenses, $7,000.") + [row] = comparison_rows(canonical) + [entry] = comparison_export_rows(canonical) + assert entry["difference"] == _difference_html(row)[1] == 2000 + assert (entry["a"]["amounts"], entry["b"]["amounts"]) == ([5000], [7000]) + assert entry["change_index"] == row["change_index"] and entry["location"] == "SEC. 5" + + def test_the_comparison_export_says_why_there_is_no_difference(self): + from deltatrack.formatters.financial_views import comparison_export_rows + + canonical = self._canonical( + "Section 5 of the Act is amended to read as follows: (a) The limit is $1,500.", + "Section 5 of the Act is amended to read as follows: (a) The limit is $2,000.", + ) + [entry] = comparison_export_rows(canonical) + assert entry["difference"] is None and entry["difference_note"] == "not given-out money" + assert (entry["a"]["amended_law_amounts"], entry["b"]["amended_law_amounts"]) == ([1500], [2000]) diff --git a/tests/test_financial_corpus.py b/tests/test_financial_corpus.py new file mode 100644 index 00000000..bb27eda3 --- /dev/null +++ b/tests/test_financial_corpus.py @@ -0,0 +1,197 @@ +"""The ledger on real bills: PDF sections give the research notebook's values (ADR 0023). + +The notebook proved its financial rows on XML (`docs/research/financial-semantics/`). The +product reads PDF, so the bar is that a version's PDF sections give the same rows: same +clauses, same types, same amounts, same review flags, in the same order. The rows are frozen +in `tests/data/financial_rows/` under the classifier version that produced them, which is +what makes the version a gate: rules that move any row need a new version (see +`deltatrack.financial.CLASSIFIER`). + +Regenerate the frozen rows after a deliberate classifier change, with its version bumped: + + UPDATE_FINANCIAL_ROWS=1 uv run pytest tests/test_financial_corpus.py + +The same run refuses to write different rows under an unchanged version. + +Marked @slow: it parses committed corpus PDFs, the three-bill Senate amendment among them. +""" + +from __future__ import annotations + +import difflib +import json +import os +import re +import sys +from functools import lru_cache + +import jsonschema +import pytest + +from deltatrack.bill_tree import normalize_bill +from deltatrack.compare.pdf import _build_canonical +from deltatrack.compare.xml import compare_xml +from deltatrack.diff_pdf import diff_pdfs +from deltatrack.financial import CLASSIFIER +from deltatrack.formatters.diff_html import format_diff_html +from deltatrack.formatters.financial_views import comparison_rows +from tests.corpus_paths import DATA_DIR, FIXTURES_DIR, PROJECT_ROOT +from tests.pdf_corpus import cached_pages + +pytestmark = pytest.mark.slow + +BILL = "118-hr-4366" +#: Version -> the next version it is diffed against (the ledger read is one side's only). +PAIRS = { + "1_reported-in-house": "2_engrossed-in-house", + "4_engrossed-amendment-senate": "5_engrossed-amendment-house", +} +ROWS_DIR = DATA_DIR / "financial_rows" +REGENERATE = "UPDATE_FINANCIAL_ROWS=1 uv run pytest tests/test_financial_corpus.py" + +#: Where PDF and XML still differ on the Senate amendment, and why: the PDF parser does not +#: start a section at a lettered number (`SEC. 109A.`, `SEC. 119A.`, `SEC. 119B.`), so those +#: sections' clauses ride in the section before them. Each such row is flagged on the page +#: ("may hold more than one section"). Pinned exactly: a parser fix raises it, and should. +PDF_ROWS_MATCHED = {"1_reported-in-house": 108, "4_engrossed-amendment-senate": 619} +PDF_FLAGGED_SECTIONS = {"1_reported-in-house": [], "4_engrossed-amendment-senate": ["SEC. 109", "SEC. 119"]} + + +@lru_cache(maxsize=None) +def canonical(kind: str, stem: str) -> dict: + old = FIXTURES_DIR / BILL / f"{stem}.{kind}" + new = FIXTURES_DIR / BILL / f"{PAIRS[stem]}.{kind}" + if kind == "xml": + return compare_xml(old.read_bytes(), new.read_bytes()) + old_pages, new_pages = cached_pages(old), cached_pages(new) + return _build_canonical(diff_pdfs(old_pages, new_pages), old_pages, new_pages, stem, PAIRS[stem]) + + +def rows(ledger: dict) -> list[list[list]]: + """Per section, per clause: [level, type, amount, needs_review].""" + return [[[c["level"], c["type"], c["amount"], c["needs_review"]] for c in s["clauses"]] for s in ledger["sections"]] + + +def flat(sections: list[list[list]]) -> list[tuple]: + return [tuple(c) for s in sections for c in s] + + +def frozen(stem: str) -> dict: + return json.loads((ROWS_DIR / f"{BILL}-{stem}.json").read_text(encoding="utf-8")) + + +@pytest.mark.skipif(os.environ.get("UPDATE_FINANCIAL_ROWS") != "1", reason="not in rows-update mode") +@pytest.mark.parametrize("stem", PAIRS) +def test_regenerate_the_frozen_rows(stem): + live = rows(canonical("xml", stem)["financial"]["v1"]) + path = ROWS_DIR / f"{BILL}-{stem}.json" + if path.exists(): + was = json.loads(path.read_text(encoding="utf-8")) + assert was["classifier"] != CLASSIFIER or was["sections"] == live, ( + f"the ledger's rows changed but deltatrack.financial.CLASSIFIER is still {CLASSIFIER!r}: " + "bump it and add its changelog line, then regenerate" + ) + ROWS_DIR.mkdir(exist_ok=True) + # One clause per line, so a classifier change reviews as a line diff of the rows it moved. + sections = ",\n".join( + " [\n" + ",\n".join(f" {json.dumps(c)}" for c in s) + "\n ]" if s else " []" for s in live + ) + head = json.dumps({"bill": BILL, "version": stem, "classifier": CLASSIFIER})[:-1] + path.write_text(f'{head}, "sections": [\n{sections}\n]}}\n', encoding="utf-8", newline="\n") + assert json.loads(path.read_text(encoding="utf-8"))["sections"] == live + + +@pytest.mark.parametrize("stem", PAIRS) +def test_the_frozen_rows_are_the_running_classifiers(stem): + assert frozen(stem)["classifier"] == CLASSIFIER, ( + f"deltatrack.financial.CLASSIFIER is {CLASSIFIER!r} but the frozen rows are " + f"{frozen(stem)['classifier']!r}'s; regenerate them with {REGENERATE}" + ) + + +@pytest.mark.parametrize("stem", PAIRS) +def test_the_xml_ledger_gives_the_frozen_rows(stem): + assert rows(canonical("xml", stem)["financial"]["v1"]) == frozen(stem)["sections"] + + +@pytest.mark.parametrize("stem", PAIRS) +def test_the_pdf_ledger_gives_the_frozen_rows(stem): + live, want = flat(rows(canonical("pdf", stem)["financial"]["v1"])), flat(frozen(stem)["sections"]) + matched = sum(b.size for b in difflib.SequenceMatcher(None, live, want, autojunk=False).get_matching_blocks()) + assert matched == PDF_ROWS_MATCHED[stem], ( + f"{matched} of {len(want)} clause rows match; pinned at {PDF_ROWS_MATCHED[stem]}" + ) + + def appropriated(clauses): + return sum(c[2] or 0 for c in clauses if c[1] == "appropriation") + + assert appropriated(live) == appropriated(want) + + +@pytest.mark.parametrize("stem", PAIRS) +def test_the_rows_that_may_hold_several_sections_are_flagged(stem): + sections = canonical("pdf", stem)["financial"]["v1"]["sections"] + flagged = [s["path"][-1][0] for s in sections if "may_hold_several_sections" in s["flags"]] + assert flagged == PDF_FLAGGED_SECTIONS[stem] + + +def test_the_frozen_rows_at_1_0_are_the_notebooks_own(): + """Provenance of the handoff point: version 1.0 is the notebook's rules, moved unchanged, + so at 1.0 the frozen rows are exactly what the notebook computes from the XML.""" + if CLASSIFIER != "1.0": + pytest.fail("the classifier has moved past 1.0: retire this test and record its last rows in the changelog") + sys.path.insert(0, str(PROJECT_ROOT / "docs" / "research" / "financial-semantics")) + import classify_bill as notebook + + for stem in PAIRS: + groups = [] + for n in normalize_bill(FIXTURES_DIR / BILL / f"{stem}.xml").nodes: + if not notebook.DOLLAR.search(n.body_text or ""): + continue + section_type = notebook.classify_text(n.body_text) + clauses = [] + for clause, level in notebook.split_clauses(n.body_text): + if not notebook.DOLLAR.search(clause): + continue + m = notebook.DOLLAR.search(notebook.CAP_AMOUNT_RE.sub("", clause)) or notebook.DOLLAR.search(clause) + value = float(m.group(1).replace(",", "")) + clauses.append( + [ + level, + section_type if level == "primary" else notebook.classify_text(clause), + int(value) if value.is_integer() else value, + len(notebook.non_cap_amounts(clause)) > 1, + ] + ) + groups.append(clauses) + assert groups == frozen(stem)["sections"], stem + + +@pytest.mark.parametrize("kind", ["pdf", "xml"]) +@pytest.mark.parametrize("stem", PAIRS) +def test_every_dollar_amount_in_the_text_is_on_the_page(kind, stem): + for side in ("v1", "v2"): + ledger = canonical(kind, stem)["financial"][side] + shown = sum(len(c["amounts"]) for s in ledger["sections"] for c in s["clauses"]) + assert shown == ledger["amounts_in_text"], f"{kind} {side}: {shown} of {ledger['amounts_in_text']}" + + +@pytest.mark.parametrize("kind", ["pdf", "xml"]) +def test_the_document_carries_a_valid_ledger(kind): + schema = json.loads((PROJECT_ROOT / "schema" / "canonical-diff.schema.json").read_text(encoding="utf-8")) + jsonschema.validate(canonical(kind, "1_reported-in-house"), schema) + + +@pytest.mark.parametrize("kind", ["pdf", "xml"]) +def test_the_report_renders_the_three_views_wired_to_the_changes(kind): + doc = canonical(kind, "1_reported-in-house") + html = format_diff_html(doc) + for view in ("fin-a", "fin-b", "fin-compare"): + assert f'data-view="{view}"' in html and f"view-{view}" in html + for side in ("v1", "v2"): + table = html.split(f'', 1)[1].split("
", 1)[0] + assert table.count('class="fin-row"') == len(doc["financial"][side]["sections"]) + # Every comparison row opens onto a card the Changes view has: no matching of its own. + wired = re.findall(r'', 1)[1].split('
Financial Summary") == 2 # Version A and Version B, nowhere else + assert html.count("Financial Summary") == 2 diff --git a/tests/test_formatters_canonical.py b/tests/test_formatters_canonical.py index 23a8a80b..2871ad27 100644 --- a/tests/test_formatters_canonical.py +++ b/tests/test_formatters_canonical.py @@ -31,10 +31,11 @@ ) from deltatrack.parsers.pdf_anchors import Anchor -# Local pin (guard against unintended bumps). 3.0 removed `amount_entries` (#671), so -# a change object carries no money field at all; 2.0 had removed the deprecated -# `amounts` before it (#274); 1.3 added the optional `tree` field (#108). -SCHEMA_VERSION = "3.0" +# Local pin (guard against unintended bumps). 3.1 added the optional top-level +# `financial` ledger (ADR 0023); 3.0 removed `amount_entries` (#671), so a change object +# carries no money field at all; 2.0 had removed the deprecated `amounts` before it +# (#274); 1.3 added the optional `tree` field (#108). +SCHEMA_VERSION = "3.1" # ---------- XML producer ------------------------------------------------------ diff --git a/tests/test_formatters_diff_html_skeleton.py b/tests/test_formatters_diff_html_skeleton.py index 1e0ee1ca..ec6dee4d 100644 --- a/tests/test_formatters_diff_html_skeleton.py +++ b/tests/test_formatters_diff_html_skeleton.py @@ -63,14 +63,12 @@ def test_canonical_h1_format(): def test_versions_line_with_version_numbers(): - """When version numbers are present, labels are prefixed "v1: " / "v2: ". - Separator is the literal · (Unicode middot).""" + """One row per version, "Before" then "After", each carrying its version number + when there is one; the Congress on a row of its own.""" html = format_diff_html(_empty()) - assert "v1: Reported in House" in html - assert "v2: Engrossed in House" in html - # Literal middot, not the · entity. - assert "·" in html - assert "118th Congress" in html + assert "Before (v1): Reported in House" in html + assert "After (v2): Engrossed in House" in html + assert '118th Congress' in html def test_versions_line_omits_congress_when_unknown(): @@ -84,15 +82,15 @@ def test_versions_line_omits_congress_when_unknown(): def test_versions_line_without_version_numbers(): - """When both version numbers are None (e.g. PDF inputs), no v1:/v2: prefix.""" + """When both version numbers are None (e.g. PDF inputs), no (v1)/(v2).""" html = format_diff_html(_empty(v1_version_number=None, v2_version_number=None)) # Look at the versions div specifically — "v1:" also appears in the citation CSS. versions_marker = '
' start = html.index(versions_marker) + len(versions_marker) end = html.index("
", start) versions_block = html[start:end] - assert "v1:" not in versions_block - assert "v2:" not in versions_block + assert "(v1)" not in versions_block + assert "(v2)" not in versions_block assert "Reported in House" in versions_block assert "Engrossed in House" in versions_block diff --git a/tests/test_frontend_browser.py b/tests/test_frontend_browser.py index 1f2101a4..96152762 100644 --- a/tests/test_frontend_browser.py +++ b/tests/test_frontend_browser.py @@ -1339,3 +1339,182 @@ def test_csp_base_uri_none_ignores_base_tag(live_url, chromium): f"CSP enforcement is not working. currentSrc: {result_with_csp['currentSrc']}" ) page.close() + + +# --- Each view keeps its own scroll position ---------------------------------------------- +# The views share one page, so a view used to come back at wherever the last one left the +# page. `showView` now remembers each view's position and restores it inside the switch, so +# anything that scrolls after it (a sidebar #change-N jump, a search's first match, a +# financial section link) still wins. Built on a real report: the committed H.R. 4366 XML +# pair, whose financial views are long enough to scroll. + + +def _scroll_report_path(tmp_path_factory): + from deltatrack.compare.xml import compare_xml_html + from tests.corpus_paths import fixture_path + + old = fixture_path("118-hr-4366", "1_reported-in-house.xml").read_bytes() + new = fixture_path("118-hr-4366", "2_engrossed-in-house.xml").read_bytes() + path = tmp_path_factory.mktemp("scroll") / "report.html" + path.write_text(compare_xml_html(old, new, start_label="Before", end_label="After"), encoding="utf-8") + return path + + +@pytest.fixture(scope="module") +def scroll_report(tmp_path_factory): + return _scroll_report_path(tmp_path_factory) + + +def _open(chromium, report): + page = chromium.new_page(viewport={"width": 1280, "height": 800}) + page.goto(report.as_uri(), wait_until="domcontentloaded") + return page + + +def _tab(page, view): + page.locator(f'.view-toggle__btn[data-view="{view}"]').click() + page.wait_for_timeout(100) + + +def _in_viewport(page, locator): + box = locator.bounding_box() + return box is not None and 0 <= box["y"] <= page.viewport_size["height"] - 10 + + +def test_a_view_comes_back_where_the_reader_left_it(chromium, scroll_report): + page = _open(chromium, scroll_report) + _tab(page, "fin-compare") + page.locator(".view-fin-compare tr.fin-crow").nth(8).scroll_into_view_if_needed() + left_at = page.evaluate("window.scrollY") + assert left_at > 400, "the comparison should be scrolled well down for this to mean anything" + + _tab(page, "changes") + page.evaluate("window.scrollTo(0, 150)") + _tab(page, "fin-compare") + assert abs(page.evaluate("window.scrollY") - left_at) <= 1 + page.close() + + +def test_a_section_link_and_back_returns_to_the_comparison_row(chromium, scroll_report): + page = _open(chromium, scroll_report) + _tab(page, "fin-compare") + row = page.locator(".view-fin-compare tr.fin-crow").nth(6) + row.scroll_into_view_if_needed() + left_at = page.evaluate("window.scrollY") + + link = row.locator('.fin-jump[data-side="v1"]') + group = link.get_attribute("data-group") + link.click() + page.wait_for_timeout(900) # the section link scrolls smoothly + target = page.locator(f'.view-fin-a tr.fin-row[data-group="{group}"]') + assert page.locator(".view-toggle__btn.is-active").get_attribute("data-view") == "fin-a" + assert target.get_attribute("aria-expanded") == "true" + assert _in_viewport(page, target) + assert row.get_attribute("aria-expanded") == "false", "the link must not open its own comparison row" + + _tab(page, "fin-compare") + assert abs(page.evaluate("window.scrollY") - left_at) <= 1 + assert _in_viewport(page, row) + page.close() + + +def test_a_sidebar_change_link_still_lands_on_its_card_from_another_view(chromium, scroll_report): + # The browser's jump to #change-N runs after the click handler switches views; the + # restore happens inside the switch, so the jump must still win. + page = _open(chromium, scroll_report) + _tab(page, "changes") + page.evaluate("window.scrollTo(0, 0)") + _tab(page, "fin-b") + page.evaluate("window.scrollTo(0, document.body.scrollHeight)") + link = page.locator(".sidebar-changes a[href^='#change-']").nth(3) + # Sidebar groups start collapsed; open this link's, as a reader would before clicking it. + link.evaluate( + "el => { for (let d = el.closest('details'); d; d = d.parentElement.closest('details')) d.open = true; }" + ) + card = page.locator(link.get_attribute("href")) + link.click() + page.wait_for_timeout(300) + assert page.locator(".view-toggle__btn.is-active").get_attribute("data-view") == "changes" + assert _in_viewport(page, card) + page.close() + + +def test_a_first_visit_opens_at_the_top_of_the_view(chromium, scroll_report): + page = _open(chromium, scroll_report) + page.evaluate("window.scrollTo(0, document.body.scrollHeight)") + _tab(page, "fin-b") + heading = page.locator(".view-fin-b h2") + assert _in_viewport(page, heading) + page.close() + + +def test_an_active_search_still_goes_to_its_first_match_on_a_view_switch(chromium, scroll_report): + page = _open(chromium, scroll_report) + _tab(page, "fin-a") + page.evaluate("window.scrollTo(0, 0)") + _tab(page, "changes") + page.locator("#find-input").fill("specially adapted housing") + page.wait_for_timeout(400) + _tab(page, "fin-a") + page.wait_for_timeout(1200) + hit = page.locator("mark.find-hit--current").first + assert hit.count() and _in_viewport(page, hit) + page.close() + + +# --- Jumps land below the sticky action bar ----------------------------------------------- +# The bar is two rows tall when the financial views are present (view switcher, then find and +# navigation), so a jump that put its target at the window's top hid it under the bar. +# The targets' scroll-margin-top follows the bar's measured height (--sticky-bar-height), so every +# anchor jump and scrollIntoView lands below it. + + +def _below_the_bar(page, locator): + bar = page.locator(".action-bar").bounding_box() + box = locator.bounding_box() + return box is not None and box["y"] >= bar["y"] + bar["height"] - 1 + + +def test_a_full_bill_toc_link_lands_below_the_bar(chromium, scroll_report): + page = _open(chromium, scroll_report) + _tab(page, "full") + # A target the page can actually bring to the top: one near the end stops short (there is + # no more page to scroll), which would pass whether or not the bar is cleared. + href = page.evaluate( + "() => { const room = document.documentElement.scrollHeight - 2 * innerHeight;" + " const links = [...document.querySelectorAll('.sidebar-toc a[href^=\"#\"]')];" + " const ok = links.find(a => { const t = document.getElementById(a.getAttribute('href').slice(1));" + " const y = t && t.getBoundingClientRect().top + scrollY; return y > 1500 && y < room; });" + " return ok && ok.getAttribute('href'); }" + ) + assert href, "no table-of-contents target with room to scroll to the top" + link = page.locator(f".sidebar-toc a[href='{href}']").first + link.evaluate( + "el => { for (let d = el.closest('details'); d; d = d.parentElement.closest('details')) d.open = true; }" + ) + target = page.locator(link.get_attribute("href")) + link.click() + page.wait_for_timeout(300) + assert _below_the_bar(page, target), "the section the link points at is hidden under the sticky bar" + page.close() + + +def test_change_navigation_lands_below_the_bar(chromium, scroll_report): + page = _open(chromium, scroll_report) + _tab(page, "changes") + for _ in range(4): + page.locator("#btn-next").click() + page.wait_for_timeout(900) # smooth scroll + # The navigator steps visible cards only (some sit in collapsed groups), so which card it + # reached is not "the fourth in the page". Whatever it reached is aligned to the top of the + # visible area; no card may start under the bar. + bar = page.locator(".action-bar").bounding_box() + bar_bottom = bar["y"] + bar["height"] + tops = page.evaluate( + "() => [...document.querySelectorAll('.view-changes .change-card')]" + ".filter(c => c.offsetParent !== null).map(c => c.getBoundingClientRect().top)" + ) + under = [t for t in tops if -2 <= t < bar_bottom - 1] + assert not under, f"a card starts under the sticky bar (tops {under}, bar ends at {bar_bottom})" + assert any(bar_bottom - 1 <= t < bar_bottom + 40 for t in tops), "no card was brought to the top" + page.close() diff --git a/tests/test_ledger_location_scorer.py b/tests/test_ledger_location_scorer.py new file mode 100644 index 00000000..98bd1bbd --- /dev/null +++ b/tests/test_ledger_location_scorer.py @@ -0,0 +1,113 @@ +"""The ledger scorer (ADR 0022): whole-path tiers, and the per-amount check. + +``tier`` grades where the PDF files one dollar amount against where the XML files it. These pin it +on hand-built paths. Each row that grades a higher level fails under the scorer this replaced, +which compared only the account and the heading directly above it: a wrong, made-up, reordered or +missing ancestor there graded T0. + +``compare_amounts`` follows each amount from one parser's ledger to another's. It must name an +amount that got worse even when another got better by the same amount, which is the case the +pinned tier totals cannot see. +""" + +from __future__ import annotations + +import pytest + +from tests.ledger_location import compare_amounts, tier + +XML = ("title v", "independent agencies", "united states tax court", "salaries and expenses") + + +@pytest.mark.parametrize( + ("pdf", "expected"), + [ + pytest.param(XML, "T0", id="identical"), + pytest.param( + ( + "division z", + "title ix", + "department of make believe", + "united states tax court", + "salaries and expenses", + ), + "T3", + id="made-up ancestors above the parent", + ), + pytest.param( + ("title v", "department of energy", "united states tax court", "salaries and expenses"), + "T3", + id="one ancestor wrong", + ), + pytest.param( + ("independent agencies", "title v", "united states tax court", "salaries and expenses"), + "T3", + id="ancestors reordered", + ), + pytest.param(("title v", "united states tax court", "salaries and expenses"), "T2", id="an ancestor missing"), + pytest.param(("salaries and expenses",), "T2", id="only the account"), + pytest.param( + ("title v", "independent agencies", "election assistance commission", "salaries and expenses"), + "T3", + id="wrong parent", + ), + pytest.param( + ("title v", "independent agencies", "extra heading", "united states tax court", "salaries and expenses"), + "T3", + id="an extra level", + ), + pytest.param( + ("title v", "independent agencies", "u.s. tax court", "salaries and expenses"), + "T3", + id="a parent too unlike to be a variant", + ), + pytest.param( + ("title v", "independent agencies", "united states tax court", "salaries & expenses"), + "T1", + id="a label variant", + ), + pytest.param( + ("title v", "independent agencies", "united states tax court", "payment to the postal service fund"), + "T4", + id="a different account", + ), + ], +) +def test_tier_compares_every_level(pdf, expected): + assert tier(XML, pdf) == expected + + +ACCOUNT = ("title i", "department of energy", "salaries and expenses") +OTHER = ("title i", "department of energy", "science") + + +def test_an_amount_that_got_worse_is_named_when_the_totals_do_not_move(): + before = { + "bill/v": [ + (100, ACCOUNT, ("title i", "salaries and expenses")), + (200, OTHER, ("title i", "wrong heading", "science")), + ] + } + after = { + "bill/v": [ + (100, ACCOUNT, ("title i", "wrong heading", "salaries and expenses")), + (200, OTHER, ("title i", "science")), + ] + } + # One amount T2 -> T3, the other T3 -> T2: the version's tier totals are identical. + moved, worse, skipped = compare_amounts(before, after) + assert moved == {"worse": 1, "better": 1} + assert [(key, change, values) for (key, change, *_), values in worse.items()] == [("bill/v", "T2->T3", [100])] + assert not skipped + + +def test_a_version_whose_xml_side_differs_is_not_compared(): + before = {"bill/v": [(100, ACCOUNT, ACCOUNT)]} + after = {"bill/v": [(100, OTHER, ACCOUNT)]} + assert compare_amounts(before, after) == ({}, {}, ["bill/v"]) + + +def test_a_miss_is_graded_and_compared(): + moved, worse, _ = compare_amounts({"bill/v": [(100, ACCOUNT, ACCOUNT)]}, {"bill/v": [(100, ACCOUNT, None)]}) + assert moved == {"worse": 1} + assert [change for (_, change, *_) in worse] == ["T0->MISS"] diff --git a/tests/test_pdf_anchor_golden.py b/tests/test_pdf_anchor_golden.py index 4f64c7ca..e212cbe3 100644 --- a/tests/test_pdf_anchor_golden.py +++ b/tests/test_pdf_anchor_golden.py @@ -23,7 +23,8 @@ import pytest -from deltatrack.parsers.pdf_anchors import derive_size_bands, extract_anchors +from deltatrack.parsers.pdf_anchors import breadcrumb_for, derive_size_bands, extract_anchors +from deltatrack.parsers.pdf_blocks import _flatten, _group_into_blocks from tests.conftest import assert_manifest_committed from tests.corpus_paths import DATA_DIR, fixture_path from tests.pdf_corpus import cached_pages @@ -105,6 +106,26 @@ def test_no_account_regressions_vs_legacy_baseline(self): assert "FEDERAL PROTECTIVE SERVICE" in (new - legacy) +class TestHeadingsAfterUnpunctuatedProse: + """118-s-4928 p.117: the paragraph above ends "(Public Law 111–241)" with no period. The + unfinished-sentence filter (ADR 0022, pass 2) used to read the two centred headings below it + as part of that sentence and drop them, filing the Inspector General's $274,000,000 under the + previous account, PAYMENT TO THE POSTAL SERVICE FUND.""" + + PDF = fixture_path("118-s-4928", "1_reported-in-senate.pdf") + + def test_the_headings_and_their_money_stay_together(self): + pages = cached_pages(self.PDF) + anchors = extract_anchors(pages) + on_page = {(a.line_number, a.text) for a in anchors if a.page_number == 117} + assert (3, "OFFICE OF INSPECTOR GENERAL") in on_page + assert (4, "SALARIES AND EXPENSES") in on_page + block = next(b for b in _group_into_blocks(_flatten(pages), anchors) if "$274,000,000" in b.text) + crumb = breadcrumb_for(block.anchor, anchors) + assert "OFFICE OF INSPECTOR GENERAL" in crumb + assert "PAYMENT TO THE POSTAL SERVICE FUND" not in crumb + + class TestNonAppropsGeneralization: """The whole point of the change: size detection works on general legislation that has no 'For necessary expenses' language. Pinned so a future change can't @@ -554,13 +575,12 @@ class TestCorpusAccountPrecision: Complements the exact golden snapshots (which pin three bills) with a tolerant net over the appropriations corpus. The floors ARE enforced minima — the assertions below - fail under them — but they are corpus-wide smoke floors rather than tight per-bill - regression expectations. They catch a large degradation and leave substantial unused - margin on the easier bills: 118-hr-8752 scores 1.000/1.000 and would still pass having - lost 40% of its accounts. + fail under them — and one floor serves all nine bills, so a bill scoring 1.000 keeps + some margin (it would still pass having lost 5% of its accounts). The ledger-location + pin (`tests/test_pdf_ledger_location.py`) is the exact per-version check. - That looseness is deliberate for now. #489 carries the full nine-bill measurement and - the per-heading breakdown; the load-bearing conclusions are: + #489 carries the nine-bill measurement from before ADR 0022 and the per-heading + breakdown; the load-bearing conclusions are: - The oracle is LEVEL-CORRECT, and since #499 it reads the tree's EFFECTIVE heading (the `display_path` leaf) rather than the raw `header_text`. It previously asked for @@ -585,25 +605,21 @@ class TestCorpusAccountPrecision: manifest, not only the cases this module collects. The per-case skip below is a defensive fallback, not the mechanism that keeps these fixtures available. - Why precision is well under 1.0 even when correct — the residual misses are - KNOWN and accepted, deferred to #54, NOT bugs to chase here: - - Provision-group headers (ADMINISTRATIVE PROVISIONS, GENERAL PROVISIONS, - SPENDING REDUCTION ACCOUNT) — real block headers mislabeled `account`. - - Wrapped agency-name fragments (e.g. "FAMILY HOUSING CONSTRUCTION, AIR - FORCE" wrapping onto a line read as "FORCE") — correct labeling needs the - leveled tree. - - Real account names whose GPO casing/wording normalizes differently than the - XML header (counted as a vocab miss though the anchor is right). - The SEC.-catchline-continuation class is NOT among the accepted residue — it is - fixed (see TestSectionCatchlineContinuation); a regression there would lower - these numbers, but the targeted test catches it first. - - That list predates the per-heading breakdown on #489, and the third bullet is where it - needed qualifying: on the one bill measured heading by heading (118-hr-4820), the - dominant cause was not normalization disagreement but the oracle reading the wrong - field, which #499 fixed. The first two bullets held up — the wrapped-fragment class is - the 17 misses recorded on #489. The other eight bills were measured in aggregate only, - so how the causes divide on them is unknown rather than assumed to match. + What the remaining gap to 1.0 is made of, after ADR 0022's heading passes: + - 115-hr-5895: `TITLE 17 INNOVATIVE TECHNOLOGY LOAN GUARANTEE` / `PROGRAM` stays + split. The name opens with the word TITLE, which the passes read as the structural + token and so never join into a heading run. + - 118-s-4795: `FISHERMEN’S CONTINGENCY FUND` is right but scores as a miss (the oracle + does not fold the curly apostrophe), and NASA's `SCIENCE` account is read under a + `SCIENCE` major. + The SEC.-catchline-continuation class is NOT among the residue — it is fixed (see + TestSectionCatchlineContinuation); a regression there would lower these numbers, but the + targeted test catches it first. + + History: before ADR 0022 the gap was provision-group headers (ADMINISTRATIVE PROVISIONS, + SPENDING REDUCTION ACCOUNT) read as accounts and wrapped-name fragments ("FAMILY HOUSING + CONSTRUCTION, AIR" / "FORCE"), the 17 misses recorded on #489 and #524's 14 on + 118-hr-4820; #499 had already fixed the oracle reading the wrong field. """ # Appropriations bills with a paired XML; (bill id, pdf rel path, xml rel path). @@ -633,19 +649,17 @@ class TestCorpusAccountPrecision: ("118-hr-8774", "tests/corpus/118-hr-8774", None), ("118-s-4795", "tests/data/BILLS-118s4795rs.pdf", "tests/corpus/118-s-4795/1_reported-in-senate.xml"), ] - # Set below the lowest measured values across the nine bills. Since #499 corrected the - # oracle to read the tree's effective heading, the floor-setting bill is 117-hr-4502 - # (vrec 0.744 / vprec 0.750) rather than 118-hr-4820, whose precision moved 0.538 -> - # 0.782. Recall and precision rose on every bill, so no floor here is a relaxation. - # The class docstring carries what the remaining gap to 1.0 is made of. + # Set below the lowest measured values across the nine bills, re-derived from the parser + # revision of ADR 0022 as #524's verification asks. The floor-setting bill is 118-s-4795 + # (vrec 0.967 / vprec 0.967); seven of the nine score 1.000 / 1.000. The class docstring + # carries what the remaining gap to 1.0 is made of. # - # Both floor-setting bills were named as `bills/` until they were committed, so CI - # collected no case for either and the floors were calibrated on a bill CI could not - # measure. The precision figure recorded here had drifted to 0.46 in the meantime and - # nothing could catch it, because the only run that could check it was a developer's. - # Now that the pair is committed, these two numbers are re-derivable from the suite. - RECALL_FLOOR = 0.70 - PRECISION_FLOOR = 0.70 + # History: 0.70 / 0.70 under 117-hr-4502 (0.744 / 0.750) before ADR 0022's heading passes + # joined wrapped account names; 118-hr-4820's unmatched account headings went 17 -> 0. + # Before that, both floor-setting bills were named as `bills/`, so CI collected no case + # for either and a recorded precision drifted to 0.46 unnoticed (#489). + RECALL_FLOOR = 0.95 + PRECISION_FLOOR = 0.95 @staticmethod def _pair(spec) -> tuple[Path, Path] | None: diff --git a/tests/test_pdf_compare.py b/tests/test_pdf_compare.py index a0de52d1..38d3d6e3 100644 --- a/tests/test_pdf_compare.py +++ b/tests/test_pdf_compare.py @@ -137,7 +137,7 @@ def test_csp_header_on_generated_report(): fake_html = "" + ("report " * 20_000) + "" monkeypatch = pytest.MonkeyPatch() - monkeypatch.setitem(app_module._COMPARE, "pdf", (".pdf", lambda *a, **kw: fake_html, lambda *a, **kw: {})) + monkeypatch.setitem(app_module._COMPARE, "pdf", (lambda *a, **kw: fake_html, lambda *a, **kw: {})) try: resp = _client().post( "/api/compare", @@ -207,7 +207,7 @@ def test_generated_report_response_is_gzipped(monkeypatch): import web.app as app_module fake_html = "" + ("report " * 20_000) + "" - monkeypatch.setitem(app_module._COMPARE, "pdf", (".pdf", lambda *a, **kw: fake_html, lambda *a, **kw: {})) + monkeypatch.setitem(app_module._COMPARE, "pdf", (lambda *a, **kw: fake_html, lambda *a, **kw: {})) resp = _client().post( "/api/compare", files={ @@ -596,13 +596,20 @@ def test_compare_pdfs_html_returns_standalone_report(): assert "Reported in House" in html assert "Engrossed in House" in html # #671, on the PDF entry point the web app calls. The bill's own dollar figures - # still appear in the card bodies; what is gone is the report presenting any of - # them as a change. Paired with the presence assertions above so a report that + # still appear in the card bodies; what is gone is the Changes view presenting any + # of them as a change. Paired with the presence assertions above so a report that # failed to render at all cannot pass this as an absence. assert "financial-table" not in html assert "financial-callout" not in html assert "data-financial" not in html - assert "Financial Summary" not in html + # The financial views (ADR 0023) bring money back in tabs of their own, per version + # and typed, with the research notebook's "Financial Summary" heading on Version A / + # B. The #671 line holds where it was drawn: the Changes view carries none of it. + changes_view = html.split('
', 1)[1].split('
Financial Summary") == 2 # Version A and Version B, nowhere else + assert html.count("Financial Summary") == 2 @pytest.mark.slow @@ -633,3 +640,27 @@ def test_derive_congress_from_cover(): # No cover match → empty (renderer then omits the "th Congress" suffix). assert _derive_congress([Page(1, (Line(None, "AN ACT"),))]) == "" assert _derive_congress([]) == "" + + +def test_derived_title_is_whole(): + # The heading wraps, so the title is kept whole, as the XML path's `bill_title` keeps + # it: a 140-character cut hid the fiscal year on H.R. 4366's. + from deltatrack.compare.pdf import _derive_bill_title + + long_title = ( + "Making appropriations for military construction, the Department of Veterans Affairs, " + "and related agencies for the fiscal year ending September 30, 2024, and for other purposes." + ) + canonical = {"full_text": {"v1": "", "v2": f"H. R. 4366\nAN ACT\n{long_title}\nBe it enacted"}} + assert _derive_bill_title(canonical) == f"H.R. 4366 — {long_title}" + + +def test_upload_label_is_the_filename_as_given(): + # Nothing is inferred from a filename (people name files their own way), so the label + # is the whole name, extension included; only client-sent path parts are dropped. + from web.app import _label_from_filename + + assert _label_from_filename("BILLS-118hr4366rh - BEFORE.pdf", "Start version") == "BILLS-118hr4366rh - BEFORE.pdf" + assert _label_from_filename(r"C:\Users\me\draft v3.XML", "Start version") == "draft v3.XML" + assert _label_from_filename("../../etc/after.pdf", "End version") == "after.pdf" + assert _label_from_filename("", "End version") == "End version" diff --git a/tests/test_pdf_heading_passes.py b/tests/test_pdf_heading_passes.py new file mode 100644 index 00000000..bd61913b --- /dev/null +++ b/tests/test_pdf_heading_passes.py @@ -0,0 +1,582 @@ +"""The ordered heading passes and the scope rules they feed (ADR 0022). + +Each class pins one rule on SYNTHETIC lines laid out like a GPO print: body prose runs the +full column at body size; a heading line is centered at heading size, and carries the case +pattern ``pdf_text`` reads from its letters (``initial_caps``: True for title case in small +caps, the agency style; False for even letters, the account style). Every test is a mutation +proof: it fails when its rule is removed. The corpus-level measurement against the XML twin is +``test_pdf_ledger_location.py``. +""" + +from __future__ import annotations + +from deltatrack.parsers import pdf_anchors +from deltatrack.parsers.pdf_anchors import Anchor, breadcrumb_for, extract_anchors +from deltatrack.parsers.pdf_heading_passes import _Stream, converge_headings +from deltatrack.parsers.pdf_text import Line, LineGeom, Page + +BODY = 14.0 +HEAD = 11.2 +LEFT = 100.0 +RIGHT = 439.0 # a 339 pt body column, as on the corpus prints +CHAR = 7.0 # synthetic heading glyph advance, points + + +def body(n: int, text: str) -> tuple[int, str, float, LineGeom]: + return (n, text, BODY, LineGeom(LEFT, RIGHT, LEFT + 30, initial_caps=False, size_min=BODY, size_max=BODY)) + + +def para(n: int, text: str) -> tuple[int, str, float, LineGeom]: + """The first line of a paragraph: body prose indented as GPO sets it (about 28 pt).""" + return (n, text, BODY, LineGeom(LEFT + 28, RIGHT, LEFT + 58, initial_caps=False, size_min=BODY, size_max=BODY)) + + +def head(n: int, text: str, caps: bool | None, width: float | None = None, size: float = HEAD): + """A centered heading line ``width`` points wide (default: from its length).""" + w = width if width is not None else len(text) * CHAR + left = LEFT + (RIGHT - LEFT - w) / 2 + first = left + len(text.split()[0]) * CHAR + return (n, text, size, LineGeom(left, left + w, first, initial_caps=caps, size_min=size, size_max=size)) + + +def page(number: int, rows) -> Page: + return Page(number, tuple(Line(n, t, s, g) for n, t, s, g in rows)) + + +def anchors_of(rows, *more_pages) -> list[Anchor]: + return extract_anchors([page(1, rows), *more_pages]) + + +def texts(anchors, kind: str) -> list[str]: + return [a.text for a in anchors if a.kind == kind] + + +# Body prose long enough for the size bands to find the body mode. +PROSE = [body(n, f"for necessary expenses of the program, line {n}.") for n in range(40, 52)] + + +class TestLinesThatCannotBeHeadings: + def test_a_line_inside_an_unfinished_sentence_is_not_a_heading(self): + rows = [ + body(1, "For payments to the fund established under the"), + head(2, "UNITED STATES MINT", caps=False), + body(3, "Public Enterprise Fund, $10,000,000."), + head(4, "OPERATIONS AND SUPPORT", caps=False), + body(5, "For necessary expenses, $5,000,000."), + *PROSE, + ] + accounts = texts(anchors_of(rows), "account") + assert "UNITED STATES MINT" not in accounts + assert "OPERATIONS AND SUPPORT" in accounts + + def test_headings_after_an_unpunctuated_paragraph_end_are_kept(self): + # 118-s-4928 p.117: the paragraph above ends "(Public Law 111–241)" with no period. The + # headings are centered and a new, indented paragraph follows them, so they are set apart + # from the prose; missing punctuation alone must not drop them and file the $274M below + # them under the previous account. + rows = [ + body(1, "Semipostal Stamp Act of 2010 (Public Law 111–241)"), + head(2, "OFFICE OF INSPECTOR GENERAL", caps=True), + head(3, "SALARIES AND EXPENSES", caps=False), + para(4, "For necessary expenses of the Office of Inspector General, $274,000,000."), + *PROSE, + ] + found = {a.text for a in anchors_of(rows)} + assert "OFFICE OF INSPECTOR GENERAL" in found + assert "SALARIES AND EXPENSES" in found + + def test_headings_inside_an_interrupted_enacting_clause_are_kept(self): + # 118-hr-4394 p.2: "Be it enacted ... assembled, That" is genuinely unfinished, and GPO + # prints the title and its headings inside it before the prose resumes. + rows = [ + body(1, "tives of the United States of America in Congress assembled,"), + body(2, "That"), + head(3, "TITLE I", caps=None, size=BODY), + head(4, "CORPS OF ENGINEERS—CIVIL", caps=False, size=BODY), + head(5, "DEPARTMENT OF THE ARMY", caps=False, size=BODY), + head(6, "CORPS OF ENGINEERS—CIVIL", caps=True), + para(7, "The following appropriations shall be expended under the direction of the Secretary."), + *PROSE, + ] + found = [a.text for a in anchors_of(rows)] + assert "DEPARTMENT OF THE ARMY" in found + assert found.count("CORPS OF ENGINEERS—CIVIL") == 2 + + def test_a_quoted_heading_is_not_a_heading(self): + # A reconciliation bill amends other laws by quoting them; the quoted law's headings + # print like headings but belong to the quotation. + rows = [ + body(1, "(a) Chapter 1 is amended by adding at the end the following:"), + head(2, "‘‘PROHIBITED USES OF FUNDS", caps=False), + body(3, "‘‘No funds may be used for the purpose described.’’."), + head(4, "OPERATIONS AND SUPPORT", caps=False), + body(5, "For necessary expenses, $5,000,000."), + *PROSE, + ] + all_text = {a.text for a in anchors_of(rows)} + assert not any("PROHIBITED USES" in t for t in all_text) + assert "OPERATIONS AND SUPPORT" in all_text + + def test_a_multi_paragraph_quote_closes_at_its_one_closing_mark(self): + # GPO opens every quoted paragraph with ‘‘ and closes the block once: the heading after + # the block is outside the quote even though openings outnumber closings. + rows = [ + body(1, "(a) Section 5 is amended to read as follows:"), + body(2, "‘‘(a) IN GENERAL.—The Secretary shall make grants."), + body(3, "‘‘(b) LIMITATION.—No grant may exceed $1,000.’’."), + head(4, "OPERATIONS AND SUPPORT", caps=False), + body(5, "For necessary expenses, $5,000,000."), + *PROSE, + ] + assert "OPERATIONS AND SUPPORT" in texts(anchors_of(rows), "account") + + def test_quote_state_resets_at_a_section_line(self): + # A closing mark lost in the extracted text (here the quote opened on line 1 never + # closes) must not silence the rest of the bill: the SEC. line closes it. + stream = _Stream( + [ + page( + 1, + [ + body(1, "the term ‘‘covered program means a program."), + body(2, "SEC. 102. Amounts made available shall remain available."), + head(3, "OPERATIONS AND SUPPORT", caps=False), + ], + ) + ] + ) + assert stream.in_quote == set() + + +class TestSegmentation: + def test_letters_printed_alike_join_a_wrapped_name(self): + # Both lines even small caps; the upper line fills the heading measure, so the break is + # a wrap: one account, not an agency fragment over an account fragment. + rows = [ + body(1, "budget for the current fiscal year for such corporation."), + head(2, "LIMITATION ON ADMINISTRATIVE EXPENSES, FEDERAL", caps=False, width=300), + head(3, "PRISON INDUSTRIES, INCORPORATED", caps=False), + body(4, "Not to exceed $2,700,000 shall be available."), + head(5, "SHORT CENTERED HEADING", caps=False, width=300), + body(6, "For necessary expenses, $1,000,000."), + *PROSE, + ] + anchors = anchors_of(rows) + assert "LIMITATION ON ADMINISTRATIVE EXPENSES, FEDERAL PRISON INDUSTRIES, INCORPORATED" in texts( + anchors, "account" + ) + assert texts(anchors, "agency") == [] + + def test_letters_printed_differently_split(self): + # Agency style over account style: two headings even where the upper line is full. + rows = [ + body(1, "budget for the current fiscal year for such corporation."), + head(2, "STATE AND LOCAL LAW ENFORCEMENT ACTIVITIES", caps=True, width=300), + head(3, "VIOLENCE AGAINST WOMEN PREVENTION AND", caps=False, width=300), + head(4, "PROSECUTION PROGRAMS", caps=False), + body(5, "For grants, contracts, and cooperative agreements, $1,000."), + *PROSE, + ] + anchors = anchors_of(rows) + assert texts(anchors, "agency") == ["STATE AND LOCAL LAW ENFORCEMENT ACTIVITIES"] + assert "VIOLENCE AGAINST WOMEN PREVENTION AND PROSECUTION PROGRAMS" in texts(anchors, "account") + + def test_a_short_upper_line_is_a_deliberate_break(self): + # Letters alike, but the next line's first word would have fitted on the upper line: + # the printer broke it on purpose, so these are two stacked headings. + rows = [ + body(1, "budget for the current fiscal year for such corporation."), + head(2, "LIMITATION ON ADMINISTRATIVE EXPENSES, FEDERAL", caps=False, width=300), + head(3, "PRISON INDUSTRIES, INCORPORATED", caps=False), + body(4, "Not to exceed $2,700,000 shall be available."), + head(5, "BUREAU OF PRISONS", caps=False, width=120), + head(6, "BUILDINGS AND FACILITIES", caps=False), + body(7, "For planning, acquisition of sites, and construction, $1,000."), + *PROSE, + ] + anchors = anchors_of(rows) + assert "BUREAU OF PRISONS" in texts(anchors, "agency") + assert "BUILDINGS AND FACILITIES" in texts(anchors, "account") + + def test_a_name_that_repeats_its_own_words_stays_one_heading(self): + # 118-s-4928 p.35. A veto once split any lower line whose words all appear above it, but + # names repeat words: on 118 PDFs that veto changed 9 headings and was wrong in all 9 + # (this council three times, the Council on Environmental Quality six), so it is gone. + rows = [ + body(1, "budget for the current fiscal year for such corporation."), + head(2, "NATIONAL SECURITY COUNCIL AND HOMELAND", caps=True, width=300), + head(3, "SECURITY COUNCIL", caps=True), + head(4, "SALARIES AND EXPENSES", caps=False), + body(5, "For necessary expenses of the National Security Council, $1,000."), + *PROSE, + ] + assert "NATIONAL SECURITY COUNCIL AND HOMELAND SECURITY COUNCIL" in texts(anchors_of(rows), "agency") + + def test_a_line_ending_in_a_small_word_continues(self): + # 115-hr-5895 p.159: an account name broken early for balance. The next word would have + # fitted on the upper line, so the fullness veto reads a deliberate break, but a heading + # cannot end in OF. On 118 PDFs, widening the AND/OR rule to the small words changed 7 + # headings, all toward the XML. + rows = [ + body(1, "budget for the current fiscal year for such corporation."), + head(2, "LIMITATION ON ADMINISTRATIVE EXPENSES, FEDERAL", caps=False, width=300), + head(3, "PRISON INDUSTRIES, INCORPORATED", caps=False), + body(4, "Not to exceed $2,700,000 shall be available."), + head(5, "GRANTS FOR CONSTRUCTION OF", caps=False), + head(6, "STATE EXTENDED CARE FACILITIES", caps=False), + body(7, "For grants to assist States to acquire or construct State nursing homes, $1,000."), + *PROSE, + ] + assert "GRANTS FOR CONSTRUCTION OF STATE EXTENDED CARE FACILITIES" in texts(anchors_of(rows), "account") + + def test_a_lower_line_standing_alone_elsewhere_is_its_own_heading(self): + rows = [ + body(1, "budget for the current fiscal year for such corporation."), + head(2, "LIMITATION ON ADMINISTRATIVE EXPENSES, FEDERAL", caps=False, width=300), + head(3, "PRISON INDUSTRIES, INCORPORATED", caps=False), + body(4, "Not to exceed $2,700,000 shall be available."), + head(5, "OPERATIONS AND SUPPORT", caps=False), + body(6, "For necessary expenses, $1,000."), + head(7, "OPERATIONS AND SUPPORT", caps=False), + body(8, "For necessary expenses, $2,000."), + head(9, "FEDERAL LAW ENFORCEMENT TRAINING CENTERS", caps=False, width=300), + head(10, "OPERATIONS AND SUPPORT", caps=False), + body(11, "For necessary expenses, $3,000."), + *PROSE, + ] + accounts = texts(anchors_of(rows), "account") + assert accounts.count("OPERATIONS AND SUPPORT") == 3 + + def test_a_line_break_hyphen_joins_whatever_the_letters_say(self): + rows = [ + body(1, "budget for the current fiscal year for such corporation."), + head(2, "NATIONAL AERONAUTICS AND SPACE ADMINIS-", caps=True, width=300), + head(3, "TRATION", caps=False), + head(4, "SCIENCE", caps=False), + body(5, "For necessary expenses, $1,000."), + *PROSE, + ] + assert "NATIONAL AERONAUTICS AND SPACE ADMINISTRATION" in texts(anchors_of(rows), "agency") + + def test_a_trailing_conjunction_joins_whatever_the_letters_say(self): + rows = [ + body(1, "budget for the current fiscal year for such corporation."), + head(2, "DEFENSE ENVIRONMENTAL CLEANUP AND", caps=True, width=150), + head(3, "DECONTAMINATION", caps=False), + body(4, "For necessary expenses, $1,000."), + *PROSE, + ] + assert "DEFENSE ENVIRONMENTAL CLEANUP AND DECONTAMINATION" in texts(anchors_of(rows), "account") + + +class TestNoWordingDecidesALineBreak: + """ADR 0018: two stacked lines printed alike, with no veto, stay one heading whatever they + say. Recognising a familiar sub-account or department name would split them; that is a + known residual (ADR 0022), not a case for a word list.""" + + def test_a_familiar_subaccount_name_does_not_split_a_stack(self): + rows = [ + body(1, "budget for the current fiscal year for such corporation."), + head(2, "TREASURY INSPECTOR GENERAL FOR TAX ADMINISTRATION", caps=False, width=320), + head(3, "SALARIES AND EXPENSES", caps=False), + body(4, "For necessary expenses, $1,000."), + *PROSE, + ] + assert texts(anchors_of(rows), "account") == [ + "TREASURY INSPECTOR GENERAL FOR TAX ADMINISTRATION SALARIES AND EXPENSES" + ] + + def test_a_department_name_does_not_split_a_stack(self): + rows = [ + body(1, "budget for the current fiscal year for such corporation."), + head(2, "DEPARTMENT OF DEFENSE", caps=False, width=320), + head(3, "MILITARY UNACCOMPANIED HOUSING IMPROVEMENT FUND", caps=False), + body(4, "For necessary expenses, $1,000."), + *PROSE, + ] + assert texts(anchors_of(rows), "account") == [ + "DEPARTMENT OF DEFENSE MILITARY UNACCOMPANIED HOUSING IMPROVEMENT FUND" + ] + + +class TestHangingIndent: + def test_a_hanging_indent_block_is_one_heading(self): + # First line at the left margin running full width, the next line starting where the + # following paragraph starts: one heading, even though the letters differ. + para = LEFT + 20 + rows = [ + body(1, "budget for the current fiscal year for such corporation."), + ( + 2, + "SUBSTANCE ABUSE AND MENTAL HEALTH SERVICES ADMINISTRATION", + HEAD, + LineGeom(LEFT, RIGHT, LEFT + 70, initial_caps=True, size_min=HEAD, size_max=HEAD), + ), + ( + 3, + "PROGRAMS OF REGIONAL AND NATIONAL SIGNIFICANCE", + HEAD, + LineGeom(para, para + 250, para + 60, initial_caps=False, size_min=HEAD, size_max=HEAD), + ), + ( + 4, + "For necessary expenses, $1,000.", + BODY, + LineGeom(para, RIGHT, para + 20, initial_caps=False, size_min=BODY, size_max=BODY), + ), + *PROSE, + ] + accounts = texts(anchors_of(rows), "account") + assert ( + "SUBSTANCE ABUSE AND MENTAL HEALTH SERVICES ADMINISTRATION PROGRAMS OF REGIONAL AND NATIONAL SIGNIFICANCE" + in accounts + ) + + +class TestTitleNameRunningOn: + def test_the_rest_of_a_title_line_is_not_a_heading(self): + # 118-hr-4665 p.296: the title's number and name share a line and the name runs on, + # hyphenated, over two more uncentred lines before the first centred heading. Those lines + # finish the title's name; read as a heading they became "ENCE OF THE PEOPLE'S REPUBLIC + # OF CHINA". + full = LineGeom(LEFT, RIGHT, LEFT + 60, initial_caps=False, size_min=BODY, size_max=BODY) + rows = [ + body(1, "Emergency Deficit Control Act of 1985."), + (2, "TITLE VIII—COUNTERING THE MALIGN INFLU-", BODY, full), + (3, "ENCE OF THE PEOPLE’S REPUBLIC OF", BODY, full), + (4, "CHINA", BODY, LineGeom(LEFT, LEFT + 50, LEFT + 50, initial_caps=False, size_min=BODY, size_max=BODY)), + head(5, "BILATERAL ECONOMIC ASSISTANCE", caps=False, size=BODY), + head(6, "FUNDS APPROPRIATED TO THE PRESIDENT", caps=True), + head(7, "ECONOMIC SUPPORT FUND", caps=False), + para(8, "For an additional amount for the Economic Support Fund, $1,000."), + *PROSE, + ] + assert not any("ENCE OF" in a.text or a.text == "CHINA" for a in anchors_of(rows)) + + +class TestDepartmentHeadingsMidTitle: + def test_a_body_size_capitals_line_mid_title_is_a_major(self): + rows = [ + body(1, "TITLE II"), + head(2, "SECURITY, ENFORCEMENT, AND INVESTIGATIONS", caps=False, size=BODY), + head(3, "OPERATIONS AND SUPPORT", caps=False), + body(4, "For necessary expenses, $1,000."), + head(5, "UNITED STATES SECRET SERVICE", caps=False, size=BODY), + head(6, "PROCUREMENT, CONSTRUCTION, AND IMPROVEMENTS", caps=False), + body(7, "For necessary expenses, $2,000."), + *PROSE, + ] + rows[0] = (1, "TITLE II", BODY, LineGeom(250, 290, 290, initial_caps=None, size_min=BODY, size_max=BODY)) + anchors = anchors_of(rows) + account = next(a for a in anchors if a.text.startswith("PROCUREMENT")) + assert breadcrumb_for(account, anchors)[:2] == ("TITLE II", "UNITED STATES SECRET SERVICE") + + def test_a_structural_token_line_is_not_a_department(self): + # A bill with no heading band (every line at body size, like the reconciliation bill + # 119-hr-1) gets no account detection, so this pass is what finds its departments. A + # CHAPTER line, or the wrapped name of a TITLE, is its own level, not a department. + rows = [ + body(1, "budget for the current fiscal year for such corporation."), + head(2, "CHAPTER 2—GENERAL PROVISIONS", caps=False, size=BODY), + body(3, "For necessary expenses, $1,000."), + ( + 4, + "TITLE II—AGRICULTURE AND", + BODY, + LineGeom(200, 340, 250, initial_caps=None, size_min=BODY, size_max=BODY), + ), + head(5, "NUTRITION PROGRAMS", caps=False, size=BODY), + body(6, "For necessary expenses, $1,000."), + head(7, "DEPARTMENT OF AGRICULTURE", caps=False, size=BODY), + body(8, "For necessary expenses, $1,000."), + *PROSE, + ] + assert texts(anchors_of(rows), "major") == ["DEPARTMENT OF AGRICULTURE"] + + +class TestFailsClosed: + def test_no_case_pattern_leaves_the_parser_reading(self, monkeypatch): + # Without the letters' case pattern the passes keep the detectors' split. + rows = [ + body(1, "budget for the current fiscal year for such corporation."), + head(2, "LIMITATION ON ADMINISTRATIVE EXPENSES, FEDERAL", caps=None, width=300), + head(3, "PRISON INDUSTRIES, INCORPORATED", caps=None), + body(4, "Not to exceed $2,700,000 shall be available."), + *PROSE, + ] + converged = [(a.kind, a.text) for a in anchors_of(rows)] + monkeypatch.setattr(pdf_anchors, "converge_headings", lambda pages, anchors: anchors) + detected = [(a.kind, a.text) for a in anchors_of(rows)] + assert ("account", "PRISON INDUSTRIES, INCORPORATED") in detected + assert converged == detected + + def test_no_anchors_in_no_anchors_out(self): + assert converge_headings([page(1, PROSE)], []) == [] + + +class TestKeepsWhatDetectorsDecided: + def test_the_department_line_under_a_bare_title_is_left_as_detected(self): + # Directly under TITLE n the major detector reads the department heading (#105); the + # passes re-segment only the heading lines after it, so it is never glued to them. + rows = [ + (1, "TITLE I", BODY, LineGeom(250, 290, 290, initial_caps=None, size_min=BODY, size_max=BODY)), + head(2, "DEPARTMENTAL MANAGEMENT", caps=False, width=300, size=BODY), + head(3, "OPERATIONS AND SUPPORT", caps=False), + body(4, "For necessary expenses, $1,000."), + *PROSE, + ] + anchors = anchors_of(rows) + assert texts(anchors, "major") == ["DEPARTMENTAL MANAGEMENT"] + assert texts(anchors, "account") == ["OPERATIONS AND SUPPORT"] + + def test_a_wrapped_subsection_title_tail_is_not_a_heading(self): + rows = [ + body(1, "budget for the current fiscal year for such corporation."), + head(2, "LIMITATION ON AMOUNTS FOR CERTAIN PURPOSES.—", caps=False), + body(3, "Of the amounts made available, not more than $1,000 may be used."), + *PROSE, + ] + assert not any(a.text.endswith(".—") for a in anchors_of(rows)) + + def test_heading_anchors_record_their_case_pattern(self): + rows = [ + body(1, "budget for the current fiscal year for such corporation."), + head(2, "BUREAU OF PRISONS", caps=True, width=120), + head(3, "BUILDINGS AND FACILITIES", caps=False), + body(4, "For planning, acquisition of sites, and construction, $1,000."), + *PROSE, + ] + caps = {a.text: a.caps for a in anchors_of(rows)} + assert caps == {"BUREAU OF PRISONS": True, "BUILDINGS AND FACILITIES": False} + + +def _a(kind: str, text: str, line: int, caps: bool | None = None) -> Anchor: + return Anchor(1, line, kind, text, caps=caps) + + +class TestLevelOfASplitPiece: + def test_a_piece_printed_like_its_account_is_on_the_account_level(self): + # 118-s-4690 p.72: HEALTH AND HUMAN SERVICES (title case) / FOOD AND DRUG ADMINISTRATION + # (even) / SALARIES AND EXPENSES (even). GPO tags FDA at the account level, a name over its + # own account. Defaulted to an agency, FDA ended HHS's reach, so HHS's later accounts lost + # it; on 118 PDFs taking the level from the print moved 202 amounts, all toward the XML. + rows = [ + body(1, "budget for the current fiscal year for such corporation."), + head(2, "DEPARTMENT OF HEALTH AND HUMAN SERVICES", caps=True, width=300), + head(3, "FOOD AND DRUG ADMINISTRATION", caps=False), + head(4, "SALARIES AND EXPENSES", caps=False), + para(5, "For necessary expenses of the Food and Drug Administration, $1,000."), + head(6, "BUILDINGS AND FACILITIES", caps=False), + para(7, "For plans, construction, repair, and improvement of facilities, $2,000."), + *PROSE, + ] + anchors = anchors_of(rows) + later = next(a for a in anchors if a.text == "BUILDINGS AND FACILITIES") + assert "DEPARTMENT OF HEALTH AND HUMAN SERVICES" in breadcrumb_for(later, anchors) + assert "FOOD AND DRUG ADMINISTRATION" not in texts(anchors, "agency") + + +class TestAHeadingThatIntroducesAccounts: + """ADR 0012, boundary 1: a title-case heading followed by prose. The level is decided on the + joined heading as a whole, from what follows it. With no dollar amount in its text and an + account-style heading next, it introduces accounts: an agency. On 118 PDFs this moved 183 + amounts toward the XML and none away.""" + + def test_an_authority_paragraph_with_no_money_makes_the_heading_an_agency(self): + # 118-hr-4820 p.62: the name wraps, its paragraph authorizes spending but appropriates + # nothing, and OPERATIONS AND MAINTENANCE below it carries the money. + rows = [ + body(1, "1986 (26 U.S.C. 9503(e)(4))."), + head(2, "GREAT LAKES ST. LAWRENCE SEAWAY DEVELOPMENT", caps=True, width=300), + head(3, "CORPORATION", caps=True), + para(4, "The Corporation is hereby authorized to make such expenditures, within the limits"), + body(5, "of funds available to it, as may be necessary."), + head(6, "OPERATIONS AND MAINTENANCE", caps=False), + para(7, "For necessary expenses to conduct the operations, $38,000,000."), + *PROSE, + ] + anchors = anchors_of(rows) + account = next(a for a in anchors if a.text == "OPERATIONS AND MAINTENANCE") + assert "GREAT LAKES ST. LAWRENCE SEAWAY DEVELOPMENT CORPORATION" in breadcrumb_for(account, anchors) + + def test_a_heading_that_holds_money_stays_an_account(self): + # A title-case account name with its own appropriation (FAMILY HOUSING OPERATION AND + # MAINTENANCE, ARMY) must not become the parent of the account after it. Promoting on + # title case alone did exactly that: 265 amounts the wrong way against 8. + rows = [ + body(1, "budget for the current fiscal year for such corporation."), + head(2, "FAMILY HOUSING OPERATION AND MAINTENANCE,", caps=True, width=300), + head(3, "ARMY", caps=True), + para(4, "For expenses of family housing for the Army, $436,000,000."), + head(5, "OPERATIONS AND MAINTENANCE", caps=False), + para(6, "For necessary expenses to conduct the operations, $1,000."), + *PROSE, + ] + anchors = anchors_of(rows) + assert "FAMILY HOUSING OPERATION AND MAINTENANCE, ARMY" not in texts(anchors, "agency") + account = next(a for a in anchors if a.text == "OPERATIONS AND MAINTENANCE") + assert "FAMILY HOUSING OPERATION AND MAINTENANCE, ARMY" not in breadcrumb_for(account, anchors) + + +class TestAgencyScope: + def test_a_department_ends_the_agency_above_it(self): + anchors = [ + _a("title", "TITLE II", 1), + _a("major", "DEPARTMENTAL MANAGEMENT", 2), + _a("agency", "MANAGEMENT DIRECTORATE", 3, caps=True), + _a("account", "OPERATIONS AND SUPPORT", 4, caps=False), + _a("major", "UNITED STATES SECRET SERVICE", 6), + _a("account", "PROCUREMENT", 7, caps=False), + ] + assert breadcrumb_for(anchors[-1], anchors) == ("TITLE II", "UNITED STATES SECRET SERVICE", "PROCUREMENT") + + def test_an_agency_printed_like_the_heading_below_it_does_not_carry_over(self): + # Two stacked account-style lines are peers (or a wrap), not agency over account. + anchors = [ + _a("title", "TITLE I", 1), + _a("agency", "FEDERAL BUILDINGS FUND", 2, caps=False), + _a("account", "LIMITATIONS ON AVAILABILITY OF REVENUE", 3, caps=False), + _a("account", "WORKING CAPITAL FUND", 6, caps=False), + ] + assert breadcrumb_for(anchors[-1], anchors) == ("TITLE I", "WORKING CAPITAL FUND") + + def test_an_agency_style_account_ends_the_agency(self): + anchors = [ + _a("title", "TITLE I", 1), + _a("agency", "BUREAU OF PRISONS", 2, caps=True), + _a("account", "SALARIES AND EXPENSES", 3, caps=False), + _a("account", "OFFICE OF THE INSPECTOR GENERAL", 6, caps=True), + ] + assert breadcrumb_for(anchors[-1], anchors) == ("TITLE I", "OFFICE OF THE INSPECTOR GENERAL") + + def test_an_agency_style_heading_between_ends_the_agency(self): + anchors = [ + _a("title", "TITLE I", 1), + _a("agency", "BUREAU OF PRISONS", 2, caps=True), + _a("account", "SALARIES AND EXPENSES", 3, caps=False), + _a("account", "OFFICE OF THE INSPECTOR GENERAL", 6, caps=True), + _a("account", "WORKING CAPITAL FUND", 9, caps=False), + ] + assert breadcrumb_for(anchors[-1], anchors) == ("TITLE I", "WORKING CAPITAL FUND") + + def test_a_real_agency_carries_over_its_accounts(self): + anchors = [ + _a("title", "TITLE I", 1), + _a("agency", "BUREAU OF PRISONS", 2, caps=True), + _a("account", "SALARIES AND EXPENSES", 3, caps=False), + _a("account", "BUILDINGS AND FACILITIES", 6, caps=False), + ] + assert breadcrumb_for(anchors[-1], anchors) == ("TITLE I", "BUREAU OF PRISONS", "BUILDINGS AND FACILITIES") + + def test_unknown_case_pattern_carries_over(self): + anchors = [ + _a("title", "TITLE I", 1), + _a("agency", "BUREAU OF PRISONS", 2), + _a("account", "SALARIES AND EXPENSES", 3), + _a("account", "OFFICE OF THE INSPECTOR GENERAL", 6), + ] + assert breadcrumb_for(anchors[-1], anchors) == ( + "TITLE I", + "BUREAU OF PRISONS", + "OFFICE OF THE INSPECTOR GENERAL", + ) diff --git a/tests/test_pdf_ledger_location.py b/tests/test_pdf_ledger_location.py new file mode 100644 index 00000000..202f60ae --- /dev/null +++ b/tests/test_pdf_ledger_location.py @@ -0,0 +1,76 @@ +"""Where the PDF files each dollar amount, pinned per version against the XML twin (ADR 0022). + +``tests/ledger_location.py`` scores one bill version: every amount in the XML ledger, aligned +with the same amount in the PDF ledger, lands in a tier from ``T0`` (the same path, every level) +to ``T4`` (filed under a different heading), or is a ``MISS``. This module pins those tier counts for +every committed version that has both a PDF and its XML twin, in +``tests/data/ledger_location_baseline.json``. + +The pin is exact. A version whose not-tolerated count (``T3``, ``T4``, ``MISS``) rises or whose +true hits (``T0``) fall is a regression. A version that improves also fails, so the improvement +is locked in by regenerating rather than leaving slack for a later regression to hide in. +Regenerating is a claim that the ledger *should* have moved, made with the measurement in the +same pull request (the discipline the canonical baselines follow): + + UPDATE_LEDGER_BASELINE=1 uv run pytest tests/test_pdf_ledger_location.py + +Enrolled versions are left out: their unnumbered layout is declined by the PDF pipeline. + +Marked @slow: parses every committed dual-format version in both formats. +""" + +from __future__ import annotations + +import json +import os + +import pytest + +from tests.corpus_paths import DATA_DIR +from tests.ledger_location import NOT_TOLERATED, TIERS, committed_versions, score + +pytestmark = pytest.mark.slow + +BASELINE = DATA_DIR / "ledger_location_baseline.json" +REGENERATE = "UPDATE_LEDGER_BASELINE=1 uv run pytest tests/test_pdf_ledger_location.py" + + +def _not_tolerated(tally: dict[str, int]) -> int: + return sum(tally[t] for t in NOT_TOLERATED) + + +@pytest.mark.skipif(os.environ.get("UPDATE_LEDGER_BASELINE") != "1", reason="not in baseline-update mode") +def test_regenerate_the_ledger_baseline(): + live = {key: score(xml, pdf) for key, (xml, pdf) in sorted(committed_versions().items())} + BASELINE.write_text(json.dumps(live, indent=1) + "\n", encoding="utf-8", newline="\n") + + +def test_every_committed_dual_format_version_is_pinned(): + pinned = set(json.loads(BASELINE.read_text(encoding="utf-8"))) + live = set(committed_versions()) + assert live == pinned, ( + f"unpinned versions {sorted(live - pinned)}, pinned but absent {sorted(pinned - live)}. " + f"Regenerate with: {REGENERATE}" + ) + + +def test_ledger_location_matches_the_pinned_tiers(): + pinned = json.loads(BASELINE.read_text(encoding="utf-8")) + worse, better = [], [] + for key, (xml, pdf) in sorted(committed_versions().items()): + if key not in pinned: + continue # reported by the coverage test above + live, was = score(xml, pdf), pinned[key] + if live == was: + continue + regressed = _not_tolerated(live) > _not_tolerated(was) or live["T0"] < was["T0"] + (worse if regressed else better).append(f"{key}: {was} -> {live}") + assert not worse, "ledger location regressed against the XML twin:\n " + "\n ".join(worse) + assert not better, f"ledger location improved; lock the improvement in with {REGENERATE}\n " + "\n ".join(better) + + +def test_the_baseline_carries_every_tier(): + # A tally missing a tier would compare as unequal forever, or hide a tier from the sums. + pinned = json.loads(BASELINE.read_text(encoding="utf-8")) + assert pinned, "empty baseline" + assert all(set(tally) == set(TIERS) for tally in pinned.values()) diff --git a/tests/test_pdf_round1_revocation.py b/tests/test_pdf_round1_revocation.py index b9082f7f..ae1033e9 100644 --- a/tests/test_pdf_round1_revocation.py +++ b/tests/test_pdf_round1_revocation.py @@ -133,8 +133,9 @@ def test_moving_the_threshold_moves_the_split_population() -> None: pairing already below production's cutoff, in which case raising the threshold changes nothing there and the control reports a false alarm. The first draft did exactly that. - Measured: 0.2 → 192, 0.3 → 214, 0.4 → 230, 0.6 → 257, 0.9 → 402. Under the censored - evidence this slice shipped with, 0.2 and 0.3 both returned 230 — the two sub-production + Measured: 0.2 → 195, 0.3 → 215, 0.4 → 231, 0.6 → 259, 0.9 → 404 (the heading passes of ADR + 0022 move block boundaries; before them, 192 / 214 / 230 / 257 / 402). Under the censored + evidence this slice shipped with, 0.2 and 0.3 both returned the production count — the two sub-production points collapsing onto the cutoff is precisely the signature this control now detects. """ points = (0.2, 0.3, SIMILARITY_THRESHOLD, 0.6, 0.9) @@ -187,8 +188,8 @@ def test_evidence_reports_the_true_overlap_for_every_aligned_pair() -> None: @pytest.mark.slow -def test_the_revocation_population_splits_as_224_accepted_plus_6_declined() -> None: - """Reconciles this module's 230 with the research record's §3.2 figure of 224. +def test_the_revocation_population_splits_as_217_accepted_plus_6_declined() -> None: + """Reconciles this module's 223 with the research record's §3.2 figure of 224. §3.2 counted the split population over the pairs a user can actually reach; this module sweeps every adjacent committed pair, including the six ``compare.pdf`` declines. The @@ -196,8 +197,12 @@ def test_the_revocation_population_splits_as_224_accepted_plus_6_declined() -> N pair, rather than from ``compare.pdf._is_unnumbered_layout`` — the same choice ``test_pdf_canonical_baseline`` makes, and it keeps this off a private cross-module import. - Measured and pinned rather than left as a plausible explanation: 224 + 6 = 230, with the - 224 landing exactly on §3.2's number. + Measured and pinned rather than left as a plausible explanation: 217 + 6 = 223. §3.2's 224 + was measured on the parser before ADR 0022; its heading passes move block boundaries on three + accepted pairs (115-hr-5895 v3→v4 −2, 118-hr-4366 v3→v4 +2, 118-hr-4366 v4→v5 +1), and the + heading breaks read from print and grammar move two of them again (115-hr-5895 v3→v4 −5, + 118-hr-4366 v3→v4 −3): headings that now agree across versions leave fewer aligned blocks + to split. The declined side is unchanged. """ baseline = json.loads((DATA_DIR / "pdf_canonical_baseline.json").read_text()) accepted = declined = 0 @@ -209,9 +214,9 @@ def test_the_revocation_population_splits_as_224_accepted_plus_6_declined() -> N else: accepted += revoked - assert (accepted, declined) == (224, 6), ( + assert (accepted, declined) == (217, 6), ( f"the split population partitions as {accepted} accepted + {declined} declined, not " - "224 + 6. §3.2's figure and this module's now describe different populations for a " + "217 + 6. §3.2's figure and this module's now describe different populations for a " "reason that is no longer the admissibility split." ) diff --git a/tests/test_pdf_size_detection.py b/tests/test_pdf_size_detection.py index 06ca2699..da4169ae 100644 --- a/tests/test_pdf_size_detection.py +++ b/tests/test_pdf_size_detection.py @@ -19,6 +19,10 @@ BODY = 14.0 HEAD = 11.2 +# A body line that ends a paragraph before a heading ends a sentence (`.`, `:`), as GPO prints +# it: a line that follows unfinished prose continues that sentence and is never a heading +# (ADR 0022). History: fixtures before ADR 0022 left the period off. + # Synthetic page geometry for the major detector's stacked-vs-wrapped split (#130), # which reads each line's horizontal extent. COLUMN is the justified body width GPO # prints (335-339 pt on the corpus subcommittee prints); the body-prose lines below @@ -177,7 +181,7 @@ def test_section_catchline_continuation_not_account(self): rows = [ (1, "SEC. 5. ACTIONS TO PROMOTE FREEDOM OF THE PRESS", HEAD), (2, "AND ASSEMBLY IN HAITI.", HEAD), - (3, "body prose of the section follows here", BODY), + (3, "body prose of the section follows here.", BODY), (4, "OPERATIONS AND SUPPORT", HEAD), (5, "the real account body prose runs here", BODY), ] @@ -195,7 +199,7 @@ def test_real_heading_after_section_with_body_still_emitted(self): rows = [ (1, "SEC. 5. ACTIONS TO PROMOTE FREEDOM OF THE PRESS", HEAD), (2, "AND ASSEMBLY IN HAITI.", HEAD), # catchline continuation (suppressed) - (3, "body prose of the section runs here now", BODY), + (3, "body prose of the section runs here now.", BODY), (4, "OPERATIONS AND SUPPORT", HEAD), # real account, separated by body (5, "the account body prose follows here", BODY), ] @@ -418,7 +422,7 @@ def test_carryover_agency_spans_multiple_accounts(self): (1, "TITLE I", BODY), (2, "MANAGEMENT DIRECTORATE", HEAD), # agency (3, "OPERATIONS AND SUPPORT", HEAD), # account A - (4, "the body prose for account A here", BODY), + (4, "the body prose for account A here.", BODY), (5, "PROCUREMENT, CONSTRUCTION, AND IMPROVEMENTS", HEAD), # account B (6, "the body prose for account B here", BODY), ] @@ -439,7 +443,7 @@ def test_second_agency_overrides_first(self): (1, "TITLE I", BODY), (2, "MANAGEMENT DIRECTORATE", HEAD), # agency A (3, "OPERATIONS AND SUPPORT", HEAD), # account under A - (4, "the body prose for account A here", BODY), + (4, "the body prose for account A here.", BODY), (5, "FEDERAL EMERGENCY MANAGEMENT AGENCY", HEAD), # agency B (6, "PROCUREMENT, CONSTRUCTION, AND IMPROVEMENTS", HEAD), # account under B (7, "the body prose for account B here", BODY), @@ -523,10 +527,10 @@ def test_agency_scope_resets_at_grouping_boundary(self): (1, "TITLE I", BODY), (2, "MANAGEMENT DIRECTORATE", HEAD), # agency (3, "OPERATIONS AND SUPPORT", HEAD), # account under the agency - (4, "the body prose for the account here", BODY), + (4, "the body prose for the account here.", BODY), (5, "ADMINISTRATIVE PROVISIONS", HEAD), # grouping header (6, "SEC. 101. (a) The Secretary shall act here.", BODY), - (7, "more body prose for the section here", BODY), + (7, "more body prose for the section here.", BODY), (8, "WORKING CAPITAL FUND", HEAD), # title-level account after the grouping (9, "the body prose for the later account", BODY), ] @@ -871,7 +875,7 @@ def test_breadcrumb_title_major_grouping_section(self): (5, "For necessary expenses of the office, $100.", BODY), (6, "ADMINISTRATIVE PROVISIONS", HEAD), # grouping header (7, "SEC. 101. (a) The Secretary shall act here.", BODY), - (8, "more body prose for the section here", BODY), + (8, "more body prose for the section here.", BODY), ] anchors = extract_anchors([_page(1, rows)]) sec = next(a for a in anchors if a.kind == "section" and a.text == "SEC. 101") @@ -891,10 +895,10 @@ def test_breadcrumb_post_grouping_account_keeps_major(self): (2, "DEPARTMENTAL MANAGEMENT", BODY), # major (3, "MANAGEMENT DIRECTORATE", HEAD), # agency (4, "OPERATIONS AND SUPPORT", HEAD), # account under the agency - (5, "the body prose for the account here", BODY), + (5, "the body prose for the account here.", BODY), (6, "ADMINISTRATIVE PROVISIONS", HEAD), # grouping header (7, "SEC. 101. (a) The Secretary shall act here.", BODY), - (8, "more body prose for the section here", BODY), + (8, "more body prose for the section here.", BODY), (9, "WORKING CAPITAL FUND", HEAD), # title-level account after grouping (10, "the body prose for the later account", BODY), ] @@ -941,7 +945,7 @@ def test_major_and_agency_span_multiple_accounts(self): (2, "DEPARTMENTAL MANAGEMENT", BODY), # major (3, "MANAGEMENT DIRECTORATE", HEAD), # agency (4, "OPERATIONS AND SUPPORT", HEAD), # account A - (5, "the body prose for account A here", BODY), + (5, "the body prose for account A here.", BODY), (6, "PROCUREMENT, CONSTRUCTION, AND IMPROVEMENTS", HEAD), # account B (7, "the body prose for account B here", BODY), ] @@ -963,7 +967,7 @@ def test_earlier_major_does_not_bleed_into_later_title(self): (1, "TITLE I", BODY), (2, "DEPARTMENTAL MANAGEMENT", BODY), # major A (3, "OPERATIONS AND SUPPORT", HEAD), # account under A - (4, "the body prose for account A here", BODY), + (4, "the body prose for account A here.", BODY), (5, "TITLE II", BODY), (6, "SECURITY, ENFORCEMENT, AND INVESTIGATIONS", BODY), # major B (7, "PROCUREMENT, CONSTRUCTION, AND IMPROVEMENTS", HEAD), # account under B diff --git a/tests/test_pdf_text.py b/tests/test_pdf_text.py index e00876d3..30da39e3 100644 --- a/tests/test_pdf_text.py +++ b/tests/test_pdf_text.py @@ -8,6 +8,7 @@ Line, Page, _first_word_right, + _initial_caps, _merge_print_lines, _page_glyph_sizes, _parse_print_lines, @@ -235,6 +236,18 @@ def test_falls_back_to_wide_gap_when_no_space_glyph(self): right = [(0.0, gap_start, gap_start + 6.0, ord("OF"[i]), 11.0) for i in range(2)] assert _first_word_right(left + right) == left[-1][2] + def test_a_narrow_word_gap_with_no_space_glyph_still_ends_the_word(self): + # 114-hr-2029 (Senate) p.78: SECURITY INVESTMENT PROGRAM carries no space glyph, and its + # word gap is under 0.25 of the size (0.5% of heading word gaps are). Read with that + # threshold the first word ran on to INVESTMENT, the fullness veto thought the lower line's + # first word could not have fitted, and NORTH ATLANTIC TREATY ORGANIZATION over it merged. + left = self._glyphs("SECURITY") + gap_start = left[-1][2] + 0.22 * 11.0 # a real word gap: over 0.2, under 0.25 of the size + right = [ + (0.0, gap_start + i * 6.5, gap_start + i * 6.5 + 6.0, ord(c), 11.0) for i, c in enumerate("INVESTMENT") + ] + assert _first_word_right(left + right) == left[-1][2] + def test_skips_leading_space_glyph(self): glyphs = self._glyphs(" RELATED") # stray leading space assert _first_word_right(glyphs) == glyphs[-1][2] # 'RELATED' right edge @@ -304,6 +317,80 @@ def test_sizes_attached_to_lines_after_extract(self): assert by_num[12].glyph_size < by_num[13].glyph_size +def _glyphs( + words: list[tuple[str, list[float]]], *, spaces: bool = False, letter_gap: float = 0.8, word_gap: float = 4.5 +): + """Synthetic x-ordered content glyphs (bottom, left, right, cp, size) for `_initial_caps`: + each word with a size per letter, 7 pt advance per letter. GPO's small-caps headings usually + carry no space glyph; `spaces=True` adds one between words, as some prints do.""" + out, x = [], 100.0 + for w, (text, sizes) in enumerate(words): + if w: + if spaces: + out.append((0.0, x, x, 32, sizes[0])) + x += word_gap - letter_gap + for ch, size in zip(text, sizes, strict=True): + out.append((0.0, x, x + 7.0, ord(ch), size)) + x += 7.0 + letter_gap + return out + + +SMALL, CAP = 11.2, 14.0 + + +class TestInitialCaps: + """The case pattern of a small-caps heading line (ADR 0022): title case, even, or unknown.""" + + def test_a_title_case_line_with_no_space_glyphs_that_opens_with_a_small_word(self): + # 118-s-4928 p.75 line 4, "and Efficiency": judged by the whole line's first letter, the + # small "A" of "and", it read as even and split the council's name in two. + line = _glyphs([("AND", [SMALL] * 3), ("EFFICIENCY", [CAP] + [SMALL] * 9)]) + assert _initial_caps(line) is True + + def test_an_even_line_with_no_space_glyphs(self): + line = _glyphs([("SALARIES", [SMALL] * 8), ("AND", [SMALL] * 3), ("EXPENSES", [SMALL] * 8)]) + assert _initial_caps(line) is False + + def test_the_letters_of_one_word_stay_one_word(self): + # Only the first letter is large; were the word cut between its letters, a fragment + # starting with a small letter would vote "even". + assert _initial_caps(_glyphs([("INTEGRITY", [CAP] + [SMALL] * 8)])) is True + + def test_space_glyphs_still_split_words(self): + line = _glyphs( + [ + ("OFFICE", [CAP] + [SMALL] * 5), + ("OF", [SMALL] * 2), + ("THE", [SMALL] * 3), + ("SECRETARY", [CAP] + [SMALL] * 8), + ], + spaces=True, + word_gap=0.8, + ) + assert _initial_caps(line) is True + + def test_an_all_capital_acronym_does_not_vote(self): + # "FDA" and "NIH" print entirely in large capitals: no case pattern of their own. Counted + # as even they outvote the one title-case word and the line reads even. + line = _glyphs([("FDA", [CAP] * 3), ("NIH", [CAP] * 3), ("INNOVATION", [CAP] + [SMALL] * 9)], spaces=True) + assert _initial_caps(line) is True + + def test_the_corpus_line_that_split_the_council_name(self): + import pypdfium2 as pdfium + + pdf = pdfium.PdfDocument(str(fixture_path("118-s-4928", "1_reported-in-senate.pdf"))) + try: + tp = pdf[74].get_textpage() # page 75 + try: + geoms = {ln: geom for ln, (_size, geom) in _page_glyph_sizes(tp, tp.get_text_range()).items()} + finally: + tp.close() + finally: + pdf.close() + # 3 COUNCIL OF THE INSPECTORS GENERAL ON INTEGRITY / 4 AND EFFICIENCY / 5 SALARIES AND EXPENSES + assert (geoms[3].initial_caps, geoms[4].initial_caps, geoms[5].initial_caps) == (True, True, False) + + class TestPageRangeText: def test_concatenates_pages_in_range(self): pages = [_page(1, "first"), _page(2, "second"), _page(3, "third")] diff --git a/tests/test_pipeline_parity.py b/tests/test_pipeline_parity.py index db59d6c9..d353b736 100644 --- a/tests/test_pipeline_parity.py +++ b/tests/test_pipeline_parity.py @@ -73,14 +73,16 @@ # signature as #96's PDF recalibration. PDF totals untouched. "118-hr-8752": ( (39, 39), - (37, 37), - "clean; XML +2 = bare subsections of added SEC.s, which the PDF folds into the section block (#188)", + (38, 38), + "clean; XML +2 = bare subsections of added SEC.s, which the PDF folds into the section block (#188); " + "PDF +1 = the added SPENDING REDUCTION ACCOUNT heading, its own block since ADR 0022", ), "118-hr-8774": ((30, 33), (31, 36), "PDF over-segments a few blocks (segmentation granularity)"), "117-hr-4502": ( (1390, 1445), (1430, 1520), - "PDF over-segments a large added block; XML +304 added from subsection nodes (#188) — gap narrowed +389→+85", + "PDF over-segments a large added block; XML +304 added from subsection nodes (#188) — gap narrowed +389→+85; " + "ADR 0022 joins wrapped account names into one block, +85→+41", ), "115-hr-5895": ( (290, 315), diff --git a/tests/test_structure_vocabulary_gate.py b/tests/test_structure_vocabulary_gate.py index e64f3a4d..591d4909 100644 --- a/tests/test_structure_vocabulary_gate.py +++ b/tests/test_structure_vocabulary_gate.py @@ -50,6 +50,11 @@ # parser consumes result-bearingly, so it gets a narrower guard of its own below -- # see `test_the_amount_primitive_reads_only_the_amendment_vocabulary`. "amounts.py": "financial layer — the (increased|reduced|decreased) by $X primitive", + # The typed ledger (ADR 0023): its classifier reads appropriations wording to say what + # an amount is. It takes its sections from the structure tree and never decides one. + "financial.py": "financial layer — the ledger's clause classifier (ADR 0023)", + # The report's financial views: they name the ledger's money types (and style them). + "formatters/financial_views.py": "financial layer — the ledger's views (ADR 0023)", # Parses the comparative-statement TABLE in a committee report, a different # document used as independent ground truth (ADR 0009), not bill structure. "parsers/committee_report.py": "committee-report table parser (ADR 0009 carve-out)", diff --git a/web/app.py b/web/app.py index e530f2d6..cc264569 100644 --- a/web/app.py +++ b/web/app.py @@ -50,10 +50,10 @@ # throttled. COMPARE_RATE_LIMIT_PER_MINUTE = 10 -# Format → (label-extension, html entry point, json entry point). +# Format → (html entry point, json entry point). _COMPARE = { - "pdf": (".pdf", compare_pdfs_html, compare_pdfs), - "xml": (".xml", compare_xml_html, compare_xml), + "pdf": (compare_pdfs_html, compare_pdfs), + "xml": (compare_xml_html, compare_xml), } app = FastAPI( @@ -245,16 +245,20 @@ async def _read_upload(upload: UploadFile, field: str, fmt: str) -> bytes: return data -def _label_from_filename(name: str | None, fallback: str, ext: str) -> str: - """Derive a human label from the uploaded filename, defensively (strip any - path components a client might send, drop the format extension).""" +def _label_from_filename(name: str | None, fallback: str) -> str: + """The uploaded filename as the version's label, defensively (strip any path + components a client might send). + + Kept whole, extension included: people name files by their own conventions, so the + name says nothing the app can rely on (a stage, which version is newer), and only + the uploader knows what it means. The report header shows it exactly as they chose it. + The same label also reaches the financial views, the CSV exports and diff.json; how + it should read there was not part of that decision and is open to revisit. + """ if not name: return fallback - stem = name.rsplit("/", 1)[-1].rsplit("\\", 1)[-1] - if stem.lower().endswith(ext): - stem = stem[: -len(ext)] - stem = stem.strip() - return stem or fallback + label = name.rsplit("/", 1)[-1].rsplit("\\", 1)[-1].strip() + return label or fallback @app.post("/api/compare") @@ -264,12 +268,12 @@ async def compare( output: str = Query("html", pattern="^(html|json)$"), fmt: str = Query("pdf", alias="format", pattern="^(pdf|xml)$"), ): - ext, html_fn, json_fn = _COMPARE[fmt] + html_fn, json_fn = _COMPARE[fmt] start_bytes = await _read_upload(start_file, "start_file", fmt) end_bytes = await _read_upload(end_file, "end_file", fmt) - start_label = _label_from_filename(start_file.filename, "Start version", ext) - end_label = _label_from_filename(end_file.filename, "End version", ext) + start_label = _label_from_filename(start_file.filename, "Start version") + end_label = _label_from_filename(end_file.filename, "End version") compare_fn = html_fn if output == "html" else json_fn diff --git a/web/webapp/sample/example.html b/web/webapp/sample/example.html index 1ec03259..f224bfc1 100644 --- a/web/webapp/sample/example.html +++ b/web/webapp/sample/example.html @@ -19,6 +19,8 @@ --gold: #c9944e; --destructive: #c04040; --success: #3d9b6d; + --info: #e3f2f8; + --info-foreground: #0b4f63; --diff-add: #d3f0e2; --diff-add-foreground: #1a6647; --diff-remove: #f5ddd8; @@ -27,7 +29,22 @@ --diff-modified-foreground: #8a6320; --diff-moved: #eef0f8; --diff-moved-foreground: #2c2c5c; + --fin-transfer: #cffafe; + --fin-transfer-foreground: #155e75; + --fin-authorization: #e0f2fe; + --fin-authorization-foreground: #0c4a6e; + --fin-fee: #ffedd5; + --fin-fee-foreground: #9a3412; + --fin-directive: #e0e7ff; + --fin-directive-foreground: #3730a3; + --fin-earmark: #ede9fe; + --fin-earmark-foreground: #5b21b6; + --fin-availability: #d1fae5; + --fin-availability-foreground: #065f46; + --fin-sub-allocation: #dbeafe; + --fin-sub-allocation-foreground: #1e40af; --radius: 0.625rem; + --sticky-bar-height: 64px; --font-sans: ui-sans-serif, system-ui, -apple-system, 'Segoe UI', Roboto, Helvetica, Arial, sans-serif; --font-serif: ui-serif, Georgia, 'Times New Roman', serif; --font-mono: ui-monospace, 'SF Mono', Menlo, Consolas, monospace; @@ -88,11 +105,13 @@ .filter-empty[hidden] { display: none; } /* Main content */ -.main { margin-left: 280px; padding: 28px 36px; max-width: 940px; flex: 1; } +.main { margin-left: 280px; padding: 28px 36px; max-width: 940px; flex: 1; min-width: 0; } /* Header */ .report-header h1 { font-size: 24px; margin-bottom: 4px; } -.report-header .versions { color: var(--muted-foreground); font-size: 15px; margin-bottom: 16px; } +.report-header .versions { color: var(--muted-foreground); font-size: 15px; margin-bottom: 16px; line-height: 1.5; } +.versions__row { display: block; overflow-wrap: anywhere; } +.versions__row b { color: var(--foreground); font-weight: 600; } .summary-bar { display: flex; gap: 10px; margin-bottom: 24px; flex-wrap: wrap; } .summary-item { display: inline-flex; align-items: center; gap: 6px; padding: 4px 12px; border-radius: 999px; font-size: 13px; background: var(--secondary); } @@ -164,16 +183,25 @@ border-bottom: 1px solid var(--border); padding: 10px 0; margin-bottom: 16px; } .action-bar__left { display: flex; align-items: center; gap: 10px; flex-wrap: wrap; } .action-bar__group { display: flex; align-items: center; gap: 10px; flex-wrap: wrap; } -.find-bar { display: inline-flex; align-items: center; gap: 4px; } -.find-bar input { padding: 5px 10px; border: 1px solid var(--border); border-radius: var(--radius); - font: inherit; font-family: var(--font-sans); font-size: 13px; width: 180px; background: var(--card); } -.find-bar button { padding: 5px 9px; border: 1px solid var(--border); border-radius: var(--radius); - background: var(--card); cursor: pointer; font-family: var(--font-sans); font-size: 13px; } +.action-bar__left { flex: 1 1 auto; } +/* The page's one search: it reaches text that is collapsed, which the browser's own find + cannot, so it is sized and marked to be found first. */ +.find-bar { display: inline-flex; align-items: center; gap: 8px; flex: 1 1 380px; max-width: 640px; } +.find-field { position: relative; display: flex; flex: 1; min-width: 0; } +.find-icon { position: absolute; left: 14px; top: 50%; width: 22px; height: 22px; transform: translateY(-50%); + fill: none; stroke: var(--muted-foreground); stroke-width: 2.2; stroke-linecap: round; pointer-events: none; } +.find-bar input { width: 100%; padding: 10px 14px 10px 46px; border: 1px solid var(--border); + border-radius: var(--radius); font: inherit; font-family: var(--font-sans); font-size: 20px; + background: var(--card); color: var(--foreground); } +.find-bar input:focus { outline: 2px solid var(--primary); outline-offset: 1px; border-color: var(--primary); } +.find-bar button { padding: 8px 12px; border: 1px solid var(--border); border-radius: var(--radius); + background: var(--card); cursor: pointer; font-family: var(--font-sans); font-size: 15px; } .find-bar button:hover { background: var(--secondary); } .find-bar button[disabled] { opacity: 0.4; cursor: default; } .find-counter { font-variant-numeric: tabular-nums; font-size: 12px; color: var(--muted-foreground); min-width: 3.5em; text-align: center; } -mark.find-hit { background: var(--accent); color: inherit; border-radius: 2px; scroll-margin-top: 64px; } +mark.find-hit { background: var(--accent); color: inherit; border-radius: 2px; + scroll-margin-top: calc(var(--sticky-bar-height) + 8px); } mark.find-hit--current { background: var(--gold); color: #fff; } .nav-controls { display: inline-flex; align-items: center; gap: 4px; } .nav-controls button { padding: 6px 12px; border: 1px solid var(--border); border-radius: var(--radius); @@ -244,9 +272,13 @@ .prompt-copy:hover { background: var(--accent); } .prompt-text { line-height: 1.5; } -/* Nav targets clear the sticky action bar when scrolled to via Prev/Next */ +/* Nav targets clear the sticky action bar when scrolled to via Prev/Next or a link: + --sticky-bar-height is the bar's measured height (padForStickyBar), plus a small gap. A + fixed 64px cleared the one-row bar but not the two-row one. On the targets, not as the + page's scroll-padding: that would also count the bar's own controls as covered, so + clicking or tabbing to one would scroll the page. */ .change-card, .full-bill [id^="attr-"], .full-bill [id^="sec-"], .full-bill [id^="fb-off-"], -.removed-block { scroll-margin-top: 64px; } +.removed-block { scroll-margin-top: calc(var(--sticky-bar-height) + 8px); } /* Full-bill section TOC (sidebar variant) */ .sidebar-changes[hidden], .sidebar-toc[hidden] { display: none; } @@ -295,7 +327,25 @@ .find-bar { position: fixed; left: 0; right: 0; bottom: 46px; z-index: 35; justify-content: center; background: var(--card); border-top: 1px solid var(--border); padding: 8px 16px; } - .find-bar input { flex: 1; max-width: 320px; } + .find-bar { max-width: none; } + body.outside-changes { padding-bottom: 62px; } + body.outside-changes .find-bar { bottom: 0; padding-bottom: calc(8px + env(safe-area-inset-bottom)); } +} +/* Change navigation and the changes export serve Changes and Full bill only. */ +body.outside-changes .nav-controls, body.outside-changes #export-open { display: none; } + +/* Five view buttons get a row of their own when the financial views are present (ADR 0023). + The views' own styles live with them, in formatters/financial_views.py. */ +.action-bar--stacked { display: grid; grid-template-columns: 1fr auto; row-gap: 10px; align-items: center; } +.action-bar--stacked .action-bar__left { display: contents; } +.action-bar--stacked .view-toggle { grid-column: 1 / -1; justify-self: start; } +.action-bar--stacked .find-bar { grid-column: 1; } +.action-bar--stacked .action-bar__group { grid-column: 2; } +.fin-scroll { overflow-x: auto; } +@media (max-width: 820px) { + .action-bar--stacked { display: flex; flex-direction: column; align-items: stretch; } + .view-toggle { display: flex; flex-wrap: wrap; } + .view-toggle__btn { flex: 1 1 auto; } } /* Print */ @@ -304,6 +354,133 @@ .main { margin-left: 0; } .change-card { break-inside: avoid; } } +/* Financial views (ADR 0023): Version A / B, and the comparison */ +/* The three financial tabs use the full window on a wide screen: their tables are columns, + cramped at the report's 940px reading width. Changes and Full bill keep that width (their + cards are prose). `body[data-active-view]` is set by the report's view switch. */ +@media (min-width: 821px) { + body[data-active-view="fin-a"] .main, body[data-active-view="fin-b"] .main, + body[data-active-view="fin-compare"] .main { + max-width: none; padding-right: 10%; } +} +.fin-head { display: flex; align-items: baseline; justify-content: space-between; gap: 12px; flex-wrap: wrap; } +.view-fin-a h2, .view-fin-b h2, .view-fin-compare h2 { font-size: 20px; margin-bottom: 2px; } +.fin-version { color: var(--muted-foreground); font-size: 14px; margin-bottom: 12px; } +.fin-coverage, .fin-note, .fin-explain { font-size: 13px; color: var(--muted-foreground); margin-bottom: 10px; } +/* "About these figures": an information alert, left-accented, above each view's content. */ +.fin-alert { display: flex; gap: 12px; align-items: flex-start; margin: 4px 0 16px; padding: 12px 16px; + background: var(--info); color: var(--info-foreground); border: 1px solid var(--info); + border-left: 4px solid var(--info-foreground); border-radius: var(--radius); font-size: 14px; line-height: 1.55; } +.fin-alert__icon { flex: none; width: 20px; height: 20px; margin-top: 1px; fill: none; + stroke: var(--info-foreground); stroke-width: 2; stroke-linecap: round; } +.fin-alert__dot { fill: var(--info-foreground); } +.fin-alert a { color: inherit; font-weight: 600; text-decoration: underline; } +.fin-alert__version { display: block; margin-top: 4px; font-size: 12px; opacity: 0.85; } +.fin-coverage--gap { color: var(--destructive); font-weight: 600; } +.fin-empty { color: var(--muted-foreground); padding: 16px 2px; font-size: 14px; } +.view-fin-a h3, .view-fin-b h3 { font-size: 18px; margin: 18px 0 10px; } +.fin-key { border: 1px solid var(--border); border-radius: var(--radius); padding: 12px 14px; + background: var(--card); margin-bottom: 8px; } +.fin-key__title { font-weight: 600; font-size: 14px; margin-bottom: 8px; } +.fin-key__item { display: flex; align-items: baseline; gap: 8px; padding: 2px 0; } +.fin-key__count { color: var(--muted-foreground); font-size: 12px; min-width: 32px; display: inline-block; } +.fin-key__desc { color: var(--muted-foreground); font-size: 13px; } +.fin-type { display: inline-block; padding: 2px 7px; border-radius: 999px; font-size: 11px; font-weight: 600; + white-space: nowrap; background: var(--muted); color: var(--muted-foreground); } +.fin-type--appropriation { background: var(--diff-add); color: var(--diff-add-foreground); } +.fin-type--rescission, .fin-type--restriction { background: var(--diff-remove); color: var(--diff-remove-foreground); } +.fin-type--cap { background: var(--diff-modified); color: var(--diff-modified-foreground); } +.fin-type--transfer { background: var(--fin-transfer); color: var(--fin-transfer-foreground); } +.fin-type--authorization { background: var(--fin-authorization); color: var(--fin-authorization-foreground); } +.fin-type--fee { background: var(--fin-fee); color: var(--fin-fee-foreground); } +.fin-type--directive { background: var(--fin-directive); color: var(--fin-directive-foreground); } +.fin-type--earmark { background: var(--fin-earmark); color: var(--fin-earmark-foreground); } +.fin-type--availability { background: var(--fin-availability); color: var(--fin-availability-foreground); } +.fin-type--sub_allocation { background: var(--fin-sub-allocation); color: var(--fin-sub-allocation-foreground); } +.fin-sort { display: flex; gap: 8px; align-items: center; margin-bottom: 12px; font-size: 13px; } +.fin-sort span { color: var(--muted-foreground); } +.fin-sort__btn { padding: 4px 12px; border: 1px solid var(--border); border-radius: 999px; background: var(--card); + cursor: pointer; font: inherit; font-family: var(--font-sans); font-size: 13px; color: var(--foreground); } +.fin-sort__btn:hover { background: var(--secondary); } +.fin-sort__btn.is-active { background: var(--foreground); border-color: var(--foreground); color: var(--card); } +.fin-table { width: 100%; border-collapse: collapse; background: var(--card); border-radius: var(--radius); + box-shadow: var(--shadow-soft); overflow: hidden; font-size: 14px; table-layout: fixed; } +.fin-table th { background: var(--accent); text-align: left; padding: 9px 12px; border-bottom: 2px solid var(--border); + font-weight: 600; font-size: 13px; } +.fin-table td { padding: 8px 12px; border-bottom: 1px solid var(--border); vertical-align: middle; } +.fin-col-name { width: 72%; } .fin-col-type { width: 13%; } .fin-col-amount { width: 15%; text-align: right; } +.fin-amount { text-align: right; font-family: var(--font-mono); font-variant-numeric: tabular-nums; + white-space: nowrap; } +.fin-row, .fin-crow { cursor: pointer; } +.fin-row { background: var(--muted); font-weight: 600; } +.fin-row:hover td, .fin-crow:hover td { background: var(--secondary); } +.fin-row__name::before { content: "\25b6"; font-size: 10px; margin-right: 8px; color: var(--muted-foreground); + display: inline-block; transition: transform 0.15s; } +.fin-row.is-open .fin-row__name::before { transform: rotate(90deg); } +.fin-count { font-size: 11px; color: var(--muted-foreground); font-weight: 400; margin-left: 6px; } +.fin-review { background: var(--diff-modified); color: var(--diff-modified-foreground); font-size: 11px; + padding: 1px 6px; border-radius: 999px; margin-left: 6px; font-weight: 500; white-space: nowrap; } +/* The warning mark (`_WARNING_ICON`): drawn in the surrounding text's colour and size. */ +.fin-icon { display: inline-block; width: 1.15em; height: 1.15em; vertical-align: -0.2em; margin-right: 0.25em; + fill: none; stroke: currentColor; stroke-width: 2; stroke-linejoin: round; stroke-linecap: round; } +.fin-icon__dot { fill: currentColor; } +.fin-review[role="img"] .fin-icon { margin-right: 0; } +.fin-chip { font-size: 11px; font-weight: 600; border-radius: 4px; padding: 0 5px; margin-left: 6px; } +.fin-chip--law { background: var(--muted); color: var(--muted-foreground); border: 1px dashed var(--border); } +.fin-detail > td { padding: 0; background: var(--card); } +.fin-detail__inner { display: flex; } +.fin-detail__amounts { min-width: 220px; max-width: 260px; padding: 12px; border-right: 1px solid var(--border); + display: flex; flex-direction: column; gap: 10px; } +.fin-clause { display: flex; flex-direction: column; gap: 3px; align-items: flex-start; } +.fin-clause__amount { font-family: var(--font-mono); font-size: 15px; font-variant-numeric: tabular-nums; } +.fin-detail__text { padding: 12px 16px; font-size: 13px; line-height: 1.9; flex: 1; font-weight: 400; } +.fin-piece { border-radius: 3px; padding: 1px 3px; } +.fin-piece.fin-type--unknown { background: var(--muted); color: var(--foreground); } +.fin-proviso { color: var(--muted-foreground); } +.fin-card { background: var(--card); border: 1px solid var(--border); border-radius: var(--radius); overflow: hidden; + box-shadow: var(--shadow-soft); } +.fin-card .fin-table { box-shadow: none; border-radius: 0; table-layout: auto; } +.fin-summary { display: flex; gap: 18px; flex-wrap: wrap; padding: 10px 14px; border-bottom: 1px solid var(--border); + font-size: 13px; color: var(--muted-foreground); } +.fin-summary b { color: var(--foreground); font-variant-numeric: tabular-nums; } +.fin-table--compare td { vertical-align: top; } +.fin-table--compare th { background: var(--secondary); font-size: 12px; text-transform: uppercase; + letter-spacing: 0.04em; color: var(--muted-foreground); border-bottom: 1px solid var(--border); padding: 8px 10px; } +.fin-table--compare td { padding: 8px 10px; } +.fin-table--compare .fin-amount { font-family: var(--font-sans); } +.fin-stack__tag { white-space: nowrap; } +.fin-caret { color: var(--muted-foreground); width: 16px; display: inline-block; } +.fin-loc { font-weight: 600; } +.fin-loc small { display: block; font-weight: 400; color: var(--muted-foreground); font-size: 12px; } +/* Links from a comparison row to its sections in Version A / B. */ +.fin-jumps { display: flex; flex-wrap: wrap; gap: 4px 14px; margin-top: 6px; } +.fin-jump { background: none; border: 0; padding: 0; font: inherit; font-size: 12px; color: var(--primary); + text-decoration: underline; text-underline-offset: 2px; cursor: pointer; } +.fin-jump:hover { color: var(--foreground); } +.fin-jump--none { color: var(--muted-foreground); text-decoration: none; cursor: default; } +.fin-row.is-target > td { background: var(--accent); transition: background 0.6s ease; } +.fin-stack { display: grid; grid-template-columns: auto 1fr auto; gap: 2px 8px; align-items: center; } +.fin-ab { font-size: 11px; font-weight: 700; border-radius: 4px; padding: 0 5px; } +.fin-ab--a { background: var(--secondary); color: var(--primary); } +.fin-ab--b { background: var(--accent); color: var(--gold); } +.fin-stack__amt { text-align: right; font-variant-numeric: tabular-nums; white-space: nowrap; } +.fin-stack__amt--old { color: var(--muted-foreground); } +.fin-stack__amt--new { font-weight: 600; } +.fin-stack__none { color: var(--muted-foreground); font-style: italic; text-align: right; } +.fin-up { color: var(--diff-add-foreground); } .fin-down { color: var(--diff-remove-foreground); } +.fin-same { color: var(--muted-foreground); } +.fin-same small, .fin-up small, .fin-down small { display: block; font-size: 11px; } +.fin-cdetail > td { background: var(--background); padding: 12px 14px; } +.fin-subhead { font-size: 13px; font-weight: 600; color: var(--muted-foreground); padding: 14px 14px 6px; + border-top: 1px solid var(--border); } +.fin-table--plain { box-shadow: none; } +@media (max-width: 820px) { + .fin-table--compare { min-width: 640px; } + .fin-detail__inner { flex-direction: column; } + .fin-detail__amounts { max-width: none; border-right: 0; border-bottom: 1px solid var(--border); } + .fin-col-name { width: 55%; } .fin-col-type { width: 20%; } .fin-col-amount { width: 25%; } +} + @@ -317,24 +494,24 @@
- +
- +

H.R. 8752 — Making appropriations for the Department of Homeland Security for the fiscal year ending September 30, 2025, and for other purposes.

-
v1: reported-in-house → v2: engrossed-in-house · 118th Congress
-
modified 18added 16removed 2moved 1
+
Before (v1): reported-in-houseAfter (v2): engrossed-in-house118th Congress
+
modified 18added 17removed 2moved 1
-
+
-
- +
+
- +

Changes

@@ -571,21 +748,33 @@

TITLE IV > RESEARCH, DEVELOPMENT, TRAINING, AND SERVICES > ADMINISTRAT
TITLE V
GENERAL PROVISIONS -
SEC. 552 +
SPENDING REDUCTION ACCOUNT
added -

TITLE V > GENERAL PROVISIONS > SEC. 552

+

TITLE V > GENERAL PROVISIONS > SPENDING REDUCTION ACCOUNT

+
+
— (new in v2)p.97 L13
+
SPENDING REDUCTION ACCOUNT
+
+
+
+
SPENDING REDUCTION ACCOUNT +
SEC. 552 +
+
+added +

TITLE V > SPENDING REDUCTION ACCOUNT > SEC. 552

— (new in v2)p.97 L14
SEC. 552. $0 (increased by $10,000,000).
SEC. 553 -
+
added -

TITLE V > GENERAL PROVISIONS > SEC. 553

+

TITLE V > SPENDING REDUCTION ACCOUNT > SEC. 553

— (new in v2)p.97 L15 – p.100 L6
SEC. 553. (a) None of the funds made available by @@ -652,10 +841,10 @@

TITLE V > GENERAL PROVISIONS > SEC. 553

SEC. 554 -
+
added -

TITLE V > GENERAL PROVISIONS > SEC. 554

+

TITLE V > SPENDING REDUCTION ACCOUNT > SEC. 554

— (new in v2)p.100 L7 – p.100 L12
SEC. 554. None of the funds made available by this @@ -666,10 +855,10 @@

TITLE V > GENERAL PROVISIONS > SEC. 554

SEC. 555 -
+
added -

TITLE V > GENERAL PROVISIONS > SEC. 555

+

TITLE V > SPENDING REDUCTION ACCOUNT > SEC. 555

— (new in v2)p.100 L13 – p.100 L16
SEC. 555. None of the funds appropriated or otherwise made available by this Act may be used to implement, @@ -678,10 +867,10 @@

TITLE V > GENERAL PROVISIONS > SEC. 555

SEC. 556 -
+
added -

TITLE V > GENERAL PROVISIONS > SEC. 556

+

TITLE V > SPENDING REDUCTION ACCOUNT > SEC. 556

— (new in v2)p.100 L17 – p.100 L20
SEC. 556. None of the funds made available in this @@ -691,10 +880,10 @@

TITLE V > GENERAL PROVISIONS > SEC. 556

SEC. 557 -
+
added -

TITLE V > GENERAL PROVISIONS > SEC. 557

+

TITLE V > SPENDING REDUCTION ACCOUNT > SEC. 557

— (new in v2)p.100 L21 – p.100 L22
SEC. 557. None of the funds made available by this @@ -702,10 +891,10 @@

TITLE V > GENERAL PROVISIONS > SEC. 557

SEC. 558 -
+
added -

TITLE V > GENERAL PROVISIONS > SEC. 558

+

TITLE V > SPENDING REDUCTION ACCOUNT > SEC. 558

— (new in v2)p.100 L23 – p.100 L25
SEC. 558. None of the funds made available by this @@ -714,10 +903,10 @@

TITLE V > GENERAL PROVISIONS > SEC. 558

SEC. 559 -
+
added -

TITLE V > GENERAL PROVISIONS > SEC. 559

+

TITLE V > SPENDING REDUCTION ACCOUNT > SEC. 559

— (new in v2)p.101 L1 – p.101 L2
SEC. 559. None of the funds made available by this @@ -725,10 +914,10 @@

TITLE V > GENERAL PROVISIONS > SEC. 559

SEC. 560 -
+
added -

TITLE V > GENERAL PROVISIONS > SEC. 560

+

TITLE V > SPENDING REDUCTION ACCOUNT > SEC. 560

— (new in v2)p.101 L4 – p.101 L5
SEC. 560. None of the funds made available under @@ -736,10 +925,10 @@

TITLE V > GENERAL PROVISIONS > SEC. 560

SEC. 561 -
+
added -

TITLE V > GENERAL PROVISIONS > SEC. 561

+

TITLE V > SPENDING REDUCTION ACCOUNT > SEC. 561

— (new in v2)p.101 L8 – p.101 L11
SEC. 561. None of the funds appropriated or otherwise made available by this Act may be made available @@ -748,10 +937,10 @@

TITLE V > GENERAL PROVISIONS > SEC. 561

SEC. 562 -
+
added -

TITLE V > GENERAL PROVISIONS > SEC. 562

+

TITLE V > SPENDING REDUCTION ACCOUNT > SEC. 562

— (new in v2)p.101 L12 – p.102 L9
SEC. 562. None of the funds made available by this @@ -771,10 +960,10 @@

TITLE V > GENERAL PROVISIONS > SEC. 562

SEC. 563 -
+
added -

TITLE V > GENERAL PROVISIONS > SEC. 563

+

TITLE V > SPENDING REDUCTION ACCOUNT > SEC. 563

— (new in v2)p.102 L10 – p.102 L12
SEC. 563. The salary of Nejwa Ali, U.S. Citizenship @@ -783,10 +972,10 @@

TITLE V > GENERAL PROVISIONS > SEC. 563

SEC. 564 -
+
added -

TITLE V > GENERAL PROVISIONS > SEC. 564

+

TITLE V > SPENDING REDUCTION ACCOUNT > SEC. 564

— (new in v2)p.102 L13 – p.102 L16
SEC. 564. None of the funds made available by this @@ -796,10 +985,10 @@

TITLE V > GENERAL PROVISIONS > SEC. 564

SEC. 565 -
+
added -

TITLE V > GENERAL PROVISIONS > SEC. 565

+

TITLE V > SPENDING REDUCTION ACCOUNT > SEC. 565

— (new in v2)p.102 L17 – p.102 L20
SEC. 565. None of the funds made available by this @@ -809,10 +998,10 @@

TITLE V > GENERAL PROVISIONS > SEC. 565

SEC. 566 -
+
added -

TITLE V > GENERAL PROVISIONS > SEC. 566

+

TITLE V > SPENDING REDUCTION ACCOUNT > SEC. 566

— (new in v2)p.102 L21 – p.102 L25
SEC. 566. None of the funds made available by this @@ -856,10 +1045,10 @@

TITLE V > GENERAL PROVISIONS > SEC. 552

Committed to the Committee of the Whole House on the State of the Union and ordered to be printed
-
+
added -

TITLE V > GENERAL PROVISIONS > SEC. 567

+

TITLE V > SPENDING REDUCTION ACCOUNT > SEC. 567

— (new in v2)p.103 L1 – p.104
SEC. 567. None of the funds appropriated or otherwise made available by this Act may be used to make payments to the Department of Veterans Affairs for medical @@ -876,36 +1065,77 @@

TITLE V > GENERAL PROVISIONS > SEC. 567

Homeland Security for the fiscal year ending
-

+ 16 labor is, or services are, of a temporary or seasonal nature.
- + \ No newline at end of file