Repository navigation
feat(nowcast): grade final-demand column totals against BEA by year - #1159
Conversation
Adds bedrock/analysis/nowcasting/final_demand_vs_published.py. For each year of a named balanced build (default: the canonical config's nowcast_mut_vintage, fetched from GCS if not local), it sums each final-demand column of the balanced detail Use SUT and compares it with the same column of BEA's published summary Use SUT. --check exits non-zero when a column-year is outside both 2% and $1bn; --reference latest compares against the 1997-2024 workbook for every year (#1157). On v0.5.0_a166975 it fails on exports (F040, 2.6-10.3% short every year: the S00900 rest-of-world adjustment is zero, #1158), on inventories (F030, $2-28bn), and on F07S in 2023 (-27%). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
bl-young
left a comment
bl-young
left a comment
There was a problem hiding this comment.
Approve -- the check is apples-to-apples (same $M units, commodity set, and code[:4] FD rollup as use_summary_sut column totals), and I reproduced the failures locally on v0.5.0_a166975.
Inline notes below are docstring / tolerance clarity, not blockers.
| (default: the canonical config's ``nowcast_mut_vintage``) and fetches it from | ||
| GCS when it is not local, so the answer is about the tables the model ships. | ||
|
|
||
| ⚠️ **Most final-demand columns cannot fail this check by construction.** The |
There was a problem hiding this comment.
Nit on "cannot fail by construction": conditioning only scales reachable same-sign groups against the as_read workbook (nowcast_fd_conditioning empty-group / sign-mismatch guards). F07S 2023 and --reference latest (e.g. F02S 2022 in the docstring below) are the escapes. Worth softening to that.
There was a problem hiding this comment.
Softened in 06f2417: the docstring now says the conditioned columns rarely fail, and names the two ways out: the empty-group and sign-mismatch guards that hold a factor at 1.0 (F07S 2023), and --reference latest (F02S 2022).
| larger column, so the same dollars stay under the 2% bar there: ``F040`` | ||
| fails, ``F010`` does not. | ||
|
|
||
| ❌ **``F07S`` federal nondefense software investment is 27% short in 2023 |
There was a problem hiding this comment.
F07S is federal nondefense structures, not software (that is F07N / IPP).
Also worth naming the cause: on v0.5.0_a166975 the 2023 gap is almost entirely the Used summary group (ours about -$5.5bn vs published +$0.16bn) -- the FD conditioning sign-mismatch guard holding the factor at 1.0 -- not a broad structures miss.
There was a problem hiding this comment.
Fixed in 06f2417: F07S is federal nondefense structures. The docstring also names the cause: the Used group (ours about -$5.5bn vs published +$0.16bn), with the sign-mismatch guard holding the factor at 1.0.
| table['relative_gap'] = table['gap'] / table['published'].abs().where( | ||
| table['published'] != 0 | ||
| ) | ||
| table['fails'] = (table['gap'].abs() > ABSOLUTE_TOLERANCE_USD_M) & ( |
There was a problem hiding this comment.
The AND rule lets F010 pass with a ~$200bn gap every year (+1.5% < 2%). Same S00900 dollars as the F040 fail -- so --check encodes "F040 fails, F010 does not" by the relative bar, not because PCE is clean.
Fine if intentional for the Phase 3 review bar; worth one sentence in the Tolerance section so a reader does not treat a green F010 as "matched".
There was a problem hiding this comment.
Intentional for the Phase 3 bar. The Tolerance section now says so: F010 passes with about $200bn every year on v0.5.0_a166975, the same S00900 dollars that fail F040, so a pass on a large column is not a match. #1171 moves those dollars in Step 6.
| codes = [ | ||
| c for c in USA_2017_SUMMARY_FINAL_DEMAND_CODES if c in published.columns | ||
| ] | ||
| theirs = published.reindex(index=list(USA_2017_SUMMARY_COMMODITY_CODES))[ |
There was a problem hiding this comment.
Optional harden: reindex silently drops missing rows. Asserting Used and Other are present would keep a BEA label change from zeroing mass without error.
There was a problem hiding this comment.
Added in 06f2417: the check raises a KeyError naming the year if the published Use has no Used or Other row, before the reindex. It runs clean on both references for v1.0.0_5514c3b.
- Say when conditioned columns can still fail (empty-group and sign-mismatch guards; --reference latest) instead of "by construction". - F07S is federal nondefense structures; its 2023 miss is the Used group. - Note that a pass on a large column is not a match (F010 passes ~200bn). - Raise if the published Use loses its Used or Other row. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
What this adds
bedrock/analysis/nowcasting/final_demand_vs_published.py: a check that grades each final-demand column total of a balanced nowcast build against BEA's published summary Use table, for every year 2017–2024.--build), defaulting to the canonical config'snowcast_mut_vintage. It fetches the balanced Use tables from GCS (flowsa/BalancedSUT/) when they are not on disk. The existinguse_summary_sutsection insections.pygrades cells, has no pass/fail, and reads whichever build on disk is newest by modification time.--checkexits non-zero when any column-year is outside both 2% of BEA's total and $1bn. The 2% bar is the one set for the Phase 3 review; the $1bn floor keeps rounding on near-zero columns from failing.--reference latestcompares against the 1997–2024 BEA workbook for every year instead of the workbook the build reads (see Nowcast 2017-2022 mixes BEA releases: summary supply-use controls are pinned to an older workbook while gross output is on the 2025 release ($1.3T of 2022 revisions) #1157).No unit tests, per the convention for analysis scripts;
--checkis the runnable verification.What it finds on
v0.5.0_a166975(the v1.0 build)F040exportsS00900is zero in every nowcast year (#1158).F010PCEF030inventoriesF07Sfederal nondefense software investmentSo
--checkfails onmaintoday, by design: it records three open defects, not a regression.How to run
Refs #1158, #1157, #528, #658
🤖 Generated with Claude Code