feat: consumption-basis water supply with AWARE scarcity and groundwater - #33
Merged
koen-vg merged 12 commits intoJul 22, 2026
Merged
Conversation
koen-vg
force-pushed
the
water-supply-restructure
branch
4 times, most recently
from
July 22, 2026 20:19
13dd5b5 to
3f155b9
Compare
Replace the single per-region growing-season water store, sized from Huang et al. withdrawals, with a consumption-basis water system. Crops draw their GAEZ net irrigation requirement E from per-region field buses; a delivery link draws C = E / eta_c from the regional pool, with eta_c calibrated at build time against observed WaterGAP irrigation consumption; withdrawal W is reported via the consumed fraction. The three quantities the literature routinely conflates are now distinct. Availability, monthly timing and within-region basin allocation come from WaterGAP 2.2e; AWARE 2.0 supplies the convex scarcity curve. Water supply and demand can be resolved at intra-year periods (water.temporal_resolution, a divisor of 12), so a season whose surface cannot meet its demand draws groundwater endogenously instead of being rescued by annual averaging. Demand is placed into periods by the observed MIRCA-OS irrigated crop calendar, retimed by iterative proportional fitting to WaterGAP's monthly requirement. Renewable and non-renewable groundwater bands are annual per-region resources (an aquifer can be pumped in any period) sitting additively on top of the period-bound surface supply, so mining emerges endogenously rather than being capped at renewable availability. The supply.mode ladder (single/scarcity/groundwater) is retired in favour of two independent switches, supply.scarcity_tiers and supply.groundwater: the ladder welded two orthogonal decisions to one enum and had no rung for collapsed tiers with the groundwater split, which is what main wants by default. The groundwater bands are derived from the pre-collapse curve, since renewable groundwater borrows the region's scarcest surface CF and collapsed tiers carry none. Defaults are chosen for cost, not fidelity: annual resolution and collapsed tiers, with the groundwater split kept. Both caveats are documented -- at annual resolution the whole year's pool serves every season, so the groundwater bands go nearly inert and reported depletion is an artefact of the resolution; and the AWARE pool is a looser constraint than the previous binding present-day withdrawal cap, so results move. tests/config/test.yaml pins quarterly resolution with tiers and groundwater on so CI still covers the full-fidelity path. New solve-time levers, both off by default: water_scarcity and groundwater_depletion pricing/capping. Scarcity pricing raises if scarcity_tiers is off rather than silently pricing an identically zero signal. Analysis gains per-region water metrics. Also fixes a GAEZ artefact where a few cells carry a negative net irrigation requirement, which flipped those crop links into spurious water producers, and removes the dead Water Footprint Network availability path along with the build inputs that went unused when the per-region water store was replaced.
The deleted WFN availability module left a dangling autodoc entry, and moving the water narrative into docs/water.rst orphaned the Hoekstra citation in crop_production.rst while duplicating the Huang one.
The cost-calibration and doc-figure configs still pinned the removed water.supply_scenario key, which the schema now rejects outright.
Two defects a model review surfaced in the water restructure. The rice-methane sensitivity dial hardcoded bus4/efficiency4 on crop_production. Rice CH4 now sits at bus(3 + T), and more than half of it moved to crop_production_multi, which the carrier filter never covered at any resolution. On the quarterly test build the mask matched zero links, so GSA sweeps of the CH4 factor silently left rice cultivation -- about a fifth of the CH4 budget -- unscaled. The dial now sweeps every secondary port of both crop carriers, as the GHG attribution extractor already did; verified to scale rice CH4 exactly 2.00x under a 2x factor. solve_and_analyze_model never got the water params, so any config with solving.inline_analysis (both GSA configs) died with AttributeError the moment run_solve read them. Mirrored from solve_model; a static cross-check now confirms every water param run_solve reads is defined in both solve rules and in the cluster manifest. Also from the review: blank a crop water bus when its requirement is non-finite rather than attaching one with a NaN efficiency (latent, no rows hit it today); raise instead of silently doing nothing when groundwater depletion is priced or capped with the bands switched off, mirroring the scarcity guard; and state plainly in the irrigation docstring that the eta_c feasibility floor and its warning are annual, so at finer resolution a period's demand can still exceed that period's surface and is met from the annual groundwater bands.
Second review round on the rebased branch: - Implement the static cross-check promised in the previous fix commit: tests/test_solve_rule_params.py parses run_solve's smk.params reads and fails when a param is missing from solve_model, solve_and_analyze_model, or the cluster manifest (build_scenario_entry). - Warn when a water-scarcity cap runs with groundwater on and mining neither priced nor capped: nonrenewable_cf only applies under scarcity pricing, so a bare cap can be met by CF-free mining. Documented in docs/water.rst and the cap docstring. - Fail loudly when an irrigated multi-cropping link references a region without water buses, which would silently grant free irrigation at solve time (consistency checks are skipped there). - Blank zero-demand period water buses on multi-cropping links, matching the single-crop path; port offsets stay reserved. - Drop dead generality: the tier-link block in primary_resources handled groundwater sources the tier table never carries (now asserted); remove the unread water.data.groundwater config knob; remove the backward-compatibility source-column fallback in extract_water_metrics. - Report cf as NaN for non-renewable rows in the per-tier water table; their efficiency2 is a volume tally, not a characterisation factor. - Remove the orphaned Water Footprint Network availability pipeline (download/extract/process rules, scripts and doc figure); nothing references it since the AWARE restructure. - Docs: correct the nonrenewable_cf default (100, not 30), note the cross-sector renewable-groundwater subtraction, document water_metrics.parquet in the analysis chapter, fix the supply-chain diagram label, and require water_scarcity/groundwater_depletion at the schema top level.
The water restructure declared region_water_tiers.csv as an output of build_region_water_current_use and documented the current_use source as a single zero-CF tier, but process_huang_irrigation_water.py never wrote the file. Nothing exercised the path -- the test config uses aware -- until cost calibration (which pins availability: current_use) failed on the missing output. Write the flat zero-scarcity tier per region-month in the shared schema.
Full tools/calibrate chains against the restructured water system (default set from config/default.yaml, gbd-anchored from config/gsa.yaml). The previous stamps recorded the pre-restructure water keys, so every non-test run failed the provenance check. Stability converged in 3 (default) / 4 (gbd-anchored) Broyden iterations with L1 costs near the previous centres -- default: cropland 1.47, grassland 0.24, feed 0.062; gbd-anchored: cropland 1.62, grassland 0.24, feed 0.056 -- so no residual supply/demand mismatch is hiding in the penalty.
…ismatch The regenerated cost artefacts are the first fit on the merged basin-aware region map, whose review fixes shifted the 750-region geometry: banana:CHL and sugarcane:CHL no longer build any link there (no actual-yield coverage on the new regions), so the artefact carries no rows for them. The 200-region test config still builds those links with a positive baseline, and the missing-correction check made every test build fail. The old artefact only passed by luck -- it predates the region-map fixes. Cross-resolution coverage gaps are inherent to consuming an artefact fit on a different region map, which is exactly what calibration.accept_provenance_mismatch declares. Such configs now get a warning and zero corrections for the missing pairs; provenance-matched configs keep the hard staleness error.
extract_mirca_os_subcrop_monthly only unpacked Rice2/Rice3, so the irrigated monthly grids build_mirca_crop_calendar needs had no producer in a clean checkout -- the CI dryrun failed with MissingInputException while local runs passed on manually extracted files. The rule now unpacks every calendar subcrop grid (single-sourced list shared with water.smk's input function) in the same single archive pass.
Resolves the open design questions from review: - Each basin's AWARE CF curve now spans the joint renewable envelope (WaterGAP surface delivery plus renewable groundwater) and is split at the basin's surface fraction: the lower slice is period-bound surface, the upper slice becomes annual per-region renewable-groundwater CF bands. This matches AWARE's own methodology (availability is basin discharge including baseflow; CF application is source-agnostic) and replaces the flat band at the region's scarcest surface CF, which saturated at the AWARE cutoff in 86% of regions and drifted with the temporal resolution. On the test build the renewable-GW CF now spans a real distribution (volume-weighted mean 35 vs uniformly ~100 before). - water.supply.groundwater is removed; groundwater is always part of the aware supply (constrain at solve time for mining-free systems), and the current_use source emits no bands -- its observed-withdrawal pool already contains groundwater, so additive bands double-counted it. - Irrigation's share of the groundwater-storage depletion trend is attributed by pirrusegw/ptotusegw (new WaterGAP download), so basins mined by municipal or industrial pumping no longer zero irrigation's renewable band. - A scarcity cap with nonrenewable_cf set is now a joint constraint charging each mined m3 CF-fold against the cap, mirroring pricing; the porosity warning remains for null-cf caps. - run_solve gates the groundwater levers on the availability source (water_availability param replaces water_groundwater).
…design Full tools/calibrate chains on the final tree (default from config/default.yaml, gbd-anchored from config/gsa.yaml), restamping the provenance for the removed water.supply.groundwater key and the merged renewable envelope. L1 centres are essentially unchanged (default: cropland 1.465, grassland 0.234, feed 0.062, 2 iterations; gbd-anchored converged on the warm start), confirming the water redesign does not disturb the validation-mode calibration solves.
koen-vg
force-pushed
the
water-supply-restructure
branch
from
July 22, 2026 21:39
3f155b9 to
2ee930e
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rebuilds the model's water system on a consumption basis, replacing the single per-region growing-season store sized from Huang et al. withdrawals.
What changes
Three quantities, no longer conflated. Crops need a net irrigation requirement
E(GAEZ). The resource that is scarce is consumptionC. What gets reported in the literature is withdrawalW. The model now separates them: crop links drawEfrom per-region field buses, a delivery link drawsC = E / eta_cfrom the regional pool witheta_ccalibrated at build time against observed WaterGAP consumption, andWis reported via the consumed fraction.Availability and scarcity from AWARE 2.0 on WaterGAP 2.2e volumes. The regional pool, its monthly timing and its within-region basin allocation come from WaterGAP's irrigation surface field; AWARE supplies the convex scarcity curve. The Water Footprint Network "sustainable" branch and the
water.supply_scenariokey are gone; the source is nowwater.data.availability(aware|current_use).Optional intra-year resolution.
water.temporal_resolution(a divisor of 12) splits the year into periods that each balance their own surface availability against the demand landing in them, so a season whose surface falls short draws groundwater endogenously rather than being rescued by annual averaging. Demand is placed into periods by the observed MIRCA-OS irrigated crop calendar, retimed by iterative proportional fitting to WaterGAP's monthly requirement.Groundwater as an annual resource. Renewable and non-renewable bands (annual, per region — an aquifer can be pumped in any period) sit additively on top of the period-bound surface supply, so mining emerges endogenously wherever surface plus recharge fall short, instead of being capped at renewable availability.
Two new solve-time levers, both off by default:
water_scarcity(price and/or cap accumulated AWARE scarcity) andgroundwater_depletion(price and/or cap accumulated mining). Analysis gains awater_metricsoutput: per-region withdrawal, scarcity, renewable groundwater and depletion.Defaults chosen for main (deliberately cheaper than the study branch)
The water-aware branch runs quarterly resolution with the full scarcity curve. That is the right setting for a water study and the wrong default for everyone else, so main ships:
I retired the
supply.modeladder (single→scarcity→groundwater) in favour of two independent switches. The ladder welded two orthogonal decisions to one enum and had no rung for "collapsed tiers with the groundwater split" — the combination main wants. All four combinations are now valid andprimary_resourceswas always mode-agnostic, so this costs nothing downstream. One subtlety this exposed: the groundwater bands must be derived from the pre-collapse curve, since renewable groundwater borrows the region's scarcest surface CF, and with collapsed tiers every surface CF is zero — passing the collapsed table would have silently made renewable groundwater free.Two honest caveats about the default, both stated in the docs and changelog:
temporal_resolution: 1a region's whole annual pool is available to every season — exactly the averaging the seasonal design exists to remove. The groundwater bands consequently go nearly inert and reported depletion falls to near zero. That near-zero is an artefact of the resolution, not a finding. Any study about water should raise it.data.availability: awareis a looser water system than main's current binding present-day withdrawal cap, so results will move. It is still the right default — better dataset, andeta_canchors the baseline draw to observed consumption regardless of pool size — but it deserves a sanity check on the first recalibrated solve.tests/config/test.yamlpinstemporal_resolution: 4with tiers and groundwater on, so CI keeps exercising the full-fidelity path that the shipped default does not.Also
scarcity_tiersis off, rather than silently pricing a signal that is identically zero.blue_water_availability/growing_season_waterbuild inputs, which went dead when the per-region water store was replaced.docs/water.rst;crop_production.rst's water section now points at it.Verification
Full pipeline builds, solves and analyzes on the test config at
T=4with tiers and groundwater on. Global totals are plausible for the test config's 9-crop subset: 863 km³/yr irrigation consumption (1488 km³ reported withdrawal), 227 km³ renewable groundwater, 27 km³ depletion, draw-weighted mean CF 25 on AWARE's 0.1–100 scale. The objective-breakdown reconciliation — which is a hard identity check — passes with the new water carriers. 42 water unit tests plus the full suite pass.Calibration
This PR carries the single
tools/calibrateregeneration for the whole structural series (multi-cropping anchoring → basin-aware regions → this), for both tracked artefact sets. See the final commit.Review follow-up
An independent model review validated the water chain empirically against the built and solved test network: every water,
water_fieldandgroundwaterbus balances (max imbalance 2.6e-3 Mm³), water is an input on every crop port, per-period shares sum to the annual requirement,sum_p(CF_p × draw_p)reproduces the scarcity store to 8 digits, delivery draw matches tier draw to 1e-3 Mm³, andeta_cstays in range with no division blow-up. All fourscarcity_tiers × groundwatercombinations were exercised at T ∈ {1,2,3,4,6,12}: collapsed surface conserves capacity to 1.5e-11 Mm³, and the CF fix is confirmed necessary — bands built from the collapsed table give renewable-groundwater CF = 0 for every region, bands built from the pre-collapse curve give the real CF.It found two real bugs, both now fixed in the last commit:
solve_and_analyze_modelwas missing all the water params, so any config withsolving.inline_analysis— both GSA configs — would have died withAttributeErrorthe momentrun_solveread them. There is now a static cross-check that every water paramrun_solvereads is defined in both solve rules and in the cluster manifest.bus4; rice CH₄ now sits atbus(3 + T), and over half of it moved tocrop_production_multi, which the carrier filter never covered at any resolution. On the quarterly test build the old mask matched zero links, so GSA sweeps of the CH₄ factor left rice cultivation — about a fifth of the CH₄ budget — unscaled. Fixed and verified at exactly 2.00× under a 2× dial.Two open questions for a maintainer, deliberately not decided here
scarcity_mm3_eq— and anything a scarcity price or cap acts on — is dominated by a saturated constant rather than AWARE's spatial signal. A volume-weighted or high-percentile CF would be more defensible and stable in T. I left it alone because the water study computes its headline numbers with this exact definition, and changing it here would silently diverge main from the study.eta_cfeasibility floor is annual while the binding constraint is per-period. At T=4, 259 of 703 region-periods with baseline demand exceed that period's surface cap; groundwater currently rescues all of them, so the default solve is clean, but withsupply.groundwater: falsethat baseline quietly lands on slack instead of being flagged. I corrected the docstring, which claimed a property the code does not deliver; making the floor and warning per-region-period is the right follow-up.