Skip to content

feat: consumption-basis water supply with AWARE scarcity and groundwater - #33

Merged
koen-vg merged 12 commits into
Sustainable-Solutions-Lab:mainfrom
koen-vg:water-supply-restructure
Jul 22, 2026
Merged

feat: consumption-basis water supply with AWARE scarcity and groundwater#33
koen-vg merged 12 commits into
Sustainable-Solutions-Lab:mainfrom
koen-vg:water-supply-restructure

Conversation

@koen-vg

@koen-vg koen-vg commented Jul 21, 2026

Copy link
Copy Markdown
Member

Stacked on #32 (basin-aware regions), which is stacked on #31 (multi-cropping baseline). Review the last commit only. This is the final structural PR of the water-aware extraction series, and it carries the single recalibration for all three — #31 and #32 must not merge without it.

Rebuilds the model's water system on a consumption basis, replacing the single per-region growing-season store sized from Huang et al. withdrawals.

What changes

Three quantities, no longer conflated. Crops need a net irrigation requirement E (GAEZ). The resource that is scarce is consumption C. What gets reported in the literature is withdrawal W. The model now separates them: crop links draw E from per-region field buses, a delivery link draws C = E / eta_c from the regional pool with eta_c calibrated at build time against observed WaterGAP consumption, and W is reported via the consumed fraction.

Availability and scarcity from AWARE 2.0 on WaterGAP 2.2e volumes. The regional pool, its monthly timing and its within-region basin allocation come from WaterGAP's irrigation surface field; AWARE supplies the convex scarcity curve. The Water Footprint Network "sustainable" branch and the water.supply_scenario key are gone; the source is now water.data.availability (aware | current_use).

Optional intra-year resolution. water.temporal_resolution (a divisor of 12) splits the year into periods that each balance their own surface availability against the demand landing in them, so a season whose surface falls short draws groundwater endogenously rather than being rescued by annual averaging. Demand is placed into periods by the observed MIRCA-OS irrigated crop calendar, retimed by iterative proportional fitting to WaterGAP's monthly requirement.

Groundwater as an annual resource. Renewable and non-renewable bands (annual, per region — an aquifer can be pumped in any period) sit additively on top of the period-bound surface supply, so mining emerges endogenously wherever surface plus recharge fall short, instead of being capped at renewable availability.

Two new solve-time levers, both off by default: water_scarcity (price and/or cap accumulated AWARE scarcity) and groundwater_depletion (price and/or cap accumulated mining). Analysis gains a water_metrics output: per-region withdrawal, scarcity, renewable groundwater and depletion.

Defaults chosen for main (deliberately cheaper than the study branch)

The water-aware branch runs quarterly resolution with the full scarcity curve. That is the right setting for a water study and the wrong default for everyone else, so main ships:

water:
  temporal_resolution: 1      # annual
  supply:
    scarcity_tiers: false     # one flat availability cap per region-period
    groundwater: true         # surface/groundwater split retained

I retired the supply.mode ladder (singlescarcitygroundwater) in favour of two independent switches. The ladder welded two orthogonal decisions to one enum and had no rung for "collapsed tiers with the groundwater split" — the combination main wants. All four combinations are now valid and primary_resources was always mode-agnostic, so this costs nothing downstream. One subtlety this exposed: the groundwater bands must be derived from the pre-collapse curve, since renewable groundwater borrows the region's scarcest surface CF, and with collapsed tiers every surface CF is zero — passing the collapsed table would have silently made renewable groundwater free.

Two honest caveats about the default, both stated in the docs and changelog:

  1. At temporal_resolution: 1 a region's whole annual pool is available to every season — exactly the averaging the seasonal design exists to remove. The groundwater bands consequently go nearly inert and reported depletion falls to near zero. That near-zero is an artefact of the resolution, not a finding. Any study about water should raise it.
  2. data.availability: aware is a looser water system than main's current binding present-day withdrawal cap, so results will move. It is still the right default — better dataset, and eta_c anchors the baseline draw to observed consumption regardless of pool size — but it deserves a sanity check on the first recalibrated solve.

tests/config/test.yaml pins temporal_resolution: 4 with tiers and groundwater on, so CI keeps exercising the full-fidelity path that the shipped default does not.

Also

  • Scarcity pricing/capping now raises if scarcity_tiers is off, rather than silently pricing a signal that is identically zero.
  • Fixed a GAEZ artefact where a few cells carry a negative net irrigation requirement, which flipped those crop links into spurious water producers.
  • Removed the dead WFN availability script and the now-unused blue_water_availability / growing_season_water build inputs, which went dead when the per-region water store was replaced.
  • New chapter docs/water.rst; crop_production.rst's water section now points at it.

Verification

Full pipeline builds, solves and analyzes on the test config at T=4 with tiers and groundwater on. Global totals are plausible for the test config's 9-crop subset: 863 km³/yr irrigation consumption (1488 km³ reported withdrawal), 227 km³ renewable groundwater, 27 km³ depletion, draw-weighted mean CF 25 on AWARE's 0.1–100 scale. The objective-breakdown reconciliation — which is a hard identity check — passes with the new water carriers. 42 water unit tests plus the full suite pass.

Calibration

This PR carries the single tools/calibrate regeneration for the whole structural series (multi-cropping anchoring → basin-aware regions → this), for both tracked artefact sets. See the final commit.

Review follow-up

An independent model review validated the water chain empirically against the built and solved test network: every water, water_field and groundwater bus balances (max imbalance 2.6e-3 Mm³), water is an input on every crop port, per-period shares sum to the annual requirement, sum_p(CF_p × draw_p) reproduces the scarcity store to 8 digits, delivery draw matches tier draw to 1e-3 Mm³, and eta_c stays in range with no division blow-up. All four scarcity_tiers × groundwater combinations were exercised at T ∈ {1,2,3,4,6,12}: collapsed surface conserves capacity to 1.5e-11 Mm³, and the CF fix is confirmed necessary — bands built from the collapsed table give renewable-groundwater CF = 0 for every region, bands built from the pre-collapse curve give the real CF.

It found two real bugs, both now fixed in the last commit:

  • solve_and_analyze_model was missing all the water params, so any config with solving.inline_analysis — both GSA configs — would have died with AttributeError the moment run_solve read them. There is now a static cross-check that every water param run_solve reads is defined in both solve rules and in the cluster manifest.
  • The rice-CH₄ sensitivity dial was silently inert. It hardcoded bus4; rice CH₄ now sits at bus(3 + T), and over half of it moved to crop_production_multi, which the carrier filter never covered at any resolution. On the quarterly test build the old mask matched zero links, so GSA sweeps of the CH₄ factor left rice cultivation — about a fifth of the CH₄ budget — unscaled. Fixed and verified at exactly 2.00× under a 2× dial.

Two open questions for a maintainer, deliberately not decided here

  1. Renewable groundwater borrows the region's maximum surface CF, which saturates: 86% of regions land exactly at CF_MAX = 100, and the statistic drifts with resolution (mean 95.5 at T=1, 99.98 at T=12). In the solved network renewable groundwater is 26% of volume but 70% of reported scarcity, so scarcity_mm3_eq — and anything a scarcity price or cap acts on — is dominated by a saturated constant rather than AWARE's spatial signal. A volume-weighted or high-percentile CF would be more defensible and stable in T. I left it alone because the water study computes its headline numbers with this exact definition, and changing it here would silently diverge main from the study.
  2. The eta_c feasibility floor is annual while the binding constraint is per-period. At T=4, 259 of 703 region-periods with baseline demand exceed that period's surface cap; groundwater currently rescues all of them, so the default solve is clean, but with supply.groundwater: false that baseline quietly lands on slack instead of being flagged. I corrected the docstring, which claimed a property the code does not deliver; making the floor and warning per-region-period is the right follow-up.

@koen-vg
koen-vg force-pushed the water-supply-restructure branch 4 times, most recently from 13dd5b5 to 3f155b9 Compare July 22, 2026 20:19
koen-vg added 12 commits July 22, 2026 14:03
Replace the single per-region growing-season water store, sized from
Huang et al. withdrawals, with a consumption-basis water system.

Crops draw their GAEZ net irrigation requirement E from per-region field
buses; a delivery link draws C = E / eta_c from the regional pool, with
eta_c calibrated at build time against observed WaterGAP irrigation
consumption; withdrawal W is reported via the consumed fraction. The
three quantities the literature routinely conflates are now distinct.
Availability, monthly timing and within-region basin allocation come
from WaterGAP 2.2e; AWARE 2.0 supplies the convex scarcity curve.

Water supply and demand can be resolved at intra-year periods
(water.temporal_resolution, a divisor of 12), so a season whose surface
cannot meet its demand draws groundwater endogenously instead of being
rescued by annual averaging. Demand is placed into periods by the
observed MIRCA-OS irrigated crop calendar, retimed by iterative
proportional fitting to WaterGAP's monthly requirement. Renewable and
non-renewable groundwater bands are annual per-region resources (an
aquifer can be pumped in any period) sitting additively on top of the
period-bound surface supply, so mining emerges endogenously rather than
being capped at renewable availability.

The supply.mode ladder (single/scarcity/groundwater) is retired in
favour of two independent switches, supply.scarcity_tiers and
supply.groundwater: the ladder welded two orthogonal decisions to one
enum and had no rung for collapsed tiers with the groundwater split,
which is what main wants by default. The groundwater bands are derived
from the pre-collapse curve, since renewable groundwater borrows the
region's scarcest surface CF and collapsed tiers carry none.

Defaults are chosen for cost, not fidelity: annual resolution and
collapsed tiers, with the groundwater split kept. Both caveats are
documented -- at annual resolution the whole year's pool serves every
season, so the groundwater bands go nearly inert and reported depletion
is an artefact of the resolution; and the AWARE pool is a looser
constraint than the previous binding present-day withdrawal cap, so
results move. tests/config/test.yaml pins quarterly resolution with
tiers and groundwater on so CI still covers the full-fidelity path.

New solve-time levers, both off by default: water_scarcity and
groundwater_depletion pricing/capping. Scarcity pricing raises if
scarcity_tiers is off rather than silently pricing an identically zero
signal. Analysis gains per-region water metrics.

Also fixes a GAEZ artefact where a few cells carry a negative net
irrigation requirement, which flipped those crop links into spurious
water producers, and removes the dead Water Footprint Network
availability path along with the build inputs that went unused when the
per-region water store was replaced.
The deleted WFN availability module left a dangling autodoc entry, and
moving the water narrative into docs/water.rst orphaned the Hoekstra
citation in crop_production.rst while duplicating the Huang one.
The cost-calibration and doc-figure configs still pinned the removed
water.supply_scenario key, which the schema now rejects outright.
Two defects a model review surfaced in the water restructure.

The rice-methane sensitivity dial hardcoded bus4/efficiency4 on
crop_production. Rice CH4 now sits at bus(3 + T), and more than half of
it moved to crop_production_multi, which the carrier filter never
covered at any resolution. On the quarterly test build the mask matched
zero links, so GSA sweeps of the CH4 factor silently left rice
cultivation -- about a fifth of the CH4 budget -- unscaled. The dial now
sweeps every secondary port of both crop carriers, as the GHG
attribution extractor already did; verified to scale rice CH4 exactly
2.00x under a 2x factor.

solve_and_analyze_model never got the water params, so any config with
solving.inline_analysis (both GSA configs) died with AttributeError the
moment run_solve read them. Mirrored from solve_model; a static
cross-check now confirms every water param run_solve reads is defined in
both solve rules and in the cluster manifest.

Also from the review: blank a crop water bus when its requirement is
non-finite rather than attaching one with a NaN efficiency (latent, no
rows hit it today); raise instead of silently doing nothing when
groundwater depletion is priced or capped with the bands switched off,
mirroring the scarcity guard; and state plainly in the irrigation
docstring that the eta_c feasibility floor and its warning are annual,
so at finer resolution a period's demand can still exceed that period's
surface and is met from the annual groundwater bands.
Second review round on the rebased branch:

- Implement the static cross-check promised in the previous fix commit:
  tests/test_solve_rule_params.py parses run_solve's smk.params reads and
  fails when a param is missing from solve_model, solve_and_analyze_model,
  or the cluster manifest (build_scenario_entry).
- Warn when a water-scarcity cap runs with groundwater on and mining
  neither priced nor capped: nonrenewable_cf only applies under scarcity
  pricing, so a bare cap can be met by CF-free mining. Documented in
  docs/water.rst and the cap docstring.
- Fail loudly when an irrigated multi-cropping link references a region
  without water buses, which would silently grant free irrigation at solve
  time (consistency checks are skipped there).
- Blank zero-demand period water buses on multi-cropping links, matching
  the single-crop path; port offsets stay reserved.
- Drop dead generality: the tier-link block in primary_resources handled
  groundwater sources the tier table never carries (now asserted); remove
  the unread water.data.groundwater config knob; remove the
  backward-compatibility source-column fallback in extract_water_metrics.
- Report cf as NaN for non-renewable rows in the per-tier water table;
  their efficiency2 is a volume tally, not a characterisation factor.
- Remove the orphaned Water Footprint Network availability pipeline
  (download/extract/process rules, scripts and doc figure); nothing
  references it since the AWARE restructure.
- Docs: correct the nonrenewable_cf default (100, not 30), note the
  cross-sector renewable-groundwater subtraction, document
  water_metrics.parquet in the analysis chapter, fix the supply-chain
  diagram label, and require water_scarcity/groundwater_depletion at the
  schema top level.
The water restructure declared region_water_tiers.csv as an output of
build_region_water_current_use and documented the current_use source as a
single zero-CF tier, but process_huang_irrigation_water.py never wrote the
file. Nothing exercised the path -- the test config uses aware -- until cost
calibration (which pins availability: current_use) failed on the missing
output. Write the flat zero-scarcity tier per region-month in the shared
schema.
Full tools/calibrate chains against the restructured water system (default
set from config/default.yaml, gbd-anchored from config/gsa.yaml). The
previous stamps recorded the pre-restructure water keys, so every
non-test run failed the provenance check.

Stability converged in 3 (default) / 4 (gbd-anchored) Broyden iterations
with L1 costs near the previous centres -- default: cropland 1.47,
grassland 0.24, feed 0.062; gbd-anchored: cropland 1.62, grassland 0.24,
feed 0.056 -- so no residual supply/demand mismatch is hiding in the
penalty.
…ismatch

The regenerated cost artefacts are the first fit on the merged
basin-aware region map, whose review fixes shifted the 750-region
geometry: banana:CHL and sugarcane:CHL no longer build any link there
(no actual-yield coverage on the new regions), so the artefact carries
no rows for them. The 200-region test config still builds those links
with a positive baseline, and the missing-correction check made every
test build fail. The old artefact only passed by luck -- it predates the
region-map fixes.

Cross-resolution coverage gaps are inherent to consuming an artefact fit
on a different region map, which is exactly what
calibration.accept_provenance_mismatch declares. Such configs now get a
warning and zero corrections for the missing pairs; provenance-matched
configs keep the hard staleness error.
extract_mirca_os_subcrop_monthly only unpacked Rice2/Rice3, so the
irrigated monthly grids build_mirca_crop_calendar needs had no producer
in a clean checkout -- the CI dryrun failed with MissingInputException
while local runs passed on manually extracted files. The rule now
unpacks every calendar subcrop grid (single-sourced list shared with
water.smk's input function) in the same single archive pass.
Resolves the open design questions from review:

- Each basin's AWARE CF curve now spans the joint renewable envelope
  (WaterGAP surface delivery plus renewable groundwater) and is split at
  the basin's surface fraction: the lower slice is period-bound surface,
  the upper slice becomes annual per-region renewable-groundwater CF
  bands. This matches AWARE's own methodology (availability is basin
  discharge including baseflow; CF application is source-agnostic) and
  replaces the flat band at the region's scarcest surface CF, which
  saturated at the AWARE cutoff in 86% of regions and drifted with the
  temporal resolution. On the test build the renewable-GW CF now spans a
  real distribution (volume-weighted mean 35 vs uniformly ~100 before).
- water.supply.groundwater is removed; groundwater is always part of the
  aware supply (constrain at solve time for mining-free systems), and
  the current_use source emits no bands -- its observed-withdrawal pool
  already contains groundwater, so additive bands double-counted it.
- Irrigation's share of the groundwater-storage depletion trend is
  attributed by pirrusegw/ptotusegw (new WaterGAP download), so basins
  mined by municipal or industrial pumping no longer zero irrigation's
  renewable band.
- A scarcity cap with nonrenewable_cf set is now a joint constraint
  charging each mined m3 CF-fold against the cap, mirroring pricing;
  the porosity warning remains for null-cf caps.
- run_solve gates the groundwater levers on the availability source
  (water_availability param replaces water_groundwater).
…design

Full tools/calibrate chains on the final tree (default from
config/default.yaml, gbd-anchored from config/gsa.yaml), restamping the
provenance for the removed water.supply.groundwater key and the merged
renewable envelope. L1 centres are essentially unchanged (default:
cropland 1.465, grassland 0.234, feed 0.062, 2 iterations; gbd-anchored
converged on the warm start), confirming the water redesign does not
disturb the validation-mode calibration solves.
@koen-vg
koen-vg force-pushed the water-supply-restructure branch from 3f155b9 to 2ee930e Compare July 22, 2026 21:39
@koen-vg
koen-vg merged commit 43fd189 into Sustainable-Solutions-Lab:main Jul 22, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant