Skip to content

perf: accelerate crop raster aggregation - #34

Merged
koen-vg merged 4 commits into
Sustainable-Solutions-Lab:mainfrom
koen-vg:perf/build-crop-yields
Jul 22, 2026
Merged

perf: accelerate crop raster aggregation#34
koen-vg merged 4 commits into
Sustainable-Solutions-Lab:mainfrom
koen-vg:perf/build-crop-yields

Conversation

@koen-vg

@koen-vg koen-vg commented Jul 22, 2026

Copy link
Copy Markdown
Member

Summary

  • precompute exact region, resource-class, and raster-cell coverage once per configuration
  • reuse that mapping for vectorized crop-yield and harvested-area aggregation while retaining one crop/water-supply per Snakemake job
  • validate raster shape, transform, and CRS against the cached mapping
  • reduce rule memory requests and document the shared intermediate artifact
  • add focused tests for fractional polygon coverage and empty-group behavior

Performance

On the full-resolution central wheat-irrigated case, pinned to one core:

  • crop yields: about 11 s and 879 MB peak RSS -> about 1.2 s and 320 MB
  • harvested area: about 2.8 s and 477 MB peak RSS -> about 0.7 s and 299 MB
  • one-time mapping build: about 1-2.5 s and 300 MB peak RSS (18 MB artifact)

This is about 9x faster with 64% less peak memory for crop yields, and about 4x faster with 37% less peak memory for harvested area. The mapping cost is amortized across all crop and water-supply jobs in a configuration.

Correctness

Optimized CSVs are byte-identical to the baseline for:

  • irrigated potential wheat
  • rainfed actual wheat
  • irrigated potential sugarcane, including its unit-conversion override
  • irrigated wheat harvested area
  • rainfed alfalfa harvested area, including the fodder aggregate
  • rainfed olive harvested area, including the oil-crop deflation
  • rainfed banana harvested area, including the fruit residual

The implementation retains exact fractional polygon coverage. A center-cell region mapping was rejected because it materially changes boundary allocation.

Batching crops into fewer Snakemake jobs was also rejected: after removing the repeated spatial work, independent jobs are faster in aggregate under normal parallel execution and retain finer failure, scheduling, and invalidation granularity.

Validation

  • pytest -q tests/test_region_class_aggregation.py (5 passed)
  • pytest -q tests/test_integration.py::test_workflow_dryrun
  • byte-for-byte comparisons for representative yield and harvested-area outputs
  • ruff, formatting, snakefmt, REUSE, Sphinx warnings-as-errors, and repository hooks

@koen-vg koen-vg changed the title perf: accelerate crop yield preparation perf: accelerate crop raster aggregation Jul 22, 2026
@koen-vg
koen-vg marked this pull request as ready for review July 22, 2026 19:31
@koen-vg
koen-vg merged commit bd9c735 into Sustainable-Solutions-Lab:main Jul 22, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant