Skip to content

perf: accelerate model build workflow - #35

Merged
koen-vg merged 3 commits into
Sustainable-Solutions-Lab:mainfrom
koen-vg:perf/build-workflow
Jul 22, 2026
Merged

perf: accelerate model build workflow#35
koen-vg merged 3 commits into
Sustainable-Solutions-Lab:mainfrom
koen-vg:perf/build-workflow

Conversation

@koen-vg

@koen-vg koen-vg commented Jul 22, 2026

Copy link
Copy Markdown
Member

Summary

  • reuse exact region/resource-class cell mappings across spatial preprocessing
  • vectorize LUC, land-area, water, MIRCA, multi-cropping, and crop-link operations
  • replace repeated LUIcube grassland polygon traversals with mapped reductions

Impact

On a clean default build through build_model with -j8, wall time fell from 150.94 seconds to 90.91 seconds and peak reported RSS fell from 5.58 GB to 4.70 GB. Downloads were excluded from both runs. The subsequent LUIcube optimization reduced that rule from 11.41 seconds and 2.37 GB RSS to 0.63 seconds and 711 MB, with byte-identical output; this improvement is not yet included in the full-run figure.

Validation

  • full 261-job workflow through build_model
  • strict equality of the 305,411-link PyPSA network
  • byte-identical LUC, land-area, MIRCA, multi-cropping, and water outputs
  • xarray-identical resource-class output
  • 89 focused tests and the integration dry-run
  • Ruff, Snakefmt, REUSE, and pre-push hooks

@koen-vg
koen-vg marked this pull request as ready for review July 22, 2026 20:49
@koen-vg
koen-vg merged commit 5c7f977 into Sustainable-Solutions-Lab:main Jul 22, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant