Live-data docs site, marimo notebooks, animated GIFs + weekly CI refresh#6
Open
MaykThewessen wants to merge 98 commits into
Open
Live-data docs site, marimo notebooks, animated GIFs + weekly CI refresh#6MaykThewessen wants to merge 98 commits into
MaykThewessen wants to merge 98 commits into
Conversation
Replaces the slow, fragile curl-based MaStR API scrape with the curated Zenodo dataset (DOI 10.5281/zenodo.18697247, version 2026-02-19), which provides corrected coordinates and metadata for 40,455 wind turbines. Changes: - wind.ipynb: replace parser.py/JSON loading cell with direct Zenodo CSV load; maps Zenodo columns (nettonennleistung, datum_inbetriebnahme, lon_x/lat_y, bundesland) to the field names expected by the rest of the notebook - build_kreise_gpkg.py: build germany_kreise.gpkg from Eurostat GISCO NUTS-3 boundaries instead of requiring a 4 GB OSM planet file + osmfilter; city-states (Berlin, Hamburg, Bremen) sourced from NUTS-1 as admin_level=4 - run_notebook.py: standalone runnable Python equivalent of wind.ipynb - export_capacity_per_region.py: export current installed wind capacity (MW) per Landkreis and offshore sea area to a named CSV - .gitignore: exclude data files, generated outputs, and Python cache Closes emmericp#2 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…egion
- fetch_mastr.py: new script to fetch wind/solar units from MaStR API
- incremental mode (default): sorts descending by install date, stops
at cutoff — retrieves only turbines newer than Zenodo snapshot in ~1 page
- full mode: downloads all entries (wind ~2 pages; warns for solar ~237 pages)
- supports --energy wind|solar via Energieträger code (wind=2497, solar=2495)
- export_capacity_per_region.py: merge Zenodo base + API incremental files
- merges Zenodo dataset with any incremental API JSON files
- spatial join via germany_kreise.gpkg for API entries missing Landkreis
- fills missing Bundesland from Zenodo landkreis→bundesland lookup
- offshore fallback: coordinate bounding-box (Nordsee lon 3–9 / Ostsee 9–15)
- historical capacity columns: capacity_MW_2013, _2024, _2025, _latest
- output validated: 313 regions, no duplicates, 2013=28.8 GW → latest=78.6 GW
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- pv.py + wind.py: reactive marimo explorers (date slider, type filter, bin count, colormap picker, summary stats, growth trend). - mastr_plot.py: shared helpers (load_records, load_admin_units, aggregate_by_unit, jenks_bins, plot_choropleth). Falls back to a synthetic demo dataset when no scrape or kreise polygons are present. - docs/: Read-the-Docs-styled single-page site (sticky sidebar, search filter, scroll-spy, 404, .nojekyll) with embedded SVG diagrams and rendered HTML exports of both notebooks. Served from /docs on main. - fig/: pipeline + architecture + enum-decoding + timeline SVGs, plus sample PV/wind choropleths and growth charts rendered from demo data. - fig/.gitignore: allowlist *.svg so diagrams ship with the repo. - .gitignore: ignore macOS junk, __pycache__, __marimo__, regenerable scrape/OSM artifacts. - CLAUDE.md: conventions for Claude Code agents working in this tree. - README.md: rewritten with badges, embedded SVGs, notebook section.
- Live scrape of 7 non-PV pages (169 174 rows: wind, biomass, gas, hydro)
and 5 PV pages (125 000 of 6.1 M registry plants).
- Wind map now shows the full registry slice at 2025-01-01: 31 404
active turbines, 63 GW total. Real Kreis polygons (434 units from
isellsoap/deutschlandGeoJSON) drive the choropleth.
- PV map shows a 125 k-plant sample — captures the spatial pattern
even though it covers ~0.7 % of installed capacity.
- mastr_plot.load_records: merge every default data dir (data/, data-pv/,
non-pv-data/) so a single load covers wind + PV when both have been
scraped.
- README: document the PV-scrape command + the MaStR filter quirk
(silent drop of second clause when ANDed across fields).
- docs/index.html: figure captions updated, "demo data" labels removed.
- docs/{pv,wind}.html: re-exported from notebooks reading real data
(~520 KB each, up from ~160 KB demo).
…lice Switched the PV scrape from "first 125 000 rows by default sort" to "top 50 000 by Bruttoleistung-desc". Every plant in the new slice is ≥ 200 kW; combined capacity is ~68.8 GW — essentially every commercial and utility-scale PV plant in Germany. Trade-off: the long tail (6 M rooftop and balcony installs below 200 kW) is excluded, but the spatial choropleth signal is dramatically stronger since utility parks dominate per-Kreis aggregates anyway. - README.md: update PV scrape command (sort=Bruttoleistung-desc, 2 pages) and the sample-render caption. - docs/index.html: figure captions updated, "0.7%" / "125k-plant" wording replaced. - fig/sample-pv-*.svg + docs/assets/sample-pv-*.svg: re-rendered. - docs/pv.html + docs/wind.html: re-exported from notebooks reading the refreshed slice.
Three additions wired into both README and the docs site: - fig/wind-2005-2025.gif (300 KB, 21 yearly frames 2005→2025). Built from the live registry, fixed Jenks bins so colors stay comparable across years. Drops onto the README as the marquee image. - fig/sample-by-bundesland.svg — horizontal stacked bar of wind + utility-scale PV by state. Niedersachsen leads wind (12.7 GW); Bayern leads PV (12.0 GW). Live-data spatial join through the rebuilt GeoPackage. - germany_kreise.gpkg now carries a `bundesland` column (NAME_1 from deutschlandGeoJSON), enabling per-state aggregations without re-running the OSM extract. - pixi.toml: reproducible env spec + tasks (scrape-non-pv, scrape-pv-top, fetch-kreise, pv-edit, wind-edit, docs-build). - fig/.gitignore: allow wind-*.gif alongside SVGs. - .gitignore: skip fig/frames/ (per-PNG dump regenerable from the notebooks).
Three follow-ups landed together because they share infrastructure (the new scripts/ directory). Top-Kreise table (docs) - scripts/build_kreise_json.py produces docs/assets/kreise.json with one row per Kreis (434 total): wind GW + count, PV ≥200 kW GW + count, combined total. Snapshot is 2025-01-01. - docs/assets/kreise-table.js renders a sortable, filterable HTML table mounted at #kreise-table. Click columns to re-sort, type in the filter box to search, drop-down for row count. - docs/index.html: new "Top Kreise by installed capacity" section plus sidebar link. Linear / log color-scale picker - mastr_plot.plot_choropleth() grows a `scale=` argument (jenks | linear | log) selecting BoundaryNorm vs. Normalize vs. SymLogNorm respectively. - pv.py + wind.py add an mo.ui.radio for the scale and pass it through to plot_choropleth, then re-export to docs/pv.html and docs/wind.html. GitHub Actions weekly refresh - .github/workflows/refresh-docs.yml runs Sun 03:17 UTC (or manually via workflow_dispatch). Scrapes 7 non-PV pages + 2 top-sorted PV pages, regenerates the polygons, runs the three helper scripts, re-exports the notebooks, commits anything that changed back to main. Uses prefix-dev/setup-pixi@v0.8.10 for reproducible env. - scripts/render_samples.py renders all 5 sample SVGs. - scripts/render_wind_gif.py renders 21 frames + 9 hold + ffmpeg assemble. - scripts/build_kreise_json.py rebuilds the per-Kreis dataset. Bonus - mastr_plot.find_data_dirs() now returns every dir that has *.json scrapes, so wind + PV scrapes are merged automatically when both have been pulled. find_data_dir() retained for single-dir callers.
- fig/sample-top-operators.svg + docs section "operators": top 30 owners by combined wind + PV. Offshore project vehicles dominate the very top (every farm is its own LLC). - fig/sample-offshore-windparks.svg + docs section "offshore": top 25 offshore wind operators, colored by sea. 1 637 turbines / 9.2 GW total (Nordsee 7.4 + Ostsee 1.8). - docs/index.html: full Open Graph + Twitter Card meta tags so link previews show the wind animation. Theme color set. Canonical URL added. New SVG favicon (turbine + sun). - scripts/render_samples.py: now also re-renders the two new charts on every CI run. - README: new sections for both charts.
The repo's marquee was wind-only — now PV gets the same treatment.
True "all solar PV" needs ~6 M plants (~24 GB JSON), so the practical
approximation is "all plants ≥ 49 kW" = top 200 000 sorted by capacity.
That covers ~58 GW active 2025-01-01: every utility park, every
commercial rooftop, every parking-lot canopy. The long tail of
sub-50 kW residential and balcony installs is excluded.
- fig/pv-2005-2025.gif (474 KB, 21 yearly frames). Sits next to the
wind GIF in the README as a side-by-side comparison.
- scripts/render_wind_gif.py: generalised with PRESETS for "wind" and
"pv" plus a CLI arg ("wind" | "pv" | "both", default "wind").
- pixi.toml: new tasks (render-samples, render-gifs, render-wind,
render-pv, build-kreise) + scrape-pv-top now fetches 8 pages.
- .github/workflows/refresh-docs.yml: defaults bumped to 8 PV pages,
GIF step runs `both`, commit-back add-list extended for pv-*.gif.
- docs/index.html: new section + figure for the PV animation.
Figure numbers renumbered to avoid the collision the new section
introduced.
- .gitignore + fig/.gitignore: allow pv-*.gif, ignore the new
frames-pv/ scratch dir.
…CITATION Five additions that share infrastructure (the scripts/ directory). Energy-type mix chart - fig/sample-energy-mix.svg + docs section "energy-mix": national installed capacity per Energieträger. Coal (31 GW) still rivals utility-scale PV (58 GW). Adds context — until now the docs only showed Wind + PV. Year-over-year additions choropleth - fig/sample-2024-additions.svg + docs section "yoy-additions": new plants registered during 2024, aggregated per Kreis. Top: Leipziger Land (565 MW utility solar in the former-lignite belt). Bulk downloads - scripts/build_downloads.py: produces docs/data/mastr-snapshot.parquet (8.4 MB, zstd), docs/data/mastr-snapshot.csv.gz (9.8 MB), and a per-Kreis × energy_type roll-up. Each row is joined to its Kreis + Bundesland via spatial point-in-polygon. - docs section "downloads" with a table of formats + recommended use. Parser unit tests - tests/test_parser.py: 52 pytest cases covering parse_dotnet_date and every enum decoder in PowerPlant.from_json. Runs in 0.54 s. - pixi tasks: `pixi run test`. - CI runs the suite as part of the weekly refresh. CITATION.cff - Standard citation metadata for the software + upstream MaStR data + OSM polygons. Picked up automatically by GitHub's "Cite this repository" button. Wiring - pixi.toml: added pyarrow + pytest deps; new tasks (build-downloads, test, tests). - .github/workflows/refresh-docs.yml: runs build_downloads.py and the pytest suite, then includes docs/data/ artefacts in the commit-back. - scripts/render_samples.py: now also re-renders the energy mix + 2024-additions charts.
Two new charts complete the time-series narrative on the docs site. Per-Bundesland build-out ramp - fig/sample-state-ramp.svg: monthly cumulative wind + PV ≥ 49 kW per state, 2000 → 2025. Bayern (22 GW), Niedersachsen (19 GW), Brandenburg (18 GW) lead. Sharp inflection around 2010 = PV boom. - Spatial join attaches each plant to its Bundesland once, then pivots monthly to a wide table for matplotlib's stackplot. PV by installation type - fig/sample-pv-by-type.svg: cumulative GW per ArtDerSolaranlage bucket (building / free / parking_lot / water / balkonkraftwerk), plus a pie of the latest share. 16 k free-standing plants hold ~ 43 GW; 184 k building rooftops hold ~ 30 GW. Wiring - scripts/render_samples.py: two new render_* functions, both invoked from main() so the weekly CI keeps them fresh. - docs/index.html: new sidebar entries + sections. - README.md: figures inlined under the existing data narrative.
Author
|
Update: pushed three more commits onto the PR head since the original
Live preview rebuilt at https://maykthewessen.github.io/marktstammdatenplotter/. Happy to split into smaller PRs or rebase if any of this is out of scope — |
… templates Density (MW/km²) reshuffles the absolute-capacity ranking - fig/sample-wind-density.svg + fig/sample-pv-density.svg. Wind: Dithmarschen (1.64 MW/km²), Nordfriesland (1.14), Paderborn (1.05). PV: Straubing Städte (1.49), Amberg Städte (1.20), Leipziger Land (0.94). - units.area_km2 is computed via ETRS89 / UTM32N (epsg=25832), the standard equal-area projection for Germany. Top 10 largest individual plants - docs/assets/largest-plants.json — JSON dataset for potential future client-side rendering. - docs/index.html: two inline HTML tables (top-10 PV + top-10 wind by single-unit Bruttoleistung). PV list reveals the Leipziger Land / Märkisch-Oderland utility- solar cluster. Wind top-10 is dominated by the new 15 MW offshore turbines (Nordseecluster A+B, Windanker). README badges - Added: Refresh-CI status, parser-tests badge (52 passing), pixi badge, marimo badge. Existing badges (docs, Python, data source) restyled with logos. .github templates - bug_report.md and feature_request.md issue templates plus a PULL_REQUEST_TEMPLATE.md with the standard test-plan checklist (parser tests + render-samples + docs-build). Wiring - scripts/render_samples.py: two new render_density_map functions + export_largest_plants(). Both called from main() so CI keeps them fresh.
PV orientation analysis - fig/sample-pv-orientation.svg: capacity-weighted distribution of panel facing (pie) + tilt angle (horizontal bar). South dominates at 62 % of installed PV; east-west flat-mount is second at 14 %. Trackers rare (0.5 GW). Most plants tilt 0-19° (flat, utility-park footprints). Wind fleet age + repowering signal - fig/sample-wind-age.svg: two-panel chart. Top: installs vs. decommissions per year, decommission wave starting 2019 as the first EEG cohort hits its 20-year boundary. Bottom: mean rotor MW by commission year — fleet has upsized from ~0.5 MW in the late 90s to ~7 MW in 2025 (15× growth at the unit level, driven by new 15 MW offshore class). Project hygiene - CHANGELOG.md: Keep-a-Changelog format, 0.1.0 → 0.5.0 plus an Unreleased bucket. Captures every commit in the public history. - CONTRIBUTING.md: quickstart, repo layout, "how to add a chart" + "how to add a parser enum" recipes, style + commit rules. Wiring - scripts/render_samples.py: two new render_* functions wired into main(). Both honour the same dual-write pattern (fig/ + docs/assets/).
The registry already carries 4 981 plants with install_date in 2026 (354 wind / 2.3 GW + 4 148 PV / 3.5 GW); the latest install_date is 2026-05-15. Adding a final May-2026 snapshot to both animations captures the year-to-date build-out without waiting for year-end. - scripts/render_wind_gif.py: introduced snapshot_dates() helper plus a FINAL_FRAME constant (date(2026, 5, 1)). Frame list is now the 21 yearly Jan-1 dates plus the final cap. Trivial to extend to finer cadence later (monthly, quarterly). - fig/wind-2005-may2026.gif (302 KB, was 300 KB / 21 frames → now 31 frames including the 9-frame hold) and fig/pv-2005-may2026.gif (509 KB). - Old GIFs removed: fig/wind-2005-2025.gif, fig/pv-2005-2025.gif, docs/assets/ mirrors. - README + docs/index.html + Open Graph meta tags updated to point at the new filenames. - CHANGELOG.md: Unreleased bucket records the change.
Density choropleths now match the final frame of the animation GIFs. The same bump propagates through every snapshot-derived artefact so the docs site reads as one consistent point-in-time. Headline numbers at 2026-05-01: - Wind: 32 107 turbines active, 79.7 GW (was 31 404 / 63 GW @ 2025-01-01) - PV (≥ 49 kW): 195 591 plants active, 72.8 GW (was 174 934 / 58 GW) - Offshore: 1 732 turbines / 10.4 GW (Nordsee 8.6 + Ostsee 1.8) Code changes - scripts/render_samples.py: SNAP → 2026-05-01. - scripts/build_kreise_json.py: SNAP → 2026-05-01. - scripts/build_downloads.py: SNAP → 2026-05-01; active column renamed `active_2025_01_01` → `active_at_snap`. Regenerated artefacts (all on the new snapshot) - fig/sample-*.svg + docs/assets/sample-*.svg (all 14 charts). - docs/assets/kreise.json (434 rows). - docs/data/mastr-snapshot.parquet + .csv.gz + mastr-by-kreis.csv.gz. Caption fixes - README.md and docs/index.html: every "2025-01-01" replaced with "2026-05-01"; stale absolute figures (63 GW, 58 GW, 31 404 turbines, 1 637 offshore) updated to match the new snapshot. CHANGELOG.md: entry expanded under Unreleased / Changed.
…ization claim Three bugs found by a coverage audit: A. Choropleth titles used the per-Kreis aggregated sum `agg["power_gw"].sum()`, which silently excludes plants that fall outside every Kreis polygon. For wind that's 3 098 turbines / 11.3 GW of offshore + slightly-misaligned coastal capacity. Titles now use the active-set total `active["power"].sum() / 1e6` and add a "X GW offshore / out-of-Kreis" footnote when the gap > 1 GW. Fixed in: scripts/render_samples.py (map + density), scripts/ render_wind_gif.py (yearly + YTD frames). B. `active_at_snap` in build_downloads.py used `install_date.isna() | install_date <= snap`, which silently includes NaT-install plants (~15 k rows / 71 GW of legacy conventional). Every other filter in the codebase excludes them. The parquet + CSV downloads were inconsistent with the rendered charts. Now matches `mastr_plot.aggregate_by_unit`. C. README, CLAUDE.md, and docs/index.html previously claimed offshore-turbine coordinates are anonymised. Reality: all 1 909 offshore rows carry real Laengengrad / Breitengrad (verified against the raw scrape — EnBW Baltic 1 at 54.605, 12.63 etc.). Only the StandortAnonymisiert string is a "Nordsee…" / "Ostsee…" label. The original notebook's synthetic-sea-rectangle hack was never required. Docs corrected. README additions - New "What this method does not track" table enumerating every systematically-excluded category (sub-49 kW PV, NaT install, NaN coords, offshore in choropleths, battery storage, heat-only, pre-commissioning, behind-the-meter, decommissioned). CHANGELOG.md: entries under Unreleased / Fixed. Re-renders - All 16 sample SVGs (titles now match the active-set totals). - Both animation GIFs (wind, pv) regenerated with corrected titles. - Parquet + CSV downloads regenerated with the consistent flag. - 52 parser tests still green (0.53 s).
scripts/render_wind_gif.py prepended the snapshot date to its title,
then mastr_plot.plot_choropleth appended it again. Net result: the
date showed up twice in every frame ("Installed wind capacity in
Germany — 2026-05-01 / 32,107 turbines · 79.7 GW / 2026-05-01").
plot_choropleth is responsible for the date suffix; callers shouldn't
duplicate it. Stripped the leading "— {snap.isoformat()}" from the
GIF script's title template.
Re-rendered both animations (wind: 310 KB, pv: 502 KB) — date now
appears exactly once per frame.
User feedback: prefer the date on row 1 (next to the chart name)
rather than as a trailing line under the stats.
- mastr_plot.plot_choropleth no longer auto-appends the date.
Callers control title formatting verbatim.
- scripts/render_wind_gif.py: keeps date in row 1
("Installed wind capacity in Germany — 2026-05-01\n…").
- scripts/render_samples.py render_map: date now lives on row 1
("Wind capacity — 2026-05-01\n32,107 plants · 79.7 GW").
- pv.py + wind.py marimo notebooks: same pattern; the date_slider
value is interpolated into the first row.
Re-rendered all 16 sample SVGs, both animation GIFs, and re-exported
both marimo HTML pages.
Source PNG dpi 110 → 150 so we render at higher pixel density (946×1105 → 1290×1505 per frame), then ffmpeg downscales via Lanczos to 1200 px height (was 900 px). Palette grows 128 → 192 colors for smoother Jenks-bin transitions. Result - wind-2005-may2026.gif: 320 KB → 576 KB (1027×1200) - pv-2005-may2026.gif: 576 KB → 876 KB (1027×1200) Both still well under LinkedIn's 5 MB GIF cap. Choropleth edges + text are noticeably sharper at 1027×1200 — good for embed in LinkedIn/Slack/Twitter previews where the platform itself doesn't upscale.
Source PNG dpi 150 → 180, ffmpeg target height 1200 → 1440, palette 192 → 224 colors. The animations are the only raster outputs in the repo (every chart, diagram, and favicon is already SVG) so the GIFs are where resolution bumps actually matter. Result - wind-2005-may2026.gif: 576 KB → 688 KB - pv-2005-may2026.gif: 876 KB → 1.1 MB Both still comfortably under LinkedIn / Slack / X 5 MB caps. Note on PNG → SVG: no PNG files exist in the repo's published outputs. fig/frames*/ holds intermediate raster PNGs but those are gitignored. GIF is a raster format by definition, so the animations themselves can't be SVG without switching to animated SVG (SMIL/CSS) or WebM/MP4 — both lose the universal-autoplay property that makes GIFs work in README badges.
WebP intermediates (lossless)
- scripts/render_wind_gif.py renders frames as lossless WebP via
matplotlib + Pillow (pil_kwargs={"lossless": True, "method": 4}).
ffmpeg reads them natively via the webp decoder.
- Smaller than PNG at the same dpi, identical pixels into ffmpeg.
Modern output: H.264 MP4 next to the GIF
- New assemble_mp4() emits libx264 yuv420p MP4 with crf=18, preset=slow,
faststart, fps=10. Same 1233×1440 px target as the GIF; ~30 % smaller
file at notably sharper color reproduction (h264 isn't capped by GIF's
256-color palette).
- LinkedIn, Slack, X all autoplay MP4 inline. Major win for sharing.
- docs/index.html embeds via <video autoplay loop muted playsinline>
with the GIF as <img> fallback + poster. Both formats also linked
in the figure captions.
Sizes
| Output | Wind | PV |
|--------|------|----|
| GIF | 688K | 1.1M |
| MP4 | 476K | 916K |
Wiring
- fig/.gitignore: allows wind-*.mp4 + pv-*.mp4 alongside the GIFs.
- .github/workflows/refresh-docs.yml: commit-back add-list extended
for fig/*.mp4 + docs/assets/*.mp4.
- README + CHANGELOG entries.
MaStR carries 2.5 M storage records under Energieträger=2496 (Speicher) on the same Stromerzeugung endpoint we already scrape. Top 200 000 by Bruttoleistung captures every grid-scale + commercial unit down to ~10 kW home batteries. Snapshot 2026-05-01 (commissioned only) - 193 678 units · 7.86 GW · 98.9 GWh active - Power split ~50/50 between Pumpspeicher (6.48 GW, 41 sites) and Batterie (6.49 GW, 193 599 units). Energy ~100× asymmetric — Li-ion sits in the 1-2 h duration band, PSH is multi-hour. Pipeline (incl. planned-and-permitted) - 25 GW / 2 565 GWh by ~2028. Headlines: LEAG GigaBattery Jänschwalde 1 GW / 4 GWh, GigaBattery Boxberg 400 MW, ECO POWER 1-6 (300 MW each). What landed - parser.py: new BatteryUnit dataclass + parser.load_bess(). Energy capacity, storage tech, planned commissioning, voltage level, status preserved. Kreis + Bundesland are pre-joined in source JSON so no spatial join needed inside the parser layer. - mastr_plot.py: load_bess(), aggregate_bess_by_unit() with optional include_planned=True for pipeline view. - scripts/render_samples.py: render_bess_charts() emits 5 SVGs in one pass (power map, energy map, duration histogram, tech mix, growth). - scripts/render_wind_gif.py: PRESETS gains a "bess" entry; CLI takes `bess` and `all` arguments. Same WebP/GIF/MP4 pipeline. - tests/test_parser.py: 5 new BatteryUnit cases. 57 / 57 green. - pixi.toml: new tasks scrape-bess + render-bess; render-gifs upgraded from `both` → `all`. - .github/workflows/refresh-docs.yml: BESS scrape step, commit-back add-list extended for bess-*.gif/.mp4. - docs/index.html + README.md: new "Battery + electricity storage (BESS)" section with 5 charts + animation video. - fig/.gitignore + .gitignore: allow bess-*.gif/mp4, ignore data-bess/ + fig/frames-bess/. - CHANGELOG.md: extensive entry under Unreleased / Added.
…tion) The German + EU storage-reporting standard splits BESS into three sectors by usable energy capacity: | Sector | Threshold | Source convention | |--------|-----------|-------------------| | HSS Heimspeicher | < 30 kWh | battery-charts.de, BVES | | CSS Gewerblich | 30 kWh – 1 MWh | battery-charts.de, BVES | | LSS Großspeicher | ≥ 1 MWh | battery-charts.de, EASE, EU SET-Plan | Snapshot 2026-05-01 (active only) - HSS 176 441 units · 2.47 GW · 2.54 GWh (1-h residential PV+battery) - CSS 16 659 units · 0.68 GW · 1.20 GWh - LSS 569 units · 9.81 GW · 933.15 GWh (Pumpspeicher dominates) Pipeline (planned-and-permitted) - + 8.2 GW LSS Li-ion by ~ 2028 (Jänschwalde 1 GW, Boxberg 400 MW, ECO POWER 1-6 = 6 × 300 MW, Westfalen 538 MW, …) What landed - mastr_plot.bess_sector() classifier + BESS_SECTORS constant. - mastr_plot.load_bess() decorates each row with a `sector` column. - scripts/render_samples.py: new render_bess_sector_charts() emits three SVGs in one pass (summary bars, stacked-area cumulative growth, per-sector duration histogram). Wired into main(). - tests/test_parser.py: 4 new TestBessSector cases (HSS/CSS/LSS bounds, unknown for None/0/negative). 61 / 61 tests green. - README + docs/index.html: new sector subsection cross-referencing battery-charts.de (RWTH Aachen), BVES, EASE, EU SET-Plan. Why this matters The headline "12.97 GW BESS" hides three very different stories: 176k households doing 1-h self-consumption, ~17k commercial sites, and 569 grid-services + Pumpspeicher units carrying most of the energy capacity. Sector reporting lets the docs site speak the same language as the German + EU storage-market trackers.
Seven log-spaced bins on a shared axis across all three techs:
0-10 kW · 10-100 kW · 100 kW-1 MW · 1-10 MW · 10-100 MW · 100-1000 MW · 1 GW+
BESS gets two panels (GW + GWh per bin) because storage carries both
power and energy. Wind + PV charts are single-panel (GW only).
What it reveals (snapshot 2026-05-01)
Wind
- 75.4 GW of the 79.7 GW total sits in the 1-10 MW bin
(26 174 turbines). 10-100 MW = 118 of the new 14-15 MW
offshore class (1.5 GW). Everything else is rounding.
PV (top-200k slice)
- 100 kW-1 MW: 73 569 plants / 22.7 GW (commercial rooftop)
- 1-10 MW: 7 765 plants / 24.9 GW (utility ground-mount)
- 10-100 MW: 787 plants / 15.6 GW (large utility)
- 100-1000 MW: 6 plants / 0.8 GW (mega-parks)
- Sub-49 kW residential excluded by the scrape's sort floor.
BESS
- 10-100 kW: 192 075 units · 2.9 GW · 3.5 GWh (residential)
- 1-10 MW: 356 units · 1.1 GW · 2.1 GWh (grid-scale Li-ion)
- 10-100 MW: 94 units · 2.4 GW · 165 GWh (utility Li-ion +
smaller Pumpspeicher)
- 100-1000 MW: 30 units · 6.2 GW · 765 GWh (Pumpspeicher +
biggest Li-ion)
- 1 GW+: 0 units (Jänschwalde still In Planung)
Code
- scripts/render_samples.py: SIZE_BIN_EDGES_KW + SIZE_BIN_LABELS,
_size_bin_stats(), _bar_with_labels(), render_bess_by_size(),
render_generation_by_size(). Numpy import was missing — fixed.
- Three new sample SVGs (one per tech).
- Embedded in docs/index.html (under wind/PV section + BESS section)
and README.md.
CHANGELOG entry under Unreleased / Added.
Three concurrent changes shipped together because they share the
size-bin pipeline.
1. Pumped-hydro storage (PSH) is now its own section
- PSH is reported separately from batteries everywhere serious
(battery-charts.de / BVES / EASE / EU SET-Plan): different
technology, different lifecycle (50+ yr vs 10-15), different
grid-services profile.
- mastr_plot.split_bess_storage() returns three disjoint slices
(batteries / psh / other).
- scripts/render_samples.py: new render_psh_charts() emits a map,
per-Bundesland + duration summary, and top-15 sites bar chart.
- Battery-only numbers in the BESS section now drop from the
combined 12.97 GW / 936.9 GWh to the true Li-ion-and-friends
number: 6.49 GW / 9.4 GWh.
- PSH numbers: 41 sites · 6.48 GW · 927.5 GWh, median duration
~8 h vs ~1 h for residential Li-ion.
2. Wind size-bin chart now aggregates per project, not per turbine
- render_generation_by_size() grows an aggregate_by="owner_name"
parameter. For wind, project ≈ Windpark SPV, so summing per
owner before binning surfaces the real distribution: offshore
farms now appear in the 100-1000 MW bin (was hidden because
individual turbines are 6-15 MW each).
3. Size-bin bars no longer print unit-count annotations
- Dropped count_series from _bar_with_labels; value is enough.
Cleanup
- Removed the now-stale BESS tech-mix subchart + sample-bess-tech-mix.svg
(post-split it'd collapse to a single 'Batterie' bar).
- README + docs/index.html: BESS section rewritten with batteries-only
numbers; new dedicated PSH section + sidebar entry; figure numbers
renumbered to keep continuity.
- 61 / 61 parser tests still green.
…outputs Reads the complete open-MaStR bulk export (Zenodo 14783581) for all three technologies, selects key columns, parses dates, and saves as snappy-compressed parquet. Includes stats on capacity by Bundesland, year-over-year additions, battery chemistry breakdown, and usable storage capacity joined from the storage_units sub-table. Solar.parquet excluded from git (126 MB > GitHub 100 MB limit); regenerate locally via process_mastr_zenodo.py.
Regenerated 31 WebP frames via render_wind_gif.py pv — now driven by open-MaStR bulk parquet (load_records prefer_bulk=True), covering the full ~4.86 M PV registry including residential and balcony plants, not just the previous top-200k ≥49 kW slice.
Remove shapely simplify(0.005) from load_admin_units. Re-render all sample SVGs at full vertex count (~88,869 vertices, ~2.47MB per map). Restore previously deleted _gradient.svg variants.
- render_wind_gif.py: add agg_unit/legend_label GWp to PV preset - render_samples.py: add unit param to render_map, render_growth, render_density_map, render_generation_by_size (default "GW"); pass unit="GWp" at all PV call sites. Directly update PV-only functions: render_pv_by_type, render_pv_orientation, render_pv_orientation_polar, render_pv_orientation_polar_snapshot - Re-render all sample SVGs with GWp labels at full polygon resolution (no geometry.simplify — reverted in previous commit)
- Physical 6.3 GB SQLite now lives at data/mastr/open-mastr.db (gitignored) - Symlink at ~/.open-MaStR/data/sqlite/open-mastr.db preserves open-mastr's default refresh path, so `pixi run db-mastr-core` writes to the repo copy - mastr_db.DB_PATH prefers the in-repo file, falls back to home path - .gitignore: exclude data/mastr/ - CLAUDE.md: document the path + symlink
- mastr_plot.py: add KREIS_NAME_TO_AGS5 lookup (23 city/Landkreis pairs) and comma-suffix regex for BKG VG2500 names; resolves Passau/München/etc. collisions when aggregating anonymised MaStR rows via municipality_key prefix - scripts/render_wind_gif.py: minor pipeline tweak - pixi.toml: trivial whitespace - docs/assets + fig: re-render all sample SVGs and per-tech animations (bess, pv, wind 2005-may2026 GIF/MP4 + extra 2010/2018 starts)
Drop docs/assets/sample-pv-density_gradient.svg and sample-wind-density_gradient.svg (orphans from removed render path). Add planning note for open-mastr integration.
- mastr_db.load_for_pipeline(tech) returns DataFrame matching the
Zenodo-parquet column contract that mastr_plot.load_from_bulk consumes:
* common: mastr_id, einheit_name, gross_capacity_kw, commissioning_date,
decommissioning_date, longitude, latitude, landkreis, bundesland,
municipality_key, owner_name (via JOIN to market_actors.Firmenname)
* wind: + location_type (WindAnLandOderAufSee), seelage, wind_park
* pv: + location_type (mapped ArtDerSolaranlage), usage_type (translated
Nutzungsbereich), orientation, tilt (mapped Neigungswinkel tuple)
* bess: + usable_capacity_kwh, planned_commissioning_date, storage_technology
- Strict mode: _ensure_market_actors() raises RuntimeError when the
market_actors table is empty so callers get a clear diagnosis rather
than silently NULL owner_name across all rows
- pixi.toml: add db-mastr-market task; db-mastr-core now also pulls market
- tests/test_open_mastr_parity.py: 19 tests covering schema contract,
row-count floors, capacity sum bounds, enum value translation, and the
strict-mode raise path; all skip cleanly when DB is absent
load_from_bulk(tech, bulk_dir=None, source="auto") gains a new selector:
- "auto" prefers SQLite when data/mastr/open-mastr.db is populated,
falls back to the Zenodo parquet dir otherwise (default)
- "sqlite" forces open-mastr via mastr_db.load_for_pipeline
- "zenodo" forces the legacy parquet path
A new helper, _resolve_bulk_source(), encapsulates the path/DB discovery so
the main function body stays linear. The downstream rename + decoration
block at lines 477+ is unchanged: both backends produce Zenodo-style column
names so the same translation logic applies.
Backward compatibility: existing callers (no source= argument) inherit the
"auto" behavior. When the SQLite DB is absent, behavior is identical to
pre-change. When present, the SQLite path returns the newer registry data
(e.g. wind: 42,500 vs Zenodo 40,815 rows).
- scripts/build_full_bess.py: collapse the Zenodo-base + JSON-API delta merge
into a single mastr_db.load_for_pipeline("bess") call. Output parquet path
unchanged for backward compatibility with CI artefact uploads.
- mastr_db: BESS adapter now LEFT JOINs storage_units for
NutzbareSpeicherkapazitaet — open-mastr stores it there, not in
storage_extended where the column is always NULL. Falls back gracefully
when storage_units is unpopulated.
- mastr_plot.load_records / load_bess: accept new `source` parameter and
rely on load_from_bulk's source resolution. Drops the explicit
find_bulk_dir() probe — load_from_bulk handles SQLite-vs-parquet auto.
Notebooks (wind.py, pv.py) call load_records() with no args and now
transparently get the daily-fresh SQLite data when present.
- pixi: add db-mastr-storage-units task; db-mastr-core extended to fetch
it alongside wind/solar/storage/market so energy capacity is populated.
Verified end-to-end: 2.53 M BESS rows, 36.83 GW + 751.25 GWh active. 19/19
parity tests green.
- README: new Data sources table summarising primary (SQLite) vs fallback (Zenodo parquet, JSON API top-N) paths and showing the source= switch - CHANGELOG: document the SQLite addition, mastr_db.load_for_pipeline, source= switch in load_from_bulk, 19 parity tests, 7 new pixi tasks, and the build_full_bess.py rewrite - .github/workflows/refresh-docs.yml: add a "Refresh open-mastr SQLite snapshot" step that runs pixi run db-mastr-core before the render stages, with SQLITE_DATABASE_PATH env pointed at repo path so mastr_db resolves to _REPO_DB. ZIP cleanup after parse keeps runner disk under 9 GB. All 80 tests green (19 parity + 61 parser).
- All sample SVGs regenerated using mastr_plot.load_records / load_bess → open-mastr SQLite (wind 42.5k, PV 6.13M, BESS 2.53M) - scripts/render_samples.py: BESS load now uses the default prefer_bulk=True path so BESS charts pull the full 2.5M-unit registry via SQLite instead of the 200k JSON-scrape slice. Falls back to JSON if SQLite + parquet are both absent. AGS5 disambiguation note (already in place since 3158604): KREIS_NAME_TO_AGS5 dict (46 entries / 23 city-Landkreis pairs) maps BKG VG2500 name → Destatis 5-digit Kreisschlüssel. Both aggregate_by_unit (mastr_plot.py:875) and aggregate_bess_by_unit (mastr_plot.py:791) use municipality_key[:5] via combine_first as the canonical override of the landkreis_norm name-fallback for anonymised rows.
Risk 1 (BESS energy 3x gap): open-mastr's XML drops ~85 % of pumped-hydro
reservoir energy (e.g. Goldisthal 9.6 GWh per turbine → NULL). Added
_backfill_psh_energy_from_zenodo() that maps mastr_id → usable_capacity_kwh
from BNetzA_MaStR/storage.parquet for rows where SQLite returned NULL.
load_for_pipeline("bess", psh_backfill=True) wires it in by default;
psh_backfill=False yields pure open-mastr output. Result: 751 → 2,586 GWh
total (within 1.4 % of Zenodo's 2,551 GWh baseline; matches all 115 PSH
units now non-null vs 17/115 before).
Risk 2 (render_samples GWh trajectory stale parquet read): replaced the
direct pd.read_parquet(bulk_dir / 'storage.parquet') call at line 1605
with mastr_plot.load_bess() so the chart pulls from SQLite (2.5 M BESS
units) instead of the 1.75 M Zenodo snapshot. Column references switched
to the post-rename schema (energy_kwh, install_date, planned_date,
removal_date — all tz-naive). New chart numbers: 29.0 GWh historical +
18.1 GWh planned, up from 19.2 + 4.9.
Risk 3 (PV installation_type lossy): documented in mastr_db.py comment.
0.002 % of registry (~126 rows) — Großparkplatz / Gewässer collapsed
into "Bauliche Anlagen (Sonstige)". Use source="zenodo" if granularity
matters.
Risk 4 (full-storage.parquet consumer audit): no breakage. All removed
columns either rename-resilient (storage_technology, usable_capacity_kwh,
commissioning_date are preserved by load_from_bulk) or unused downstream
(status, voltage_level, feed_in_mode, net_capacity_kw, battery_technology_code).
New: cadence-aware GIF rendering (--cadence yearly|halfyear|quarterly|monthly).
22 → 43 → 85 → 253 frames over 2005-2026 with proportionally faster framerate
so wall-clock playback stays ~8–12 s. Six new pixi tasks render-{wind,pv,bess}-
{halfyear,monthly} expose the smoother variants without changing the default.
Smoother frame-to-frame growth using the new --cadence monthly option. GIF 1.3 MB / MP4 886 KB at 24 fps display rate. fig/ + docs/assets/.
- fig/bess-2018-may2026.{gif,mp4}: monthly cadence render (106 frames)
- docs/assets: replace stale bess-2017-* with bess-2018-* (BESS start_year=2018)
- docs/index.html: bump poster + source URLs to match new filename
- snapshot_dates(): walk up to FINAL_FRAME instead of END_YEAR/Dec, then ensure FINAL_FRAME is the closing frame regardless of step alignment. Monthly cadence now includes Jan/Feb/Mar/Apr 2026 + May 2026 YTD instead of jumping from Dec 2025 straight to May 2026. - PRESETS[bess]: start_year 2018 -> 2020; basename bess-2018-may2026 -> bess-2020-may2026 (battery growth before 2020 is negligible) - docs/index.html: update poster + source URLs accordingly
- wind 2005→may2026: 266 frames (was 262; +4 for 2026 Jan-Apr) - pv 2010→may2026: 206 frames; rename from pv-2005-* to pv-2010-* to match PRESETS[pv].start_year=2010 (older years were noise — flat at zero) - bess 2020→may2026: 86 frames; start year 2018 → 2020 per user request - Drop stale bess-2017-*, bess-2018-*, pv-2005-* artefacts - docs/index.html + README.md: update gif/mp4 hrefs to new filenames
astype(str) coerces NaN to the literal 'nan', which can never match an AGS5 key, so the AGS5 override was silently skipped for any anonymised row whose municipality_key was also missing. Use .str[:5] directly so NaN stays NaN and combine_first falls through to the name-based join cleanly. Applies to both aggregate_by_unit (PV/Wind) and aggregate_bess_by_unit (BESS). All 19 parity tests still pass; wind 2026-05 aggregate unchanged at 69.22 GW.
Hardcoded date(2026, 5, 1) becomes a moving target. Use date.today() by default; override with env MASTR_FINAL_FRAME=YYYY-MM-DD for reproducible renders. Monthly cadence now adds today's date as the closing frame on top of the regular month-start sequence.
Aggregates installed battery power/energy to the 16 German federal states at a configurable snapshot date (default 2026-02-28) and applies a €/MW/yr revenue benchmark (default 130 k€, near Modo Energy index median) for an order-of-magnitude annual-income estimate per state. Bundesland polygons fetched from the geoBoundaries ADM1 API and cached under data/geo/ (gitignored); used to spatially recover the few rows where MaStR has Bundesland NULL but coordinates are present.
~400 unique Kreis names vs millions of MaStR rows — apply() re-runs the regex chain per row. New normalise_kreis_series() normalises only on unique values and maps back. Behaviour-identical incl. NaN handling.
Refresh open-mastr.db to Gesamtdatenexport_20260614 (all techs) and re-render wind/pv/bess monthly choropleth animations on current data. Latest commissioning: wind 2026-06-11, solar/storage 2026-06-14.
From claude/german-battery-income-analysis. Clean merge onto current main (base 9ed7a12); adds the income-by-Bundesland BESS snapshot script.
… sources Add BESS animation to the marquee (was wind+pv only); refresh caption to monthly/2026-06. New At-a-glance table with verified live totals (wind 80.3 GW, PV 124.3 GWp, storage 29.0 GW @ 2026-06-14). Rework Data sources table with Retrieval-mode + Freshness columns (bulk full-snapshot vs JSON incremental delta vs Zenodo frozen). Fix BKG-VG2500-vs-OSM contradiction.
process_mastr_zenodo.py and convert_storage_to_parquet.py only read the bnetza_mastr_*_raw.csv bulk dump, now deleted. The DB path replaces them: load_from_bulk(source="auto") resolves to sqlite and reads wind/pv/bess straight from open-mastr.db via mastr_db.load_for_pipeline; build_full_bess rebuilds full-storage.parquet from the same DB. Verified all three techs load clean (42,617 / 6,204,610 / 2,580,464 rows).
run_notebook.py and wind.ipynb both switch the wind loader from the frozen
Zenodo CSV (record 18697247, v2026_02_19) to mastr_db.load("wind", ...),
which reads the daily-refreshable open-mastr SQLite snapshot. off_shore now
comes straight from the decoded Seelage column (Nordsee / Ostsee / None),
replacing the bundesland-string apply() heuristic with a vectorised assign.
open-mastr's two bulk-download paths report progress without a usable transfer rate: the subset path (db-mastr-core etc.) drives a per-file tqdm bar whose rate and ETA swing wildly because file sizes differ by orders of magnitude, and the full-download path labels its 1 MiB-chunk bar with unit="" so the rate prints as a bare /s. Add scripts/mastr_download.py, a thin wrapper that patches both paths before invoking Mastr().download: - Subset path: re-drive the bar by bytes (summing each member's compressed size from the remote zip's central directory) for a real MB/s rate and a byte-accurate ETA. Extraction logic mirrors upstream. - Full path: relabel the chunk bar's unit to MB (each update already consumes one 1 MiB chunk). Repoint all db-mastr-* pixi tasks to the wrapper. Task names are unchanged, so docs and README stay valid. Keeping the patch in-repo makes it durable across pixi reinstalls instead of editing site-packages.
render_wind_gif.py extends the snapshot loop past today to PLANNED_HORIZON (default 2028-12-31, env MASTR_PLANNED_HORIZON) and folds In-Planung units into those future frames only, via effective_date. Future frames carry an "incl. planned pipeline" title note; historical frames stay commissioned-only and unchanged. aggregate_by_unit gains an include_planned flag (mirrors the BESS aggregator); wind + PV load_for_pipeline now select GeplantesInbetriebnahmedatum. load_from_bulk derives wind off_shore from the authoritative MaStR seelage column (Nordsee / Ostsee) when present, instead of a longitude split; the column was already fetched by load_for_pipeline but left unused. mastr_db.load() / load_for_pipeline() convert only the date columns actually present in the result, replacing read_sql(parse_dates=...) which silently ignored names absent from a column whitelist. Rename animations *-may2026 to *-2028-planned and re-render all three at monthly cadence. Update README, docs/index.html, CHANGELOG.
…alues
open-mastr's replace_mastr_katalogeintraege casts catalog columns with
.astype("float"); WeitereBrennstoffe can hold comma-separated codes
(e.g. "2442, 2442") that crash the cast and silently drop the entire
combustion table (gas/coal/oil) to 0 rows, logged as ERROR only. Re-drive
the catalog casts through pd.to_numeric(errors="coerce") so the
unparseable multi-code value (unused downstream) is dropped and the table
parses. Restores ~93.6k combustion units. Verified via full db-mastr-all.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A larger-than-typical PR — splits cleanly into 7 commits if you'd prefer to
cherry-pick a subset. Live preview at
https://maykthewessen.github.io/marktstammdatenplotter/.
What's in here
d45260a/docs, four SVG pipeline diagrams,CLAUDE.md,mastr_plot.pyhelper module, two marimo reactive notebooks (pv.py,wind.py) with controls for date / installation type / on-offshore / colormap / bin count. Static HTML exports check in asdocs/pv.html+docs/wind.html.b8e424b3cef76cBruttoleistung-desc". Same number of rows, ~20× the signal.ac4801epixi.tomlfor reproducible env across macOS / Linux / Windows.germany_kreise.gpkgis now built fromisellsoap/deutschlandGeoJSON(~1 MB) instead of the 3.5 GB OSM extract, and carries abundeslandcolumn for state aggregations.318974ddocs/assets/kreise.json+kreise-table.js). New `scale=jenks|linear|log` picker on the marimo maps. GitHub Actions workflow (`.github/workflows/refresh-docs.yml`) runs Sundays 03:17 UTC: re-scrapes, re-renders, re-commits docs back to main. Helper scripts in `scripts/`.cca0c0f54e4d64Why a single PR
The pieces share infrastructure: the docs site reads the same SVG output dir
that the marimo notebooks write to, the CI workflow re-runs those same
notebooks, and the operator / offshore / Bundesland charts all consume the
unified DataFrame from
mastr_plot.load_records(). Splitting them woulddouble the diff churn. Happy to break into smaller PRs if you'd rather
review them in isolation.
Things I did NOT change
parser.pyis untouched — all six MaStR enum decoders preserved.wind.ipynbJupyter notebook is preserved as-is; the newreactive notebooks live alongside as
wind.py+pv.py.helpers are additive.
Caveats
of national capacity. True 6 M-plant scrape is ~24 GB JSON and impractical
for a GitHub Pages-hosted demo. CLI flag lets you go bigger.
LICENSEadded — the upstream repo carries no license file, so Ididn't unilaterally apply one. Happy to add MIT / CC-BY-SA / Apache-2.0
on request.
the branch. The CI workflow keeps them current; running manually is one
`pixi run docs-build` away.
Live preview
Cheers.