Skip to content

feat(downloader): ENTSO-E gap repair + resume/cooldown fixes (country_code default DE) #382

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
bartzbeielstein merged 1 commit into from
Jun 13, 2026
Merged

feat(downloader): ENTSO-E gap repair + resume/cooldown fixes (country_code default DE) #382

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions _quarto.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -445,6 +445,10 @@ website:
file: docs/reference/downloader.entsoe.download_day_ahead_price.qmd
- text: "merge_build_manual"
file: docs/reference/downloader.entsoe.merge_build_manual.qmd
- text: "repair_data_gaps"
file: docs/reference/downloader.entsoe.repair_data_gaps.qmd
- text: "find_missing_intervals"
file: docs/reference/downloader.entsoe.find_missing_intervals.qmd
- text: "download_zone_loads"
file: docs/reference/downloader.entsoe.download_zone_loads.qmd
- text: "assemble_zone_loads"
Expand Down Expand Up @@ -832,6 +836,8 @@ quartodoc:
- downloader.entsoe.download_renewable_forecast
- downloader.entsoe.download_day_ahead_price
- downloader.entsoe.merge_build_manual
- downloader.entsoe.repair_data_gaps
- downloader.entsoe.find_missing_intervals
- downloader.entsoe.download_zone_loads
- downloader.entsoe.assemble_zone_loads
- downloader.entsoe.ZoneResult
Expand Down
34 changes: 18 additions & 16 deletions docs/reference/downloader.entsoe.download_new_data.qmd
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,12 +3,13 @@
```python
downloader.entsoe.download_new_data(
api_key,
country_code='FR',
country_code='DE',
start=None,
end=None,
force=False,
keep_forecast_future=False,
timeout=60.0,
on_unavailable='raise',
)
```

Expand All @@ -25,23 +26,24 @@ automatically on the next incremental download.

## Parameters {.doc-section .doc-section-parameters}

| Name | Type | Description | Default |
|----------------------|---------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------|
| api_key | [str](`str`) | The ENTSO-E API key. | _required_ |
| country_code | [str](`str`) | The country code to query (e.g., 'FR', 'DE'). Defaults to "FR". | `'FR'` |
| start | [str](`str`) \| None | Start date in 'YYYYMMDDHH00' format. | `None` |
| end | [str](`str`) \| None | End date in 'YYYYMMDDHH00' format. | `None` |
| force | [bool](`bool`) | If True, bypass the 24h cooldown check. | `False` |
| keep_forecast_future | [bool](`bool`) | If True, retain rows after the current UTC moment when building the interim file, preserving ENTSO-E's day-ahead `Forecasted Load` for tomorrow. Defaults to False (future rows are dropped, the leakage-free input for training). The flag only keeps rows the query window already covers, so pass an ``end`` that reaches into the target day (e.g. tomorrow) as well. See `merge_build_manual` for details. | `False` |
| timeout | [Optional](`typing.Optional`)\[[float](`float`)\] | Per-socket-operation read timeout in seconds passed to the ENTSO-E client. Kills stalled connections (raises ``requests.exceptions.Timeout``, caught by the existing retry loop, then ``RuntimeError`` after retries) without bounding long live transfers. ``None`` disables the timeout. Defaults to ``60.0``. | `60.0` |
| Name | Type | Description | Default |
|----------------------|------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------|
| api_key | [str](`str`) | The ENTSO-E API key. | _required_ |
| country_code | [str](`str`) | The country code to query (e.g., 'DE', 'FR'). Defaults to "DE". | `'DE'` |
| start | [str](`str`) \| [pd](`pandas`).[Timestamp](`pandas.Timestamp`) \| None | Start of the query window ('YYYYMMDDHH00' or a timestamp). If None, resumes from the last observed timestamp in the interim file, extending back to heal still-missing actuals (bounded by ``_MAX_BACKFILL_DAYS``) and falling back to seven days ago when no prior data exists. | `None` |
| end | [str](`str`) \| [pd](`pandas`).[Timestamp](`pandas.Timestamp`) \| None | End of the query window ('YYYYMMDDHH00' or a timestamp). If None, the current UTC moment is used. | `None` |
| force | [bool](`bool`) | If True, bypass the cooldown that skips a download when the last successful download is younger than ``_COOLDOWN_HOURS`` (24) hours. The cooldown is bypassed automatically when the requested window overlaps a known gap in the interim data, so gap backfills are never silently skipped. | `False` |
| keep_forecast_future | [bool](`bool`) | If True, retain rows after the current UTC moment when building the interim file, preserving ENTSO-E's day-ahead `Forecasted Load` for tomorrow. Defaults to False (future rows are dropped, the leakage-free input for training). The flag only keeps rows the query window already covers, so pass an ``end`` that reaches into the target day (e.g. tomorrow) as well. See `merge_build_manual` for details. | `False` |
| timeout | [Optional](`typing.Optional`)\[[float](`float`)\] | Per-socket-operation read timeout in seconds passed to the ENTSO-E client. Kills stalled connections (raises ``requests.exceptions.Timeout``, caught by the existing retry loop, then ``RuntimeError`` after retries) without bounding long live transfers. ``None`` disables the timeout. Defaults to ``60.0``. | `60.0` |
| on_unavailable | [OnUnavailable](`spotforecast2_safe.downloader.entsoe.OnUnavailable`) | What to do when ENTSO-E stays unreachable after the retry budget is exhausted. ``"raise"`` (default) raises ``RuntimeError``; ``"use_existing"`` logs how stale the interim data is and returns so the caller can continue on existing data (requires an existing interim file, else ``RuntimeError``). | `'raise'` |

## Raises {.doc-section .doc-section-raises}

| Name | Type | Description |
|--------|--------------------------------|----------------------------------------------------------------|
| | [ImportError](`ImportError`) | If the Python package 'entsoe-py' is not installed. |
| | [ValueError](`ValueError`) | If ``start`` or ``end`` cannot be parsed as a valid timestamp. |
| | [RuntimeError](`RuntimeError`) | If data fetching fails after ``_MAX_RETRIES`` attempts. |
| Name | Type | Description |
|--------|--------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| | [ImportError](`ImportError`) | If the Python package 'entsoe-py' is not installed. |
| | [ValueError](`ValueError`) | If ``start`` or ``end`` cannot be parsed as a valid timestamp, if both are given explicitly but ``end`` is not after ``start``, or if ``on_unavailable`` is not a recognized value. |
| | [RuntimeError](`RuntimeError`) | If data fetching fails after ``_MAX_RETRIES`` attempts and ``on_unavailable='raise'``. |

## Notes {.doc-section .doc-section-notes}

Expand Down Expand Up @@ -79,7 +81,7 @@ download_new_data(
)

# Incremental download (automatically resumes from last data point)
download_new_data(api_key="YOUR_API_KEY", country_code="FR")
download_new_data(api_key="YOUR_API_KEY", country_code="DE")

# Forced download bypassing the 24-hour cooldown check
download_new_data(
Expand Down
44 changes: 44 additions & 0 deletions docs/reference/downloader.entsoe.find_missing_intervals.qmd
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
# downloader.entsoe.find_missing_intervals { #spotforecast2_safe.downloader.entsoe.find_missing_intervals }

```python
downloader.entsoe.find_missing_intervals(index)
```

Find interior gaps in a datetime index.

Compares the index against the complete date range spanning its first
and last timestamp -- the same completeness notion as
`spotforecast2_safe.preprocessing.curate_data.basic_ts_checks`, which
raises on an incomplete index; this function reports the gaps instead
so callers can repair them. The expected spacing is the most common
step between consecutive timestamps, so the function works for hourly
as well as 15-minute ENTSO-E data.

## Parameters {.doc-section .doc-section-parameters}

| Name | Type | Description | Default |
|--------|--------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------|------------|
| index | [pd](`pandas`).[DatetimeIndex](`pandas.DatetimeIndex`) | Sorted, duplicate-free datetime index to inspect. Indexes with fewer than 3 entries cannot contain a detectable interior gap and yield an empty list. | _required_ |

## Returns {.doc-section .doc-section-returns}

| Name | Type | Description |
|--------|--------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| | [list](`list`)\[[tuple](`tuple`)\[[pd](`pandas`).[Timestamp](`pandas.Timestamp`), [pd](`pandas`).[Timestamp](`pandas.Timestamp`)\]\] | list[tuple[pd.Timestamp, pd.Timestamp]]: One ``(first_missing, last_missing)`` pair per contiguous gap, both bounds inclusive. Empty list when the index is complete. |

## Raises {.doc-section .doc-section-raises}

| Name | Type | Description |
|--------|----------------------------|-------------------------------------------------|
| | [ValueError](`ValueError`) | If ``index`` is not sorted in increasing order. |

## Examples {.doc-section .doc-section-examples}

```{python}
import pandas as pd
from spotforecast2_safe.downloader.entsoe import find_missing_intervals

full = pd.date_range("2026-06-01", periods=96, freq="h", tz="UTC")
gappy = full.delete(list(range(48, 72))) # June 3rd is missing
print(find_missing_intervals(gappy))
```
Loading
Loading