diff --git a/documentation/api/introduction.rst b/documentation/api/introduction.rst index caa047d716..bac858783c 100644 --- a/documentation/api/introduction.rst +++ b/documentation/api/introduction.rst @@ -116,18 +116,18 @@ See Other (303) --------------- Some API responses return ``HTTP status 303 (See Other)`` to redirect the client to a different resource. -This happens, for example, when a scheduling job fails and a fallback schedule has been computed instead. +This can happen when a custom scheduler defines a fallback scheduler, the original scheduling job fails, and the fallback schedule has been computed instead. In that case, the response includes a ``Location`` header pointing to the fallback schedule endpoint, so clients can automatically retrieve the fallback result. The response body will contain a JSON message with a ``status`` field set to ``"UNKNOWN_SCHEDULE"`` and a ``message`` field explaining the reason for the redirect. .. note:: - The fallback schedule mechanism activates when the main scheduler encounters an infeasible problem (i.e. when constraints cannot be satisfied). - This is less likely to happen when ``"relax-constraints": true`` is set in the ``flex-context``, as constraint relaxation softens most infeasibility-causing constraints. - The hard constraints that remain even after constraint relaxation are ``soc-min``, ``soc-max``, ``soc-targets`` and ``power-capacity`` in the ``flex-model``, and ``site-power-capacity`` in the ``flex-context``. + FlexMeasures' built-in storage scheduler no longer computes a fallback schedule for infeasible problems. + Instead, ``soc-minima`` and ``soc-maxima`` are relaxed by default through ``"relax-soc-constraints": true``, while ``soc-min``, ``soc-max`` and ``soc-targets`` remain hard constraints. + If hard constraints cannot be satisfied, the scheduling job fails and clients receive the failure reason when requesting the schedule. - Server administrators can configure whether clients receive a 303 redirect (``FLEXMEASURES_FALLBACK_REDIRECT = True``) or whether FlexMeasures follows the fallback automatically and returns the fallback schedule directly (``FLEXMEASURES_FALLBACK_REDIRECT = False``, the default). + For custom schedulers that still define a fallback scheduler, server administrators can configure whether clients receive a 303 redirect (``FLEXMEASURES_FALLBACK_REDIRECT = True``) or whether FlexMeasures follows the fallback automatically and returns the fallback schedule directly (``FLEXMEASURES_FALLBACK_REDIRECT = False``, the default). Here is a client-side code example in Python for handling 303 redirects (this merely follows the redirect and should be revised to make use of the client's monitoring tools): diff --git a/documentation/changelog.rst b/documentation/changelog.rst index 6e8d585933..a0b0e26f84 100644 --- a/documentation/changelog.rst +++ b/documentation/changelog.rst @@ -14,6 +14,7 @@ New features * Sensor references in flex-model and flex-context support various ways of filtering by source [see `PR #2209 `_] * Let storage scheduling infer missing ``power-capacity`` from directional device capacities before falling back to site capacity, and default the missing opposite capacity to zero when only a non-zero ``consumption-capacity`` or ``production-capacity`` is configured [see `PR #2222 `_] * CLI support for adding/editing account attributes [see `PR #2242 `_] +* Relax storage SoC constraints by default and report infeasible storage schedules directly instead of saving fallback schedules [see `PR #2252 `_] Infrastructure / Support diff --git a/documentation/configuration.rst b/documentation/configuration.rst index 1b823aa585..99f9195ce2 100644 --- a/documentation/configuration.rst +++ b/documentation/configuration.rst @@ -787,7 +787,9 @@ Default: ``None`` (defaults are set internally for each sunset API version, e.g. FLEXMEASURES_FALLBACK_REDIRECT ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Control how the API handles a failed scheduling job when a fallback schedule has been computed. +Control how the API handles a failed scheduling job when a custom scheduler has computed a fallback schedule. + +FlexMeasures' built-in storage scheduler no longer computes fallback schedules, but custom schedulers may still define fallback schedulers. If ``True``, the API returns ``HTTP status 303 (See Other)`` with a ``Location`` header pointing to the fallback schedule endpoint. Clients must follow this redirect themselves to obtain the fallback schedule (see :ref:`api_see_other`). diff --git a/documentation/features/scheduling.rst b/documentation/features/scheduling.rst index 1643330a6d..4fef6110bb 100644 --- a/documentation/features/scheduling.rst +++ b/documentation/features/scheduling.rst @@ -268,13 +268,12 @@ However, here are some tips to model a buffer correctly: - Set ``charging-efficiency`` to the sensor describing the :abbr:`COP (coefficient of performance)` values. - Set ``storage-efficiency`` to a value below 100% to model (heat) loss. -What happens if the flex model describes an infeasible problem for the storage scheduler? Excellent question! -It is highly important for a robust operation that these situations still lead to a somewhat good outcome. -From our practical experience, we derived a ``StorageFallbackScheduler``. -It simplifies an infeasible situation by just starting to charge, discharge, or do neither, -depending on the first target state of charge and the capabilities of the asset. +If the flex model describes an infeasible problem for the storage scheduler, the failure should remain visible. +By default, ``soc-minima`` and ``soc-maxima`` are relaxed into soft constraints, so the scheduler can still return a useful schedule when these boundaries cannot be fully met. +Exact ``soc-targets`` and physical ``soc-min`` / ``soc-max`` bounds remain hard constraints. +If those hard constraints make the problem infeasible, the scheduling job fails instead of producing a fallback schedule. -Of course, we also log a failure in the scheduling job, so it's important to take note of these failures. Often, mis-configured flex models are the reason. +It is important to take note of these failures. Often, misconfigured flex models are the reason. For a hands-on tutorial on using some of the storage flex-model fields, head over to :ref:`tut_v2g` use case and `the API documentation for triggering schedules <../api/v3_0.html#post--api-v3_0-assets-id-schedules-trigger>`_. diff --git a/flexmeasures/api/v3_0/tests/test_sensor_schedules.py b/flexmeasures/api/v3_0/tests/test_sensor_schedules.py index 6928c452ab..392c2fb12b 100644 --- a/flexmeasures/api/v3_0/tests/test_sensor_schedules.py +++ b/flexmeasures/api/v3_0/tests/test_sensor_schedules.py @@ -340,7 +340,8 @@ def test_trigger_and_get_schedule_with_unknown_prices( @pytest.mark.parametrize( "requesting_user", ["test_prosumer_user@seita.nl"], indirect=True ) -def test_get_schedule_fallback( +@pytest.mark.parametrize("fallback_redirect", [True, False]) +def test_get_schedule_infeasible_storage_job_without_fallback( app, add_battery_assets, add_market_prices, @@ -349,13 +350,14 @@ def test_get_schedule_fallback( keep_scheduling_queue_empty, requesting_user, db, + fallback_redirect, ): """ - Test if the fallback job is created after a failing StorageScheduler call. This test - is based on flexmeasures/data/models/planning/tests/test_solver.py + Test that a failing StorageScheduler call reports the failure without creating a fallback job. + + This test is based on flexmeasures/data/models/planning/tests/test_solver.py. """ - assert app.config["FLEXMEASURES_FALLBACK_REDIRECT"] is False - app.config["FLEXMEASURES_FALLBACK_REDIRECT"] = True + app.config["FLEXMEASURES_FALLBACK_REDIRECT"] = fallback_redirect target_soc = 9 charging_station_name = "Test charging station" @@ -375,7 +377,7 @@ def test_get_schedule_fallback( assert capacity == 2 assert charging_station.get_attribute("consumption-price") == {"sensor": epex_da.id} - # check that no Fallback schedule has been saved before + # check that no storage fallback schedule has been saved before models = [ source.model for source in charging_station.search_beliefs().sources.unique() ] @@ -386,6 +388,7 @@ def test_get_schedule_fallback( "start": start, "duration": "PT24H", "resolution": "PT15M", # just schedule in the original sensor resolution + "force-new-job-creation": True, "flex-model": { "soc-at-start": 10, "soc-min": charging_station.get_attribute("min_soc_in_mwh", 0), @@ -439,193 +442,26 @@ def test_get_schedule_fallback( # Make sure the resolution shows up in the job kwargs assert job.kwargs.get("resolution") == pd.Timedelta(message["resolution"]) - # the callback creates the fallback job which is still pending - assert len(app.queues["scheduling"]) == 1 - fallback_job_id = Job.fetch( - job_id, connection=app.queues["scheduling"].connection - ).meta.get("fallback_job_id") - - # check that the fallback_job_id is stored on the metadata of the original job - assert app.queues["scheduling"].get_job_ids()[0] == fallback_job_id - assert fallback_job_id != job_id + # no storage fallback job is created + assert len(app.queues["scheduling"]) == 0 + assert job.meta.get("fallback_job_id") is None get_schedule_response = client.get( url_for("SensorAPI:get_schedule", id=charging_station.id, uuid=job_id), ) - # requesting the original job redirects to the fallback job - assert ( - get_schedule_response.status_code == 303 - ) # Status code for redirect ("See other") - assert ( + assert get_schedule_response.status_code == 400 + assert "Scheduling job failed with InfeasibleProblemException: infeasible." in ( get_schedule_response.json["message"] - == "Scheduling job failed with InfeasibleProblemException: infeasible. StorageScheduler was used." ) + assert "StorageScheduler was used." in get_schedule_response.json["message"] assert get_schedule_response.json["status"] == "UNKNOWN_SCHEDULE" assert get_schedule_response.json["result"] == "Rejected" - # check that the redirection location points to the fallback job - assert ( - get_schedule_response.headers["location"] - == f"http://localhost/api/v3_0/sensors/{charging_station.id}/schedules/{fallback_job_id}" - ) - - # run the fallback job - work_on_rq( - app.queues["scheduling"], - exc_handler=handle_scheduling_exception, - max_jobs=1, - ) - - # check that the queue is empty - assert len(app.queues["scheduling"]) == 0 - - # get the fallback schedule - fallback_schedule = client.get( - get_schedule_response.headers["location"], - json={"duration": "PT24H"}, - ).json - - # check that the fallback schedule has the right status and start dates - assert fallback_schedule["status"] == "PROCESSED" - assert parse_datetime(fallback_schedule["start"]) == parse_datetime(start) - models = [ source.model for source in charging_station.search_beliefs().sources.unique() ] - assert "StorageFallbackScheduler" in models - - app.config["FLEXMEASURES_FALLBACK_REDIRECT"] = False - - -@pytest.mark.parametrize( - "requesting_user", ["test_prosumer_user@seita.nl"], indirect=True -) -def test_get_schedule_fallback_not_redirect( - app, - add_battery_assets, - add_market_prices, - battery_soc_sensor, - add_charging_station_assets, - keep_scheduling_queue_empty, - requesting_user, - db, -): - """ - Test if the fallback scheduler is returned directly after a failing StorageScheduler call. This test - is based on flexmeasures/data/models/planning/tests/test_solver.py - """ - app.config["FLEXMEASURES_FALLBACK_REDIRECT"] = False - - target_soc = 9 - charging_station_name = "Test charging station" - - start = "2015-01-02T00:00:00+01:00" - epex_da = get_test_sensor(db) - charging_station = get_sensor_by_name( - add_charging_station_assets[charging_station_name], "power" - ) - - capacity = charging_station.get_attribute( - "capacity_in_mw", - ur.Quantity(charging_station.get_attribute("site-power-capacity")) - .to("MW") - .magnitude, - ) - assert capacity == 2 - assert charging_station.get_attribute("consumption-price") == {"sensor": epex_da.id} - - # create a scenario that yields an infeasible problem (unreachable target SOC at 2am) - message = { - "start": start, - "duration": "PT24H", - "flex-model": { - "soc-at-start": 10, - "soc-min": charging_station.get_attribute("min_soc_in_mwh", 0), - "soc-max": charging_station.get_attribute("max-soc-in-mwh", target_soc), - "roundtrip-efficiency": charging_station.get_attribute( - "roundtrip-efficiency", 1 - ), - "storage-efficiency": charging_station.get_attribute( - "storage-efficiency", 1 - ), - "soc-targets": [ - { - "value": target_soc, - "start": "2015-01-02T02:00:00+01:00", - "duration": "PT0H", - } - ], - }, - } - - with app.test_client() as client: - # trigger storage scheduler - trigger_schedule_response = client.post( - url_for("SensorAPI:trigger_schedule", id=charging_station.id), - json=message, - ) - - # check that the call is successful - assert trigger_schedule_response.status_code == 200 - job_id = trigger_schedule_response.json["schedule"] - - # look for scheduling jobs in queue - assert ( - len(app.queues["scheduling"]) == 1 - ) # only 1 schedule should be made for 1 asset - job = app.queues["scheduling"].jobs[0] - assert job.kwargs["asset_or_sensor"]["id"] == charging_station.id - assert job.kwargs["start"] == parse_datetime(message["start"]) - assert job.id == job_id - - # process only the job that runs the storage scheduler (max_jobs=1) - work_on_rq( - app.queues["scheduling"], - exc_handler=handle_scheduling_exception, - max_jobs=1, - ) - - # check that the job is failing - job = Job.fetch(job_id, connection=app.queues["scheduling"].connection) - assert job.is_failed - - # Make sure that the db flex_context shows up in the job kwargs - assert "flex-context" not in message and job.kwargs.get("flex_context") - - # the callback creates the fallback job which is still pending - assert len(app.queues["scheduling"]) == 1 - - fallback_job_id = Job.fetch( - job_id, connection=app.queues["scheduling"].connection - ).meta.get("fallback_job_id") - - # check that the fallback_job_id is stored on the metadata of the original job - assert app.queues["scheduling"].get_job_ids()[0] == fallback_job_id - assert fallback_job_id != job_id - - get_schedule_response = client.get( - url_for("SensorAPI:get_schedule", id=charging_station.id, uuid=job_id), - ) - - work_on_rq( - app.queues["scheduling"], - exc_handler=handle_scheduling_exception, - max_jobs=1, - ) - - get_schedule_response = client.get( - url_for("SensorAPI:get_schedule", id=charging_station.id, uuid=job_id), - ) - - assert get_schedule_response.status_code == 200 - - schedule = get_schedule_response.json - - # check that the fallback schedule has the right status and start dates - assert schedule["status"] == "PROCESSED" - assert parse_datetime(schedule["start"]) == parse_datetime(start) - assert schedule["scheduler_info"]["scheduler"] == "StorageFallbackScheduler" + assert "StorageFallbackScheduler" not in models app.config["FLEXMEASURES_FALLBACK_REDIRECT"] = False diff --git a/flexmeasures/api/v3_0/tests/test_sensors_api_freshdb.py b/flexmeasures/api/v3_0/tests/test_sensors_api_freshdb.py index 18596d64aa..2a6a61d34f 100644 --- a/flexmeasures/api/v3_0/tests/test_sensors_api_freshdb.py +++ b/flexmeasures/api/v3_0/tests/test_sensors_api_freshdb.py @@ -246,7 +246,7 @@ def test_upload_sensor_data_with_unit_conversion_success( len(beliefs) == expected_num_beliefs ), f"Fetched {len(beliefs)} beliefs from the database, expecting {expected_num_beliefs}." - assert [b.event_value for b in beliefs] == expected_event_values + assert [b.event_value for b in beliefs] == pytest.approx(expected_event_values) @pytest.mark.parametrize( @@ -327,7 +327,7 @@ def test_upload_sensor_data_floors_offclock_datetimes( pd.testing.assert_index_equal( bdf.event_starts, pd.DatetimeIndex(expected_event_starts, name="event_start") ) - assert bdf["event_value"].to_list() == expected_event_values + assert bdf["event_value"].to_list() == pytest.approx(expected_event_values) @pytest.mark.parametrize( diff --git a/flexmeasures/data/models/planning/storage.py b/flexmeasures/data/models/planning/storage.py index 4c59a0ffa9..75705f06d6 100644 --- a/flexmeasures/data/models/planning/storage.py +++ b/flexmeasures/data/models/planning/storage.py @@ -3,7 +3,6 @@ import re import copy from datetime import datetime, timedelta -from typing import Type import pandas as pd import numpy as np @@ -26,7 +25,6 @@ initialize_series, initialize_df, get_power_values, - fallback_charging_policy, get_continuous_series_sensor_or_quantity, ) from flexmeasures.data.models.planning.exceptions import InfeasibleProblemException @@ -1582,81 +1580,10 @@ def _ensure_variable_quantity( return q -class StorageFallbackScheduler(MetaStorageScheduler): - __version__ = "3" - __author__ = "Seita" - - def compute(self, skip_validation: bool = False) -> SchedulerOutputType: - """Schedule a battery or Charge Point by just starting to charge, discharge, or do neither, - depending on the first target state of charge and the capabilities of the Charge Point. - For the resulting consumption schedule, consumption is defined as positive values. - - Note that this ignores any cause of the infeasibility. - - :param skip_validation: If True, skip validation of constraints specified in the data. - :returns: The computed schedule. - """ - - ( - sensors, - start, - end, - resolution, - soc_at_start, - device_constraints, - ems_constraints, - commitments, - ) = self._prepare(skip_validation=skip_validation) - - # Fallback policy if the problem was unsolvable - storage_schedule = { - sensor: fallback_charging_policy( - sensor, device_constraints[d], start, end, resolution - ) - for d, sensor in enumerate(sensors) - if sensor is not None - } - - # Convert each device schedule to the unit of the device's power sensor - storage_schedule = { - sensor: convert_units( - storage_schedule[sensor], - "MW", - sensor.unit, - event_resolution=sensor.event_resolution, - ) - for sensor in sensors - if sensor is not None - } - - # Round schedule - if self.round_to_decimals: - storage_schedule = { - sensor: storage_schedule[sensor].round(self.round_to_decimals) - for sensor in sensors - if sensor is not None - } - - if self.return_multiple: - return [ - { - "name": "storage_schedule", - "sensor": sensor, - "data": storage_schedule[sensor], - } - for sensor in sensors - if sensor is not None - ] - else: - return storage_schedule[sensors[0]] - - class StorageScheduler(MetaStorageScheduler): __version__ = "8" __author__ = "Seita" - fallback_scheduler_class: Type[Scheduler] = StorageFallbackScheduler - @staticmethod def _build_soc_schedule( flex_model: list[dict], diff --git a/flexmeasures/data/models/planning/tests/test_solver.py b/flexmeasures/data/models/planning/tests/test_solver.py index e9e61da848..8c37ff05c2 100644 --- a/flexmeasures/data/models/planning/tests/test_solver.py +++ b/flexmeasures/data/models/planning/tests/test_solver.py @@ -498,22 +498,18 @@ def test_charging_station_solver_day_2( (15, "Test charging station (bidirectional)"), ], ) -def test_fallback_to_unsolvable_problem( +def test_storage_scheduler_reports_unsolvable_problem_without_fallback( target_soc, charging_station_name, setup_planning_test_data, db ): """Starting with a state of charge 10 kWh, within 2 hours we should be able to reach any state of charge in the range [10, 14] kWh for a unidirectional station, or [6, 14] for a bidirectional station, given a charging capacity of 2 kW. Here we test target states of charge outside that range, ones that we should be able - to get as close to as 1 kWh difference. - We want our scheduler to handle unsolvable problems like these with a sensible fallback policy. - - The StorageScheduler raises an Exception which triggers the creation of a new job to compute a fallback - schedule. + The StorageScheduler should report this infeasible problem without hiding it behind + a fallback schedule. """ soc_at_start = 10 duration_until_target = timedelta(hours=2) - expected_gap = 1 epex_da = get_test_sensor(db) charging_station = setup_planning_test_data[charging_station_name].sensors[0] @@ -578,26 +574,8 @@ def test_fallback_to_unsolvable_problem( # calling the scheduler with an infeasible problem raises an Exception with pytest.raises(InfeasibleProblemException): - consumption_schedule = scheduler.compute(skip_validation=True) - - # check that the fallback scheduler provides a sensible fallback policy - fallback_scheduler = scheduler.fallback_scheduler_class(**kwargs) - fallback_scheduler.config_deserialized = True - consumption_schedule = fallback_scheduler.compute(skip_validation=True) - - soc_schedule = integrate_time_series( - consumption_schedule, soc_at_start, decimal_precision=6 - ) - - # Check if constraints were met - assert min(consumption_schedule.values) >= capacity * -1 - assert max(consumption_schedule.values) <= capacity - print(consumption_schedule.head(12)) - print(soc_schedule.head(12)) - assert ( - abs(abs(soc_schedule.loc[target_soc_datetime] - target_soc) - expected_gap) - < TOLERANCE - ) + scheduler.compute(skip_validation=True) + assert scheduler.fallback_scheduler_class is None @pytest.mark.parametrize( diff --git a/flexmeasures/data/models/planning/tests/test_storage.py b/flexmeasures/data/models/planning/tests/test_storage.py index ed8c09be82..a5a47d6bc6 100644 --- a/flexmeasures/data/models/planning/tests/test_storage.py +++ b/flexmeasures/data/models/planning/tests/test_storage.py @@ -174,7 +174,8 @@ def test_battery_relaxation(add_battery_assets, db): """Check that resolving SoC breaches is more important than resolving device power breaches. The battery is still charging with 25 kW between noon and 4 PM, when the consumption capacity is supposed to be 0. - It is still charging because resolving the still unmatched SoC minima takes precedence (via breach prices). + It is still charging because resolving the still unmatched SoC minima takes precedence + through the default SoC breach prices. """ _, battery = get_sensors_from_db( db, add_battery_assets, battery_name="Test battery" @@ -252,7 +253,6 @@ def test_battery_relaxation(add_battery_assets, db): "site-peak-production-price": series_to_ts_specs( pd.Series(260, production_prices.index), unit="EUR/MW" ), - "soc-minima-breach-price": "6000 EUR/kWh", # high breach price (to mimic a hard constraint) "consumption-breach-price": f"{device_power_breach_price} EUR/kW", # lower breach price (thus prioritizing minimizing soc breaches) "production-breach-price": f"{device_power_breach_price} EUR/kW", # lower breach price (thus prioritizing minimizing soc breaches) }, diff --git a/flexmeasures/data/models/planning/utils.py b/flexmeasures/data/models/planning/utils.py index cfea6eba58..7cd79f7156 100644 --- a/flexmeasures/data/models/planning/utils.py +++ b/flexmeasures/data/models/planning/utils.py @@ -220,83 +220,6 @@ def get_power_values( return -series -def fallback_charging_policy( - sensor: Sensor, - device_constraints: pd.DataFrame, - start: datetime, - end: datetime, - resolution: timedelta, -) -> pd.Series: - """This fallback charging policy is to just start charging or discharging, or do neither, - depending on the first target state of charge and the capabilities of the Charge Point. - Note that this ignores any cause of the infeasibility and, - while probably a decent policy for Charge Points, - should not be considered a robust policy for other asset types. - """ - max_charge_capacity = ( - device_constraints[["derivative max", "derivative equals"]].min().min() - ) - max_discharge_capacity = ( - -device_constraints[["derivative min", "derivative equals"]].max().max() - ) - charge_power = max_charge_capacity if sensor.get_attribute("is_consumer") else 0 - discharge_power = ( - -max_discharge_capacity if sensor.get_attribute("is_producer") else 0 - ) - - charge_schedule = initialize_series(charge_power, start, end, resolution) - discharge_schedule = initialize_series(discharge_power, start, end, resolution) - idle_schedule = initialize_series(0, start, end, resolution) - if ( - device_constraints["equals"].first_valid_index() is not None - and device_constraints["equals"][ - device_constraints["equals"].first_valid_index() - ] - > 0 - ): - # start charging to get as close as possible to the next target - return idle_after_reaching_target(charge_schedule, device_constraints["equals"]) - if ( - device_constraints["equals"].first_valid_index() is not None - and device_constraints["equals"][ - device_constraints["equals"].first_valid_index() - ] - < 0 - ): - # start discharging to get as close as possible to the next target - return idle_after_reaching_target( - discharge_schedule, device_constraints["equals"] - ) - if ( - device_constraints["max"].first_valid_index() is not None - and device_constraints["max"][device_constraints["max"].first_valid_index()] < 0 - ): - # start discharging to try and bring back the soc below the next max constraint - return idle_after_reaching_target(discharge_schedule, device_constraints["max"]) - if ( - device_constraints["min"].first_valid_index() is not None - and device_constraints["min"][device_constraints["min"].first_valid_index()] > 0 - ): - # start charging to try and bring back the soc above the next min constraint - return idle_after_reaching_target(charge_schedule, device_constraints["min"]) - # stand idle - return idle_schedule - - -def idle_after_reaching_target( - schedule: pd.Series, - target: pd.Series, - initial_state: float = 0, -) -> pd.Series: - """Stop planned (dis)charging after target is reached (or constraint is met).""" - first_target = target[target.first_valid_index()] - if first_target > initial_state: - schedule[schedule.cumsum() > first_target] = 0 - else: - schedule[schedule.cumsum() < first_target] = 0 - return schedule - - def get_series_from_quantity_or_sensor( variable_quantity: Sensor | SensorReference | list[dict] | ur.Quantity, unit: ur.Quantity | str, diff --git a/flexmeasures/data/schemas/scheduling/__init__.py b/flexmeasures/data/schemas/scheduling/__init__.py index ad9dc79d17..97a2029b2a 100644 --- a/flexmeasures/data/schemas/scheduling/__init__.py +++ b/flexmeasures/data/schemas/scheduling/__init__.py @@ -180,7 +180,7 @@ class FlexContextSchema(Schema): # Dev fields relax_soc_constraints = fields.Bool( data_key="relax-soc-constraints", - load_default=False, + load_default=True, metadata=metadata.RELAX_SOC_CONSTRAINTS.to_dict(), ) relax_capacity_constraints = fields.Bool( @@ -358,7 +358,9 @@ def check_prices(self, data: dict, original_data: dict, **kwargs): for field_var, field in self.declared_fields.items() } if any( - field_map[field] in data and data[field_map[field]] + field in original_data + and field_map[field] in data + and data[field_map[field]] for field in ( "soc-minima-breach-price", "soc-maxima-breach-price", @@ -391,8 +393,7 @@ def check_prices(self, data: dict, original_data: dict, **kwargs): # Fill in default soc breach prices when asked to relax SoC constraints, unless already set explicitly. if ( - data["relax_soc_constraints"] - or data["relax_constraints"] + (data["relax_soc_constraints"] or data["relax_constraints"]) and not data.get("soc_minima_breach_price") and not data.get("soc_maxima_breach_price") ): @@ -404,8 +405,7 @@ def check_prices(self, data: dict, original_data: dict, **kwargs): # Fill in default capacity breach prices when asked to relax capacity constraints, unless already set explicitly. if ( - data["relax_capacity_constraints"] - or data["relax_constraints"] + (data["relax_capacity_constraints"] or data["relax_constraints"]) and not data.get("consumption_breach_price") and not data.get("production_breach_price") ): @@ -417,8 +417,7 @@ def check_prices(self, data: dict, original_data: dict, **kwargs): # Fill in default site capacity breach prices when asked to relax site capacity constraints, unless already set explicitly. if ( - data["relax_site_capacity_constraints"] - or data["relax_constraints"] + (data["relax_site_capacity_constraints"] or data["relax_constraints"]) and not data.get("ems_consumption_breach_price") and not data.get("ems_production_breach_price") ): @@ -764,6 +763,11 @@ def _to_currency_per_mwh(price_unit: str) -> str: class DBFlexContextSchema(FlexContextSchema, NoTimeSeriesSpecs): + relax_soc_constraints = fields.Bool( + data_key="relax-soc-constraints", + load_default=False, + metadata=metadata.RELAX_SOC_CONSTRAINTS.to_dict(), + ) commitments = fields.Nested( DBCommitmentSchema, data_key="commitments", required=False, many=True diff --git a/flexmeasures/data/schemas/scheduling/metadata.py b/flexmeasures/data/schemas/scheduling/metadata.py index 21c7765f18..bfa2db6be7 100644 --- a/flexmeasures/data/schemas/scheduling/metadata.py +++ b/flexmeasures/data/schemas/scheduling/metadata.py @@ -145,13 +145,14 @@ def to_dict(self): 2. Avoid not meeting SoC minima/maxima. 3. Avoid breaching the desired device consumption/production capacity. -We recommend to set this field to ``True`` to enable the default prices and associated priorities as defined by FlexMeasures. +SoC minima/maxima are already relaxed by default through ``relax-soc-constraints``. +Set this field to ``True`` to also enable the default site and device capacity breach prices and associated priorities as defined by FlexMeasures. For tighter control over prices and priorities, the breach prices can also be set explicitly (the relevant fields have ``breach-price`` in their name). """, example=True, ) RELAX_SOC_CONSTRAINTS = MetaData( - description="If True, avoids not meeting SoC minima/maxima as a relaxed constraint.", + description="If True (default), avoids not meeting SoC minima/maxima as relaxed constraints. Set this to False to keep SoC minima/maxima as hard constraints unless breach prices are supplied explicitly.", example=True, ) RELAX_CAPACITY_CONSTRAINTS = MetaData( @@ -246,8 +247,9 @@ def to_dict(self): ) SOC_MINIMA = MetaData( description="""Set points that form lower boundaries, e.g. to target a full car battery in the morning. -If a ``soc-minima-breach-price`` is defined, the ``soc-minima`` become soft constraints in the optimization problem. -Otherwise, they become hard constraints. [#maximum_overlap]_. Both single points in time and ranges are possible, see example.""", +The ``soc-minima`` are soft constraints in the optimization problem by default, because ``relax-soc-constraints`` defaults to ``True`` and supplies a default ``soc-minima-breach-price``. +Set ``relax-soc-constraints`` to ``False`` to keep them as hard constraints unless ``soc-minima-breach-price`` is supplied explicitly [#maximum_overlap]_. +Both single points in time and ranges are possible, see example.""", example=[ {"datetime": "2024-02-05T08:00:00+01:00", "value": "8.2 kWh"}, { @@ -259,8 +261,8 @@ def to_dict(self): ) SOC_MAXIMA = MetaData( description="""Set points that form upper boundaries at certain times, e.g. to target an empty heat buffer before a maintenance window. -If a ``soc-maxima-breach-price`` is defined, the ``soc-maxima`` become soft constraints in the optimization problem. -Otherwise, they become hard constraints. [#minimum_overlap]_""", +The ``soc-maxima`` are soft constraints in the optimization problem by default, because ``relax-soc-constraints`` defaults to ``True`` and supplies a default ``soc-maxima-breach-price``. +Set ``relax-soc-constraints`` to ``False`` to keep them as hard constraints unless ``soc-maxima-breach-price`` is supplied explicitly. [#minimum_overlap]_""", example=[ { "value": "51 kWh", diff --git a/flexmeasures/data/schemas/tests/test_scheduling.py b/flexmeasures/data/schemas/tests/test_scheduling.py index 3d4450580e..4522d97484 100644 --- a/flexmeasures/data/schemas/tests/test_scheduling.py +++ b/flexmeasures/data/schemas/tests/test_scheduling.py @@ -517,6 +517,48 @@ def test_flex_context_schema( check_schema_loads_data(schema=schema, data=flex_context, fails=fails) +def test_flex_context_schema_relaxes_soc_constraints_by_default(): + loaded_flex_context = FlexContextSchema().load({"consumption-price": "1 EUR/MWh"}) + + assert loaded_flex_context["relax_soc_constraints"] is True + assert loaded_flex_context["soc_minima_breach_price"].to( + "EUR/MWh" + ).magnitude == pytest.approx(1_000_000) + assert loaded_flex_context["soc_maxima_breach_price"].to( + "EUR/MWh" + ).magnitude == pytest.approx(1_000_000) + assert "consumption_breach_price" not in loaded_flex_context + assert "production_breach_price" not in loaded_flex_context + assert "ems_consumption_breach_price" not in loaded_flex_context + assert "ems_production_breach_price" not in loaded_flex_context + + +def test_flex_context_schema_preserves_explicit_soc_breach_prices(): + loaded_flex_context = FlexContextSchema().load( + { + "consumption-price": "1 EUR/MWh", + "soc-minima-breach-price": "5 EUR/kWh", + "soc-maxima-breach-price": "7 EUR/kWh", + } + ) + + assert loaded_flex_context["relax_soc_constraints"] is True + assert loaded_flex_context["soc_minima_breach_price"].to( + "EUR/kWh" + ).magnitude == pytest.approx(5) + assert loaded_flex_context["soc_maxima_breach_price"].to( + "EUR/kWh" + ).magnitude == pytest.approx(7) + + +def test_db_flex_context_schema_does_not_relax_soc_constraints_by_default(): + loaded_flex_context = DBFlexContextSchema().load({}) + + assert loaded_flex_context["relax_soc_constraints"] is False + assert "soc_minima_breach_price" not in loaded_flex_context + assert "soc_maxima_breach_price" not in loaded_flex_context + + def check_schema_loads_data(schema, data, fails): if fails: with pytest.raises(ValidationError) as e_info: diff --git a/flexmeasures/data/schemas/tests/test_sensor.py b/flexmeasures/data/schemas/tests/test_sensor.py index 7d3c06fe4b..ec574b0bcf 100644 --- a/flexmeasures/data/schemas/tests/test_sensor.py +++ b/flexmeasures/data/schemas/tests/test_sensor.py @@ -20,6 +20,11 @@ def serialize_variable_quantity(value): return VariableQuantityDumpSchema().dump({"value": value})["value"] +def assert_quantity_equals(quantity, expected_quantity): + assert str(quantity.units) == str(expected_quantity.units) + assert quantity.magnitude == pytest.approx(expected_quantity.magnitude) + + @pytest.mark.parametrize( "src_quantity, dst_unit, fails, exp_dst_quantity", [ @@ -93,11 +98,11 @@ def test_quantity_or_sensor_deserialize( try: dst_quantity = schema.deserialize(src_quantity) if isinstance(src_quantity, (ur.Quantity, int, float)): - assert dst_quantity == ur.Quantity(exp_dst_quantity) - assert str(dst_quantity) == exp_dst_quantity + assert_quantity_equals(dst_quantity, ur.Quantity(exp_dst_quantity)) elif isinstance(src_quantity, list): - assert dst_quantity[0]["value"] == ur.Quantity(exp_dst_quantity) - assert str(dst_quantity[0]["value"]) == exp_dst_quantity + assert_quantity_equals( + dst_quantity[0]["value"], ur.Quantity(exp_dst_quantity) + ) assert not fails except ValidationError as e: assert fails, e diff --git a/flexmeasures/data/tests/test_scheduling_sequential.py b/flexmeasures/data/tests/test_scheduling_sequential.py index 53f67fa8c9..b36c547aaa 100644 --- a/flexmeasures/data/tests/test_scheduling_sequential.py +++ b/flexmeasures/data/tests/test_scheduling_sequential.py @@ -141,13 +141,13 @@ def test_create_sequential_jobs(db, app, flex_description_sequential, smart_buil assert total_cost == -2.1775, f"Total cost should be -2.1775 €, got {total_cost} €" -def test_create_sequential_jobs_fallback( +def test_create_sequential_jobs_without_storage_fallback( db, app, flex_description_sequential, smart_building ): - """Test fallback scheduler in a chain of sequential scheduling (sub)jobs. + """Test an infeasible first subjob in a chain of sequential scheduling jobs. - Checks execution of a sequential scheduling job, where 1 of the subjobs is set up to fail and trigger its fallback. - The deferred subjobs should still succeed after the fallback succeeds, even though the first subjob fails. + Checks that no storage fallback job is created. The deferred subjobs should remain + deferred because the first subjob failed. """ assets, sensors, _ = smart_building queue = app.queues["scheduling"] @@ -166,53 +166,51 @@ def test_create_sequential_jobs_fallback( storage_module = "flexmeasures.data.models.planning.storage" with patch(f"{storage_module}.StorageScheduler.persist_flex_model"): - with patch(f"{storage_module}.StorageFallbackScheduler.persist_flex_model"): - with patch( - f"{storage_module}.StorageScheduler.compute", - side_effect=iter([InfeasibleProblemException(), [], []]), - ): - create_sequential_scheduling_job( - asset=assets["Test Site"], - scheduler_specs=scheduler_specs, - enqueue=True, - force_new_job_creation=True, # otherwise the cache might kick in due to sub-jobs already created in other tests - **flex_description_sequential, - ) - - # There should be 3 jobs: - # 2 jobs scheduling the 2 flexible devices in the flex-model, plus 1 'done job' to wrap things up - queued_jobs = app.queues["scheduling"].jobs - deferred_jobs = [ - Job.fetch(job_id, connection=queue.connection) - for job_id in app.queues[ - "scheduling" - ].deferred_job_registry.get_job_ids() - ] - # Sort deferred_jobs by their created_at attribute - deferred_jobs = sorted(deferred_jobs, key=lambda job: job.created_at) - assert ( - len(queued_jobs) == 1 - ), "Only the job for scheduling the first device sequentially should be queued." - assert ( - len(deferred_jobs) == 2 - ), "The job for scheduling the second device, and the wrap-up job, should be deferred." - - # Work on jobs - work_on_rq(queue, exc_handler=handle_scheduling_exception) - - # Refresh jobs so that the fallback_job_id (which should be set by now) can be read - for job in queued_jobs: - job.refresh() - - finished_jobs = queue.finished_job_registry.get_job_ids() - failed_jobs = queue.failed_job_registry.get_job_ids() - - # Original job failed - assert queued_jobs[0].id in failed_jobs - - # The fallback job ran successfully - assert queued_jobs[0].meta["fallback_job_id"] in finished_jobs - - # The deferred jobs ran successfully - assert deferred_jobs[0].id in finished_jobs - assert deferred_jobs[1].id in finished_jobs + with patch( + f"{storage_module}.StorageScheduler.compute", + side_effect=InfeasibleProblemException(), + ): + create_sequential_scheduling_job( + asset=assets["Test Site"], + scheduler_specs=scheduler_specs, + enqueue=True, + force_new_job_creation=True, # otherwise the cache might kick in due to sub-jobs already created in other tests + **flex_description_sequential, + ) + + # There should be 3 jobs: + # 2 jobs scheduling the 2 flexible devices in the flex-model, plus 1 'done job' to wrap things up + queued_jobs = app.queues["scheduling"].jobs + deferred_jobs = [ + Job.fetch(job_id, connection=queue.connection) + for job_id in app.queues[ + "scheduling" + ].deferred_job_registry.get_job_ids() + ] + # Sort deferred_jobs by their created_at attribute + deferred_jobs = sorted(deferred_jobs, key=lambda job: job.created_at) + assert ( + len(queued_jobs) == 1 + ), "Only the job for scheduling the first device sequentially should be queued." + assert ( + len(deferred_jobs) == 2 + ), "The job for scheduling the second device, and the wrap-up job, should be deferred." + + # Work on jobs + work_on_rq(queue, exc_handler=handle_scheduling_exception) + + for job in queued_jobs: + job.refresh() + for job in deferred_jobs: + job.refresh() + + finished_jobs = queue.finished_job_registry.get_job_ids() + failed_jobs = queue.failed_job_registry.get_job_ids() + + # Original job failed and no fallback job was created + assert queued_jobs[0].id in failed_jobs + assert queued_jobs[0].meta.get("fallback_job_id") is None + + # The deferred jobs should not run when their dependency fails without fallback + assert deferred_jobs[0].id not in finished_jobs + assert deferred_jobs[1].id not in finished_jobs diff --git a/flexmeasures/ui/static/openapi-specs.json b/flexmeasures/ui/static/openapi-specs.json index fd62b861ac..9457e828d7 100644 --- a/flexmeasures/ui/static/openapi-specs.json +++ b/flexmeasures/ui/static/openapi-specs.json @@ -4583,13 +4583,13 @@ "relax-constraints": { "type": "boolean", "default": false, - "description": "If True (default is False), several constraints are relaxed by setting default breach prices within the optimization problem, leading to the default priority:\n\n1. Avoid breaching the site consumption/production capacity.\n2. Avoid not meeting SoC minima/maxima.\n3. Avoid breaching the desired device consumption/production capacity.\n\nWe recommend to set this field to True to enable the default prices and associated priorities as defined by FlexMeasures.\nFor tighter control over prices and priorities, the breach prices can also be set explicitly (the relevant fields have breach-price in their name).\n", + "description": "If True (default is False), several constraints are relaxed by setting default breach prices within the optimization problem, leading to the default priority:\n\n1. Avoid breaching the site consumption/production capacity.\n2. Avoid not meeting SoC minima/maxima.\n3. Avoid breaching the desired device consumption/production capacity.\n\nSoC minima/maxima are already relaxed by default through relax-soc-constraints.\nSet this field to True to also enable the default site and device capacity breach prices and associated priorities as defined by FlexMeasures.\nFor tighter control over prices and priorities, the breach prices can also be set explicitly (the relevant fields have breach-price in their name).\n", "example": true }, "relax-soc-constraints": { "type": "boolean", - "default": false, - "description": "If True, avoids not meeting SoC minima/maxima as a relaxed constraint.", + "default": true, + "description": "If True (default), avoids not meeting SoC minima/maxima as relaxed constraints. Set this to False to keep SoC minima/maxima as hard constraints unless breach prices are supplied explicitly.", "example": true }, "relax-capacity-constraints": { @@ -6124,7 +6124,7 @@ "example": true }, "soc-maxima": { - "description": "Set points that form upper boundaries at certain times, e.g. to target an empty heat buffer before a maintenance window.\nIf a soc-maxima-breach-price is defined, the soc-maxima become soft constraints in the optimization problem.\nOtherwise, they become hard constraints.", + "description": "Set points that form upper boundaries at certain times, e.g. to target an empty heat buffer before a maintenance window.\nThe soc-maxima are soft constraints in the optimization problem by default, because relax-soc-constraints defaults to True and supplies a default soc-maxima-breach-price.\nSet relax-soc-constraints to False to keep them as hard constraints unless soc-maxima-breach-price is supplied explicitly.", "example": [ { "value": "51 kWh", @@ -6135,7 +6135,7 @@ "$ref": "#/components/schemas/VariableQuantityOpenAPI" }, "soc-minima": { - "description": "Set points that form lower boundaries, e.g. to target a full car battery in the morning.\nIf a soc-minima-breach-price is defined, the soc-minima become soft constraints in the optimization problem.\nOtherwise, they become hard constraints.. Both single points in time and ranges are possible, see example.", + "description": "Set points that form lower boundaries, e.g. to target a full car battery in the morning.\nThe soc-minima are soft constraints in the optimization problem by default, because relax-soc-constraints defaults to True and supplies a default soc-minima-breach-price.\nSet relax-soc-constraints to False to keep them as hard constraints unless soc-minima-breach-price is supplied explicitly.\nBoth single points in time and ranges are possible, see example.", "example": [ { "datetime": "2024-02-05T08:00:00+01:00",