Feat/retire fallback scheduler

documentation/api/introduction.rst

@@ -116,18 +116,18 @@ See Other (303)
 ---------------
 
 Some API responses return ``HTTP status 303 (See Other)`` to redirect the client to a different resource.
-This happens, for example, when a scheduling job fails and a fallback schedule has been computed instead.
+This can happen when a custom scheduler defines a fallback scheduler, the original scheduling job fails, and the fallback schedule has been computed instead.
 In that case, the response includes a ``Location`` header pointing to the fallback schedule endpoint, so clients can automatically retrieve the fallback result.
 
 The response body will contain a JSON message with a ``status`` field set to ``"UNKNOWN_SCHEDULE"`` and a ``message`` field explaining the reason for the redirect.
 
 .. note::
 
-    The fallback schedule mechanism activates when the main scheduler encounters an infeasible problem (i.e. when constraints cannot be satisfied).
-    This is less likely to happen when ``"relax-constraints": true`` is set in the ``flex-context``, as constraint relaxation softens most infeasibility-causing constraints.
-    The hard constraints that remain even after constraint relaxation are ``soc-min``, ``soc-max``, ``soc-targets`` and ``power-capacity`` in the ``flex-model``, and ``site-power-capacity`` in the ``flex-context``.
+    FlexMeasures' built-in storage scheduler no longer computes a fallback schedule for infeasible problems.
+    Instead, ``soc-minima`` and ``soc-maxima`` are relaxed by default through ``"relax-soc-constraints": true``, while ``soc-min``, ``soc-max`` and ``soc-targets`` remain hard constraints.
+    If hard constraints cannot be satisfied, the scheduling job fails and clients receive the failure reason when requesting the schedule.
 
-    Server administrators can configure whether clients receive a 303 redirect (``FLEXMEASURES_FALLBACK_REDIRECT = True``) or whether FlexMeasures follows the fallback automatically and returns the fallback schedule directly (``FLEXMEASURES_FALLBACK_REDIRECT = False``, the default).
+    For custom schedulers that still define a fallback scheduler, server administrators can configure whether clients receive a 303 redirect (``FLEXMEASURES_FALLBACK_REDIRECT = True``) or whether FlexMeasures follows the fallback automatically and returns the fallback schedule directly (``FLEXMEASURES_FALLBACK_REDIRECT = False``, the default).
 
 Here is a client-side code example in Python for handling 303 redirects (this merely follows the redirect and should be revised to make use of the client's monitoring tools):
 

documentation/changelog.rst

@@ -14,6 +14,7 @@ New features
 * Sensor references in flex-model and flex-context support various ways of filtering by source [see `PR #2209 <https://www.github.com/FlexMeasures/flexmeasures/pull/2209>`_]
 * Let storage scheduling infer missing ``power-capacity`` from directional device capacities before falling back to site capacity, and default the missing opposite capacity to zero when only a non-zero ``consumption-capacity`` or ``production-capacity`` is configured [see `PR #2222 <https://www.github.com/FlexMeasures/flexmeasures/pull/2222>`_]
 * CLI support for adding/editing account attributes [see `PR #2242 <https://www.github.com/FlexMeasures/flexmeasures/pull/2242>`_]
+* Relax storage SoC constraints by default and report infeasible storage schedules directly instead of saving fallback schedules [see `PR #2252 <https://www.github.com/FlexMeasures/flexmeasures/pull/2252>`_]
 
 
 Infrastructure / Support

documentation/configuration.rst

@@ -787,7 +787,9 @@ Default: ``None`` (defaults are set internally for each sunset API version, e.g.
 FLEXMEASURES_FALLBACK_REDIRECT
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-Control how the API handles a failed scheduling job when a fallback schedule has been computed.
+Control how the API handles a failed scheduling job when a custom scheduler has computed a fallback schedule.
+
+FlexMeasures' built-in storage scheduler no longer computes fallback schedules, but custom schedulers may still define fallback schedulers.
 
 If ``True``, the API returns ``HTTP status 303 (See Other)`` with a ``Location`` header pointing to the fallback schedule endpoint.
 Clients must follow this redirect themselves to obtain the fallback schedule (see :ref:`api_see_other`).

documentation/features/scheduling.rst

@@ -268,13 +268,12 @@ However, here are some tips to model a buffer correctly:
    - Set ``charging-efficiency`` to the sensor describing the :abbr:`COP (coefficient of performance)` values.
    - Set ``storage-efficiency`` to a value below 100% to model (heat) loss.
 
-What happens if the flex model describes an infeasible problem for the storage scheduler? Excellent question!
-It is highly important for a robust operation that these situations still lead to a somewhat good outcome.
-From our practical experience, we derived a ``StorageFallbackScheduler``.
-It simplifies an infeasible situation by just starting to charge, discharge, or do neither,
-depending on the first target state of charge and the capabilities of the asset.
+If the flex model describes an infeasible problem for the storage scheduler, the failure should remain visible.
+By default, ``soc-minima`` and ``soc-maxima`` are relaxed into soft constraints, so the scheduler can still return a useful schedule when these boundaries cannot be fully met.
+Exact ``soc-targets`` and physical ``soc-min`` / ``soc-max`` bounds remain hard constraints.
+If those hard constraints make the problem infeasible, the scheduling job fails instead of producing a fallback schedule.
 
-Of course, we also log a failure in the scheduling job, so it's important to take note of these failures. Often, mis-configured flex models are the reason.
+It is important to take note of these failures. Often, misconfigured flex models are the reason.
 
 For a hands-on tutorial on using some of the storage flex-model fields, head over to :ref:`tut_v2g` use case and `the API documentation for triggering schedules <../api/v3_0.html#post--api-v3_0-assets-id-schedules-trigger>`_.
 

flexmeasures/api/v3_0/tests/test_sensor_schedules.py

@@ -340,7 +340,8 @@ def test_trigger_and_get_schedule_with_unknown_prices(
 @pytest.mark.parametrize(
     "requesting_user", ["test_prosumer_user@seita.nl"], indirect=True
 )
-def test_get_schedule_fallback(
+@pytest.mark.parametrize("fallback_redirect", [True, False])
+def test_get_schedule_infeasible_storage_job_without_fallback(
     app,
     add_battery_assets,
     add_market_prices,
@@ -349,13 +350,14 @@ def test_get_schedule_fallback(
     keep_scheduling_queue_empty,
     requesting_user,
     db,
+    fallback_redirect,
 ):
     """
-    Test if the fallback job is created after a failing StorageScheduler call. This test
-    is based on flexmeasures/data/models/planning/tests/test_solver.py
+    Test that a failing StorageScheduler call reports the failure without creating a fallback job.
+
+    This test is based on flexmeasures/data/models/planning/tests/test_solver.py.
     """
-    assert app.config["FLEXMEASURES_FALLBACK_REDIRECT"] is False
-    app.config["FLEXMEASURES_FALLBACK_REDIRECT"] = True
+    app.config["FLEXMEASURES_FALLBACK_REDIRECT"] = fallback_redirect
 
     target_soc = 9
     charging_station_name = "Test charging station"
@@ -375,7 +377,7 @@ def test_get_schedule_fallback(
     assert capacity == 2
     assert charging_station.get_attribute("consumption-price") == {"sensor": epex_da.id}
 
-    # check that no Fallback schedule has been saved before
+    # check that no storage fallback schedule has been saved before
     models = [
         source.model for source in charging_station.search_beliefs().sources.unique()
     ]
@@ -386,6 +388,7 @@ def test_get_schedule_fallback(
         "start": start,
         "duration": "PT24H",
         "resolution": "PT15M",  # just schedule in the original sensor resolution
+        "force-new-job-creation": True,
         "flex-model": {
             "soc-at-start": 10,
             "soc-min": charging_station.get_attribute("min_soc_in_mwh", 0),
@@ -439,193 +442,26 @@ def test_get_schedule_fallback(
         # Make sure the resolution shows up in the job kwargs
         assert job.kwargs.get("resolution") == pd.Timedelta(message["resolution"])
 
-        # the callback creates the fallback job which is still pending
-        assert len(app.queues["scheduling"]) == 1
-        fallback_job_id = Job.fetch(
-            job_id, connection=app.queues["scheduling"].connection
-        ).meta.get("fallback_job_id")
-
-        # check that the fallback_job_id is stored on the metadata of the original job
-        assert app.queues["scheduling"].get_job_ids()[0] == fallback_job_id
-        assert fallback_job_id != job_id
+        # no storage fallback job is created
+        assert len(app.queues["scheduling"]) == 0
+        assert job.meta.get("fallback_job_id") is None
 
         get_schedule_response = client.get(
             url_for("SensorAPI:get_schedule", id=charging_station.id, uuid=job_id),
         )
-        # requesting the original job redirects to the fallback job
-        assert (
-            get_schedule_response.status_code == 303
-        )  # Status code for redirect ("See other")
-        assert (
+        assert get_schedule_response.status_code == 400
+        assert "Scheduling job failed with InfeasibleProblemException: infeasible." in (
             get_schedule_response.json["message"]
-            == "Scheduling job failed with InfeasibleProblemException: infeasible. StorageScheduler was used."
         )
+        assert "StorageScheduler was used." in get_schedule_response.json["message"]
         assert get_schedule_response.json["status"] == "UNKNOWN_SCHEDULE"
         assert get_schedule_response.json["result"] == "Rejected"
 
-        # check that the redirection location points to the fallback job
-        assert (
-            get_schedule_response.headers["location"]
-            == f"http://localhost/api/v3_0/sensors/{charging_station.id}/schedules/{fallback_job_id}"
-        )
-
-        # run the fallback job
-        work_on_rq(
-            app.queues["scheduling"],
-            exc_handler=handle_scheduling_exception,
-            max_jobs=1,
-        )
-
-        # check that the queue is empty
-        assert len(app.queues["scheduling"]) == 0
-
-        # get the fallback schedule
-        fallback_schedule = client.get(
-            get_schedule_response.headers["location"],
-            json={"duration": "PT24H"},
-        ).json
-
-        # check that the fallback schedule has the right status and start dates
-        assert fallback_schedule["status"] == "PROCESSED"
-        assert parse_datetime(fallback_schedule["start"]) == parse_datetime(start)
-
         models = [
             source.model
             for source in charging_station.search_beliefs().sources.unique()
         ]
-        assert "StorageFallbackScheduler" in models
-
-        app.config["FLEXMEASURES_FALLBACK_REDIRECT"] = False
-
-
-@pytest.mark.parametrize(
-    "requesting_user", ["test_prosumer_user@seita.nl"], indirect=True
-)
-def test_get_schedule_fallback_not_redirect(
-    app,
-    add_battery_assets,
-    add_market_prices,
-    battery_soc_sensor,
-    add_charging_station_assets,
-    keep_scheduling_queue_empty,
-    requesting_user,
-    db,
-):
-    """
-    Test if the fallback scheduler is returned directly after a failing StorageScheduler call. This test
-    is based on flexmeasures/data/models/planning/tests/test_solver.py
-    """
-    app.config["FLEXMEASURES_FALLBACK_REDIRECT"] = False
-
-    target_soc = 9
-    charging_station_name = "Test charging station"
-
-    start = "2015-01-02T00:00:00+01:00"
-    epex_da = get_test_sensor(db)
-    charging_station = get_sensor_by_name(
-        add_charging_station_assets[charging_station_name], "power"
-    )
-
-    capacity = charging_station.get_attribute(
-        "capacity_in_mw",
-        ur.Quantity(charging_station.get_attribute("site-power-capacity"))
-        .to("MW")
-        .magnitude,
-    )
-    assert capacity == 2
-    assert charging_station.get_attribute("consumption-price") == {"sensor": epex_da.id}
-
-    # create a scenario that yields an infeasible problem (unreachable target SOC at 2am)
-    message = {
-        "start": start,
-        "duration": "PT24H",
-        "flex-model": {
-            "soc-at-start": 10,
-            "soc-min": charging_station.get_attribute("min_soc_in_mwh", 0),
-            "soc-max": charging_station.get_attribute("max-soc-in-mwh", target_soc),
-            "roundtrip-efficiency": charging_station.get_attribute(
-                "roundtrip-efficiency", 1
-            ),
-            "storage-efficiency": charging_station.get_attribute(
-                "storage-efficiency", 1
-            ),
-            "soc-targets": [
-                {
-                    "value": target_soc,
-                    "start": "2015-01-02T02:00:00+01:00",
-                    "duration": "PT0H",
-                }
-            ],
-        },
-    }
-
-    with app.test_client() as client:
-        # trigger storage scheduler
-        trigger_schedule_response = client.post(
-            url_for("SensorAPI:trigger_schedule", id=charging_station.id),
-            json=message,
-        )
-
-        # check that the call is successful
-        assert trigger_schedule_response.status_code == 200
-        job_id = trigger_schedule_response.json["schedule"]
-
-        # look for scheduling jobs in queue
-        assert (
-            len(app.queues["scheduling"]) == 1
-        )  # only 1 schedule should be made for 1 asset
-        job = app.queues["scheduling"].jobs[0]
-        assert job.kwargs["asset_or_sensor"]["id"] == charging_station.id
-        assert job.kwargs["start"] == parse_datetime(message["start"])
-        assert job.id == job_id
-
-        # process only the job that runs the storage scheduler (max_jobs=1)
-        work_on_rq(
-            app.queues["scheduling"],
-            exc_handler=handle_scheduling_exception,
-            max_jobs=1,
-        )
-
-        # check that the job is failing
-        job = Job.fetch(job_id, connection=app.queues["scheduling"].connection)
-        assert job.is_failed
-
-        # Make sure that the db flex_context shows up in the job kwargs
-        assert "flex-context" not in message and job.kwargs.get("flex_context")
-
-        # the callback creates the fallback job which is still pending
-        assert len(app.queues["scheduling"]) == 1
-
-        fallback_job_id = Job.fetch(
-            job_id, connection=app.queues["scheduling"].connection
-        ).meta.get("fallback_job_id")
-
-        # check that the fallback_job_id is stored on the metadata of the original job
-        assert app.queues["scheduling"].get_job_ids()[0] == fallback_job_id
-        assert fallback_job_id != job_id
-
-        get_schedule_response = client.get(
-            url_for("SensorAPI:get_schedule", id=charging_station.id, uuid=job_id),
-        )
-
-        work_on_rq(
-            app.queues["scheduling"],
-            exc_handler=handle_scheduling_exception,
-            max_jobs=1,
-        )
-
-        get_schedule_response = client.get(
-            url_for("SensorAPI:get_schedule", id=charging_station.id, uuid=job_id),
-        )
-
-        assert get_schedule_response.status_code == 200
-
-        schedule = get_schedule_response.json
-
-        # check that the fallback schedule has the right status and start dates
-        assert schedule["status"] == "PROCESSED"
-        assert parse_datetime(schedule["start"]) == parse_datetime(start)
-        assert schedule["scheduler_info"]["scheduler"] == "StorageFallbackScheduler"
+        assert "StorageFallbackScheduler" not in models
 
         app.config["FLEXMEASURES_FALLBACK_REDIRECT"] = False
 

flexmeasures/api/v3_0/tests/test_sensors_api_freshdb.py

@@ -246,7 +246,7 @@ def test_upload_sensor_data_with_unit_conversion_success(
             len(beliefs) == expected_num_beliefs
         ), f"Fetched {len(beliefs)} beliefs from the database, expecting {expected_num_beliefs}."
 
-        assert [b.event_value for b in beliefs] == expected_event_values
+        assert [b.event_value for b in beliefs] == pytest.approx(expected_event_values)
 
 
 @pytest.mark.parametrize(
@@ -327,7 +327,7 @@ def test_upload_sensor_data_floors_offclock_datetimes(
     pd.testing.assert_index_equal(
         bdf.event_starts, pd.DatetimeIndex(expected_event_starts, name="event_start")
     )
-    assert bdf["event_value"].to_list() == expected_event_values
+    assert bdf["event_value"].to_list() == pytest.approx(expected_event_values)
 
 
 @pytest.mark.parametrize(