Skip to content

Restore period index to DSD#287

Draft
idelder wants to merge 8 commits intoTemoaProject:unstablefrom
idelder:dsd_periods
Draft

Restore period index to DSD#287
idelder wants to merge 8 commits intoTemoaProject:unstablefrom
idelder:dsd_periods

Conversation

@idelder
Copy link
Copy Markdown
Collaborator

@idelder idelder commented Mar 26, 2026

Summary by CodeRabbit

  • New Features

    • Demand-specific distributions now include a period so distributions can vary by model period; loaders expose new viable-set keys.
  • Database Schema

    • demand_specific_distribution table gained a period column and added it to the primary key.
  • Updates

    • Model indexing, validation and annual/period calculations updated to use the new period dimension across constraints and retirement logic.
  • Data & Load

    • Seed and test fixtures updated with period values; manifest and loaders adjusted for period-aware imports.
  • Bug Fixes

    • Fixed period indexing in annual flow and reserve calculations for consistent results.

@coderabbitai
Copy link
Copy Markdown

coderabbitai bot commented Mar 26, 2026

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: c25fd6f5-37e4-4c1e-91f8-0139b948934c

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Walkthrough

Adds a period/time_optimize dimension to demand_specific_distribution and updates model Param indexing, all consumer lookups, data loaders, DB schema, and SQL seed/test fixtures to key distributions by (region, period, season, time_of_day, demand).

Changes

Cohort / File(s) Summary
Model Core
temoa/core/model.py
TemoaModel.demand_specific_distribution Param index now includes self.time_optimize (period) immediately after self.regions.
Constraint & Flow Consumers
temoa/_internal/table_data_puller.py, temoa/components/capacity.py, temoa/components/reserves.py, temoa/components/emissions.py
All lookups of model.demand_specific_distribution updated to include period p (indexing shifted from [r, s, ...][r, p, s, ...]), changing distribution-driven calculations in flow polling, capacity/retirement, reserve margin, and linked-emissions constraints.
Commodities & Demand Logic
temoa/components/commodities.py
Demand-distribution creation, validation, autofill, and grouping adjusted to use (r, p, s, d, dem) keys; verification grouping changed to (r, p, dem) and iteration now includes model.time_optimize.
Data I/O & Load Manifest
temoa/data_io/component_manifest.py, temoa/data_io/hybrid_loader.py
demand_specific_distribution LoadItem now expects a period column and is_period_filtered=True; HybridLoader gains/uses new viable-set containers viable_rtv_new and viable_rtv_eol.
Model Checking
temoa/model_checking/commodity_network_manager.py
Added rtv_new viable-set filter populated with (region, tech, vintage) when vintage equals the period p (int-cast match) for period-aware validation.
DB Schema & SQL Seeds
temoa/db_schema/temoa_schema_v4.sql, temoa/tutorial_assets/utopia.sql, tests/testing_data/*, tests/testing_data/utopia_data.sql
demand_specific_distribution table schema and seed files updated to include period INTEGER (FK to time_period) and make period part of the primary key; SQL fixtures updated to insert explicit period values across seeds and tests.
Test Fixtures
tests/testing_data/test_system.sql, tests/testing_data/seasonal_storage.sql, tests/testing_data/mediumville.sql, tests/testing_data/storageville.sql
Test data rows expanded/shifted to include explicit period column values (e.g., 1990/2000/2010/2020/2025/2030) for each distribution row; minor NULL/'' adjustments noted.
Other
temoa/data_io/hybrid_loader.py, temoa/data_io/component_manifest.py
Switched several LoadItem validators to viable_rtv_new and commented out some custom_loader_name entries; manifest column expectations updated (e.g., limit_seasonal_capacity_factor uses tech_or_group).

Sequence Diagram(s)

(omitted)

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

Possibly related PRs

Suggested labels

bugfix

🚥 Pre-merge checks | ✅ 3
✅ Passed checks (3 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title 'Restore period index to DSD' directly describes the main change: adding period indexing to demand_specific_distribution across the codebase.
Docstring Coverage ✅ Passed Docstring coverage is 81.82% which is sufficient. The required threshold is 80.00%.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

Copy link
Copy Markdown

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
temoa/components/commodities.py (1)

784-806: ⚠️ Potential issue | 🟠 Major

Backfill defaults per (region, period, demand), not just demand.

demands_specified is still collapsed to dem, so one custom DSD for RL in a single period marks RL as “specified” everywhere. With the restored period index, omitted (r, p, RL) combinations are never defaulted from segment_fraction, and they fail the later per-period sum check instead of inheriting the default profile.

Suggested fix
-    demands_specified = set(
-        map(
-            demand_specific_distributon_dem,
-            (i for i in demand_specific_distribution.sparse_keys()),
-        )
-    )
-    unset_demand_distributions = used_dems.difference(
-        demands_specified
-    )  # the demands not mentioned in DSD *at all*
-
-    if unset_demand_distributions:
-        unset_distributions = set(
-            cross_product(
-                model.regions,
-                model.time_optimize,
-                model.time_season,
-                model.time_of_day,
-                unset_demand_distributions,
-            )
-        )
-        for r, p, s, d, dem in unset_distributions:
-            demand_specific_distribution[r, p, s, d, dem] = value(
-                model.segment_fraction[s, d]
-            )  # DSD._constructed = True
+    specified_rp_dems = {
+        (r, p, dem)
+        for r, p, _s, _d, dem in demand_specific_distribution.sparse_keys()
+    }
+    unset_rp_dems = {
+        (r, p, dem)
+        for r, p, dem in model.demand.sparse_keys()
+        if (r, p, dem) not in specified_rp_dems
+    }
+
+    for r, p, dem in unset_rp_dems:
+        for s, d in cross_product(model.time_season, model.time_of_day):
+            demand_specific_distribution[r, p, s, d, dem] = value(
+                model.segment_fraction[s, d]
+            )  # DSD._constructed = True
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@temoa/components/commodities.py` around lines 784 - 806, demands_specified
currently collapses keys to just dem, causing a single custom DSD to mark that
demand as specified everywhere; change the computation of demands_specified to
capture the full (region, period, demand) tuple from
demand_specific_distribution.sparse_keys() (e.g., map to (r, p, dem) instead of
dem), ensure used_dems is the same shape (set of (r,p,dem)), compute
unset_demand_distributions as the difference of those tuples, and then build
unset_distributions by crossing model.regions, model.time_optimize,
model.time_season, model.time_of_day with those (r,p,dem) tuples so the loop
assigns demand_specific_distribution[r, p, s, d, dem] =
value(model.segment_fraction[s, d]) for the correct per-region/per-period
combinations.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@tests/testing_data/test_system.sql`:
- Around line 254-285: The fixture repeats the 2020 RH splits for future years
so it doesn't catch period-specific bugs; change at least one future-year row in
demand_specific_distribution (e.g., any of the ('R1',2030,...,'RH',...) or
('R2',2025,...,'RH',...) entries) to a different RH value (for example adjust a
winter night/day RH from 0.4/0.3 to 0.45/0.25 or change a spring/night 0.1 to
0.15) so the 2025/2030 profile differs from 2020 and the restored period index
is exercised.

---

Outside diff comments:
In `@temoa/components/commodities.py`:
- Around line 784-806: demands_specified currently collapses keys to just dem,
causing a single custom DSD to mark that demand as specified everywhere; change
the computation of demands_specified to capture the full (region, period,
demand) tuple from demand_specific_distribution.sparse_keys() (e.g., map to (r,
p, dem) instead of dem), ensure used_dems is the same shape (set of (r,p,dem)),
compute unset_demand_distributions as the difference of those tuples, and then
build unset_distributions by crossing model.regions, model.time_optimize,
model.time_season, model.time_of_day with those (r,p,dem) tuples so the loop
assigns demand_specific_distribution[r, p, s, d, dem] =
value(model.segment_fraction[s, d]) for the correct per-region/per-period
combinations.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: 21bfcac4-c799-4877-979a-6c9dddd65eb4

📥 Commits

Reviewing files that changed from the base of the PR and between cb703e5 and 1b81750.

📒 Files selected for processing (14)
  • temoa/_internal/table_data_puller.py
  • temoa/components/capacity.py
  • temoa/components/commodities.py
  • temoa/components/emissions.py
  • temoa/components/reserves.py
  • temoa/core/model.py
  • temoa/data_io/component_manifest.py
  • temoa/db_schema/temoa_schema_v4.sql
  • temoa/tutorial_assets/utopia.sql
  • tests/testing_data/mediumville.sql
  • tests/testing_data/seasonal_storage.sql
  • tests/testing_data/storageville.sql
  • tests/testing_data/test_system.sql
  • tests/testing_data/utopia_data.sql

Comment on lines +254 to +285
REPLACE INTO "demand_specific_distribution" VALUES('R1',2025,'spring','day','RH',0.05,'');
REPLACE INTO "demand_specific_distribution" VALUES('R1',2025,'spring','night','RH',0.1,'');
REPLACE INTO "demand_specific_distribution" VALUES('R1',2025,'summer','day','RH',0.0,'');
REPLACE INTO "demand_specific_distribution" VALUES('R1',2025,'summer','night','RH',0.0,'');
REPLACE INTO "demand_specific_distribution" VALUES('R1',2025,'fall','day','RH',0.05,'');
REPLACE INTO "demand_specific_distribution" VALUES('R1',2025,'fall','night','RH',0.1,'');
REPLACE INTO "demand_specific_distribution" VALUES('R1',2025,'winter','day','RH',0.3,'');
REPLACE INTO "demand_specific_distribution" VALUES('R1',2025,'winter','night','RH',0.4,'');
REPLACE INTO "demand_specific_distribution" VALUES('R2',2025,'spring','day','RH',0.05,'');
REPLACE INTO "demand_specific_distribution" VALUES('R2',2025,'spring','night','RH',0.1,'');
REPLACE INTO "demand_specific_distribution" VALUES('R2',2025,'summer','day','RH',0.0,'');
REPLACE INTO "demand_specific_distribution" VALUES('R2',2025,'summer','night','RH',0.0,'');
REPLACE INTO "demand_specific_distribution" VALUES('R2',2025,'fall','day','RH',0.05,'');
REPLACE INTO "demand_specific_distribution" VALUES('R2',2025,'fall','night','RH',0.1,'');
REPLACE INTO "demand_specific_distribution" VALUES('R2',2025,'winter','day','RH',0.3,'');
REPLACE INTO "demand_specific_distribution" VALUES('R2',2025,'winter','night','RH',0.4,'');
REPLACE INTO "demand_specific_distribution" VALUES('R1',2030,'spring','day','RH',0.05,'');
REPLACE INTO "demand_specific_distribution" VALUES('R1',2030,'spring','night','RH',0.1,'');
REPLACE INTO "demand_specific_distribution" VALUES('R1',2030,'summer','day','RH',0.0,'');
REPLACE INTO "demand_specific_distribution" VALUES('R1',2030,'summer','night','RH',0.0,'');
REPLACE INTO "demand_specific_distribution" VALUES('R1',2030,'fall','day','RH',0.05,'');
REPLACE INTO "demand_specific_distribution" VALUES('R1',2030,'fall','night','RH',0.1,'');
REPLACE INTO "demand_specific_distribution" VALUES('R1',2030,'winter','day','RH',0.3,'');
REPLACE INTO "demand_specific_distribution" VALUES('R1',2030,'winter','night','RH',0.4,'');
REPLACE INTO "demand_specific_distribution" VALUES('R2',2030,'spring','day','RH',0.05,'');
REPLACE INTO "demand_specific_distribution" VALUES('R2',2030,'spring','night','RH',0.1,'');
REPLACE INTO "demand_specific_distribution" VALUES('R2',2030,'summer','day','RH',0.0,'');
REPLACE INTO "demand_specific_distribution" VALUES('R2',2030,'summer','night','RH',0.0,'');
REPLACE INTO "demand_specific_distribution" VALUES('R2',2030,'fall','day','RH',0.05,'');
REPLACE INTO "demand_specific_distribution" VALUES('R2',2030,'fall','night','RH',0.1,'');
REPLACE INTO "demand_specific_distribution" VALUES('R2',2030,'winter','day','RH',0.3,'');
REPLACE INTO "demand_specific_distribution" VALUES('R2',2030,'winter','night','RH',0.4,'');
Copy link
Copy Markdown

@coderabbitai coderabbitai bot Mar 26, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick | 🔵 Trivial

Use a period-specific RH split in at least one future year.

The 2025/2030 rows repeat the 2020 profile exactly, so this fixture still passes if a consumer accidentally ignores p and treats demand_specific_distribution as period-invariant. Making one future period intentionally different here, or in a small dedicated regression fixture, would actually exercise the restored index.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@tests/testing_data/test_system.sql` around lines 254 - 285, The fixture
repeats the 2020 RH splits for future years so it doesn't catch period-specific
bugs; change at least one future-year row in demand_specific_distribution (e.g.,
any of the ('R1',2030,...,'RH',...) or ('R2',2025,...,'RH',...) entries) to a
different RH value (for example adjust a winter night/day RH from 0.4/0.3 to
0.45/0.25 or change a spring/night 0.1 to 0.15) so the 2025/2030 profile differs
from 2020 and the restored period index is exercised.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@idelder, here's a minimal change that keeps the sums valid while making the 2030 profile distinct from 2020/2025 (shifting winter demand slightly later into the night):

-REPLACE INTO "demand_specific_distribution" VALUES('R1',2030,'spring','day','RH',0.05,'');
-REPLACE INTO "demand_specific_distribution" VALUES('R1',2030,'spring','night','RH',0.1,'');
-REPLACE INTO "demand_specific_distribution" VALUES('R1',2030,'summer','day','RH',0.0,'');
-REPLACE INTO "demand_specific_distribution" VALUES('R1',2030,'summer','night','RH',0.0,'');
-REPLACE INTO "demand_specific_distribution" VALUES('R1',2030,'fall','day','RH',0.05,'');
-REPLACE INTO "demand_specific_distribution" VALUES('R1',2030,'fall','night','RH',0.1,'');
-REPLACE INTO "demand_specific_distribution" VALUES('R1',2030,'winter','day','RH',0.3,'');
-REPLACE INTO "demand_specific_distribution" VALUES('R1',2030,'winter','night','RH',0.4,'');
-REPLACE INTO "demand_specific_distribution" VALUES('R2',2030,'spring','day','RH',0.05,'');
-REPLACE INTO "demand_specific_distribution" VALUES('R2',2030,'spring','night','RH',0.1,'');
-REPLACE INTO "demand_specific_distribution" VALUES('R2',2030,'summer','day','RH',0.0,'');
-REPLACE INTO "demand_specific_distribution" VALUES('R2',2030,'summer','night','RH',0.0,'');
-REPLACE INTO "demand_specific_distribution" VALUES('R2',2030,'fall','day','RH',0.05,'');
-REPLACE INTO "demand_specific_distribution" VALUES('R2',2030,'fall','night','RH',0.1,'');
-REPLACE INTO "demand_specific_distribution" VALUES('R2',2030,'winter','day','RH',0.3,'');
-REPLACE INTO "demand_specific_distribution" VALUES('R2',2030,'winter','night','RH',0.4,'');
+REPLACE INTO "demand_specific_distribution" VALUES('R1',2030,'spring','day','RH',0.05,'');
+REPLACE INTO "demand_specific_distribution" VALUES('R1',2030,'spring','night','RH',0.1,'');
+REPLACE INTO "demand_specific_distribution" VALUES('R1',2030,'summer','day','RH',0.0,'');
+REPLACE INTO "demand_specific_distribution" VALUES('R1',2030,'summer','night','RH',0.0,'');
+REPLACE INTO "demand_specific_distribution" VALUES('R1',2030,'fall','day','RH',0.05,'');
+REPLACE INTO "demand_specific_distribution" VALUES('R1',2030,'fall','night','RH',0.1,'');
+REPLACE INTO "demand_specific_distribution" VALUES('R1',2030,'winter','day','RH',0.25,'');
+REPLACE INTO "demand_specific_distribution" VALUES('R1',2030,'winter','night','RH',0.45,'');
+REPLACE INTO "demand_specific_distribution" VALUES('R2',2030,'spring','day','RH',0.05,'');
+REPLACE INTO "demand_specific_distribution" VALUES('R2',2030,'spring','night','RH',0.1,'');
+REPLACE INTO "demand_specific_distribution" VALUES('R2',2030,'summer','day','RH',0.0,'');
+REPLACE INTO "demand_specific_distribution" VALUES('R2',2030,'summer','night','RH',0.0,'');
+REPLACE INTO "demand_specific_distribution" VALUES('R2',2030,'fall','day','RH',0.05,'');
+REPLACE INTO "demand_specific_distribution" VALUES('R2',2030,'fall','night','RH',0.1,'');
+REPLACE INTO "demand_specific_distribution" VALUES('R2',2030,'winter','day','RH',0.25,'');
+REPLACE INTO "demand_specific_distribution" VALUES('R2',2030,'winter','night','RH',0.45,'');

The sum across all slices remains 1.0 (0.05 + 0.10 + 0.00 + 0.00 + 0.05 + 0.10 + 0.25 + 0.45 = 1.0). With 2020/2025 using winter/day=0.3 and 2030 using 0.25, any code path that ignores p and uses a stale distribution will produce a wrong result that a test can catch.


🧠 Learnings used
Learnt from: idelder
Repo: TemoaProject/temoa PR: 277
File: temoa/model_checking/network_model_data.py:357-361
Timestamp: 2026-03-16T18:27:06.688Z
Learning: In TemoaProject/temoa, `NetworkModelData.demand_commodities` in `temoa/model_checking/network_model_data.py` is used only as a starting-node set for the commodity network sweep, not as a validation or filter set. Having extra period keys outside the myopic window is harmless because `available_techs` is already strictly filtered to the active myopic periods; unreachable demand entries are simply ignored during the sweep. Do not flag unfiltered demand_commodities in myopic mode as a correctness issue.

Learnt from: idelder
Repo: TemoaProject/temoa PR: 264
File: temoa/tutorial_assets/utopia.sql:1441-1449
Timestamp: 2026-03-09T13:28:25.754Z
Learning: In the Temoa project (TemoaProject/temoa), the auto-population of the `time_season_sequential` table when it is empty is intentional behavior, not a runtime fallback. For `seasonal_timeslice`/`consecutive_days` models, the sequential seasons are designed to automatically match the regular seasons defined in `time_season`. Therefore, SQL assets for such models (e.g., `temoa/tutorial_assets/utopia.sql`) intentionally omit explicit INSERT statements for `time_season_sequential`.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nah

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@idelder, understood — no changes needed here. I'll drop the suggestion.


🧠 Learnings used
Learnt from: idelder
Repo: TemoaProject/temoa PR: 277
File: temoa/model_checking/network_model_data.py:357-361
Timestamp: 2026-03-16T18:27:06.688Z
Learning: In TemoaProject/temoa, `NetworkModelData.demand_commodities` in `temoa/model_checking/network_model_data.py` is used only as a starting-node set for the commodity network sweep, not as a validation or filter set. Having extra period keys outside the myopic window is harmless because `available_techs` is already strictly filtered to the active myopic periods; unreachable demand entries are simply ignored during the sweep. Do not flag unfiltered demand_commodities in myopic mode as a correctness issue.

Learnt from: idelder
Repo: TemoaProject/temoa PR: 264
File: temoa/tutorial_assets/utopia.sql:1441-1449
Timestamp: 2026-03-09T13:28:25.754Z
Learning: In the Temoa project (TemoaProject/temoa), the auto-population of the `time_season_sequential` table when it is empty is intentional behavior, not a runtime fallback. For `seasonal_timeslice`/`consecutive_days` models, the sequential seasons are designed to automatically match the regular seasons defined in `time_season`. Therefore, SQL assets for such models (e.g., `temoa/tutorial_assets/utopia.sql`) intentionally omit explicit INSERT statements for `time_season_sequential`.

Learnt from: idelder
Repo: TemoaProject/temoa PR: 264
File: temoa/tutorial_assets/utopia.sql:898-912
Timestamp: 2026-03-09T13:29:38.792Z
Learning: In the Temoa project (TemoaProject/temoa), the `season` column on `limit_storage_level_fraction` in SQL assets (e.g., temoa/tutorial_assets/utopia.sql and data_files/example_dbs/*.sql) intentionally has no FK to `time_season`. Season validity for this table is enforced at the code/loader level rather than at the database schema level. Restoring a season_label FK table or splitting the table into seasonal/non-seasonal variants are considered worse alternatives. Do not flag the missing FK on this column as an issue.

Copy link
Copy Markdown

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
temoa/components/commodities.py (1)

779-857: ⚠️ Potential issue | 🟠 Major

Track specified DSDs by (region, period, demand).

demands_specified still collapses sparse keys to dem only. With the restored period index, one explicit DSD for a demand suppresses default population for every other (region, period) of that demand; when validation reaches one of those missing tuples, it can also fall through to max() on an empty keys list.

Proposed fix
-    demands_specified = set(
-        map(
-            demand_specific_distributon_dem,
-            (i for i in demand_specific_distribution.sparse_keys()),
-        )
-    )
-    unset_demand_distributions = used_dems.difference(
-        demands_specified
-    )  # the demands not mentioned in DSD *at all*
+    used_rp_dems = {(r, p, dem) for r, p, dem in model.demand.sparse_keys()}
+    specified_rp_dems = {
+        (r, p, dem)
+        for r, p, _s, _d, dem in demand_specific_distribution.sparse_keys()
+    }
+    unset_rp_dems = used_rp_dems.difference(
+        specified_rp_dems
+    )  # the (region, period, demand) tuples not mentioned in DSD at all
 
-    if unset_demand_distributions:
-        unset_distributions = set(
-            cross_product(
-                model.regions,
-                model.time_optimize,
-                model.time_season,
-                model.time_of_day,
-                unset_demand_distributions,
-            )
-        )
+    if unset_rp_dems:
+        unset_distributions = (
+            (r, p, s, d, dem)
+            for r, p, dem in unset_rp_dems
+            for s, d in cross_product(model.time_season, model.time_of_day)
+        )
         for r, p, s, d, dem in unset_distributions:
             demand_specific_distribution[r, p, s, d, dem] = value(
                 model.segment_fraction[s, d]
             )  # DSD._constructed = True
 
-    used_rp_dems = {(r, p, dem) for r, p, dem in model.demand.sparse_keys()}
     for r, p, dem in used_rp_dems:
         expected_key_length = len(model.time_season) * len(model.time_of_day)
         keys = [
             k
             for k in demand_specific_distribution.sparse_keys()
             if demand_specific_distribution_region(k) == r
             and demand_specific_distributon_period(k) == p
             and demand_specific_distributon_dem(k) == dem
         ]
+        if not keys:
+            raise ValueError(
+                f'No demand_specific_distribution entries found for {(r, p, dem)}'
+            )
         if len(keys) != expected_key_length:
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@temoa/components/commodities.py` around lines 779 - 857, The bug is that
demands_specified only records dem (collapsing the period), so one explicit DSD
suppresses defaults across all periods and later validation can get empty keys;
update the tracking to use the full (region, period, dem) tuple. Specifically,
change the construction of demands_specified to map each sparse key via
demand_specific_distribution_region, demand_specific_distributon_period and
demand_specific_distributon_dem to produce (r, p, dem) tuples (instead of just
dem), then compute unset_demand_distributions relative to used_rp_dems (or
rename used_dems to used_rp_dems) and use that set in the
cross_product/population loop that assigns demand_specific_distribution[r, p, s,
d, dem]. Also ensure the final validation that builds keys filters by region,
period and dem consistently using those same helper functions so keys is never
empty for a used (r,p,dem).
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@tests/testing_data/utopia_data.sql`:
- Around line 252-281: The test fixture only inserts fully explicit
period-indexed rows into demand_specific_distribution so it doesn't exercise the
mixed explicit/default code path; add a row (for example for a chosen scenario
like city 'utopia' and year 2010) where one season/time combination has an
explicit entry in demand_specific_distribution while another combination for the
same city/year/time/season is omitted so that the code falls back to
segment_fraction; update the testing data block near the existing REPLACE INTO
"demand_specific_distribution" entries to include this mixed case to ensure the
path that merges explicit period entries with segment_fraction fallback is
exercised (refer to the table demand_specific_distribution and the
segment_fraction fallback behavior).

---

Outside diff comments:
In `@temoa/components/commodities.py`:
- Around line 779-857: The bug is that demands_specified only records dem
(collapsing the period), so one explicit DSD suppresses defaults across all
periods and later validation can get empty keys; update the tracking to use the
full (region, period, dem) tuple. Specifically, change the construction of
demands_specified to map each sparse key via
demand_specific_distribution_region, demand_specific_distributon_period and
demand_specific_distributon_dem to produce (r, p, dem) tuples (instead of just
dem), then compute unset_demand_distributions relative to used_rp_dems (or
rename used_dems to used_rp_dems) and use that set in the
cross_product/population loop that assigns demand_specific_distribution[r, p, s,
d, dem]. Also ensure the final validation that builds keys filters by region,
period and dem consistently using those same helper functions so keys is never
empty for a used (r,p,dem).
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: f58b799c-b3fe-4526-bad2-a37e78215596

📥 Commits

Reviewing files that changed from the base of the PR and between 1b81750 and 8d08479.

📒 Files selected for processing (14)
  • temoa/_internal/table_data_puller.py
  • temoa/components/capacity.py
  • temoa/components/commodities.py
  • temoa/components/emissions.py
  • temoa/components/reserves.py
  • temoa/core/model.py
  • temoa/data_io/component_manifest.py
  • temoa/db_schema/temoa_schema_v4.sql
  • temoa/tutorial_assets/utopia.sql
  • tests/testing_data/mediumville.sql
  • tests/testing_data/seasonal_storage.sql
  • tests/testing_data/storageville.sql
  • tests/testing_data/test_system.sql
  • tests/testing_data/utopia_data.sql

Comment on lines +252 to +281
REPLACE INTO "demand_specific_distribution" VALUES('utopia',1990,'inter','day','RH',0.12,'');
REPLACE INTO "demand_specific_distribution" VALUES('utopia',1990,'inter','night','RH',0.06,'');
REPLACE INTO "demand_specific_distribution" VALUES('utopia',1990,'winter','day','RH',0.5467,'');
REPLACE INTO "demand_specific_distribution" VALUES('utopia',1990,'winter','night','RH',0.2733,'');
REPLACE INTO "demand_specific_distribution" VALUES('utopia',1990,'inter','day','RL',0.15,'');
REPLACE INTO "demand_specific_distribution" VALUES('utopia',1990,'inter','night','RL',0.05,'');
REPLACE INTO "demand_specific_distribution" VALUES('utopia',1990,'summer','day','RL',0.15,'');
REPLACE INTO "demand_specific_distribution" VALUES('utopia',1990,'summer','night','RL',0.05,'');
REPLACE INTO "demand_specific_distribution" VALUES('utopia',1990,'winter','day','RL',0.5,'');
REPLACE INTO "demand_specific_distribution" VALUES('utopia',1990,'winter','night','RL',0.1,'');
REPLACE INTO "demand_specific_distribution" VALUES('utopia',2000,'inter','day','RH',0.12,'');
REPLACE INTO "demand_specific_distribution" VALUES('utopia',2000,'inter','night','RH',0.06,'');
REPLACE INTO "demand_specific_distribution" VALUES('utopia',2000,'winter','day','RH',0.5467,'');
REPLACE INTO "demand_specific_distribution" VALUES('utopia',2000,'winter','night','RH',0.2733,'');
REPLACE INTO "demand_specific_distribution" VALUES('utopia',2000,'inter','day','RL',0.15,'');
REPLACE INTO "demand_specific_distribution" VALUES('utopia',2000,'inter','night','RL',0.05,'');
REPLACE INTO "demand_specific_distribution" VALUES('utopia',2000,'summer','day','RL',0.15,'');
REPLACE INTO "demand_specific_distribution" VALUES('utopia',2000,'summer','night','RL',0.05,'');
REPLACE INTO "demand_specific_distribution" VALUES('utopia',2000,'winter','day','RL',0.5,'');
REPLACE INTO "demand_specific_distribution" VALUES('utopia',2000,'winter','night','RL',0.1,'');
REPLACE INTO "demand_specific_distribution" VALUES('utopia',2010,'inter','day','RH',0.12,'');
REPLACE INTO "demand_specific_distribution" VALUES('utopia',2010,'inter','night','RH',0.06,'');
REPLACE INTO "demand_specific_distribution" VALUES('utopia',2010,'winter','day','RH',0.5467,'');
REPLACE INTO "demand_specific_distribution" VALUES('utopia',2010,'winter','night','RH',0.2733,'');
REPLACE INTO "demand_specific_distribution" VALUES('utopia',2010,'inter','day','RL',0.15,'');
REPLACE INTO "demand_specific_distribution" VALUES('utopia',2010,'inter','night','RL',0.05,'');
REPLACE INTO "demand_specific_distribution" VALUES('utopia',2010,'summer','day','RL',0.15,'');
REPLACE INTO "demand_specific_distribution" VALUES('utopia',2010,'summer','night','RL',0.05,'');
REPLACE INTO "demand_specific_distribution" VALUES('utopia',2010,'winter','day','RL',0.5,'');
REPLACE INTO "demand_specific_distribution" VALUES('utopia',2010,'winter','night','RL',0.1,'');
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick | 🔵 Trivial

Add a mixed explicit/default period fixture.

This block only exercises the fully explicit period-indexed DSD path. A case where one period is overridden and another period falls back to segment_fraction would cover the restored key shape more directly.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@tests/testing_data/utopia_data.sql` around lines 252 - 281, The test fixture
only inserts fully explicit period-indexed rows into
demand_specific_distribution so it doesn't exercise the mixed explicit/default
code path; add a row (for example for a chosen scenario like city 'utopia' and
year 2010) where one season/time combination has an explicit entry in
demand_specific_distribution while another combination for the same
city/year/time/season is omitted so that the code falls back to
segment_fraction; update the testing data block near the existing REPLACE INTO
"demand_specific_distribution" entries to include this mixed case to ensure the
path that merges explicit period entries with segment_fraction fallback is
exercised (refer to the table demand_specific_distribution and the
segment_fraction fallback behavior).

SutubraResearch added a commit to SutubraResearch/temoa that referenced this pull request Mar 31, 2026
Copy link
Copy Markdown

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@temoa/components/capacity.py`:
- Around line 317-327: The subtraction of model.v_retired_capacity[r, p, t, v]
in the fallback branch is using period-p surviving-capacity units while cap_end
has already been projected to p_next via the survival ratio; scale the
retirement term by the same survival ratio used above
(value(model.lifetime_survival_curve[r, p_next, t, v]) /
value(model.lifetime_survival_curve[r, p, t, v])) before subtracting so the
units match when updating cap_end in the branch that handles p ==
model.time_optimize.last() or p_next == v + value(model.lifetime_process[r, t,
v]).
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: cf7cf6b7-012f-42c2-bf24-26af219239b8

📥 Commits

Reviewing files that changed from the base of the PR and between 8d08479 and a0985cf.

📒 Files selected for processing (4)
  • temoa/components/capacity.py
  • temoa/data_io/component_manifest.py
  • temoa/data_io/hybrid_loader.py
  • temoa/model_checking/commodity_network_manager.py

Copy link
Copy Markdown

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
temoa/data_io/component_manifest.py (1)

321-330: ⚠️ Potential issue | 🔴 Critical

Myopic filtering is not applied when custom loader is disabled.

The custom loader _load_loan_rate handles myopic period filtering (vintage >= base_year) but is commented out. The viable_rtv_new validator filters by viability set, not by myopic period. With the custom loader disabled, is_period_filtered=False is incorrect—myopic filtering is not happening. This affects both cost_invest (same pattern) and loan_rate. Either uncomment the custom loader or handle myopic period filtering in the standard loading path.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@temoa/data_io/component_manifest.py` around lines 321 - 330, The LoadItem for
model.loan_rate (and the similar cost_invest entries) relies on the custom
loader _load_loan_rate to enforce myopic period filtering (vintage >= base_year)
but the custom_loader_name is commented out while is_period_filtered=False, so
myopic filtering is skipped; fix by either uncommenting and restoring
custom_loader_name='_load_loan_rate' for the LoadItem(s) or, if you want to use
the standard loader, set is_period_filtered=True and update the standard
loading/validation path (or the validator used, e.g., viable_rtv_new) to
explicitly apply vintage >= base_year filtering for those tables so myopic
periods are excluded.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@temoa/components/capacity.py`:
- Around line 318-334: The nested conditional under the else block can be
flattened: replace the "else:" followed by "if p == model.time_optimize.last()
or v + value(model.lifetime_process[r, t, v]) == p_end:" with a single "elif" so
the check is directly attached to the prior "if" and remove the extra
indentation; leave the cap_end assignments for both branches unchanged (the
branch that uses value(model.lifetime_survival_curve[r, p_end, t, v]) /
value(model.lifetime_survival_curve[r, p, t, v]) remains for the p==... or
v+...==p_end case, and the branch that uses model.v_capacity[r, p_end, t, v] *
value(model.lifetime_survival_curve[r, p_end, t, v]) /
value(model.process_life_frac[r, p_end, t, v]) remains for the other case).

---

Outside diff comments:
In `@temoa/data_io/component_manifest.py`:
- Around line 321-330: The LoadItem for model.loan_rate (and the similar
cost_invest entries) relies on the custom loader _load_loan_rate to enforce
myopic period filtering (vintage >= base_year) but the custom_loader_name is
commented out while is_period_filtered=False, so myopic filtering is skipped;
fix by either uncommenting and restoring custom_loader_name='_load_loan_rate'
for the LoadItem(s) or, if you want to use the standard loader, set
is_period_filtered=True and update the standard loading/validation path (or the
validator used, e.g., viable_rtv_new) to explicitly apply vintage >= base_year
filtering for those tables so myopic periods are excluded.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: 0deb05a9-65f2-4c9a-a399-4b813acef9db

📥 Commits

Reviewing files that changed from the base of the PR and between a0985cf and 48bff9d.

📒 Files selected for processing (2)
  • temoa/components/capacity.py
  • temoa/data_io/component_manifest.py

Comment on lines 318 to 334
else:
# Mid-life period, ending capacity is beginning capacity of next period
p_next = model.time_future.next(p)

if p == model.time_optimize.last() or p_next == v + value(model.lifetime_process[r, t, v]):
if p == model.time_optimize.last() or v + value(model.lifetime_process[r, t, v]) == p_end:
# No v_capacity or v_retired_capacity for next period so just continue down the
# survival curve
# survival curve. If v+l=p then eol would be dumped in the next period
cap_end = (
cap_begin
* value(model.lifetime_survival_curve[r, p_next, t, v])
* value(model.lifetime_survival_curve[r, p_end, t, v])
/ value(model.lifetime_survival_curve[r, p, t, v])
)
else:
# Get the next period's beginning capacity
cap_end = (
model.v_capacity[r, p_next, t, v]
* value(model.lifetime_survival_curve[r, p_next, t, v])
/ value(model.process_life_frac[r, p_next, t, v])
model.v_capacity[r, p_end, t, v]
* value(model.lifetime_survival_curve[r, p_end, t, v])
/ value(model.process_life_frac[r, p_end, t, v])
)
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick | 🔵 Trivial

Consider using elif for cleaner structure.

The static analysis correctly identifies that the nested if inside else could be simplified to elif for better readability.

♻️ Proposed refactor
-    if p <= v + value(model.lifetime_process[r, t, v]) < p_end:
+    if p <= v + value(model.lifetime_process[r, t, v]) < p_end:
         # EOL so capacity ends on zero
         cap_end = 0
-    else:
-        # Mid-life period, ending capacity is beginning capacity of next period
-        if p == model.time_optimize.last() or v + value(model.lifetime_process[r, t, v]) == p_end:
+    elif p == model.time_optimize.last() or v + value(model.lifetime_process[r, t, v]) == p_end:
+        # Mid-life period, ending capacity is beginning capacity of next period
+        # No v_capacity or v_retired_capacity for next period so just continue down the
+        # survival curve. If v+l=p_end then eol would be dumped in the next period
+        cap_end = (
+            cap_begin
+            * value(model.lifetime_survival_curve[r, p_end, t, v])
+            / value(model.lifetime_survival_curve[r, p, t, v])
+        )
+    else:
+        # Get the next period's beginning capacity
+        cap_end = (
+            model.v_capacity[r, p_end, t, v]
+            * value(model.lifetime_survival_curve[r, p_end, t, v])
+            / value(model.process_life_frac[r, p_end, t, v])
+        )
🧰 Tools
🪛 Ruff (0.15.7)

[warning] 318-320: Use elif instead of else then if, to reduce indentation

Convert to elif

(PLR5501)

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@temoa/components/capacity.py` around lines 318 - 334, The nested conditional
under the else block can be flattened: replace the "else:" followed by "if p ==
model.time_optimize.last() or v + value(model.lifetime_process[r, t, v]) ==
p_end:" with a single "elif" so the check is directly attached to the prior "if"
and remove the extra indentation; leave the cap_end assignments for both
branches unchanged (the branch that uses value(model.lifetime_survival_curve[r,
p_end, t, v]) / value(model.lifetime_survival_curve[r, p, t, v]) remains for the
p==... or v+...==p_end case, and the branch that uses model.v_capacity[r, p_end,
t, v] * value(model.lifetime_survival_curve[r, p_end, t, v]) /
value(model.process_life_frac[r, p_end, t, v]) remains for the other case).

@idelder idelder marked this pull request as draft April 2, 2026 21:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants