feat!: batched element models (stacked on the v8 API removals)#744
Closed
FBumann wants to merge 400 commits into
Closed
feat!: batched element models (stacked on the v8 API removals)#744FBumann wants to merge 400 commits into
FBumann wants to merge 400 commits into
Conversation
FlowsData (batched.py):
1. Added categorizations: with_flow_hours, with_load_factor
2. Renamed: size_minimum → effective_size_lower, size_maximum → effective_size_upper
3. Properties now only include relevant flows (no NaN padding):
- flow_hours_minimum/maximum → only with_flow_hours
- flow_hours_minimum/maximum_over_periods → only with_flow_hours_over_periods
- load_factor_minimum/maximum → only with_load_factor
4. Added absolute_lower_bounds, absolute_upper_bounds for all flows
5. Added _stack_values_for_subset() helper
FlowsModel (elements.py):
1. Removed hours and hours_over_periods variables - not needed
2. Simplified constraints to compute inline:
- constraint_flow_hours() - directly constrains sum_temporal(rate)
- constraint_flow_hours_over_periods() - directly constrains weighted sum
- constraint_load_factor_min/max() - compute hours inline
3. rate variable uses self.data.absolute_lower_bounds/upper_bounds directly
4. Removed obsolete bound collection methods
Benefits:
- Cleaner separation: data in FlowsData, constraints in FlowsModel
- No NaN handling needed - properties only include relevant flows
- Fewer variables in the model
- More explicit about which flows have which constraints
1. Added Status Data Properties to FlowsData (batched.py) Added new cached properties for status-related bounds: - min_uptime, max_uptime - uptime bounds for flows with uptime tracking - min_downtime, max_downtime - downtime bounds for flows with downtime tracking - startup_limit_values - startup limits for flows with startup limit - previous_uptime, previous_downtime - computed previous durations using StatusHelpers.compute_previous_duration() 2. Simplified FlowsModel Variable Creation (elements.py) Refactored uptime, downtime, and startup_count methods to use the new FlowsData properties instead of inline computation: - uptime: Now uses self.data.min_uptime, self.data.max_uptime, self.data.previous_uptime - downtime: Now uses self.data.min_downtime, self.data.max_downtime, self.data.previous_downtime - startup_count: Now uses self.data.startup_limit_values 3. Kept active_hours (Plan Adjustment) The original plan called for removing active_hours, but functional tests (test_on_total_max, test_on_total_bounds) demonstrated that active_hours is required to enforce active_hours_min and active_hours_max parameters. Without it, the optimizer would ignore those constraints. Verification All tests pass: - pytest tests/test_functional.py -v - 26 tests passed - pytest tests/test_flow.py -v -k "time_only" - 22 tests passed - pytest tests/test_component.py -v -k "time_only" - 9 tests passed
1. Combined min/max pairs - Created _uptime_bounds and _downtime_bounds cached properties that compute both min and max in a single iteration - Individual properties (min_uptime, max_uptime, etc.) now delegate to these cached tuples 2. Added helper methods - _build_status_bounds(flow_ids, min_attr, max_attr) - builds both bounds in one pass - _build_previous_durations(flow_ids, target_state, min_attr) - consolidates previous duration logic 3. Used more efficient patterns - Pre-allocated numpy arrays (np.empty, np.full) instead of Python list appends - Cached dict lookups - params = self.status_params at loop start instead of repeated self.status_params[fid] - Reduced redundant iterations - accessing min/max uptime now only iterates once instead of twice
- status_effects_per_startup → effects_per_startup
1. compute_previous_duration - Simple helper for computing previous duration (used by FlowsData)
2. add_batched_duration_tracking - Creates duration tracking constraints (used by FlowsModel)
3. create_status_features - Used by ComponentsModel (separate code path, not part of FlowsModel refactoring)
Removed:
- collect_status_effects - replaced with simpler _build_status_effects helper directly in FlowsData
The effect building is now consistent - effects_per_active_hour and effects_per_startup use the same pattern as effects_per_flow_hour:
# Simple, direct approach - no intermediate dict
def _build_status_effects(self, attr: str) -> xr.DataArray | None:
flow_factors = [
xr.concat(
[xr.DataArray(getattr(params[fid], attr).get(eff, np.nan)) for eff in effect_ids],
dim='effect',
coords='minimal',
).assign_coords(effect=effect_ids)
for fid in flow_ids
]
return concat_with_coords(flow_factors, 'flow', flow_ids)
…red element names but didn't reset the model's _is_built flag, so get_status() would still report MODEL_BUILT. Added fs.model._is_built = False when fs.model is not None.
…taArray() (a scalar with no dims) when empty, breaking downstream .dims checks. Changed to xr.DataArray(dims=['case'], coords={'case': []}) so the 'case' dimension is always present.
- PiecewiseBuilder.create_piecewise_constraints — removed the zero_point parameter entirely. It now always creates sum(inside_piece) <= 1. - Callers (FlowsModel and StoragesModel) — add the tighter <= invested constraint separately, only for optional IDs that exist in invested_var. No coord mismatch possible. - ConvertersModel — was already passing None, just cleaned up the dead code.
…erseded/math/ directory. Here's a summary of the changes made:
Summary of Updated Tests
test_flow.py (88 tests)
- Updated variable names: flow|rate, flow|size, flow|invested, flow|status, flow|active_hours
- Updated constraint names: share|temporal(costs), share|periodic(costs) instead of 'ComponentName->effect(temporal)'
- Updated uptime/downtime constraints: flow|uptime|forward, flow|uptime|backward, flow|uptime|min instead of flow|uptime|fwd/bwd/lb
- Updated switch constraints: flow|switch_transition instead of flow|switch
- Removed non-existent flow|fixed constraint check (fixed profile uses flow|invest_lb/ub)
test_storage.py (48 tests)
- Updated variable names: storage|charge, storage|netto, storage|size, storage|invested
- Updated constraint names: storage|balance, storage|netto_eq, storage|initial_equals_final
- Updated status variable from status|status to flow|status
- Updated prevent simultaneous constraint: storage|prevent_simultaneous
- Fixed effects_of_investment syntax to use dict: {'costs': 100}
test_component.py (40 tests)
- Updated status variables: component|status, flow|status instead of status|status
- Updated active_hours variables: component|active_hours
- Updated uptime variables: component|uptime
- Updated constraints: component|status|lb/ub/eq, component|uptime|initial
- Removed non-existent flow|total_flow_hours checks
test_linear_converter.py (36 tests)
- Updated constraint names: converter|conversion (no index suffix)
- Updated status variables: component|status, component|active_hours
- Updated share constraints: share|temporal(costs)
- Made piecewise tests more flexible with pattern matching
test_effect.py (26 tests)
- No changes needed - tests already working
1. Storage charge state scalar bounds (batched.py): Added .astype(float) after expand_dims().copy() to prevent silent int→float truncation when assigning final charge state overrides (0.5 was being truncated to 0 on an int64 array). 2. SourceAndSink deserialization (components.py): Convert inputs/outputs from dict to list before + concatenation in __init__, fixing TypeError: unsupported operand type(s) for +: 'dict' and 'dict' during NetCDF save/reload. 3. Legacy config leaking between test modules (test_math/conftest.py, superseded/math/conftest.py, test_legacy_solution_access.py): Converted module-level fx.CONFIG.Legacy.solution_access = True to autouse fixtures that restore the original value after each test, preventing the plotting isinstance(solution, xr.Dataset) test from failing.
* Here's a summary of everything that was done: Summary Phase 1: TransmissionsData - Added flow_ids parameter to TransmissionsData.__init__ - Moved 12 cached properties from TransmissionsModel to TransmissionsData: bidirectional_ids, balanced_ids, _build_flow_mask(), in1_mask, out1_mask, in2_mask, out2_mask, relative_losses, absolute_losses, has_absolute_losses_mask, transmissions_with_abs_losses - Updated TransmissionsModel.create_constraints() to use self.data.* - Updated BatchedAccessor.transmissions to pass flow_ids Phase 2: BusesData - Added balance_coefficients cached property to BusesData - Updated BusesModel.create_constraints() to use self.data.balance_coefficients Phase 3: ConvertersData - Added flow_ids and timesteps parameters to ConvertersData.__init__ - Moved 13 cached properties from ConvertersModel to ConvertersData: factor_element_ids, max_equations, equation_mask, signed_coefficients, n_equations_per_converter, piecewise_element_ids, piecewise_segment_counts_dict, piecewise_max_segments, piecewise_segment_mask, piecewise_flow_breakpoints, piecewise_segment_counts_array, piecewise_breakpoints - Updated ConvertersModel methods to use self.data.* - Removed unused defaultdict and stack_along_dim imports from elements.py Phase 4: ComponentsData - Added flows_data, effect_ids, timestep_duration parameters to ComponentsData.__init__ - Moved 6 cached properties from ComponentsModel to ComponentsData: with_prevent_simultaneous, status_params, previous_status_dict, status_data, flow_mask, flow_count - Moved _get_previous_status_for_component() helper to ComponentsData - Updated ComponentsModel to use self.data.* throughout Bug Fix - Discovered a stale cache issue: FlowsData.previous_states could be cached before all previous_flow_rate values were set (e.g., in from_old_results). Fixed by having ComponentsData._get_previous_status_for_component() compute previous status directly from flow attributes instead of going through the potentially-stale FlowsData.previous_states cache. * 1. _build_flow_mask exposure: Added balanced_in1_mask and balanced_in2_mask cached properties to TransmissionsData. TransmissionsModel now uses these instead of calling the private d._build_flow_mask(). 2. EffectsModel/elements: Confirmed not a bug — EffectsModel does not inherit from TypeModel and never accesses .elements on its data, so EffectsData not having elements is fine. 3. FlowsData.dim_name: Changed from @cached_property to @Property to match all other Data classes.
…classes # Conflicts: # CHANGELOG.md # tests/deprecated/test_config.py
- StatusData.with_effects_per_active_hour and StatusData.with_effects_per_startup — categorization lists that were never accessed. The actual effect values (effects_per_active_hour, effects_per_startup) are applied correctly via a different path. Removed passthrough properties (components.py — StoragesModel) 9 properties eliminated, replaced with direct self.data.X at all call sites: - with_investment, with_optional_investment, with_mandatory_investment - storages_with_investment, storages_with_optional_investment - optional_investment_ids, mandatory_investment_ids - invest_params, _investment_data Kept investment_ids — it has 4 external callers in optimization.py. Removed passthrough property (elements.py — FlowsModel) - _previous_status → replaced 3 call sites with self.data.previous_states
# Conflicts: # CHANGELOG.md # flixopt/io.py # flixopt/statistics_accessor.py # flixopt/transform_accessor.py # tests/deprecated/conftest.py # tests/deprecated/examples/03_Optimization_modes/example_optimization_modes.py # tests/test_clustering/test_cluster_reduce_expand.py # tests/test_math/test_clustering.py
- stats.flow_sizes/storage_sizes return empty arrays when the solution has no size variables instead of raising KeyError - transform.fix_sizes() accepts the element-dim DataArray that batched stats.sizes returns by splitting it into per-element variables - cyclic clustered-storage SOC assertions made degeneracy-robust (both the level and the charge timing are solver-dependent, see #733) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…tests - from_dataset: connect the network before restoring the solution — restoring first sets status SOLVED, which made connect_and_transform() skip and left bus wiring empty, crashing any rebuild of a reloaded system - solution expansion tolerates FlowSystems without a solution - main-era clustering tests adapted to batched solution variable names - manual SOC decay comparison gets an atol for near-zero values Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ure/element-data-classes-v8 # Conflicts: # CHANGELOG.md # flixopt/__init__.py # flixopt/flow_system.py # flixopt/optimization.py # flixopt/results.py # flixopt/structure.py # flixopt/topology_accessor.py # tests/io/test_io_conversion.py
Contributor
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
…s, and the docs notebooks Library fixes surfaced by executing the docs notebooks (the only coverage for several clustered/plotting paths): - InterclusterStoragesModel.create_effect_shares called the removed tuple-based InvestmentBuilder.collect_effects — clustered storage with investment effects crashed at build. Rewritten to the push-based add_periodic_contribution pattern (mirroring StoragesModel); dead collect_effects removed. - stats.storage_sizes / sizes now include intercluster storages (intercluster_storage|size, renamed onto the storage dim). - CONFIG.Legacy.solution_access defaults to True for the v8 transition: the LegacySolutionWrapper bridges old access patterns (with DeprecationWarning) by default; strict mode is one config line away. - LegacySolutionWrapper: selections drop the scalar coord (old per-element variables carried none) and cover component-level status/startup/shutdown/ uptime/downtime plus flow-level status-tracking variables. - stats.plot.storage resolves charge state from the batched variables (regular and intercluster) instead of per-element names. - stats.plot.effects(effect=...) keeps a one-element effect dim so single- effect plots render one bar instead of failing on a scalar; threshold filtering picks the breakdown dim unless there are multiple effects. - stats.plot.balance names its data so .to_dataframe() works. Notebooks adapted to the batched idioms where they taught the old API (stats.sizes.sel(element=...), flow_rates.sel(flow=...), batched linopy variable names in the custom-constraint example). Full suite: 1450 passed. All fast notebooks execute. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ure/element-data-classes-v8
Extends the v8 transition compat layer from the solution to the statistics
accessors. When CONFIG.Legacy.solution_access is on (the default),
flow_rates / flow_hours / sizes / charge_states are wrapped in
LegacyElementAccess, which keeps v7 Dataset-style patterns working with a
DeprecationWarning:
flow_rates['Boiler(Q_th)'] -> .sel(flow='Boiler(Q_th)')
flow_hours.items() -> per-element (label, DataArray) pairs
sizes.data_vars -> {label: DataArray} mapping
Everything else proxies to the wrapped DataArray (including arithmetic and
abs() via explicit dunders, since special methods bypass __getattr__).
Internal composition uses the raw arrays; fix_sizes unwraps before its
isinstance dispatch. LegacySolutionWrapper.__contains__ now answers True for
translatable legacy keys. Strict-mode contract tests toggle the flag off.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Build the mapping directly instead of routing through items(), which emits its own DeprecationWarning. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Member
Author
FBumann
added a commit
that referenced
this pull request
Jul 23, 2026
Squash of the element-data-classes work rebased onto the v8 removals.
Batched/vectorized model building — up to 67x faster for large systems:
type-level FlowsModel/StoragesModel/BusesModel replace per-element loops,
*Data containers pre-compute element parameters as element-dimensioned
DataArrays, variables use linopy masks, and the solution uses batched
naming ('flow|rate' with a flow dim instead of per-element variables).
v7 compatibility layer (CONFIG.Legacy.solution_access, default on for the
v8 transition, sunset v9):
- LegacySolutionWrapper translates old solution keys (flows, storages,
effects, component status family) with DeprecationWarnings attributed to
user code; membership checks translate too
- LegacyElementAccess keeps Dataset-style stats patterns working:
flow_rates['Boiler(Q_th)'], .items(), .data_vars
- verified against real v7 result files: load, raw access and re-optimize
work; stats accessors need one re-optimize
Also: clustered-storage investment effects fixed (dead collect_effects path),
intercluster sizes included in stats.sizes, docs notebooks adapted to the
batched idioms, expansion machinery reconciled with the tsam_xarray world.
BREAKING CHANGE: solution and linopy variables use batched names; statistics
accessors return element-dimensioned DataArrays. The legacy access layer
bridges common v7 patterns during the transition.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Stacked PR — base is
remove-deprecated-api(#741), notmain. This is a duplicate of #602's synced state with #741 merged in, so the diff here shows only the batched-modeling work on top of the v8 cleanup. #602/#610/#611/#614 remain untouched as fallback while the merge strategy is undecided.What this is
The batched/vectorized model-building rework (up to 67× faster builds): type-level
FlowsModel/StoragesModel/BusesModelreplacing per-element loops, pre-computed*Datacontainers, mask-based variables, and theflow|rate-style batched solution naming with aLegacySolutionWrapperbridge.What merging #741 into it bought
The deprecated surface #602 had been maintaining against the batched world is now deleted instead: the batched-adapted
Optimization/Resultsclasses,normalize_weightsplumbing, the network-method wrappers on the rewrittenFlowSystem, and the kwarg-rename bridge.from_old_dataset()(kept per #741) works in the batched world — verified by the real-file regression tests, including the pinned end-to-end objective.Verification
Full suite on this branch: 1450 passed, 0 failed (linopy 0.7.0).
Relationship to the existing stack
🤖 Generated with Claude Code