Skip to content

Add signal-decomposition degradation analysis - #518

Open
bmeyers wants to merge 31 commits into
developmentfrom
sigdecomp
Open

Add signal-decomposition degradation analysis#518
bmeyers wants to merge 31 commits into
developmentfrom
sigdecomp

Conversation

@bmeyers

@bmeyers bmeyers commented Jul 15, 2026

Copy link
Copy Markdown
Collaborator

Ports the CVXPY seasonal-trend decomposition prototype into a new rdtools.signal_decomposition module and wires it end-to-end through TrendAnalysis. The implementation now provides a unified framework for conventional degradation analysis, expanded lifecycle trend analysis, and SD++ soiling analysis.

New module (rdtools/signal_decomposition.py):

  • Public degradation() entry point returning (Rd_pct, Rd_CI, sd_trend_results)
  • Three trend types: linear, year-1 piecewise-linear breakpoint, and nonlinear monotone
  • Four loss functions: L2, L1, Huber, and conditional quantile
  • Optional natural-log transform; log-Huber is the recommended and default configuration
  • Missing data handled natively through a masked equality constraint
  • Moving-block bootstrap confidence intervals for overall and lifecycle-dependent degradation rates
  • Stability and conditional-quantile sweeps for diagnosing model sensitivity and degradation patterns
  • Plotting/reporting: plot_decomposition, plot_trend, plot_stability, animate_degradation, and format_degradation_report

SD++ soiling:

  • Adds a nonpositive soiling component for dry, accumulation-and-recovery behavior
  • Solves a deterministic regularization path and applies an explicit structural detection rule
  • Returns a true no-soiling model when no coherent component is detected
  • Freezes the detection decision before applying two interval-weighted IRL1 refinement steps to reduce shrinkage
  • Reports time-averaged and quarterly loss, insolation-weighted loss when available, local soiling-rate intervals, cleaning-event strength, bootstrap uncertainty, and selector diagnostics
  • Adds soiling decomposition plots and formatted reports

TrendAnalysis wiring (rdtools/analysis_chains.py):

  • _signal_decomposition_degradation private integration method
  • sensor_analysis and clearsky_analysis accept signal_decomposition in the analyses list and pass configuration through sd_kwargs
  • plot_signal_decomposition_summary method
  • Existing preprocessing can be reused explicitly across repeated analyses

Dependencies and minimum environment:

  • Python 3.11 is now the minimum supported version; Python 3.10 was removed from CI and Pixi environments
  • CVXPY >= 1.5.3 and SPCQE >= 0.0.3 are hard runtime dependencies
  • SciPy minimum raised from 1.8.1 to 1.13.0
  • pvlib minimum lowered to the tested 0.11.0 floor
  • CVXPY and SPCQE are included in the exact minimum-dependency environment
  • check-deps verifies that all direct dependency pins match their declared lower bounds
  • test-min builds the Python 3.11 lower-bound environment and runs the complete test suite
  • Pixi is pinned in CI for reproducible lockfile behavior

Tests and documentation:

  • Expanded signal_decomposition_test.py coverage for trend models, losses, bootstrap uncertainty, structural soiling selection, IRL1 refinement, null behavior, metrics, plots, and reports
  • Integration coverage in analysis_chains_test.py
  • Complete minimum-environment suite passes with 491 tests
  • Added a full signal-decomposition guide covering basic degradation, expanded trend analysis, diagnostic sweeps, and SD++ soiling
  • Added public PVDAQ examples with synthetic year-1 breakpoint and dry-soiling signals
  • Added generated decomposition and soiling figures
  • Updated api.rst and the v3.3.0 changelog

Todo list:

  • Code changes are covered by tests
  • Code changes have been evaluated for compatibility/integration with TrendAnalysis
  • [ ] New functions added to __init__.py
    This is intentionally not being done. The function should be imported as rdtools.signal_decomposition.degradation or from rdtools.signal_decomposition import degradation to avoid clashing with the existing rdtools.degradation module.
  • api.rst and the other Sphinx documentation pages are up to date
  • Example notebooks have been rerun and result differences scrutinized
  • Changelog updated

martin-springer and others added 11 commits July 14, 2026 17:15
Ports the CVXPY seasonal-trend decomposition prototype into a new
rdtools.signal_decomposition module and wires it end-to-end through
TrendAnalysis.

New module (rdtools/signal_decomposition.py):
- Public degradation() entry point returning (Rd_pct, Rd_CI, sd_trend_results)
- Three trend types: linear, pwl, monotone
- Four loss functions: l2, l1, huber, quantile
- Missing data handled natively via masked equality constraint
- Stability analysis: analyze_fit_stability, get_valid_endpoints, plot_stability
- Plotting/display: plot_decomposition, animate_degradation, format_degradation_report
- log_transform path via extract_degradation_rate_log

TrendAnalysis wiring (rdtools/analysis_chains.py):
- _signal_decomposition_degradation private method
- sensor_analysis / clearsky_analysis accept 'signal_decomposition' in analyses
  list and sd_kwargs passthrough
- plot_signal_decomposition_summary method

Dependencies:
- cvxpy >= 1.7.5 and spcqe >= 0.3.0 added as hard runtime dependencies
- scipy floor raised from 1.8.1 to 1.13.0 (required by cvxpy's transitive deps)
- cvxpy and spcqe excluded from check_dependencies.py min-pin enforcement
  (their transitive scipy requirement is incompatible with pinning an exact
  lower-bound version alongside scipy==1.13.0 on Python 3.10)

Tests and docs:
- rdtools/test/signal_decomposition_test.py (29 tests)
- Additions to analysis_chains_test.py (fixtures, plot tests, bad-case coverage)
- api.rst updated with Signal Decomposition section and new TrendAnalysis method
@bmeyers

bmeyers commented Jul 15, 2026

Copy link
Copy Markdown
Collaborator Author

Just finishing up examples and package dependency testing. Ready for general review. Open item: minimum scipy version support.

bmeyers added 7 commits July 15, 2026 14:12
…repeated calls to analysis_chains.TrendAnalysis.sensor_analysis
analyze_fit_stability and animate_degradation both call
make_problem(y_full[:n], **make_problem_kwargs). When callers pass
sd_trend_results['args'] directly (which includes 'y' because _get_kwargs
captures the full make_problem signature), this caused a "multiple values
for argument 'y'" TypeError. Strip 'y' from the dict at the top of each
function so passing args through is always safe.
@codecov-commenter

codecov-commenter commented Jul 16, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 91.24886% with 96 lines in your changes missing coverage. Please review.
✅ Project coverage is 94.99%. Comparing base (b5eb6e2) to head (d414a75).

Files with missing lines Patch % Lines
rdtools/signal_decomposition.py 91.48% 72 Missing ⚠️
rdtools/degradation.py 84.55% 21 Missing ⚠️
rdtools/plotting.py 95.31% 3 Missing ⚠️
Additional details and impacted files
@@               Coverage Diff               @@
##           development     #518      +/-   ##
===============================================
- Coverage        96.79%   94.99%   -1.81%     
===============================================
  Files               12       13       +1     
  Lines             2342     3437    +1095     
===============================================
+ Hits              2267     3265     +998     
- Misses              75      172      +97     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

bmeyers added 6 commits July 16, 2026 12:38
dict of CI arrays rather than a single [lo, hi] for the overall rate.
degradation() merges these into sd_trend_results as ci_<rate_key>
entries, giving ci_rate_pre/post_pct_yr for pwl and ci_rate_yearly_pct_yr
(shape 2×n_years) for monotone. Rd_CI is unchanged.
  - add validated convex soiling component and coherence selector
  - return a structural null when coherent soiling is not detected
  - report overall and quarterly losses and local soiling rates
  - bootstrap uncertainty using the selected terminal model
  - pass insolation through TrendAnalysis for weighted losses
  - add selector, metric, integration, and plotting tests
  - add a PVDAQ soiling comparison marimo notebook
  - ignore local agent and prototype artifacts
Refine positively detected soiling components with two deterministic cleaning-interval IRL1 solves to reduce L1 amplitude shrinkage.

Apply the same refinement to bootstrap fits, preserve the structural-null path, expose refinement diagnostics, and document the intended dry-soiling scope and upstream outage requirement.
Drop Python 3.10 support and move the minimum test environment to Python 3.11. Raise dependency floors to a coherent stack that builds and passes the full test suite, including matching minimum pins for CVXPY and SPCQE.

Regenerate the Pixi lockfile, pin the Pixi version used in CI, and update the supported-version documentation and release notes. Make the PVLib bootstrap regression test deterministic without leaking global RNG state.
@bmeyers bmeyers self-assigned this Aug 25, 2026
Fully qualify signal decomposition function names so Sphinx does not resolve them relative to the module twice during autosummary generation.
Add a first-class signal decomposition guide covering the basic trend model, loss functions, expanded trend diagnostics, and SD++ soiling.

Use log-transformed Huber loss with M=0.05 as the default SD model while preserving explicit overrides, and record the resolved transform in result metadata.

Add reproducible PVDAQ decomposition and synthetic year-1 breakpoint figures, align the marimo example defaults, and test the new preset.
Test the animation workflow, soiling insolation validation, missing-quarter warnings, and the public trend plot.

Use deterministic component doubles for animation rendering so the test exercises frame updates and writer selection without invoking an external encoder or duplicating solver tests.

Plot normalized energy directly with Matplotlib to avoid incompatible datetime converters.
Document synthetic dry-soiling recovery with a reproducible PVDAQ figure and generated degradation and soiling report.

Reduce the ordinary signal-decomposition seasonal default from six to three Fourier harmonic pairs, matching the SD++ preset while preserving explicit configuration.

Regenerate the basic and year-1 breakpoint figures under the updated default and extend the preset test.
@bmeyers

bmeyers commented Aug 25, 2026

Copy link
Copy Markdown
Collaborator Author

We updated the minimum supported environment to Python 3.11 and defined dev-min as a coherent, tested lower-bound stack. Each direct runtime dependency is pinned to its declared minimum, check-deps verifies those pins remain synchronized, and test-min builds the environment and runs the full test suite. Pixi is also pinned in CI for reproducibility.

The newest package in this minimum environment is CVXPY 1.7.5, released December 5, 2025. As of August 25, 2026, the environment is therefore about 8 months and 20 days old. Most other minimum dependencies date from 2022–2024, with a few newer requirements introduced by the signal-decomposition stack. This gives us a realistic, reproducible minimum environment rather than a collection of independently old version floors that may not build or work together.

@martin-springer
martin-springer changed the base branch from master to development August 25, 2026 19:57
@bmeyers

bmeyers commented Aug 25, 2026

Copy link
Copy Markdown
Collaborator Author

I am currently working on raising the "age" of the minimum environment, which is currently about 8 months. The new target is 2 years. Cvxpy has been downgraded successfully, and I'm now testing additional binding packages.

Lower the tested floors to CVXPY 1.5.3, pvlib 0.11.0, and SPCQE 0.0.3, then regenerate the Pixi lockfile.

The newest signal-decomposition floor is SPCQE 0.0.3, released October 28, 2024—1 year, 9 months, and 28 days old as of August 25, 2026. The complete environment’s newest direct requirement remains bayesian-filters 1.4.5, released October 20, 2025.

Confirm the complete minimum-version suite passes with 491 tests.
@bmeyers

bmeyers commented Aug 25, 2026

Copy link
Copy Markdown
Collaborator Author

Follow-up: The minimum-version review is complete. We lowered the tested floors to CVXPY 1.5.3, pvlib 0.11.0, and SPCQE 0.0.3, and regenerated the Pixi lockfile. The complete dev-min suite passes with 491 tests.

The newest signal-decomposition dependency floor is now SPCQE 0.0.3, released October 28, 2024—1 year, 9 months, and 28 days old as of August 25, 2026. This is close to the two-year target while retaining the first stable SPCQE release; the older pre-cutoff release is an alpha with an incomplete public API.

The complete environment still contains one newer direct requirement: bayesian-filters 1.4.5, released October 20, 2025. This is the maintained drop-in replacement for the unmaintained FilterPy package and has no earlier PyPI release, so reverting it solely to improve the environment-age statistic would undo an existing maintenance decision. Subject to that exception, the newly introduced signal-decomposition stack now has a practical minimum age of approximately two years.

@bmeyers

bmeyers commented Aug 25, 2026

Copy link
Copy Markdown
Collaborator Author

Last remaining open item is to update the changelog, per the PR template

@bmeyers

bmeyers commented Aug 25, 2026

Copy link
Copy Markdown
Collaborator Author

I have no more planned commits for this PR. Ready for final review!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants