Skip to content

feat(drift): integrate ADWIN drift detection - #14

Open
MichalRedm wants to merge 11 commits into
mainfrom
feat/adwin-integration
Open

MichalRedm wants to merge 11 commits into
mainfrom
feat/adwin-integration

Conversation

@MichalRedm

@MichalRedm MichalRedm commented Oct 2, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

  • Integrates the ADWIN (Adaptive Windowing) drift detector from River into the STRIDE framework and interactive Streamlit dashboard.
  • Implements DualADWIN wrapper to support dual-threshold workflows (warning and drift signals).
  • Introduces directional degradation filtering (degradation_only=True by default) in BinaryErrorDriftDescriptor, eliminating false alarms caused by model learning, convergence, or recovery.
  • Calibrates lookback boundary clamping so CUSUM, Gradient, and Threshold methods isolate the true change-point inflection rather than defaulting to the lookback window boundary.
  • Enhances BinaryErrorDriftDescriptor to safely handle single-threshold detectors without warning zones and to dynamically derive lookback windows from detector.width.
  • Synchronizes with main, resolving merge conflicts and integrating ADWIN into the newly established documentation suite (User Guide, Mathematical Formulations, API Reference, and Changelog).
  • Exposes warm-up / burn-in sample controls in the dashboard UI.
  • Adds comprehensive deterministic unit tests and pipeline integration tests across STRIDE's synthetic benchmark stream generators.

Motivation

  • Concept drift detection serves as the entry point for downstream explainable AI (xAI) analysis in STRIDE. Prior to this change, STRIDE only supported detectors from river.drift.binary (DDM, EDDM, FHDDM, HDDM).
  • ADWIN is the standard benchmark in data stream mining due to its dynamic window resizing, Hoeffding/Bernstein statistical bounds, and O(log W) complexity.
  • Because ADWIN is a two-sided statistical change detector ($|\mu_0 - \mu_1| \ge \epsilon_{\text{cut}}$) unlike one-sided error detectors (such as DDM), applying it to binary classification errors without directional filtering triggers false alarms whenever the model improves (during initial training warm-up or post-drift adaptation). Adding directional degradation filtering aligns ADWIN with the error-monitoring framework.
  • Resolves feat: Integrate ADWIN drift detection with adaptive post-processing & benchmark validation #5.

Key Changes

1. Core Framework & Packaging (src/stride/drift/, src/DDM/)

  • DualADWIN Adapter (src/stride/drift/adwin.py): Wrapper maintaining two internal ADWIN instances (delta_warn and delta_drift) for systems requiring explicit warning and drift states; exposes .estimation property delegated to drift instance; fully documented with Google-style docstrings and type annotations.
  • Directional Degradation Filtering (src/stride/drift/binary_descriptor.py): Added degradation_only=True parameter to compare window error estimations (curr_estimation < prev_estimation) and suppress drift alarms when error decreases (learning/recovery).
  • Lookback Boundary Calibration (src/stride/drift/binary_descriptor.py): Fixed lookback clamping from min(drift_start_idx, detection_idx - lookback_window) to max(min_lookback_bound, min(drift_start_idx, detection_idx)) so CUSUM inflection point localization works properly.
  • Adaptive Descriptor Safety: Replaced direct .warning_detected accesses with getattr(..., False) to prevent AttributeError on detectors lacking warning properties.
  • Public API Exports: Re-exported ADWIN and DualADWIN through stride.drift and backward-compatibility bridge DDM.

2. Streamlit Dashboard (dashboard/components/tabs/drift_detection.py)

  • Added "ADWIN" to the Detector Type selector.
  • Exposed ADWIN parameter controls: Confidence Level (delta), Clock Rate, Min Window Length, and Grace Period.
  • Added Warm-up / Burn-in (Samples) slider control and enabled directional degradation filtering for drift runs.
  • Updated tab typography and configuration handling.

3. Documentation Infrastructure Integration (docs/**, CHANGELOG.md)

  • Model Layer Guide (docs/user-guide/model-layer.md): Comprehensive documentation and code recipes for DualADWIN, dynamic lookback bounding via detector.width, and directional degradation filtering.
  • Mathematical Formulations (docs/theory/mathematical-formulations.md): Formalized the ADWIN Hoeffding bound formulation ($\epsilon_{\text{cut}} = \sqrt{\frac{1}{2m} \ln(4|W|/\delta)}$) and two-tier $\delta_{\text{warn}} &gt; \delta_{\text{drift}}$ confidence thresholding.
  • Dashboard Tour (docs/user-guide/dashboard.md): Documented the updated detector suite in Tab 2.
  • Changelog (CHANGELOG.md): Added release notes under [Unreleased].
  • CI Dependency Compatibility (pyproject.toml): Pinned pymdown-extensions < 12.2 to resolve an upstream constructor signature break with mkdocstrings.

4. Empirical Benchmark Insights

As part of resolving #5, ADWIN's detection dynamics were evaluated across STRIDE's synthetic benchmark streams:

  • Hyperplane & Random Tree Multi-Window: Strong, clean detections (t ≈ 1119 with start at 1011 on Hyperplane + MLP). Directional filtering prevents initial warm-up alarms and post-drift re-adaptation false triggers.
  • SEA & Linear Weight Inversion (LWI): Rapid online SGD adaptation can mask transient error spikes under conservative confidence (δ = 0.002). Under fixed-model loss monitoring or with δ ∈ [0.01, 0.05] (now configurable in the UI), ADWIN detects shifts promptly (t ≈ 1087).
  • RBF: Requires sufficient model capacity (e.g., MLP/kernel) so that centroid shifts induce an error rate delta rather than baseline random guessing.

5. Test Suite (tests/)

  • tests/test_adwin.py: Fast, deterministic unit tests verifying imports, DualADWIN warning/drift dynamics and estimation, descriptor AttributeError safety, directional filtering (degradation_only), and CUSUM inflection localization.
  • tests/test_adwin_drift.py: End-to-end integration tests verifying prequential pipeline execution and descriptor structural invariants across synthetic generators, plus a clean single-drift test on Hyperplane with MLP.

Verification

  • ruff check . passed with 0 errors
  • ruff format --check . passed cleanly
  • python -m unittest discover tests passed with 0 failures (57 tests)
  • mkdocs build --strict passed with 0 warnings/errors
  • Both remote CI (CI and Documentation) GitHub Actions workflows passed
  • Verified Streamlit dashboard runs and processes drift analysis with ADWIN parameter inputs and directional filtering

@MichalRedm MichalRedm changed the title feat: Integrate ADWIN drift detection feat(drift): integrate ADWIN drift detection Oct 2, 2026
…nd benchmark validation

Resolves #5 by implementing ADWIN and DualADWIN drift detectors, integrating them into the stream UI, and validating against synthetic benchmark generators.
…ion tests

- Created tests/test_adwin.py with deterministic unit tests for ADWIN imports, DualADWIN warning/drift dynamics, and BinaryErrorDriftDescriptor interface invariants.
- Refactored tests/test_adwin_drift.py into pipeline execution integration tests across the 5 synthetic stream benchmarks, focusing on data flow integrity and descriptor structure rather than brittle statistical bounds.
…k bounds

- Enable degradation_only=True by default in BinaryErrorDriftDescriptor to suppress false alarms on error decreases (model learning/adaptation).
- Fix lookback start clamping to prevent overriding CUSUM, Gradient, and Threshold change points with the maximum window boundary.
- Expose estimation property on DualADWIN.
- Add optional Warm-up / Burn-in slider in the Drift Detection dashboard tab.
- Add unit and integration tests verifying clean drift detection on Hyperplane with MLPClassifier.
@MichalRedm
MichalRedm force-pushed the feat/adwin-integration branch from 64911d2 to 19f2676 Compare October 4, 2026 10:05
@MichalRedm
MichalRedm requested a review from dka124 October 4, 2026 10:38

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat: Integrate ADWIN drift detection with adaptive post-processing & benchmark validation

1 participant