Skip to content

Detect distribution drift against each series' recent windows - #27

Open
zzylol wants to merge 1 commit into
feat/cardinalityfrom
feat/drift
Open

zzylol wants to merge 1 commit into
feat/cardinalityfrom
feat/drift

Conversation

@zzylol

@zzylol zzylol commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

Part 5 of the RFC 0001 stack: use case 2, distribution drift and baseline comparison.

The detector keeps each quantile series' last 5 KLL windows and merges them as the baseline. It computes the Kolmogorov–Smirnov distance of the current window from that baseline: the CDF is rank/count, compared at both windows' percentiles. Above the threshold (0.1) the series has drifted. Drift start and end are OTLP log events to Loki, and asap_drift_ks and asap_drift are gauges. exact_answers runs the same test on the exact values. The scenario triples http.route-3's median at minute 10.

Result (56-core x86 server, 1-minute windows)

Drift dashboard

  • Both methods flag only route-3, in the first window after the change: KS 0.505 (sketch) and 0.491 (exact), against a no-change noise level of 0.012–0.017 and 0.001–0.003. There were no false positives across the other 19 routes.
  • The KS distance decays as the new distribution enters the baseline.
  • The exact baseline's memory grows steadily because it must retain 5 windows of raw values; the sketch stages stay flat.

cargo fmt --check, cargo clippy --all-targets --features otap-engine,stream-benchmark -- -D warnings and cargo test --features otap-engine,stream-benchmark pass at this PR's head.

The detector keeps each quantile series' last N KLL windows, merges them as
the baseline and computes the Kolmogorov-Smirnov distance of the current
window from it (CDF = rank/count over both windows' percentiles). Above the
threshold (0.1) the series has drifted; start and end are logged as events.
exact_answers runs the same test on the exact values. Adds the drift scenario
(one route's median triples at minute 10) and dashboard.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant