Skip to content

Add end-to-end acceptance on real collector containers, and cost/accuracy experiment tooling - #30

Open
zzylol wants to merge 1 commit into
feat/merge-benchfrom
feat/e2e-acceptance
Open

zzylol wants to merge 1 commit into
feat/merge-benchfrom
feat/e2e-acceptance

Conversation

@zzylol

@zzylol zzylol commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

Part 8 of the RFC 0001 stack: the end-to-end acceptance, plus the tooling for experiments 1 and 2. The measured results come in the last PR of the stack.

deploy/telemetry-functions/acceptance.py

Runs all four use cases on the real collector containers (generators, branches, merge, detector, exact baseline, Prometheus, Loki). It checks the sketch answers against the exact ground truth through Prometheus and Loki, prints PASS or FAIL per check, and exits non-zero on any failure.

To keep it to a few minutes per use case, time is scaled down 6×: 10-second windows, a 2-second grace (raised back to the deployed 5 s in #31, after the first real CI run showed sketches arriving later than 2 s on a shared 4-CPU runner), changes at second 100, and growth 6× faster. Rules counted in windows (baselines, thresholds) are as deployed. Comparisons are pinned to one window answered by both methods, with PromQL's @ modifier.

Result on a 56-core x86 server: 25/25 checks passed.

Use case Checks (all passed)
all four 7/7 stages up; 400,000 points in every window, the full offered rate; sketch and exact windows hold identical point counts
latency exact p50/p99 inside the sketch's bound: 40/40
cardinality sketch and exact give every key the same class (user.id identifier-like, k8s.pod.name growing, http.route and http.method stable); distinct error ≤ 1.13%; class events in Loki
drift route-3 drift detected by both methods (KS 0.33); no other route flagged; drift event in Loki
top-k /new/hot-route in both top-10s; recall@10 = 1.00; novelty event in Loki

Informational, not a pass condition: with 10-second windows, the p99 relative error for routes with ≥ 20,000 points reached 12.8%. That is KLL's rank error in a log-normal tail (see #25).

experiments.py and report_experiments.py

  • inline-cost: branch CPU with and without each function, over offered rates. In this run the exact stage only counts points, so it cannot throttle the branch.
  • accuracy-size: error against exact as KLL k and Count-Min width change, with the bytes merge receives per window.
  • The answer paths report asap_answer_delay: seconds from a window's end to its answers.
  • report_experiments.py turns the CSVs into a Markdown report with tables and a figure.

Also includes PIPELINES and the published Loki port in compose, run.sh --build-only, and a README covering every use case.

cargo fmt --check, cargo clippy ... -D warnings and cargo test --features otap-engine,stream-benchmark pass at this PR's head.

acceptance.py runs each use case (latency, cardinality, drift, top-k) on the
real collector containers with time scaled 6x and checks sketch answers
against the exact ground truth through Prometheus and Loki: every stage up,
full rate delivered, identical points per window, bounds containing the exact
answer, classes, drift and novelty detected with no false positives, events
in Loki. Exits non-zero on any failure. experiments.py measures branch CPU
with and without each function over offered rates (inline cost) and error vs
sketch size (KLL k, Count-Min width). Answer paths report asap.answer.delay,
seconds from a window's end to its answers.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant