Skip to content

Add CMS-with-heap top-k demo and benchmark vs exact - #22

Open
zzylol wants to merge 7 commits into
feat/kll-prometheus-grafanafrom
feat/cmsheap-topk-demo
Open

zzylol wants to merge 7 commits into
feat/kll-prometheus-grafanafrom
feat/cmsheap-topk-demo

Conversation

@zzylol

@zzylol zzylol commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

Stacked on #20. Adds a Count-Min sketch with heavy-hitter heap (asap_sketchlib::CMSHeap) and a CMS + heap vs exact top-k demo and benchmark, using the same multi-process pinned topology as the KLL demo in #20.

Before → after

  • Before: the collector had CMS and CountSketch wrappers but no top-k: CountSketch::top_k returns nothing, and the OTAP carrier accepted only KLL ASAPv1 payloads.
  • After:
    • CMSHeapWrapper wraps asap_sketchlib::CMSHeap. It sends ASAPv1 envelopes (kind 0x03 0x00, CountMinSketchWithHeap) and implements FrequencySketch::top_k. Merging re-ranks both heaps on the merged counters. The OTAP carrier now accepts and validates these payloads.
    • The benchmark has two new scenarios, topk-exact and cmsheap, run on keyed traffic: Zipf(1.1) over 2^20 keys.
      • topk-exact: each branch counts keys in a hash map and ships every (key, count) entry; merge sums them.
      • cmsheap: each branch ships one sketch per window (4 x 2048 counters, 10-slot heap); merge merges sketches.
    • The backend checks every window: exact must match, and CMS + heap must reach recall@10 ≥ 0.9.
    • run-demo.sh --scenarios topk-exact cmsheap serves a new Grafana dashboard: both pipelines' current top 10 side by side, recall@10, count error, throughput, and per-process CPU, egress and memory.

Run

demos/kll-grafana/run-demo.sh --scenarios topk-exact cmsheap   # Ctrl+C stops

Open http://localhost:3000/d/asap-topk-stream.

CMS + heap vs exact top-k dashboard

56-core x86 server, 50,000 signals/s per source. Both pipelines keep up. CMS + heap uses 1.7 processor CPU-seconds per million signals; exact uses 13.0. CMS + heap returns the same ten keys in the same order, with counts within 0.6%.

Throughput: CMS + heap vs exact top-k (vs raw forwarding)

65,536-key windows, 3 runs per point, one dedicated CPU per processor, a 56-core x86 server. Median backend-validated signals/s, both sources:

Offered Raw Exact top-k CMS + heap
20,000 21,813 21,813 21,813
50,000 47,988 52,348 47,989
100,000 100,327 100,336 100,335
200,000 196,309 148,316 200,675
400,000 274,848 130,879 397,004
600,000 279,185 130,860 597,659
800,000 283,555 126,516 798,355

Validated throughput vs offered load

Largest sustainable offered rate per source (CAPACITY_DESIGN criteria): 50,000 for exact top-k, 100,000 for raw, and at least 400,000 (the highest tested) for CMS + heap. Above that, exact top-k plateaus near 130,000 signals/s.

Processor CPU and egress summed over branches, merge and estimate, at the same load:

Offered per source Exact CPU CMS + heap CPU Exact egress CMS + heap egress
10,000 0.29 cores 0.039 cores 43.9 Mbit/s 0.1 Mbit/s
25,000 0.66 cores 0.084 cores 102.9 Mbit/s 0.3 Mbit/s
50,000 1.29 cores 0.165 cores 201.1 Mbit/s 0.5 Mbit/s

Accuracy

The backend checked every window of every run:

  • Exact top-k: matched exactly.
  • CMS + heap: recall@10 = 1.0, mean relative count error 0.62% over the true top 10. Count-Min only overestimates.

Library pin

This PR moves the asap_sketchlib pin from 010457de to 3fc3442c (current main), because the CMSHeap ASAPv1 payload exists only there (asap_sketchlib#115 and follow-ups). On main, DdSketchState also gained negative and zero stores. The DDSketch wrapper now uses sketchlib's own to_proto/from_proto instead of a local field mapping. All existing sketch, codec and runtime tests pass on the new pin.

Validation

cargo test --features otap-engine,stream-benchmark   # CMSHeap wrapper, carrier round trip, top-k model tests
python3 demos/kll-grafana/verify-streaming.py --scenarios topk-exact cmsheap

All tests pass, along with fmt and Clippy. The demo ran end to end on a 56-core x86 server.

The pinned rev predates the CMSHeap ASAPv1 envelope (asap_sketchlib#115).
On main, DdSketchState also carries negative and zero stores, so the
DDSketch wrapper now uses sketchlib's own to_proto/from_proto mapping.
Count-Min sketch with a top-k heap: full ASAPv1 snapshots (kind 0x03 0x00),
merge that re-ranks both heaps on merged counters, and FrequencySketch
top_k. Envelopes name it CountMinSketchWithHeap.
The carrier validated every sketch.envelope as KLL. CountMinSketchWithHeap
envelopes are now validated as CMSHeap; other types keep the KLL check.
topk-exact ships per-key count maps and sums them; cmsheap ships one
ASAPv1 CMSHeap per source and merges sketches. Keys follow Zipf(1.1) over
2^20 ranks. The backend validates every window: exact must match, the
sketch must reach recall@10 >= 0.9. The demo exports top-k, recall and
count error next to per-process throughput, CPU and egress.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant