Repository navigation
Conversation
The pinned rev predates the CMSHeap ASAPv1 envelope (asap_sketchlib#115). On main, DdSketchState also carries negative and zero stores, so the DDSketch wrapper now uses sketchlib's own to_proto/from_proto mapping.
Count-Min sketch with a top-k heap: full ASAPv1 snapshots (kind 0x03 0x00), merge that re-ranks both heaps on merged counters, and FrequencySketch top_k. Envelopes name it CountMinSketchWithHeap.
The carrier validated every sketch.envelope as KLL. CountMinSketchWithHeap envelopes are now validated as CMSHeap; other types keep the KLL check.
topk-exact ships per-key count maps and sums them; cmsheap ships one ASAPv1 CMSHeap per source and merges sketches. Keys follow Zipf(1.1) over 2^20 ranks. The backend validates every window: exact must match, the sketch must reach recall@10 >= 0.9. The demo exports top-k, recall and count error next to per-process throughput, CPU and egress.
zzylol
force-pushed
the
feat/cmsheap-topk-demo
branch
from
October 13, 2026 00:20
d856571 to
fb6fa0f
Compare
zzylol
force-pushed
the
feat/kll-prometheus-grafana
branch
from
October 13, 2026 00:20
d31060b to
a0466df
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #20. Adds a Count-Min sketch with heavy-hitter heap (
asap_sketchlib::CMSHeap) and a CMS + heap vs exact top-k demo and benchmark, using the same multi-process pinned topology as the KLL demo in #20.Before → after
CountSketch::top_kreturns nothing, and the OTAP carrier accepted only KLL ASAPv1 payloads.CMSHeapWrapperwrapsasap_sketchlib::CMSHeap. It sends ASAPv1 envelopes (kind0x03 0x00,CountMinSketchWithHeap) and implementsFrequencySketch::top_k. Merging re-ranks both heaps on the merged counters. The OTAP carrier now accepts and validates these payloads.topk-exactandcmsheap, run on keyed traffic: Zipf(1.1) over 2^20 keys.topk-exact: each branch counts keys in a hash map and ships every (key, count) entry; merge sums them.cmsheap: each branch ships one sketch per window (4 x 2048 counters, 10-slot heap); merge merges sketches.run-demo.sh --scenarios topk-exact cmsheapserves a new Grafana dashboard: both pipelines' current top 10 side by side, recall@10, count error, throughput, and per-process CPU, egress and memory.Run
demos/kll-grafana/run-demo.sh --scenarios topk-exact cmsheap # Ctrl+C stopsOpen http://localhost:3000/d/asap-topk-stream.
56-core x86 server, 50,000 signals/s per source. Both pipelines keep up. CMS + heap uses 1.7 processor CPU-seconds per million signals; exact uses 13.0. CMS + heap returns the same ten keys in the same order, with counts within 0.6%.
Throughput: CMS + heap vs exact top-k (vs raw forwarding)
65,536-key windows, 3 runs per point, one dedicated CPU per processor, a 56-core x86 server. Median backend-validated signals/s, both sources:
Largest sustainable offered rate per source (CAPACITY_DESIGN criteria): 50,000 for exact top-k, 100,000 for raw, and at least 400,000 (the highest tested) for CMS + heap. Above that, exact top-k plateaus near 130,000 signals/s.
Processor CPU and egress summed over branches, merge and estimate, at the same load:
Accuracy
The backend checked every window of every run:
Library pin
This PR moves the
asap_sketchlibpin from010457deto3fc3442c(current main), because the CMSHeap ASAPv1 payload exists only there (asap_sketchlib#115 and follow-ups). On main,DdSketchStatealso gained negative and zero stores. The DDSketch wrapper now uses sketchlib's ownto_proto/from_protoinstead of a local field mapping. All existing sketch, codec and runtime tests pass on the new pin.Validation
All tests pass, along with fmt and Clippy. The demo ran end to end on a 56-core x86 server.