Repository navigation
Conversation
Drop the asap-otap-demo binary, the file-backed nightly and scale-sweep benchmarks, the criterion bench, the Grafana verify script and KLL dashboard, the demo guide, the multiprocess e2e test, the flat RecordBatch encode/records codec with its tests, and the adapter and control-channel stubs. Update the READMEs, benchmark docs, Compose file and run-nightly.sh (which ran the removed criterion bench) to match.
Every wrapper now wraps sketchlib's native sketch (DDSketch, ErtlMLE HyperLogLog, CountMin, Count, KLL, CMSHeap) and snapshot() returns its self-describing ASAPv1 envelope. Sketch::merge_snapshot decodes one and merges it; delta transmission (compute_delta_against, apply_delta*, DeltaResult, SnapshotCache, delta_transmission/delta_threshold), the Encoding enum, the proto path, hash_spec and KLL's replay history are gone. The OTAP codec validates sketch.envelope against the declared sketch type on every decode path, including Gauge data points. The processor accepts transmit_sketch for every sketch type and dispatches countminsketch_heap (rows/columns/k), whose estimate mode emits one Gauge per heavy hitter with key and rank labels. controller_url, agent_id and encoding are removed from the plugin config. HLL is fixed at sketchlib's precision-14 Ertl-MLE HyperLogLog, so the hll precision parameter is gone. CMS/CountSketch rows are clamped to MATRIX_MAX_ROWS; the power-of-two column rounding is no longer needed.
zzylol
force-pushed
the
feat/cmsheap-topk-demo
branch
from
October 13, 2026 00:20
d856571 to
fb6fa0f
Compare
zzylol
force-pushed
the
refactor/asapv1-only
branch
from
October 13, 2026 00:20
2bd3e0c to
974b425
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #22. Part 1 of the RFC 0001 implementation stack (sketch-based telemetry functions).
Removes everything that is not the real-container path and every pre-ASAPv1 wire format, and uses only the latest asap_sketchlib (
3fc3442c, as pinned in #22). No compatibility with older serialization formats is kept. About 8,560 lines are removed (+813 / −9,372).Before → after
asap-otap-demoand its file-based benchmark scripts, an in-process Criterion bench), and four sketches (DDSketch, HLL, CountSketch, CMS) serialized through proto envelopes and proto/msgpack deltas. Only KLL and CMS + heap could travel as ASAPv1, and the processor refused to transmit the others.snapshot()is its ASAPv1 envelope andfrom_snapshot()decodes one. TheSketchtrait keeps onlymerge_snapshot(bytes); there are no deltas. Every sketch type can transmit, and CMS + heap is in the processor dispatch (countminsketch_heap). The OTAP carrier validates each payload with the decoder for its declared type, on the Gauge path too.Removed
asap-otap-demo,src/bin/demo,tests/otap_multiprocess_e2e.rs,benches/asap_sketch_pipeline.rs, and the file-based benchmark scripts (run-file-nightly.sh,run-pipeline-component.sh,run-scale-sweep.py,summarize-stages.py,benchmarks/Dockerfile,benchmarks/nightly/).Encodingenum and every proto/delta/legacy-msgpack path,snapshot_cache.rs,delta_transmission/delta_threshold/encodingconfig,ProtoSketchEnvelope,hash_spec,resolve_hash_seed, and KLL's proto replay history.adapter.rs,control_channel.rs, the flat-Arrowotap/{encode,records,decode}.rs,controller_url/agent_id, and the Go-parity name tests.Behaviour changes to know
Validation
cargo fmt --check,cargo clippy --all-targets --features otap-engine,stream-benchmark -- -D warningsandcargo test --features otap-engine,stream-benchmarkpass at this PR's head. The in-process e2e, runtime and KLL-window tests are rewritten to ASAPv1; tests that only tested deltas or proto formats are deleted.