Skip to content

refactor(cmsheap)!: ASAPv1 only — drop the portable format, add goldens - #170

Closed
GordonYuanyc wants to merge 2 commits into
mainfrom
asapv1-only/cmsheap
Closed

GordonYuanyc wants to merge 2 commits into
mainfrom
asapv1-only/cmsheap

Conversation

@GordonYuanyc

Copy link
Copy Markdown
Collaborator

Part of the ASAPv1-only series: one PR per sketch group, each off main. This one removes the portable Count-Min-with-heap format; ASAPv1 (0x03 0x00) is the only CMSHeap format left, and it gets its first goldens.

Draft because of the heap tie-order question below.

Removed

  • message_pack_format::portable::countminsketch_topk: CountMinSketchWithHeap (incl. aggregate_topk, merge_refs, from_legacy_matrix, MessagePackCodec), CmsHeapItem, WireHeapItem, SketchlibCMSHeap, the sketchlib_cms_heap_* / new_sketchlib_cms_heap / matrix_from_sketchlib_cms_heap / heap_to_wire helpers, CountMinSketchInnerWire, CountMinSketchWithHeapWire; crate-root re-exports CountMinSketchWithHeap, CmsHeapItem.
  • CMSHeap had no proto message and no native shim.

Added

  • Five goldens (pushed to sketchlib-golden-bytes; submodule bumped), all built from fixed state, never hashed: cmsheap_i64_regular_2x3_strkeys, cmsheap_i32_fast_2x3_i64keys, cmsheap_i64_regular_2x3_i64tie, cmsheap_i64_regular_2x3_strtie, cmsheap_i64_regular_2x3_empty. The tie fixtures seat entries in two orders and check both encode to the same bytes.
  • Spec §3.5/§3.7 now state the key order inside a count tie (see below). The ASAPv1 docs no longer state a fixture count; they defer to the golden README.
  • The e2e "Count-Min bound through merge and wire" test now runs on native CMSHeap + ASAPv1.

Open question: heap key order within a count tie

Rust's key_order (shared by CMSHeap, CSHeap, Space-Saving, UnivMon) compares keys as unsigned bits: a signed key by its two's-complement pattern, so ties emit 0, 1, …, MAX, MIN, …, -1; floats by IEEE-754 bits; strings byte-wise. The spec did not say so, and a natural signed sort would have passed the old fixtures. This PR documents the current rule and pins it (…_i64tie, …_strtie) without changing the encoder. Alternative: switch to plain signed order now, before Go mirrors it and before more goldens lock it in.

Downstream

ASAPQuery-backend decodes/writes CMS-with-heap via CountMinSketchWithHeap::{from_msgpack, to_msgpack}; the equivalent native state is CMSHeap<Vector2D<i64>, RegularPath> with string keys. Behaviour differences for callers: CMSHeap::merge panics on a dimension mismatch (the portable type returned Err), and callers must round f64 weights themselves.

Verification (rustc 1.99)

cargo fmt --check; clippy -D warnings for all four CI feature sets; RUSTDOCFLAGS="-D warnings" cargo doc; gen-proto drift clean; filtered tests (asapv1_golden, msgpack_compat, e2e -- composition::, --lib -- countsketch_topk countminsketch_topk, pbt -- heaps::). The reviewer hand-decoded the goldens against spec §3.7. Implemented and reviewed by separate agents; review findings are addressed in the second commit.

🤖 Generated with Claude Code

GordonYuanyc and others added 2 commits October 2, 2026 00:50
CMSHeap serializes through ASAPv1 only. Removes
message_pack_format::portable::countminsketch_topk (CountMinSketchWithHeap,
CmsHeapItem, its wire DTOs and sketchlib_cms_heap_* helpers) and the
crate-root re-exports. The e2e merge-and-wire test runs on CMSHeap over
ASAPv1, and two CMSHeap golden fixtures are added (submodule bump).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The spec states the key comparison: signed keys by their two's-complement
bit pattern read unsigned, floats by their bits, strings and bytes
byte-wise with a proper prefix first. Three new golden fixtures pin i64
ties across zero and i64::MIN, string ties across case, length and
prefix, and the empty-heap form (submodule bump). The docs defer the
fixture list to the golden README.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@GordonYuanyc

Copy link
Copy Markdown
Collaborator Author

Outdated. ASAPv1 encodings and goldens landed in #232 and #235 against the current spec; removing the legacy formats is redone on current main in #206 and #207.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant