Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 22 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -66,6 +66,14 @@ signals a backwards-compatible change.
worker's seed and the aggregator's from one seed. The unseeded constructors
draw from the thread generator. The seed is not serialized, and the ASAPv1
payload is unchanged.
- **ASAPv1 golden fixtures for CMSHeap** (`03 00`):
`cmsheap_i64_regular_2x3_strkeys`, `cmsheap_i32_fast_2x3_i64keys`, the
key-tie fixtures `cmsheap_i64_regular_2x3_i64tie` and
`cmsheap_i64_regular_2x3_strtie`, and the empty heap
`cmsheap_i64_regular_2x3_empty`, checked by `tests/asapv1_golden.rs`.
- The ASAPv1 spec states the heap key tie order: a signed key compares as its
two's-complement bit pattern read unsigned, a float by its bits, a string or
bytes key byte-wise with a proper prefix first.

### Changed

Expand All @@ -88,6 +96,20 @@ signals a backwards-compatible change.
as an empty window like any other, and that window takes a place in
`max_windows`.

### Removed

- **CMSHeap's pre-envelope MessagePack format.** `CMSHeap` serializes through
ASAPv1 only (`serialize_to_bytes` / `deserialize_from_bytes`). Removed: the
module `message_pack_format::portable::countminsketch_topk` with
`CountMinSketchWithHeap` (and its `MessagePackCodec` impl, `merge_refs` and
`aggregate_topk`), `CmsHeapItem`, `WireHeapItem`, `SketchlibCMSHeap`,
`CountMinSketchInnerWire`, `CountMinSketchWithHeapWire`,
`new_sketchlib_cms_heap`, `sketchlib_cms_heap_from_matrix_and_heap`,
`matrix_from_sketchlib_cms_heap`, `heap_to_wire`, `sketchlib_cms_heap_update`
and `sketchlib_cms_heap_query`, plus the crate-root re-exports
`CountMinSketchWithHeap` and `CmsHeapItem`. Bytes in that format no longer
decode.

### Fixed

- `KLLDynamic` panicked on every quantile query once a NaN had entered the
Expand Down
8 changes: 3 additions & 5 deletions docs/asapv1_wire_format.md
Original file line number Diff line number Diff line change
Expand Up @@ -562,7 +562,7 @@ Two summaries have no encoding and **fail to serialize** rather than being coerc

**Empty summary.** A summary that monitors nothing has no variant to report. It emits `key_type = "u64"` with three empty arrays, so an empty summary has exactly one encoding rather than one per producer.

**Emitted order (cross-language contract).** The payload is **order-defined**: entries are written in **descending `count`**, ties broken by a **total order over the key** (variant tag first, then the value). This is required, not cosmetic: `entries()` order follows the counter arena and `top_k` order follows the bucket walk, and neither survives a rebuild, so an unordered payload would re-serialize to different bytes than it decoded from. With the order pinned, two summaries holding the same triples emit the same bytes whatever order they were seated in, and re-serializing a decoded summary reproduces its bytes exactly.
**Emitted order (cross-language contract).** The payload is **order-defined**: entries are written in **descending `count`**, ties broken by a **total order over the key** (variant tag first, then the value). The value compares as unsigned bits: a signed key by its two's-complement pattern, so every negative key sorts after every non-negative one; a float by its IEEE-754 bits; a string or bytes key byte-wise, a proper prefix first. This is required, not cosmetic: `entries()` order follows the counter arena and `top_k` order follows the bucket walk, and neither survives a rebuild, so an unordered payload would re-serialize to different bytes than it decoded from. With the order pinned, two summaries holding the same triples emit the same bytes whatever order they were seated in, and re-serializing a decoded summary reproduces its bytes exactly.

**Decode rules.** Fail **closed** on each, with an error and never a panic:

Expand Down Expand Up @@ -642,7 +642,7 @@ Wire counter types are the base Count-Min's: **`"i32"`, `"i64"` and `"f64"`**. `

**`key_type` names the exact key variant, and is never widened.** This is §3.5's rule, and the fourteen names and their `HeapItem` variants are the table there. A heap whose keys are of different variants has no single `key_type` and **fails to serialize** with an error naming the mismatch; `HeapItem::I128` / `U128` have no msgpack integer form and are not wire types. An empty heap has no variant to report: it emits `key_type = "u64"` with two empty arrays, so an empty heap has exactly one encoding rather than one per producer.

**Emitted order (cross-language contract).** `counts` is row-major. Heap entries are written in **descending `heap_counts`**, ties broken by a **total order over the key**: the variant tag first, in `HeapItem`'s declaration order, so every string sorts after every numeric key; then the value. This is required, not cosmetic: the heap's in-memory array order follows the sift path taken while it filled, and that order does not survive a rebuild, so an unordered payload would re-serialize to different bytes than it decoded from. With the order pinned, two heaps holding the same entries emit the same bytes whatever order they were seated in. Go must mirror the same comparator.
**Emitted order (cross-language contract).** `counts` is row-major. Heap entries are written in **descending `heap_counts`**, ties broken by a **total order over the key**: the variant tag first, in `HeapItem`'s declaration order, so every string sorts after every numeric key; then the value, compared as §3.5 defines it. This is required, not cosmetic: the heap's in-memory array order follows the sift path taken while it filled, and that order does not survive a rebuild, so an unordered payload would re-serialize to different bytes than it decoded from. With the order pinned, two heaps holding the same entries emit the same bytes whatever order they were seated in. Go must mirror the same comparator.

**Decode rules.** Fail **closed** on each, with an error and never a panic:

Expand All @@ -659,8 +659,6 @@ Wire counter types are the base Count-Min's: **`"i32"`, `"i64"` and `"f64"`**. `

**A producer-side constraint: a `NaN` key.** `HHHeap` finds a resident by digest and then compares with `PartialEq`, and `NaN` never equals itself, so offering the same `NaN` key twice seats **two** entries rather than rescoring one. That state is reachable in memory and has **no encoding**: the encode-side duplicate check reads the two keys' bits, sees one key twice, and fails with the same complaint the decoder would make. Nothing repairs it on either door — a producer that inserts float keys must keep `NaN` out of a heap it intends to serialize.

**Not this: `portable/countminsketch_topk.rs`.** Despite the name, that module emits no ASAPv1 bytes. It is a legacy Go-interop struct with its own msgpack shape: an `f64` matrix rounded into the `i64` one on the way in, and heap keys coerced to `String` — a `HeapItem` that is not already a `String` goes through a Rust debug format, which is lossy and reversible by nothing. `0x03 0x00` shares none of that: its matrix travels at the metadata's `counter_type` and its keys travel as the exact `HeapItem` variant `key_type` names.

### 3.8: DDSketch payload (`0x05 0x00`)

`DDSketch` maps a positive value to the logarithmic bucket index `floor(ln(v) / ln(gamma))` and keeps a dense count per bucket. The relative accuracy `alpha` is its one construction parameter and lives in the metadata, so the payload is the bucket store plus the running scalars the buckets do not determine:
Expand Down Expand Up @@ -1341,7 +1339,7 @@ Good direction (more compact, higher fidelity, less Rust-internal duplication),
2. **Golden byte-vector fixtures** in one shared repo, [`sketchlib-golden-bytes`](https://github.com/ProjectASAP/sketchlib-golden-bytes), which each implementation mounts at `asapv1_golden/`; both languages decode and re-encode them byte-identically. These replace the `portable`-as-oracle round-trip test.
3. **This registry**, mirrored, never independently allocated.

Fixtures exist for six `kind_id`s — HLL's three estimators, Count-Min, Count Sketch and compact KLL — and `sketchlib-go` mirrors those. Every other kind here has a payload and **no fixture**, so this document is its only contract; `asapv1_golden/README.md` lists the gap. `portable` carries its own Go goldens.
`asapv1_golden/README.md` lists the `kind_id`s with fixtures, which `sketchlib-go` mirrors; for every other kind this document is the only contract. `portable` carries its own Go goldens.

**Hash profile on the Go side.**
Rust derives the hash spec from a generic `HashProfile` bound on the hasher type; Go has no generic hasher type, so there is nothing to derive from.
Expand Down
7 changes: 3 additions & 4 deletions docs/message_pack_format.md
Original file line number Diff line number Diff line change
Expand Up @@ -169,10 +169,9 @@ decode → re-encode them byte-identically. The `kind_id` registry is mirrored
verbatim with Go's `wire/asapmsgpack/magic_ids.go`, never independently
allocated.

That proof covers the six `kind_id`s a fixture exists for — HLL's three
estimators, Count-Min, Count Sketch and compact KLL. Every other implemented
kind has **no ASAPv1 golden and therefore no ASAPv1 drift guard**;
[its README](https://github.com/ProjectASAP/sketchlib-golden-bytes#coverage) lists what is covered.
That proof covers only the `kind_id`s a fixture exists for, listed in
[its README](https://github.com/ProjectASAP/sketchlib-golden-bytes#coverage);
a kind without one has **no ASAPv1 drift guard**.
The older `portable` path is guarded separately, by the `sketchlib-go` goldens
of its own listed above.

Expand Down
Loading
Loading