feat(asapv1)!: delta kinds, f64 Count Sketch and heap counts, one DDSketch encoding, SetAggregator

CHANGELOG.md

@@ -68,9 +68,36 @@ signals a backwards-compatible change.
   payload is unchanged.
 - **`peek_kind_id(&[u8])`** returns an ASAPv1 envelope's `kind_id` after
   checking its magic, version, and framing, without decoding the sketch.
+- **`CountMinDelta`**, the ASAPv1 Count-Min delta (`0x02 0x80`): built with
+  `new` from changed cells, serialized like a sketch, and added with
+  `CountMin::merge_delta`, which changes nothing when it fails.
+- **`sketches::CountSketchDelta`**, the ASAPv1 Count Sketch delta (`0x04 0x80`),
+  and `f64` Count Sketch counters (`counter_type` `"f64"`): the delta is built
+  with `new` from changed cells, serialized like a sketch, and added with
+  `Count::merge_delta`, which changes nothing when it fails.
+- **`CMSHeapDelta`**, the ASAPv1 CMSHeap delta (`0x03 0x80`), and `f64`
+  CMSHeap insert and merge, with unrounded `f64` heap counts.
+- **`CSHeapDelta`**, the ASAPv1 CSHeap delta (`0x0a 0x80`), and unrounded
+  `f64` heap counts for an `f64` CSHeap (`CsHeapCounter`).
+- **`HyperLogLogDelta`**, the ASAPv1 HLL Classic and Ertl-MLE deltas
+  (`0x01 0x81`, `0x01 0x82`): built with `new` from changed registers,
+  serialized like a sketch, and merged register-wise by `max` with
+  `HyperLogLogImpl::merge_delta`.
+- **ASAPv1 for `SetAggregator` (`0x08 0x00`) and `DeltaResult` (`0x08 0x80`)**:
+  `serialize_to_bytes` / `deserialize_from_bytes` on both types.
+- **`DDSketchDelta`**, the ASAPv1 DDSketch delta (`0x05 0x80`): built with
+  `new` from changed buckets of both stores and a zero-count increment,
+  serialized like a sketch, and added with `DDSketch::merge_delta`, which
+  changes nothing when it fails.
 
 ### Changed
 
+- **DDSketch has one ASAPv1 encoding**: `metadata_version` 1 with all eight
+  payload fields, the negative store and zero count included. Bytes change for
+  positive-only and empty sketches, and `metadata_version` 2 is rejected.
+- **The CMSHeap and CSHeap ASAPv1 metadata key `k` is renamed `heap_size`**,
+  and the CMSHeap, CSHeap and UnivMon decoders reject heap entries out of
+  emitted order. `HHItem` and `HHHeap` take a count type, `i64` by default.
 - **ASAPv1 decoders reject a block with bytes after its msgpack value, and a
   float field that is not msgpack float64** (an `f32` key: not float32).
 - **The ASAPv1 golden fixtures live in their own repo**,

docs/api/api_cms_heap.md

@@ -10,6 +10,10 @@ Count-Min with integrated heavy-hitter heap (`HHHeap`) for top-k tracking.
 
 - `CMSHeap<S = Vector2D<i64>, M = RegularPath, H = DefaultXxHasher>`
 
+The heap is an `HHHeap<C>` whose count type `C` is `S::Counter`'s
+`TopKCounter::HeapCount`: `f64` over `f64` counters, `i64` over `i32`, `i64`
+and `i128`. Each entry's count is the sketch's estimate, unrounded.
+
 ## Constructors
 
 ```rust
@@ -34,7 +38,7 @@ fn estimate(&self, key: &DataInput) -> S::Counter
 fn rows(&self) -> usize
 fn cols(&self) -> usize
 fn cms(&self) -> &CountMin<S, M, H>
-fn heap(&self) -> &HHHeap
+fn heap(&self) -> &HHHeap<C>
 ```
 
 ## Merge
@@ -59,23 +63,49 @@ on every `CMSHeap`: the impl exists only for wire-eligible configs
 (only you know if the mapping is lossless).
 
 A sketch travels as the base matrix plus the heap's entries: the metadata
-carries `rows`, `cols`, `counter_type`, `mode`, the heap capacity `k` and the
-heap's `key_type`, and the payload is `[counts, keys, heap_counts]`. The heap's
-digest index is rebuilt on load, so no index reaches the wire, and `k` never
-sizes an allocation on decode. The base matrix is bound by the same
-`1 <= rows <= 20` (`MATRIX_MAX_ROWS`, the seed list length) the stand-alone
-sketch is: past that, the regular path gives row `r` and row `r + 20` the same
-seed and identical counters, so a wider matrix is refused on both sides in
-either mode.
+carries `rows`, `cols`, `counter_type`, `mode`, the heap capacity `heap_size`
+and the heap's `key_type`, and the payload is `[counts, keys, heap_counts]`,
+with `heap_counts` `f64` under `counter_type` `"f64"` and `i64` otherwise. The
+heap's digest index is rebuilt on load, so no index reaches the wire, and
+`heap_size` never sizes an allocation on decode. The base matrix is bound by
+the same `1 <= rows <= 20` (`MATRIX_MAX_ROWS`, the seed list length) the
+stand-alone sketch is: past that, the regular path gives row `r` and row
+`r + 20` the same seed and identical counters, so a wider matrix is refused on
+both sides in either mode.
 
 Heap keys are `HeapItem`s, so the key type is a runtime property: `key_type`
 names the **exact** variant (`"i32"` stays `"i32"`, never widened to `"i64"`)
 and the `keys` array is homogeneous in it. A heap whose keys mix variants, or
 holds an `I128` / `U128` key, does not serialize. An empty heap emits
 `key_type = "u64"`.
 
-Entries are emitted in descending count, ties broken by a total order over the
-key, so a decoded sketch re-serializes byte-identically.
+Entries are emitted in descending count (`f64::total_cmp` for `f64` counts),
+ties broken by a total order over the key, so a decoded sketch re-serializes
+byte-identically. The decoder rejects entries in any other order.
+
+## Deltas
+
+```rust
+CMSHeapDelta<T = i64, Mode = RegularPath, H = DefaultXxHasher>
+
+fn new(rows: usize, cols: usize, indices: Vec<u64>, deltas: Vec<T>,
+       heap: HHHeap<T::HeapCount>) -> Result<Self, String>
+fn serialize_to_bytes(&self) -> Result<Vec<u8>, RmpEncodeError>
+fn deserialize_from_bytes(bytes: &[u8]) -> Result<Self, RmpDecodeError>
+
+// on CMSHeap<Vector2D<T>, Mode, H>
+fn merge_delta(&mut self, delta: &CMSHeapDelta<T, Mode, H>)
+    -> Result<(), String>
+```
+
+A `CMSHeapDelta` is the ASAPv1 delta kind `0x03 0x80`, for the same configs as
+the state serialization: the changed cells as `CountMinDelta`'s (strictly
+ascending row-major indices below `rows * cols`, each with a non-zero value)
+and the producer's whole heap, encoded as the state's heap is. `new` and the
+decoder refuse anything else. `merge_delta` adds the cells to the counters and
+replaces the heap with the delta's, its key type included; a delta of other
+dimensions or heap capacity, or an integer add that overflows, fails and leaves
+the sketch unchanged.
 
 ## Examples
 
@@ -91,11 +121,9 @@ assert!(sk.estimate(&DataInput::Str("flow")) >= 1);
 
 - Heap can be cleared independently of sketch counters.
 - Merge requires matching dimensions and compatible type parameters.
-- **Eight built-in instances are constructible but inert.** `insert`,
-  `insert_many`, `estimate` and `merge` live in an impl bounded on
-  `S::Counter: Copy + Ord + From<i32> + Into<i64> + AddAssign`. `i128` has no
-  `Into<i64>` and `f64` has neither `Ord` nor `Into<i64>`, so
-  `CMSHeap<Vector2D<i128>>`, `CMSHeap<Vector2D<f64>>`,
+- **Six built-in instances are constructible but inert.** `insert`,
+  `insert_many`, `estimate` and `merge` need `S::Counter` to convert into its
+  heap count. `i128` has no `Into<i64>`, so `CMSHeap<Vector2D<i128>>`,
   `CMSHeap<QuickMatrixI128>` and `CMSHeap<DefaultMatrixI128>` — on both hashing
   paths — allocate a sketch and a heap and can then do nothing but report their
   dimensions. Use `CSHeap` when you need a 128-bit counter beside a top-k heap;

docs/api/api_common.md

@@ -63,7 +63,7 @@ use asap_sketchlib::common::{
     CommonHeap, CommonHeapOrder, KeepSmallest, KeepLargest,
 
     // HH helper + utilities
-    HHHeap,
+    HHHeap, HeapCount, TopKCounter,
     Nitro, compute_median_inline_f64,
 
     // Precomputed assets

docs/api/api_common_heap.md

@@ -8,26 +8,30 @@ Common heavy-hitter heap helper used by multiple sketches/frameworks.
 
 ## Type/Struct
 
-- `HHHeap`
+- `HHHeap<C: HeapCount = i64>`
 
-A capacity-bounded min-heap of `HHItem` beside an index from key digest to heap
-position, so a keyed update is a lookup and a sift rather than a scan. The index
-is patched through each sift: `update` costs one hash, one map probe and
+`C` is the count type, `i64` or `f64`; `f64` counts are ranked by
+`f64::total_cmp`. `TopKCounter` maps a matrix counter type to the heap count
+type a top-k sketch keeps beside it.
+
+A capacity-bounded min-heap of `HHItem<C>` beside an index from key digest to
+heap position, so a keyed update is a lookup and a sift rather than a scan. The
+index is patched through each sift: `update` costs one hash, one map probe and
 `O(log k)` index writes, and its rate does not fall as `k` grows. See
 [HHHeap acceleration](../hhheap_acceleration.md) for the measurements.
 
 ## Constructors
 
 ```rust
 fn new(k: usize) -> Self
-fn from_heap(other: &HHHeap) -> Self
+fn from_heap(other: &HHHeap<C>) -> Self
 ```
 
 ## Insert/Update
 
 ```rust
-fn update(&mut self, key: &DataInput, count: i64) -> bool
-fn update_heap_item(&mut self, key: &HeapItem, count: i64) -> bool
+fn update(&mut self, key: &DataInput, count: C) -> bool
+fn update_heap_item(&mut self, key: &HeapItem, count: C) -> bool
 fn clear(&mut self)
 ```
 
@@ -36,7 +40,7 @@ fn clear(&mut self)
 ```rust
 fn find(&self, key: &DataInput) -> Option<usize>
 fn find_heap_item(&self, key: &HeapItem) -> Option<usize>
-fn heap(&self) -> &[HHItem]
+fn heap(&self) -> &[HHItem<C>]
 fn len(&self) -> usize
 fn is_empty(&self) -> bool
 fn capacity(&self) -> usize

docs/api/api_common_input.md

@@ -12,7 +12,7 @@ Canonical input and ownership model shared across sketches/frameworks.
 
 - `DataInput<'a>`
 - `HeapItem`
-- `HHItem`
+- `HHItem<C: HeapCount = i64>`
 - `HydraQuery<'a>`
 - `HydraCounter`
 - `L2HH`
@@ -25,10 +25,10 @@ Canonical input and ownership model shared across sketches/frameworks.
 fn heap_item_to_sketch_input(item: &HeapItem) -> DataInput<'_>
 fn input_to_owned(input: &DataInput<'_>) -> HeapItem
 
-// HHItem
-fn new(k: DataInput, count: i64) -> Self
-fn create_item(k: HeapItem, count: i64) -> Self
-fn init_item(k: DataInput, count: i64) -> Self
+// HHItem<C>
+fn new(k: DataInput, count: C) -> Self
+fn create_item(k: HeapItem, count: C) -> Self
+fn init_item(k: DataInput, count: C) -> Self
 ```
 
 ## Insert/Update

docs/api/api_count_sketch.md

@@ -68,11 +68,10 @@ fn deserialize_from_bytes(bytes: &[u8]) -> Result<Self, RmpDecodeError>
 These produce/consume the **ASAPv1** wire envelope (kind `0x04 0x00`) — see the
 [ASAPv1 wire format spec](../asapv1_wire_format.md). They are **not** available
 on every `Count`: the impl exists only for wire-eligible configs
-`Count<Vector2D<T>, Mode, H>` where `T` is `i32` or `i64` (`CsWireCounter`),
-`Mode` is `FastPath` or `RegularPath` (`CsWireMode`), and `H: HashProfile`.
-Count Sketch counters must be signed and negatable, so there is no `f64`
-counterpart to Count-Min's; an `i128` or non-`Vector2D` sketch must be converted
-first (only you know if the mapping is lossless).
+`Count<Vector2D<T>, Mode, H>` where `T` is `i32`, `i64` or `f64`
+(`CsWireCounter`), `Mode` is `FastPath` or `RegularPath` (`CsWireMode`), and
+`H: HashProfile`. An `i128` or non-`Vector2D` sketch must be converted first
+(only you know if the mapping is lossless).
 
 `i32` is **not** widened to `i64` on the wire. The counter type is carried in
 the metadata and pinned on decode, so `i32` bytes do not decode into an `i64`
@@ -95,6 +94,28 @@ accumulator per row. `l2` is carried rather than recomputed, because
 accumulator saturates one way. The same `(counts, l2)` pair is what a `UnivMon`
 layer inlines into its own payload.
 
+## Deltas
+
+```rust
+CountSketchDelta<T = i32, Mode = RegularPath, H = DefaultXxHasher>
+
+fn new(rows: usize, cols: usize, indices: Vec<u64>, deltas: Vec<T>) -> Result<Self, String>
+fn serialize_to_bytes(&self) -> Result<Vec<u8>, RmpEncodeError>
+fn deserialize_from_bytes(bytes: &[u8]) -> Result<Self, RmpDecodeError>
+
+// on Count<Vector2D<T>, Mode, H>
+fn merge_delta(&mut self, delta: &CountSketchDelta<T, Mode, H>) -> Result<(), String>
+```
+
+A `CountSketchDelta` is the ASAPv1 delta kind `0x04 0x80`, for the same configs
+as the state serialization: the changed cells as strictly ascending row-major
+indices `row * cols + col`, below `rows * cols`, and the non-zero signed value
+added to each. `new` and the decoder refuse anything else. `merge_delta` adds
+the delta to the counters; a delta of other dimensions, or an integer add that
+overflows, fails and leaves the sketch unchanged. The crate root's
+`CountSketchDelta` is the legacy portable type; this one is
+`asap_sketchlib::sketches::CountSketchDelta`.
+
 ## Examples
 
 ```rust

docs/api/api_countmin.md

@@ -73,6 +73,26 @@ list length): past that, the regular path gives row `r` and row `r + 20` the
 same seed and identical counters, so a wider matrix is refused on both sides in
 either mode.
 
+## Deltas
+
+```rust
+CountMinDelta<T = i32, Mode = RegularPath, H = DefaultXxHasher>
+
+fn new(rows: usize, cols: usize, indices: Vec<u64>, deltas: Vec<T>) -> Result<Self, String>
+fn serialize_to_bytes(&self) -> Result<Vec<u8>, RmpEncodeError>
+fn deserialize_from_bytes(bytes: &[u8]) -> Result<Self, RmpDecodeError>
+
+// on CountMin<Vector2D<T>, Mode, H>
+fn merge_delta(&mut self, delta: &CountMinDelta<T, Mode, H>) -> Result<(), String>
+```
+
+A `CountMinDelta` is the ASAPv1 delta kind `0x02 0x80`, for the same configs
+as the state serialization: the changed cells as strictly ascending row-major
+indices `row * cols + col`, below `rows * cols`, and the non-zero value added
+to each. `new` and the decoder refuse anything else. `merge_delta` adds the
+delta to the counters; a delta of other dimensions, or an integer add that
+overflows, fails and leaves the sketch unchanged.
+
 ## Examples
 
 ```rust