From 2b2678673b56dd4bd054b745377db24720bb1610 Mon Sep 17 00:00:00 2001 From: GordonYuanyc Date: Fri, 2 Oct 2026 00:40:28 -0600 Subject: [PATCH 01/36] refactor(kll)!: drop the proto and portable formats KLL and KLLDynamic serialize only as ASAPv1 (kind_ids 0x06 0x00 / 0x06 0x01). - proto: reserve SketchEnvelope field 13 ("kll"); KLLState and CoinState stay in kll.proto because hydra.proto's HydraCell still embeds them. - portable: remove KllSketch/KllSketchData from the crate root, the sketchlib_kll_* helpers, SketchlibKll, the value-offset codec and KllProtoItems; portable::kll becomes a crate-private module holding only the per-cell type HydraKllSketch uses. - native: remove the MessagePackCodec shims for KLL and KLLDynamic. - KLL::from_portable_state is removed. - Tests: the portable KLL PBT laws and the e2e merge-and-wire rank test now run on KLL through serialize_to_bytes; proto/portable-only tests go. Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 22 + docs/asapv1_wire_format.md | 4 +- docs/e2e_testing_harness.md | 3 +- docs/features.md | 2 +- docs/message_pack_format.md | 4 +- docs/tests.md | 33 +- examples/accuracy_probe.rs | 17 - proto/sketchlib.proto | 6 +- src/lib.rs | 7 +- src/message_pack_format/native/kll.rs | 20 - src/message_pack_format/native/kll_dynamic.rs | 21 - src/message_pack_format/native/mod.rs | 2 - src/message_pack_format/portable/kll.rs | 739 +----------------- src/message_pack_format/portable/mod.rs | 8 +- src/proto/generated/sketchlib.v1.rs | 6 +- src/sketches/kll.rs | 109 +-- src/sketches/kll/wire.rs | 7 +- tests/PBT_COVERAGE.md | 2 +- tests/asapv1_golden.rs | 4 +- tests/e2e/composition.rs | 47 +- tests/msgpack_compat.rs | 37 +- tests/pbt/kll.rs | 29 +- tests/xtest_consumer.rs | 111 --- 23 files changed, 99 insertions(+), 1141 deletions(-) delete mode 100644 src/message_pack_format/native/kll.rs delete mode 100644 src/message_pack_format/native/kll_dynamic.rs diff --git a/CHANGELOG.md b/CHANGELOG.md index 80d85f98..bc0a2911 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -88,6 +88,28 @@ signals a backwards-compatible change. as an empty window like any other, and that window takes a place in `max_windows`. +### Removed + +- **KLL's proto and portable formats.** `KLL` and `KLLDynamic` serialize only + as ASAPv1 (`serialize_to_bytes` / `deserialize_from_bytes`, kind_ids + `0x06 0x00` / `0x06 0x01`). Removed: + - the `kll` field (13) of `SketchEnvelope` (now reserved) and with it + `sketch_envelope::SketchState::Kll`; `KLLState` stays in + `proto/kll/kll.proto` only as `HydraCell.kll`; + - the crate-root `KllSketch` and `KllSketchData`, and the public module + `message_pack_format::portable::kll`: `SketchlibKll`, `new_sketchlib_kll`, + `new_sketchlib_kll_with_seed`, `sketchlib_kll_update`, + `sketchlib_kll_quantile`, `sketchlib_kll_merge`, + `bytes_from_sketchlib_kll`, `sketchlib_kll_from_bytes`, + `KllSketch::{merge_refs, aggregate_kll, from_portable_state}`, the + `MessagePackCodec` impls for `KllSketch` and `KllSketchData`, the + value-offset codec (`KLL_SCALE_SWEEP`, `encode_value_offset`, + `decode_value_offset`, `KllProtoItems`), and the `unsafe impl Send/Sync + for KllSketch`. `HydraKllSketch` keeps its per-cell type crate-internally; + - `message_pack_format::native::{kll, kll_dynamic}`, the `MessagePackCodec` + impls for `KLL` and `KLLDynamic`; + - `KLL::from_portable_state`. + ### Fixed - `KLLDynamic` panicked on every quantile query once a NaN had entered the diff --git a/docs/asapv1_wire_format.md b/docs/asapv1_wire_format.md index fd14e70a..c4586bb9 100644 --- a/docs/asapv1_wire_format.md +++ b/docs/asapv1_wire_format.md @@ -458,7 +458,7 @@ Both KLL variants (the compact fixed-buffer `KLL` and the growable `KLLDynamic`) `num_levels` is `levels.len() - 1` (derived, so not stored). The `coin` is the randomized-compaction RNG; it is carried so a decoded sketch can keep compacting deterministically (a query-only consumer may ignore it). It is a nested 3-element array mirroring `sketchlib-go`'s `CoinState`. -**Item order (cross-language contract).** `levels` / `items` use the **top-most-level-first** layout, byte-for-byte matching `sketchlib-go`'s `KLLState`: index `i` in `levels` maps to compactor level `num_levels - 1 - i`, and level 0's run is in **input order**. The compact `KLL` grows its buffer leftward and stores level 0 reverse-input, so its encoder reverses level 0 back to input order (and its decoder reverses it in); `KLLDynamic` already stores this layout natively. Within a level, order past the first compaction is not guaranteed byte-identical across the two Rust variants (or across languages), but the retained set and quantiles agree — see the caveat on `KLL::wire_items`. +**Item order (cross-language contract).** `levels` / `items` use the **top-most-level-first** layout, byte-for-byte matching `sketchlib-go`: index `i` in `levels` maps to compactor level `num_levels - 1 - i`, and level 0's run is in **input order**. The compact `KLL` grows its buffer leftward and stores level 0 reverse-input, so its encoder reverses level 0 back to input order (and its decoder reverses it in); `KLLDynamic` already stores this layout natively. Within a level, order past the first compaction is not guaranteed byte-identical across the two Rust variants (or across languages), but the retained set and quantiles agree — see the caveat on `KLL::wire_items`. **Wire widths.** `levels` is `u32` on the wire, widened to `usize` in memory. The coin's `remaining_bits` is `u32` on the wire but `u8` in memory, so rule 8 below is what makes the narrowing safe; `state` and `bit_cache` are `u64` on both sides. @@ -1360,7 +1360,7 @@ Retire `portable` once the fixtures cover the rest. - **Q-SEEDS**: `seed_list` is **inlined** in v1 so the bytes self-describe the hash. Resolving seeds from `hash_profile_id` alone is a v2 space optimization. Each sketch still carries only the seed *index* it uses. - **Q-PROFILE**: the hash-spec metadata is **derived from the hasher's `HashProfile`** (`hll_metadata::` / `cms_metadata::`) and never hardcoded, so it is always truthful to the hasher. Custom hash profiles are **supported and self-describing**. Merge compatibility is hash-spec equality, so a custom-profile sketch is not mergeable with a standard one. - **Q-CMS**: Count-Min is one `kind_id` (`0x02 0x00`); counter type and mode live in the metadata, so the id stays single. The counter-type domain is `"i32"`, `"i64"` and `"f64"`, with no `i128` (no msgpack integer form); `i32` is recorded at its own width and pinned on decode. -- **Q-KLL**: KLL metadata carries **no hash-spec group** — KLL is comparison-based and never hashes, so those fields have no truthful value. Its metadata is structural-only (`metadata_version`, `k`, `m`, `item_type`, optional `seed`) and is *not* `HashProfile`-derived. The two KLL variants (compact `0x06 0x00`, dynamic `0x06 0x01`) share one payload `[levels, items, coin]` and differ only by `kind_id`. `item_type` (`"f64"`/`"i64"`) is a metadata param, not a separate `kind_id` (mirrors Q-CMS's `counter_type`). Retained samples use the top-most-level-first layout that matches `sketchlib-go`'s `KLLState`. +- **Q-KLL**: KLL metadata carries **no hash-spec group** — KLL is comparison-based and never hashes, so those fields have no truthful value. Its metadata is structural-only (`metadata_version`, `k`, `m`, `item_type`, optional `seed`) and is *not* `HashProfile`-derived. The two KLL variants (compact `0x06 0x00`, dynamic `0x06 0x01`) share one payload `[levels, items, coin]` and differ only by `kind_id`. `item_type` (`"f64"`/`"i64"`) is a metadata param, not a separate `kind_id` (mirrors Q-CMS's `counter_type`). Retained samples use the top-most-level-first layout that matches `sketchlib-go`. - **Q-KLL-SEED**: KLL records its reproducible compaction `seed` as an **optional** metadata key. It is construction config (so metadata, not payload, per the config→metadata rule), and it is the first optional key in v1: present only when the sketch carries a seed, omitted otherwise. Rationale: the payload's `coin` already carries the RNG's *current* position (enough to resume compaction), but `seed` is what a later `clear()` re-seeds from — so serializing it lets a decoded sketch keep `clear()`-reproducibility instead of falling back to wall-clock. Cost of omitting it is bounded (only a decoded-then-`clear()`ed sketch loses cross-run byte reproducibility — never correctness), but it is cheap to carry and future-proofs the checkpoint/restore path. `KLLDynamic` has no seed concept and never emits the key; the two variants are deliberately **not** forced to be symmetric here. Go carries and preserves the key without interpreting it. - **Q-CS**: Count Sketch is one `kind_id` (`0x04 0x00`) and mirrors Count-Min's metadata and `[counts]` payload, including `counter_type`. Its type domain differs: `"i32"` and `"i64"`, with no `"f64"` — not because a float fails a bound, but because **no `CountSketchCounter` impl exists for it**: the trait covers `i32`, `i64` and `i128` and nothing else, so Count-Min's `f64` has no Count Sketch counterpart to serialize — and no `i128` (no msgpack integer form), which leaves the two `CsWireCounter` covers. Structural-param order matches Q-CMS: `... matrix_seed_index, rows, cols, counter_type, mode`. **`i32` is not widened to `i64`** — unlike Count-Min, whose counters are plain numbers, a Count Sketch appears *nested* inside `HydraCounter` and `EHSketchList` as `Vector2D` and must decode back into that variant, so the width is identity and is recorded exactly. Cells are signed, so a decoder must not assume monotonicity. - **Q-CMS-DIMS**: Count-Min `rows`/`cols` are **metadata** and the payload omits them. They are configuration that shapes the payload (like HLL's `precision`), so per the config-to-metadata rule they belong in the descriptor. The payload is then just `[counts]`. Canonical structural-param order: `... matrix_seed_index, rows, cols, counter_type, mode`. diff --git a/docs/e2e_testing_harness.md b/docs/e2e_testing_harness.md index 373f6040..e7848ca8 100644 --- a/docs/e2e_testing_harness.md +++ b/docs/e2e_testing_harness.md @@ -326,8 +326,7 @@ error against its bound. - No assertion depends on unseeded randomness — no `rand::rng()`, no wall-clock seeding, no implicit RNG inside a constructor. An unseeded sketch appears only where the assertion holds for every draw. `KLL::init_kll_with_seed`, - `KLLDynamic::init_kll_with_seed`, `KllSketch::with_seed`, - `NitroBatch::with_target_and_seed`, `UniformSampling::with_seed`, + `KLLDynamic::init_kll_with_seed`, `NitroBatch::with_target_and_seed`, `UniformSampling::with_seed`, `Coco::init_with_size_and_seed` and `CocoOctoPlan::with_seed` exist for this. - An estimator whose guarantee is unbiasedness gets no per-key band: its accuracy is the mean over independent seeds, within standard errors of the diff --git a/docs/features.md b/docs/features.md index 64f296a3..d3e38a8b 100644 --- a/docs/features.md +++ b/docs/features.md @@ -112,7 +112,7 @@ MessagePack (`rmp-serde`) support. **serde support** means the type derives `Ser | EHSketchList | Yes | Yes | | ExponentialHistogram | No | Yes | -Protobuf (prost): `.proto` definitions exist for CountMin, Count, HLL, DDSketch, KLL, Elastic, Coco, Hydra, and UnivMon. Rust conversion code is in progress. +Protobuf (prost): `.proto` definitions exist for CountMin, Count, HLL, DDSketch, Elastic, Coco, Hydra, and UnivMon. Rust conversion code is in progress. ### API Stability diff --git a/docs/message_pack_format.md b/docs/message_pack_format.md index 12a2b91c..56e2da00 100644 --- a/docs/message_pack_format.md +++ b/docs/message_pack_format.md @@ -150,8 +150,8 @@ being **phased out** in favor of the per-sketch `wire.rs` + the shared `envelope.rs`. - `portable/` holds per-sketch wire types that predate the envelope. They - carry their own byte-parity goldens against `sketchlib-go` — HLL, Count-Min, - Count Sketch and KLL (both directions) — separate from `asapv1_golden/`. + carry their own byte-parity goldens against `sketchlib-go` — HLL, Count-Min + and Count Sketch — separate from `asapv1_golden/`. - `native/` is a set of thin `MessagePackCodec` shims over the sketches' `serialize_to_bytes` / `deserialize_from_bytes`. Every type it wraps emits the ASAPv1 envelope, so each shim is a pass-through. diff --git a/docs/tests.md b/docs/tests.md index ccb1571c..ed4984b7 100644 --- a/docs/tests.md +++ b/docs/tests.md @@ -149,7 +149,6 @@ Wire tests: [`src/sketches/kll/wire.rs`](../src/sketches/kll/wire.rs) | `cdf_cached_matches_uncached_across_lifecycle` | The memoized CDF agrees with the from-scratch rebuild at every lifecycle stage. | For `KLL::::init_with_seed(200, 8, 42)`, verifies `quantile_cached` equals `quantile` while empty at `{0.1, 0.5, 0.9}`, at `{0.01, 0.1, 0.25, 0.5, 0.75, 0.9, 0.99}` after 20,000 uniform samples in `[0, 1000)` (seed `0xACAC_0001`), and at `{0.1, 0.5, 0.9}` after merging a seed-43 sketch over 5,000 samples in `[-5000, -4000)` (seed `0xACAC_0002`); after `clear`, `quantile_cached(0.5)` is `0.0` and `count` is `0`. | | `cdf_cache_invalidated_by_subsequent_updates` | Updates after a cached query invalidate the cache instead of serving a stale CDF. | Takes `quantile_cached(0.5)` on a `init_with_seed(200, 8, 42)` sketch over 10,000 uniform samples in `[0, 100)` (seed `0xACAC_0003`), then inserts `999_999.0` 10,000 times and verifies the cached median rises by more than `50.0`. | | `kllwire_serde_rejects_crafted_dimensions` | The nested serde decoder refuses an out-of-range `k` before it sizes anything. | Encodes a `KLLWire` with no items, `levels` of `[0, 0]`, `k` of `usize::MAX`, `m` of `8`, `num_levels` `1`, and `Coin::from_seed(1)`, and verifies it does not decode as a `KLL` rather than overflowing `compute_max_capacity`. | -| `from_portable_state_reproduces_source_exactly` | Reconstruction from portable state is bit-exact, not a lossy replay. | Extracts the compacted `items` and `levels` from a `KLL::::init_kll(200)` sketch over 200,000 updates of `i * 0.0007 + 3.0`, and verifies `from_portable_state` reproduces the source's quantiles at `{0, 0.01, 0.1, 0.25, 0.5, 0.75, 0.9, 0.99, 1}` exactly; an empty reconstruction is verified to report `num_levels` `1` with `levels[0]` at `max_capacity`. | | `seeded_sketches_are_byte_identical` | Same seed and same input give byte-identical sketches. | Feeds 5,000 uniform samples in `[0, 1_000_000)` (seed `7`) to two `init_kll_with_seed(200, 42)` sketches and verifies their serialized bytes are equal. | | `different_seeds_produce_different_bytes` | The seed reaches `toss` rather than sitting unused on the struct. | Feeds the same 5,000 uniform samples in `[0, 1_000_000)` (seed `11`) to `init_kll_with_seed(200, 1)` and `init_kll_with_seed(200, 2)` and verifies their serialized bytes differ. | | `clear_preserves_seed_determinism` | `clear` re-seeds from the stored seed, so determinism survives a window rotation. | Burns 1,500 unrelated samples (seed `99`) through a `init_kll_with_seed(200, 1234)` sketch, clears it, then feeds it and a fresh seed-1234 twin the same 3,000 samples (seed `13`) and verifies their serialized bytes are equal. | @@ -1371,28 +1370,6 @@ Test file: [`src/message_pack_format/portable/hydra_kll.rs`](../src/message_pack | `test_aggregate_hydrakll` | `aggregate_hydrakll` builds a decodable matrix from parallel key and value arrays. | For `aggregate_hydrakll(2, 5, 200, ["a", "b", "a"], [1.0, 2.0, 3.0])`, verifies the returned bytes decode and report `rows` `2` and `cols` `5`; no quantile is checked. | | `test_aggregate_hydrakll_empty` | An empty stream produces nothing. | Verifies `aggregate_hydrakll(2, 5, 200, &[], &[])` returns `None`. | -### Portable KllSketch - -Test file: [`src/message_pack_format/portable/kll.rs`](../src/message_pack_format/portable/kll.rs) - -| test_name | test_description | what_is_tested | -| --- | --- | --- | -| `kll_data_msgpack_matches_kll_sketch` | The DTO codec is byte-identical to the facade's. | For a `k` `200` sketch fed the 500 values `0.0..=499.0`, verifies `KllSketch::to_msgpack` equals `KllSketchData { k, sketch_bytes }::to_msgpack` and that the DTO round-trips with `k` and `sketch_bytes` unchanged. | -| `test_kll_creation` | A fresh sketch is empty and keeps its `k`. | For `KllSketch::new(200)`, verifies `count()` is `0` and `k` is `200`. | -| `test_kll_update` | Every insert is counted. | Feeds `10.0`, `20.0`, and `15.0` into a `k` `200` sketch and verifies `count()` is `3`. | -| `test_kll_quantile` | The extremes are exact and the median brackets the true one. | For a `k` `200` sketch fed `1.0..=10.0`, verifies `quantile(0.0)` is `1.0`, `quantile(1.0)` is `10.0`, and the median falls in `5.0..=6.0`. | -| `test_kll_merge` | Merge conserves the count and the extremes. | Merges a `k` `200` sketch fed `6.0..=10.0` into one fed `1.0..=5.0` and verifies `count()` is `10`, `quantile(0.0)` is `1.0`, and `quantile(1.0)` is `10.0`. | -| `test_msgpack_round_trip` | The round trip preserves `k`, the count, and the extremes. | Round-trips a `k` `200` sketch fed `1.0..=5.0` and verifies the decode reports `k` `200`, `count()` `5`, `quantile(0.0)` `1.0`, and `quantile(1.0)` `5.0`. | -| `test_aggregate_kll` | `aggregate_kll` builds a decodable sketch from a value array. | For `aggregate_kll(200, [1.0, 2.0, 3.0, 4.0, 5.0])`, verifies the returned bytes decode to a sketch with `count()` `5`, `quantile(0.0)` `1.0`, and `quantile(1.0)` `5.0`. | -| `test_aggregate_kll_empty` | An empty value array produces nothing. | Verifies `aggregate_kll(200, &[])` returns `None`. | -| `test_update_then_envelope_matches_sketchlib_go_bytes` | The raw-`f64` proto envelope is byte-identical to the golden. | For `KLL::init_kll_with_seed(200, 42)` fed `1.0..=50.0`, builds a `KllState` from the `wire_k`/`wire_m`/`wire_num_levels`/`wire_levels`/`wire_items` accessors plus the `CoinState` from `wire_coin()`, leaving `offset` `0.0`, `value_scale` `0`, and `residuals` empty, wraps it in a `SketchEnvelope` with `format_version` `1`, and verifies the prost encoding equals a 423-byte inline golden. | -| `value_offset_round_trip_exact_fixed_point` | The fixed-point form round-trips integer and fixed-decimal series bit-exactly. | Verifies the 200 values `1_000_000 + 7i` encode at `value_scale` `0` and decode back equal, and that the 128 values `100.0 + 0.001i` encode at a scale in `-3..=0` and decode back equal. | -| `value_offset_guard_rejects_irrational` | A value with no exact decimal scale falls back to raw `f64`. | Verifies `encode_value_offset` returns `None` for `[0.0, 1.0, PI, 3.0]`, for the empty slice, for `[1.0, NAN]`, and for `[1.0, INFINITY]`. | -| `proto_dual_read_round_trip_both_forms` | The decoder reads the raw-`f64` and the value-offset form to the same answer. | For `KLL::init_kll_with_seed(200, 7)` fed `1.0..=5000.0`, builds one `KllState` carrying `wire_items()` and one carrying the `encode_value_offset` triple with `items` empty, and verifies `KllProtoItems::from_state` yields identical items and identical quantiles at `0.0`, `0.1`, `0.25`, `0.5`, `0.75`, `0.9`, and `1.0`. | -| `proto_rejects_both_forms_populated` | A state carrying both forms at once is refused. | Verifies a `KllState` with `k` `200`, `m` `8`, `num_levels` `1`, `levels` `[0, 2]`, `items` `[1.0, 2.0]`, and `residuals` `[1, 2]` fails to decode. | -| `golden_value_offset_envelope_from_go` | The value-offset envelope the Go side emits decodes to the right samples. | Decodes an 80-byte golden and verifies the `KllState` leaves `items` empty, carries `value_scale` `0`, `offset` `1.0`, and 50 residuals, and that the dual-read recovers `1.0..=50.0` with `quantile(0.0)` `1.0` and `quantile(1.0)` `50.0`. | -| `golden_value_offset_envelope_for_go` | The value-offset envelope this encoder emits is pinned byte for byte. | For `1.0..=50.0`, verifies `encode_value_offset` gives `offset` `1.0` and `value_scale` `0`, that the `SketchEnvelope` around a `KllState` with `k` `200`, `m` `8`, `num_levels` `1`, `levels` `[0, 50]`, and no coin encodes to the 76-byte `RUST_GOLDEN_HEX`, and that the dual-read recovers `1.0..=50.0`. | - ### Portable Sampling Rescale Test file: [`src/message_pack_format/portable/sampling.rs`](../src/message_pack_format/portable/sampling.rs) @@ -1950,7 +1927,7 @@ Test file: [`tests/e2e/composition.rs`](../tests/e2e/composition.rs) | `univmonq_with_window_bound_chooses_a_hierarchy_that_satisfies_its_own_inequality` | `with_window_bound` picks the smallest level count whose deepest stratum still fits the candidate table. | For `max_updates` in `10_000`, `1_000_000` and `100_000_000` at `delta = 1e-3`, calls `UnivMonQConfig::default().with_window_bound` and re-derives the Bernstein upper tail `mean + sqrt(2*mean*ln(1/delta)) + (2/3)*ln(1/delta)` with `mean = max_updates / 2^(levels-1)`, asserting it is strictly below `cfg.candidates`; asserts minimality by requiring the same bound at `levels - 1` to be at or above `candidates` whenever `levels > 2`; asserts `levels` is non-decreasing across the three windows; builds each chosen config and asserts `count() == 1_000` after `1_000` updates; and asserts `with_window_bound(1_000, 1.5)` is an error. | | `univmonq_multi_shard_merge_with_distinct_source_ids_covers_the_union` | Shards with distinct source ids merge into a sketch whose exact aggregates cover the union. | Builds `4` shards with `UnivMonQ::with_hasher_and_source_id(UnivMonQConfig::default(), i + 1)`, deals `40_000` uniform values over `1_000_000` (seed `0xC090_5101`) round robin by `i % 4`, asserts each shard's `source_id()` is `i + 1`, merges the other three into the first with `merge` returning `Ok`, and asserts the merged `count()` is `40_000` and `min()`/`max()` equal the whole stream's exact extremes. Only the distinct-id path is exercised: no assertion covers a merge of shards that share a source id, and no estimator is checked. | -### Composition: Portable Sketch-With-Heap Facades +### Composition: Merge and Wire Round Trips Test file: [`tests/e2e/composition.rs`](../tests/e2e/composition.rs) @@ -1958,7 +1935,7 @@ Test file: [`tests/e2e/composition.rs`](../tests/e2e/composition.rs) | --- | --- | --- | | `portable_count_min_with_heap_satisfies_the_count_min_bound_through_merge_and_wire` | The portable Count-Min-plus-heap holds Count-Min's bound and an agreeing heap through a merge and a MessagePack round trip. | Feeds `40_000` Zipf(1.1) keys over a `2_048` domain (seed `0xC090_5101`) as `"k{key}"` strings into `CountMinSketchWithHeap::new(3, 4_096, 32)`, plus even/odd halves into two more. Runs `CountMinSpec::new(3, 4_096).assert_contract` on the single sketch, asserts every `topk_heap_items()` entry's `value` equals `estimate(&item.key)` with zero violations, re-runs the contract on `CountMinSketchWithHeap::merge_refs(&[&left, &right])` against the full truth, and after `to_msgpack`/`from_msgpack` asserts the decoded sketch's `estimate` is exactly equal to the original's for every observed key and still satisfies the contract. | | `portable_count_sketch_with_heap_satisfies_the_l2_bound_through_merge_and_wire` | The portable Count-Sketch-plus-heap holds the L2 bound and an agreeing heap through a merge and a MessagePack round trip. | Feeds the same `40_000`-key Zipf(1.1) stream (domain `2_048`, seed `0xC090_5101`) as `"k{key}"` strings into `CountSketchWithHeap::new(5, 4_096, 32)`, plus even/odd halves into two more, and runs `CountSketchSpec::new(5, 4_096).assert_contract` (simultaneous L2 band at `delta = 1e-3` with zero violations, marginal `sqrt(3/4096)*\|\|f_-i\|\|_2` rate pinned at `P[Bin(5, 1/3) >= 3] = 51/243`) on the single sketch, on `merge_refs(&[&left, &right])` and on the `from_msgpack` decode; every `topk_heap_items()` entry's `value` must equal `estimate(&item.key)`. Unlike the Count-Min twin, the wire round trip is only re-checked against the band - no per-key equality across the encode/decode is asserted. | -| `portable_kll_sketch_satisfies_the_rank_error_characterization_through_merge_and_wire` | The portable KLL facade holds the rank-error characterization single-pass and after a two-shard merge, and its wire round trip is bit-for-bit. | Over `12` compaction seeds `0x5EED_0400 + t * 0x9E37_79B9_7F4A_7C15`, builds `KllSketch::with_seed(200, seed)` plus even/odd shards at `seed ^ 0xAAAA` and `seed ^ 0x5555`, feeds all of them `40_000` uniform values over `1_000_000` (seed `0xC090_5101`), and merges the shards with `merge`. Each seed records two trials - the single-pass sketch and the merged pair - of the worst normalized rank error over the grid `[0.1, 0.25, 0.5, 0.75, 0.9]` against `KllRankSpec::datasketches(200)`'s `eps = 2.446/200^0.9433 = 0.0165`, and the `24` outcomes are accepted by `assert_independent_binomial` at `p = 0.01`. Per seed it also asserts `to_msgpack`/`from_msgpack` preserves `k() == 200`, `count()`, and every grid quantile exactly, and that merging a `k = 400` sketch into the single-pass one errs. | +| `kll_satisfies_the_rank_error_characterization_through_merge_and_wire` | KLL holds the rank-error characterization single-pass and after a two-shard merge, and its ASAPv1 round trip is bit-for-bit. | Over `12` compaction seeds `0x5EED_0400 + t * 0x9E37_79B9_7F4A_7C15`, builds `KLL::::init_kll_with_seed(200, seed)` plus even/odd shards at `seed ^ 0xAAAA` and `seed ^ 0x5555`, feeds all of them `40_000` uniform values over `1_000_000` (seed `0xC090_5101`), and merges the shards with `merge`. Each seed records two trials - the single-pass sketch and the merged pair - of the worst normalized rank error over the grid `[0.1, 0.25, 0.5, 0.75, 0.9]` against `KllRankSpec::datasketches(200)`'s `eps = 2.446/200^0.9433 = 0.0165`, and the `24` outcomes are accepted by `assert_independent_binomial` at `p = 0.01`. Per seed it also asserts `serialize_to_bytes`/`deserialize_from_bytes` preserves `k() == 200`, `count()`, and every grid quantile exactly. | ### Composition: Documented Input Matrix @@ -2151,9 +2128,7 @@ Test file: [`tests/msgpack_compat.rs`](../tests/msgpack_compat.rs) | `hll_sketch_round_trip` | A portable `HllSketch` survives a msgpack round trip with its register count intact. | Updates `HllSketch::new(HllVariant::Regular, 8)` with `b"a"`, `b"b"` and `b"c"`, then verifies only that the decode's `registers` has the same length as the source's; the register values, the precision and the variant are not checked. | | `hydra_kll_decodes_go_bytes` | The test asserts nothing. | The test asserts nothing: the body is empty and the test is marked `#[ignore]` with the reason that no cross-language msgpack fixture exists in the repository. | | `hydra_kll_sketch_round_trip` | A portable `HydraKllSketch` survives a msgpack round trip with its grid shape intact. | Updates `HydraKllSketch::with_seed(2, 4, 200, 0x5EED_0900)` with `("a", 1.0)`, `("a", 2.0)` and `("b", 3.0)`, then verifies the decode reports `rows` `2` and `cols` `4`; the nested KLL payloads are not checked. | -| `hydra_kll_wire_shape` | The `HydraKllSketchWire` DTO round-trips through `rmp_serde` with its two levels of nesting intact. | Serializes a wire struct of `rows: 2`, `cols: 3` and a `2x3` grid of `KllSketchData { k: 200, sketch_bytes: vec![] }`, then verifies the `rmp_serde` decode reports `rows` `2`, `cols` `3`, an outer length of `2` and an inner length of `3`. | -| `kll_sketch_decodes_go_bytes` | The test asserts nothing. | The test asserts nothing: the body is empty and the test is marked `#[ignore]` with the reason that no cross-language msgpack fixture exists in the repository. | -| `kll_sketch_round_trip` | A portable `KllSketch` survives a msgpack round trip with its `k` and retained mass intact. | Feeds `KllSketch::with_seed(200, 0x5EED_0800)` the values `0..100` as `f64`, then verifies the decode reports `k` `200` and `count()` `100`; the retained items and the quantile answers are not checked. | +| `hydra_kll_wire_shape` | The `HydraKllSketchWire` DTO decodes a `HydraKllSketch`'s msgpack with its two levels of nesting intact. | Encodes `HydraKllSketch::new(2, 3, 200)` with `to_msgpack`, decodes the bytes as a `HydraKllSketchWire` through `rmp_serde`, and verifies it reports `rows` `2`, `cols` `3`, an outer length of `2` and an inner length of `3`. | | `set_aggregator_round_trip` | A `SetAggregator` survives a msgpack round trip with its members. | Updates a `SetAggregator::new()` with `web` and `api`, then verifies the decode holds two values, one of which is `web`. | ### Conformance Kit @@ -2202,7 +2177,7 @@ Test file: [`tests/xtest_consumer.rs`](../tests/xtest_consumer.rs) | test_name | test_description | what_is_tested | | --- | --- | --- | -| `cross_language_proto` | Protobuf envelopes read from the cross-language fixture directory answer their point queries in band. | Asserts nothing unless fixtures are available: with no `XTEST_DIR` and no discoverable producer directory holding `tests/cross_language/xtest_producer_test.go`, it prints a skip line and returns. With fixtures it decodes nine `SketchEnvelope` files and records a pass or failure for each — `countmin.pb` min-frequency for `item:42` at least `101`, `kll.pb` p50 within 5% of `5000` and p99 within 5% of `9900`, `ddsketch.pb` the same two quantiles within 2%, `hll.pb` an ErtlMLE cardinality inside `[40_000, 65_000]`, `countsketch.pb` `cs:hot` at least `200`, `coco.pb` `coco:hot` at least `500`, `elastic.pb` `elephant` at least `900`, `univmon.pb` a cardinality inside `[1_000, 15_000]`, `hydra.pb` `hydra:42` at least `51` — using the hash seeds `0xcafe3553`, `0x6a09e667` and `0xbb67ae85`, and, when `countmin_sampled.pb` and `hll_sampled.pb` exist, requires `effective_sample_p` of `0.1` and a `rescale_count` by `1/p` within 5% of `100_000` and 6% of `200_000`. A single `assert!` at the end fails if any check failed; a missing or mistyped envelope panics on the spot. | +| `cross_language_proto` | Protobuf envelopes read from the cross-language fixture directory answer their point queries in band. | Asserts nothing unless fixtures are available: with no `XTEST_DIR` and no discoverable producer directory holding `tests/cross_language/xtest_producer_test.go`, it prints a skip line and returns. With fixtures it decodes eight `SketchEnvelope` files and records a pass or failure for each — `countmin.pb` min-frequency for `item:42` at least `101`, `ddsketch.pb` p50 within 2% of `5000` and p99 within 2% of `9900`, `hll.pb` an ErtlMLE cardinality inside `[40_000, 65_000]`, `countsketch.pb` `cs:hot` at least `200`, `coco.pb` `coco:hot` at least `500`, `elastic.pb` `elephant` at least `900`, `univmon.pb` a cardinality inside `[1_000, 15_000]`, `hydra.pb` `hydra:42` at least `51` — using the hash seeds `0xcafe3553`, `0x6a09e667` and `0xbb67ae85`, and, when `countmin_sampled.pb` and `hll_sampled.pb` exist, requires `effective_sample_p` of `0.1` and a `rescale_count` by `1/p` within 5% of `100_000` and 6% of `200_000`. A single `assert!` at the end fails if any check failed; a missing or mistyped envelope panics on the spot. | ### HLL Custom Precision diff --git a/examples/accuracy_probe.rs b/examples/accuracy_probe.rs index 643bfc3e..62b3249b 100644 --- a/examples/accuracy_probe.rs +++ b/examples/accuracy_probe.rs @@ -11,7 +11,6 @@ use asap_sketchlib::message_pack_format::portable::countminsketch::CountMinSketc use asap_sketchlib::message_pack_format::portable::ddsketch::DdSketch as PortableDds; use asap_sketchlib::message_pack_format::portable::hll::{HllSketch, HllVariant}; use asap_sketchlib::message_pack_format::portable::hydra_kll::HydraKllSketch; -use asap_sketchlib::message_pack_format::portable::kll::KllSketch as PortableKll; use asap_sketchlib::{ CMSHeap, CSHeap, Count as CoreCount, CountL2HH, CountMin, DDSketch, DataInput, Hydra, HyperLogLog, KLL, UnivMon, @@ -1058,22 +1057,6 @@ fn probe_portable_wire_types() { rel_err(est, 100_000.0) < 0.03, ); - // Portable KLL. - let mut pk = PortableKll::new(200); - let mut rng = StdRng::seed_from_u64(55); - let mut vals: Vec = (0..50_000).map(|_| rng.random::() * 1e6).collect(); - vals.sort_by(|a, b| a.partial_cmp(b).unwrap()); - for v in &vals { - pk.update(*v); - } - let med = pk.quantile(0.5); - let t = vals[25_000]; - p.check( - "portable KLL median", - format!("expected ~{t:.0}, got {med:.0}"), - (med - t).abs() / t < 0.05, - ); - // HydraKllSketch: per-key quantiles, median across rows. let mut hk = HydraKllSketch::new(3, 256, 200); let mut per_key: HashMap<&str, Vec> = HashMap::new(); diff --git a/proto/sketchlib.proto b/proto/sketchlib.proto index 0a6d8193..6c618327 100644 --- a/proto/sketchlib.proto +++ b/proto/sketchlib.proto @@ -32,7 +32,6 @@ import "common/common.proto"; import "countminsketch/countminsketch.proto"; import "countsketch/countsketch.proto"; import "hll/hll.proto"; -import "kll/kll.proto"; import "ddsketch/ddsketch.proto"; import "univmon/univmon.proto"; import "hydra/hydra.proto"; @@ -69,7 +68,6 @@ message SketchEnvelope { CountMinState count_min = 10; CountSketchState count_sketch = 11; HyperLogLogState hll = 12; - KLLState kll = 13; DDSketchState ddsketch = 14; UnivMonState univmon = 15; HydraState hydra = 16; @@ -77,6 +75,8 @@ message SketchEnvelope { ElasticState elastic = 18; } + reserved 13; + reserved "kll"; reserved 19 to 31; // Reserved for future sketch types. // Geometric / NitroSketch sampling probability for the producing sketch. @@ -86,7 +86,7 @@ message SketchEnvelope { // and stores the RAW SAMPLED sketch state — never the rescaled state. The // consumer applies the `× 1/sample_p` rescale at QUERY time on the count-like // estimators (HLL cardinality, CountMin / CountSketch frequency, SUM / COUNT); - // quantile estimators (KLL, DDSketch) are scale-invariant under uniform + // quantile estimators (DDSketch) are scale-invariant under uniform // sampling so they carry `sample_p` but need no rescale. // // Lives on the envelope (not per-sketch state) so downstream consumers that diff --git a/src/lib.rs b/src/lib.rs index b062b8b9..06303512 100644 --- a/src/lib.rs +++ b/src/lib.rs @@ -10,9 +10,9 @@ //! UnivMon, tumbling windows, and batch/parallel execution helpers. //! - [`message_pack_format`]: MessagePack/proto wire format shared with //! `sketchlib-go`. Owns the wire-DTO sketch types consumed by the ASAP -//! query engine (`CountMinSketch`, `CountSketch`, `KllSketch`, `HllSketch`, -//! `DdSketch`, `HydraKllSketch`, `CountMinSketchWithHeap`, -//! `SetAggregator`, `DeltaResult`). +//! query engine (`CountMinSketch`, `CountSketch`, `HllSketch`, `DdSketch`, +//! `HydraKllSketch`, `CountMinSketchWithHeap`, `SetAggregator`, +//! `DeltaResult`). //! //! Most users can start with the crate-root re-exports such as [`DataInput`], //! [`CountMin`], [`HyperLogLog`], [`KLL`], and [`DDSketch`]. Reach for the @@ -51,7 +51,6 @@ pub use message_pack_format::portable::ddsketch::{DDSKETCH_GROW_CHUNK, DdSketch, pub use message_pack_format::portable::delta_set_aggregator::DeltaResult; pub use message_pack_format::portable::hll::{HllSketch, HllSketchDelta, HllVariant}; pub use message_pack_format::portable::hydra_kll::HydraKllSketch; -pub use message_pack_format::portable::kll::{KllSketch, KllSketchData}; pub use message_pack_format::portable::sampling::{ effective_sample_p, is_quantile_scale_invariant, rescale_count, rescale_count_with_env, sample_p_or_default, diff --git a/src/message_pack_format/native/kll.rs b/src/message_pack_format/native/kll.rs deleted file mode 100644 index e56487b7..00000000 --- a/src/message_pack_format/native/kll.rs +++ /dev/null @@ -1,20 +0,0 @@ -//! Native MessagePack codec impl for [`crate::sketches::kll::KLL`]. - -use serde::{Deserialize, Serialize}; - -use crate::common::numerical::NumericalValue; -use crate::message_pack_format::{Error, MessagePackCodec}; -use crate::sketches::kll::{KLL, KllWireItem}; - -impl MessagePackCodec for KLL -where - T: NumericalValue + KllWireItem + Serialize + for<'de> Deserialize<'de>, -{ - fn to_msgpack(&self) -> Result, Error> { - Ok(self.serialize_to_bytes()?) - } - - fn from_msgpack(bytes: &[u8]) -> Result { - Ok(Self::deserialize_from_bytes(bytes)?) - } -} diff --git a/src/message_pack_format/native/kll_dynamic.rs b/src/message_pack_format/native/kll_dynamic.rs deleted file mode 100644 index 7818a494..00000000 --- a/src/message_pack_format/native/kll_dynamic.rs +++ /dev/null @@ -1,21 +0,0 @@ -//! Native MessagePack codec impl for [`crate::sketches::kll_dynamic::KLLDynamic`]. - -use serde::{Deserialize, Serialize}; - -use crate::common::numerical::NumericalValue; -use crate::message_pack_format::{Error, MessagePackCodec}; -use crate::sketches::kll::KllWireItem; -use crate::sketches::kll_dynamic::KLLDynamic; - -impl MessagePackCodec for KLLDynamic -where - T: NumericalValue + KllWireItem + Serialize + for<'de> Deserialize<'de>, -{ - fn to_msgpack(&self) -> Result, Error> { - Ok(self.serialize_to_bytes()?) - } - - fn from_msgpack(bytes: &[u8]) -> Result { - Ok(Self::deserialize_from_bytes(bytes)?) - } -} diff --git a/src/message_pack_format/native/mod.rs b/src/message_pack_format/native/mod.rs index 0a83ef96..ad4d083b 100644 --- a/src/message_pack_format/native/mod.rs +++ b/src/message_pack_format/native/mod.rs @@ -12,7 +12,5 @@ pub mod countsketch; pub mod countsketch_topk; pub mod ddsketch; pub mod hll; -pub mod kll; -pub mod kll_dynamic; #[cfg(feature = "experimental")] pub mod kmv; diff --git a/src/message_pack_format/portable/kll.rs b/src/message_pack_format/portable/kll.rs index 11fa2ea2..96216b33 100644 --- a/src/message_pack_format/portable/kll.rs +++ b/src/message_pack_format/portable/kll.rs @@ -1,97 +1,39 @@ -//! Wire-format-aligned KLL sketch type for cross-language interop. -//! -//! `KllSketch` is a thin facade over the pure-Rust [`crate::sketches::kll::KLL`] -//! (parity already locked in by the existing -//! `test_update_then_envelope_matches_sketchlib_go_bytes` in this file -//! against `sketchlib-go::KLLSketch.SerializePortable`). The wire shape -//! also serves as a nested field for -//! [`crate::message_pack_format::portable::hydra_kll`]. +//! The per-cell KLL type of [`crate::message_pack_format::portable::hydra_kll`]. use serde::{Deserialize, Serialize}; -use crate::message_pack_format::{Error as MsgPackError, MessagePackCodec}; use crate::sketches::kll::KLL; -/// Concrete KLL type backing the wire-format `KllSketch`. -pub type SketchlibKll = KLL; - -/// Creates a fresh sketchlib KLL sketch with the requested accuracy -/// parameter `k`. -pub fn new_sketchlib_kll(k: u16) -> SketchlibKll { - KLL::init_kll(k as i32) -} - -/// [`new_sketchlib_kll`] with an explicit compaction-coin seed, so the sketch -/// replays identically across runs. -pub fn new_sketchlib_kll_with_seed(k: u16, seed: u64) -> SketchlibKll { - KLL::init_kll_with_seed(k as i32, seed) -} - -/// Updates a sketchlib KLL with one numeric observation. -pub fn sketchlib_kll_update(inner: &mut SketchlibKll, value: f64) { - inner.update(&value); -} - -/// Queries a sketchlib KLL for the value at the requested quantile. -pub fn sketchlib_kll_quantile(inner: &SketchlibKll, q: f64) -> f64 { - inner.quantile(q) -} - -/// Merges `src` into `dst`. -pub fn sketchlib_kll_merge(dst: &mut SketchlibKll, src: &SketchlibKll) { - dst.merge(src); -} - -/// Serializes a sketchlib KLL into MessagePack bytes. -pub fn bytes_from_sketchlib_kll(inner: &SketchlibKll) -> Vec { - inner.serialize_to_bytes().unwrap() -} - -/// Deserializes a sketchlib KLL from MessagePack bytes. -pub fn sketchlib_kll_from_bytes(bytes: &[u8]) -> Result> { - Ok(KLL::deserialize_from_bytes(bytes)?) -} - -/// Wire-format KLL sketch — a thin facade over [`SketchlibKll`] with a -/// fixed `k` parameter and msgpack codec via [`KllSketchData`]. +/// One `HydraKllSketch` cell: a `KLL` with its `k`. +#[derive(Clone)] pub struct KllSketch { pub k: u16, - pub(crate) backend: SketchlibKll, + pub(crate) backend: KLL, } impl KllSketch { pub fn new(k: u16) -> Self { Self { k, - backend: new_sketchlib_kll(k), + backend: KLL::init_kll(k as i32), } } /// [`KllSketch::new`] with an explicit compaction-coin seed. - /// - /// `new` seeds the coin from the wall clock, so two runs over the same - /// input produce different sketches. Pass a seed when the result has to be - /// reproducible — replaying a stream, comparing two processes, or - /// asserting an accuracy bound in a test. pub fn with_seed(k: u16, seed: u64) -> Self { Self { k, - backend: new_sketchlib_kll_with_seed(k, seed), + backend: KLL::init_kll_with_seed(k as i32, seed), } } - /// The configured `k` parameter (compactor capacity). - pub fn k(&self) -> u16 { - self.k - } - - /// Returns the raw sketch bytes (for JSON serialization, etc.). + /// The cell's ASAPv1 KLL bytes. pub fn sketch_bytes(&self) -> Vec { - bytes_from_sketchlib_kll(&self.backend) + self.backend.serialize_to_bytes().unwrap() } pub fn update(&mut self, value: f64) { - sketchlib_kll_update(&mut self.backend, value); + self.backend.update(&value); } pub fn count(&self) -> u64 { @@ -103,11 +45,11 @@ impl KllSketch { if self.count() == 0 { return 0.0; } - sketchlib_kll_quantile(&self.backend, q) + self.backend.quantile(q) } - /// Merge another KllSketch into self in place. Both operands must - /// have identical `k`. + /// Merge another cell into self in place. Both operands must have + /// identical `k`. pub fn merge( &mut self, other: &KllSketch, @@ -115,71 +57,9 @@ impl KllSketch { if self.k != other.k { return Err(format!("KllSketch k mismatch: self={}, other={}", self.k, other.k).into()); } - sketchlib_kll_merge(&mut self.backend, &other.backend); + self.backend.merge(&other.backend); Ok(()) } - - /// Merge from references without cloning. - pub fn merge_refs( - sketches: &[&Self], - ) -> Result> { - if sketches.is_empty() { - return Err("No sketches to merge".into()); - } - let k = sketches[0].k; - for s in sketches { - if s.k != k { - return Err("Cannot merge KllSketch with different k values".into()); - } - } - let mut merged = Self::new(k); - for s in sketches { - sketchlib_kll_merge(&mut merged.backend, &s.backend); - } - Ok(merged) - } - - /// One-shot aggregation: build a sketch from a slice of values. - pub fn aggregate_kll(k: u16, values: &[f64]) -> Option> { - if values.is_empty() { - return None; - } - let mut sketch = Self::new(k); - for &value in values { - sketch.update(value); - } - sketch.to_msgpack().ok() - } -} - -impl Clone for KllSketch { - fn clone(&self) -> Self { - // Structural clone: the backing `KLL` derives `Clone` (its fields - // are `Box<[..]>`/`Vec`/scalars), so copy it directly instead of a full - // msgpack serialize -> deserialize -> rebuild round-trip. Produces an - // identical sketch far more cheaply (no rmp encode/decode, far fewer - // allocations). - Self { - k: self.k, - backend: self.backend.clone(), - } - } -} - -impl KllSketch { - /// Reconstruct directly from portable wire state (k + level-ordered items + - /// level boundaries) without replaying items through `update()`. Bit-exact. - pub fn from_portable_state( - k: u16, - items: &[f64], - levels: &[usize], - num_levels: usize, - ) -> Result { - Ok(Self { - k, - backend: SketchlibKll::from_portable_state(k as usize, items, levels, num_levels)?, - }) - } } impl std::fmt::Debug for KllSketch { @@ -191,601 +71,10 @@ impl std::fmt::Debug for KllSketch { } } -// Thread safety: sketchlib's KLL is used in a single-threaded context per -// accumulator instance and only shares read-only operations across threads. -unsafe impl Send for KllSketch {} -unsafe impl Sync for KllSketch {} - -// ----- Value-offset fixed-point representation (cross-language) ----- -// -// KLL `KLLState` (proto/kll/kll.proto) carries the retained samples in one of -// two forms: -// -// * RAW F64 — `items` (field 5), 8 bytes/sample. The original v1 form. -// * VALUE-OFFSET — `offset` (7) + `value_scale` (8) + `residuals` (9), -// where `value = residual * 10^value_scale + offset`. -// -// PR2 makes BOTH languages read both forms (this is the decode-first half of -// the rollout). `sketchlib-go` gates emission of the value-offset form behind -// an exactness guard; this crate currently only reads it but the -// encode/decode helpers below are symmetric so the Rust side can produce it -// too (used by the cross-language golden test and any future Rust producer). - -/// Per-sketch scale exponents tried by [`encode_value_offset`], in priority -/// order. Mirrors `kllScaleSweep` in `sketchlib-go`. -pub const KLL_SCALE_SWEEP: [i32; 7] = [0, -1, -2, -3, -4, -5, -6]; - -/// Reconstructs retained samples from the value-offset fixed-point fields: -/// `value = residual * 10^value_scale + offset`. Caller invokes this only when -/// `residuals` is non-empty. -pub fn decode_value_offset(offset: f64, value_scale: i32, residuals: &[i64]) -> Vec { - let mul = 10f64.powi(value_scale); - residuals.iter().map(|&r| r as f64 * mul + offset).collect() -} - -/// Attempts to represent `items` exactly as `residual * 10^scale + offset` -/// using the finite [`KLL_SCALE_SWEEP`]. `offset` is the minimum value. Returns -/// `None` (the exactness-guard fallback) when no candidate scale round-trips -/// every sample bit-exactly, or when the slice is empty / contains a non-finite -/// value. Symmetric with `sketchlib-go::encodeValueOffset`. -pub fn encode_value_offset(items: &[f64]) -> Option<(f64, i32, Vec)> { - if items.is_empty() { - return None; - } - let mut offset = items[0]; - for &v in items { - if !v.is_finite() { - return None; - } - if v < offset { - offset = v; - } - } - for &sc in &KLL_SCALE_SWEEP { - let mul = 10f64.powi(-sc); - let mut out = Vec::with_capacity(items.len()); - let mut good = true; - for &v in items { - let r = ((v - offset) * mul).round(); - if !r.is_finite() || r > i64::MAX as f64 || r < i64::MIN as f64 { - good = false; - break; - } - let ri = r as i64; - // Exactness guard: reconstruct and require bit-identical recovery. - let recon = ri as f64 * 10f64.powi(sc) + offset; - if recon != v { - good = false; - break; - } - out.push(ri); - } - if good { - return Some((offset, sc, out)); - } - } - None -} - -/// Materialized retained samples + level layout decoded from a proto -/// `KLLState`, dual-reading the raw-f64 and value-offset forms. -#[derive(Debug, Clone)] -pub struct KllProtoItems { - /// Retained samples in level order (length == `levels[num_levels]`). - pub items: Vec, - /// Level boundary indices, length == `num_levels + 1`. - pub levels: Vec, - pub num_levels: usize, - pub k: u32, - pub m: u32, -} - -impl KllProtoItems { - /// Decodes a proto `KLLState`, dual-reading: when `residuals` is non-empty - /// it reconstructs from (offset, value_scale, residuals); otherwise it - /// reads raw f64 `items`. Returns an error if both forms are populated or - /// the level layout is inconsistent. - pub fn from_state( - s: &crate::proto::sketchlib::KllState, - ) -> Result> { - let items = if !s.residuals.is_empty() { - if !s.items.is_empty() { - return Err("kll: both residuals and raw items[] set".into()); - } - decode_value_offset(s.offset, s.value_scale, &s.residuals) - } else { - s.items.clone() - }; - let levels: Vec = s.levels.iter().map(|&v| v as usize).collect(); - if levels.len() < 2 { - return Err(format!("kll: invalid levels length {}", levels.len()).into()); - } - if *levels.last().unwrap() != items.len() { - return Err("kll: invalid item layout".into()); - } - Ok(Self { - items, - levels, - num_levels: s.num_levels as usize, - k: s.k, - m: s.m, - }) - } - - /// Weighted (value, weight) pairs across all levels. - pub fn weighted_samples(&self) -> Vec<(f64, u64)> { - let mut out = Vec::with_capacity(self.items.len()); - for h in 0..self.num_levels { - let weight: u64 = 1 << h; - let idx = self.num_levels - 1 - h; - if idx + 1 >= self.levels.len() { - continue; - } - let start = self.levels[idx]; - let end = self.levels[idx + 1]; - for &v in &self.items[start..end] { - out.push((v, weight)); - } - } - out - } - - /// Estimated value at quantile `q ∈ [0, 1]` from the weighted samples. - pub fn quantile(&self, q: f64) -> f64 { - let mut samples = self.weighted_samples(); - if samples.is_empty() { - return 0.0; - } - samples.sort_by(|a, b| a.0.partial_cmp(&b.0).unwrap()); - let total: u64 = samples.iter().map(|(_, w)| w).sum(); - let target = (q * total as f64).ceil() as u64; - let mut acc = 0u64; - for (v, w) in &samples { - acc += w; - if acc >= target { - return *v; - } - } - samples.last().unwrap().0 - } -} - -// ----- Wire format ----- - -/// Wire DTO for [`KllSketch`]. Also referenced as a nested field by +/// Wire DTO for one cell of /// [`crate::message_pack_format::portable::hydra_kll::HydraKllSketchWire`]. #[derive(Debug, Clone, Serialize, Deserialize)] pub struct KllSketchData { pub k: u16, pub sketch_bytes: Vec, } - -impl MessagePackCodec for KllSketch { - fn to_msgpack(&self) -> Result, MsgPackError> { - let wire = KllSketchData { - k: self.k, - sketch_bytes: self.sketch_bytes(), - }; - Ok(rmp_serde::to_vec(&wire)?) - } - - fn from_msgpack(bytes: &[u8]) -> Result { - let wire: KllSketchData = rmp_serde::from_slice(bytes)?; - let backend = KLL::deserialize_from_bytes(&wire.sketch_bytes)?; - Ok(Self { k: wire.k, backend }) - } -} - -impl MessagePackCodec for KllSketchData { - /// Encodes the `(k, sketch_bytes)` DTO directly, byte-identical to - /// [`KllSketch::to_msgpack`] (which serializes this same struct). Lets - /// producers that hold a raw sketchlib KLL backend (via - /// `serialize_to_bytes`) emit the portable KLL msgpack form without - /// constructing a [`KllSketch`] facade. - fn to_msgpack(&self) -> Result, MsgPackError> { - Ok(rmp_serde::to_vec(self)?) - } - - fn from_msgpack(bytes: &[u8]) -> Result { - Ok(rmp_serde::from_slice(bytes)?) - } -} - -#[cfg(test)] -mod tests { - use super::*; - - #[test] - fn kll_data_msgpack_matches_kll_sketch() { - let mut sk = KllSketch::new(200); - for i in 0..500 { - sk.update(i as f64); - } - let via_sketch = sk.to_msgpack().unwrap(); - let data = KllSketchData { - k: sk.k(), - sketch_bytes: sk.sketch_bytes(), - }; - let via_data = data.to_msgpack().unwrap(); - assert_eq!( - via_sketch, via_data, - "KllSketchData codec must byte-match KllSketch" - ); - let rt = KllSketchData::from_msgpack(&via_data).unwrap(); - assert_eq!(rt.k, data.k); - assert_eq!(rt.sketch_bytes, data.sketch_bytes); - } - - #[test] - fn test_kll_creation() { - let kll = KllSketch::new(200); - assert_eq!(kll.count(), 0); - assert_eq!(kll.k, 200); - } - - #[test] - fn test_kll_update() { - let mut kll = KllSketch::new(200); - kll.update(10.0); - kll.update(20.0); - kll.update(15.0); - assert_eq!(kll.count(), 3); - } - - #[test] - fn test_kll_quantile() { - let mut kll = KllSketch::new(200); - for i in 1..=10 { - kll.update(i as f64); - } - assert_eq!(kll.quantile(0.0), 1.0); - assert_eq!(kll.quantile(1.0), 10.0); - let median = kll.quantile(0.5); - assert!( - (5.0..=6.0).contains(&median), - "median should be between 5 and 6; got {median}" - ); - } - - #[test] - fn test_kll_merge() { - let mut kll1 = KllSketch::new(200); - let mut kll2 = KllSketch::new(200); - - for i in 1..=5 { - kll1.update(i as f64); - } - for i in 6..=10 { - kll2.update(i as f64); - } - - kll1.merge(&kll2).unwrap(); - assert_eq!(kll1.count(), 10); - assert_eq!(kll1.quantile(0.0), 1.0); - assert_eq!(kll1.quantile(1.0), 10.0); - } - - #[test] - fn test_msgpack_round_trip() { - let mut kll = KllSketch::new(200); - for i in 1..=5 { - kll.update(i as f64); - } - - let bytes = kll.to_msgpack().unwrap(); - let deserialized = KllSketch::from_msgpack(&bytes).unwrap(); - - assert_eq!(deserialized.k, 200); - assert_eq!(deserialized.count(), 5); - assert_eq!(deserialized.quantile(0.0), 1.0); - assert_eq!(deserialized.quantile(1.0), 5.0); - } - - #[test] - fn test_aggregate_kll() { - let values = [1.0, 2.0, 3.0, 4.0, 5.0]; - let bytes = KllSketch::aggregate_kll(200, &values).unwrap(); - let kll = KllSketch::from_msgpack(&bytes).unwrap(); - assert_eq!(kll.count(), 5); - assert_eq!(kll.quantile(0.0), 1.0); - assert_eq!(kll.quantile(1.0), 5.0); - } - - #[test] - fn test_aggregate_kll_empty() { - assert!(KllSketch::aggregate_kll(200, &[]).is_none()); - } - - /// Cross-language byte-parity guard against `sketchlib-go`'s - /// `KLLSketch.SerializePortable` output for the deterministic - /// input `(1..=50)` with `seed=42` and `k=200`. - #[test] - fn test_update_then_envelope_matches_sketchlib_go_bytes() { - use crate::proto::sketchlib::{ - CoinState, KllState, SketchEnvelope, sketch_envelope::SketchState, - }; - use prost::Message; - - let mut sk: KLL = KLL::init_kll_with_seed(200, 42); - for i in 1..=50 { - sk.update(&(i as f64)); - } - - let (state, bit_cache, remaining_bits) = sk.wire_coin(); - let kll_state = KllState { - k: sk.wire_k(), - m: sk.wire_m(), - num_levels: sk.wire_num_levels(), - levels: sk.wire_levels(), - // This golden pins the RAW-F64 wire form; the value-offset fields - // stay at their proto defaults so the bytes are unchanged. - items: sk.wire_items(), - coin: Some(CoinState { - state, - bit_cache, - remaining_bits, - }), - offset: 0.0, - value_scale: 0, - residuals: vec![], - }; - let envelope = SketchEnvelope { - format_version: 1, - producer: None, - hash_spec: None, - sample_p: 0.0, - sketch_state: Some(SketchState::Kll(kll_state)), - }; - let mut got = Vec::with_capacity(envelope.encoded_len()); - envelope.encode(&mut got).expect("prost encode"); - - const GOLDEN_HEX: &str = "08016aa20308c80110081801220200322a9003000000000000f03f000000000000004000000000000008400000000000001040000000000000144000000000000018400000000000001c40000000000000204000000000000022400000000000002440000000000000264000000000000028400000000000002a400000000000002c400000000000002e4000000000000030400000000000003140000000000000324000000000000033400000000000003440000000000000354000000000000036400000000000003740000000000000384000000000000039400000000000003a400000000000003b400000000000003c400000000000003d400000000000003e400000000000003f4000000000000040400000000000804040000000000000414000000000008041400000000000004240000000000080424000000000000043400000000000804340000000000000444000000000008044400000000000004540000000000080454000000000000046400000000000804640000000000000474000000000008047400000000000004840000000000080484000000000000049403202082a"; - let want = decode_hex(GOLDEN_HEX); - assert_eq!( - got, - want, - "KLL envelope bytes diverge from sketchlib-go golden \ - ({} bytes got vs {} bytes want)", - got.len(), - want.len(), - ); - } - - fn decode_hex(s: &str) -> Vec { - s.as_bytes() - .chunks(2) - .map(|pair| { - let high = hex_nibble(pair[0]); - let low = hex_nibble(pair[1]); - (high << 4) | low - }) - .collect() - } - - fn hex_nibble(c: u8) -> u8 { - match c { - b'0'..=b'9' => c - b'0', - b'a'..=b'f' => c - b'a' + 10, - b'A'..=b'F' => c - b'A' + 10, - _ => panic!("non-hex byte {}", c as char), - } - } - - // ----- Value-offset fixed-point representation ----- - - use crate::proto::sketchlib::{KllState, SketchEnvelope, sketch_envelope::SketchState}; - use prost::Message; - - /// Round-trip exactness for the value-offset (fixed-point) form: integer - /// and fixed-decimal series must encode then decode back bit-exactly. - #[test] - fn value_offset_round_trip_exact_fixed_point() { - // Integer-ish series (scale 0). - let ints: Vec = (0..200).map(|i| (1_000_000 + i * 7) as f64).collect(); - let (offset, scale, residuals) = - encode_value_offset(&ints).expect("integer series must be representable"); - assert_eq!(scale, 0, "integer series should pick scale 0"); - let back = decode_value_offset(offset, scale, &residuals); - assert_eq!(ints, back, "fixed-point round-trip changed integer items"); - - // Fixed-decimal series (milli resolution → scale -3). - let decis: Vec = (0..128).map(|i| 100.0 + (i as f64) * 0.001).collect(); - let (o2, s2, r2) = - encode_value_offset(&decis).expect("milli-decimal series must be representable"); - assert!( - (-3..=0).contains(&s2), - "expected a small negative scale, got {s2}" - ); - let back2 = decode_value_offset(o2, s2, &r2); - assert_eq!(decis, back2, "fixed-point round-trip changed decimal items"); - } - - /// The exactness guard must reject a value not representable at any - /// candidate scale, so the caller falls back to raw f64. - #[test] - fn value_offset_guard_rejects_irrational() { - let items = vec![0.0, 1.0, std::f64::consts::PI, 3.0]; - assert!( - encode_value_offset(&items).is_none(), - "pi is not exactly representable at any decimal scale; guard must reject" - ); - // Empty slice falls back too. - assert!(encode_value_offset(&[]).is_none()); - // Non-finite values are rejected. - assert!(encode_value_offset(&[1.0, f64::NAN]).is_none()); - assert!(encode_value_offset(&[1.0, f64::INFINITY]).is_none()); - } - - /// `KllProtoItems::from_state` must dual-read both the raw-f64 and the - /// value-offset forms and yield identical items + quantiles. - #[test] - fn proto_dual_read_round_trip_both_forms() { - let mut sk: KLL = KLL::init_kll_with_seed(200, 7); - for i in 1..=5000 { - sk.update(&(i as f64)); - } - let raw_items = sk.wire_items(); - let levels = sk.wire_levels(); - let num_levels = sk.wire_num_levels(); - - // Raw-f64 form. - let raw_state = KllState { - k: sk.wire_k(), - m: sk.wire_m(), - num_levels, - levels: levels.clone(), - items: raw_items.clone(), - coin: None, - offset: 0.0, - value_scale: 0, - residuals: vec![], - }; - let from_raw = KllProtoItems::from_state(&raw_state).expect("decode raw-f64"); - - // Value-offset form (items empty, residuals populated). - let (offset, scale, residuals) = - encode_value_offset(&raw_items).expect("integer items representable"); - let fp_state = KllState { - k: sk.wire_k(), - m: sk.wire_m(), - num_levels, - levels, - items: vec![], - coin: None, - offset, - value_scale: scale, - residuals, - }; - let from_fp = KllProtoItems::from_state(&fp_state).expect("decode value-offset"); - - assert_eq!( - from_raw.items, from_fp.items, - "raw-f64 and value-offset decode to different items" - ); - for &q in &[0.0, 0.1, 0.25, 0.5, 0.75, 0.9, 1.0] { - assert_eq!( - from_raw.quantile(q), - from_fp.quantile(q), - "quantile mismatch at p={q} between the two decode paths" - ); - } - } - - /// Decoding must error if a malformed state sets BOTH raw items and - /// residuals. - #[test] - fn proto_rejects_both_forms_populated() { - let state = KllState { - k: 200, - m: 8, - num_levels: 1, - levels: vec![0, 2], - items: vec![1.0, 2.0], - coin: None, - offset: 0.0, - value_scale: 0, - residuals: vec![1, 2], - }; - assert!(KllProtoItems::from_state(&state).is_err()); - } - - /// Cross-language golden: decode the exact envelope bytes that - /// `sketchlib-go` emits for the deterministic `(1..=50)` integer input via - /// its value-offset encoder, and confirm the retained samples + quantiles - /// match. The Go side produces this with `SerializePortable` (fixed-point, - /// scale 0, offset = 1.0). See `sketches/KLL/portable_test.go:: - /// TestGoldenValueOffsetEnvelopeForRust` in sketchlib-go, which prints this - /// hex and asserts the identical bytes. - #[test] - fn golden_value_offset_envelope_from_go() { - // SketchEnvelope { format_version: 1, kll: KLLState { k: 200, m: 8, - // num_levels: 1, levels: [0, 50], offset: 1.0, value_scale: 0, - // residuals: [0..=49] } }, producer/hash_spec cleared. - const GOLDEN_HEX: &str = GO_VALUE_OFFSET_GOLDEN_HEX; - let bytes = decode_hex(GOLDEN_HEX); - let env = SketchEnvelope::decode(bytes.as_slice()).expect("decode envelope"); - let state = match env.sketch_state { - Some(SketchState::Kll(s)) => s, - _ => panic!("envelope did not contain a KLLState"), - }; - assert!( - state.items.is_empty(), - "value-offset form must leave items[] empty" - ); - assert_eq!(state.value_scale, 0); - assert_eq!(state.offset, 1.0); - assert_eq!(state.residuals.len(), 50); - - let decoded = KllProtoItems::from_state(&state).expect("dual-read decode"); - let expected: Vec = (1..=50).map(|i| i as f64).collect(); - assert_eq!(decoded.items, expected, "value-offset decode != (1..=50)"); - // Quantile sanity: with k=200 and 50 inputs, no compaction occurred so - // these are exact order statistics. - assert_eq!(decoded.quantile(0.0), 1.0); - assert_eq!(decoded.quantile(1.0), 50.0); - } - - /// Reverse-direction cross-language golden: build the value-offset envelope - /// for (1..=50) with the Rust encoder and assert exact bytes. The hex - /// printed here is consumed by sketchlib-go `TestRustGoldenDecodesInGo`. - /// Run with `-- --nocapture` to (re)print when regenerating. - #[test] - fn golden_value_offset_envelope_for_go() { - let items: Vec = (1..=50).map(|i| i as f64).collect(); - let (offset, value_scale, residuals) = - encode_value_offset(&items).expect("integers representable"); - assert_eq!(offset, 1.0); - assert_eq!(value_scale, 0); - - let state = KllState { - k: 200, - m: 8, - num_levels: 1, - levels: vec![0, 50], - items: vec![], - // No coin: the reverse-decode in Go only checks the retained set, - // and an absent coin keeps the fixture minimal/stable. - coin: None, - offset, - value_scale, - residuals, - }; - let envelope = SketchEnvelope { - format_version: 1, - producer: None, - hash_spec: None, - sample_p: 0.0, - sketch_state: Some(SketchState::Kll(state)), - }; - let mut got = Vec::with_capacity(envelope.encoded_len()); - envelope.encode(&mut got).expect("prost encode"); - let got_hex: String = got.iter().map(|b| format!("{b:02x}")).collect(); - println!( - "RUST_VALUE_OFFSET_GOLDEN_HEX ({} bytes):\n{got_hex}", - got.len() - ); - - const RUST_GOLDEN_HEX: &str = "08016a4808c801100818012202003239000000000000f03f4a3200020406080a0c0e10121416181a1c1e20222426282a2c2e30323436383a3c3e40424446484a4c4e50525456585a5c5e6062"; - assert_eq!( - got_hex, RUST_GOLDEN_HEX, - "Rust value-offset envelope bytes drifted; update both this constant \ - and sketchlib-go's RustValueOffsetGoldenHex" - ); - - // Self-consistency: our own dual-reader recovers (1..=50). - let decoded = KllProtoItems::from_state(match &envelope.sketch_state { - Some(SketchState::Kll(s)) => s, - _ => unreachable!(), - }) - .expect("decode"); - assert_eq!(decoded.items, items); - } - - // Captured from sketchlib-go's value-offset `SerializePortable` for input - // (1..=50), k=200, producer/hash_spec cleared. Generated and asserted by - // sketchlib-go `TestGoldenValueOffsetEnvelopeForRust` (run with `-v` to - // re-print). 80 bytes: SketchEnvelope{ format_version:1, kll: KLLState{ - // k:200, m:8, num_levels:1, levels:[0,50], coin:{state:42}, offset:1.0, - // value_scale:0, residuals:[0..=49] } }. - const GO_VALUE_OFFSET_GOLDEN_HEX: &str = "08016a4c08c80110081801220200323202082a39000000000000f03f4a3200020406080a0c0e10121416181a1c1e20222426282a2c2e30323436383a3c3e40424446484a4c4e50525456585a5c5e6062"; -} diff --git a/src/message_pack_format/portable/mod.rs b/src/message_pack_format/portable/mod.rs index 97e0d372..db33e47b 100644 --- a/src/message_pack_format/portable/mod.rs +++ b/src/message_pack_format/portable/mod.rs @@ -2,9 +2,9 @@ //! //! Each per-algorithm submodule holds one sketch's pre-envelope wire type. //! These carry a cross-language contract of their own, on goldens apart -//! from ASAPv1's `asapv1_golden/`: HLL, Count-Min, Count Sketch and KLL pin -//! byte parity against Go's serializers, KLL in both directions; DDSketch's -//! golden is a Rust self-pin, not yet reconciled. New code uses ASAPv1: +//! from ASAPv1's `asapv1_golden/`: HLL, Count-Min and Count Sketch pin byte +//! parity against Go's serializers; DDSketch's golden is a Rust self-pin, not +//! yet reconciled. New code uses ASAPv1: //! prefer a sketch's own `serialize_to_bytes`; do not add callers here. pub mod countminsketch; @@ -15,6 +15,6 @@ pub mod ddsketch; pub mod delta_set_aggregator; pub mod hll; pub mod hydra_kll; -pub mod kll; +pub(crate) mod kll; pub mod sampling; pub mod set_aggregator; diff --git a/src/proto/generated/sketchlib.v1.rs b/src/proto/generated/sketchlib.v1.rs index 2c098d3f..413043cf 100644 --- a/src/proto/generated/sketchlib.v1.rs +++ b/src/proto/generated/sketchlib.v1.rs @@ -877,7 +877,7 @@ pub struct SketchEnvelope { /// and stores the RAW SAMPLED sketch state — never the rescaled state. The /// consumer applies the `× 1/sample_p` rescale at QUERY time on the count-like /// estimators (HLL cardinality, CountMin / CountSketch frequency, SUM / COUNT); - /// quantile estimators (KLL, DDSketch) are scale-invariant under uniform + /// quantile estimators (DDSketch) are scale-invariant under uniform /// sampling so they carry `sample_p` but need no rescale. /// /// Lives on the envelope (not per-sketch state) so downstream consumers that @@ -892,7 +892,7 @@ pub struct SketchEnvelope { /// The sketch payload. Exactly one field must be set. #[prost( oneof = "sketch_envelope::SketchState", - tags = "10, 11, 12, 13, 14, 15, 16, 17, 18" + tags = "10, 11, 12, 14, 15, 16, 17, 18" )] pub sketch_state: ::core::option::Option, } @@ -907,8 +907,6 @@ pub mod sketch_envelope { CountSketch(super::CountSketchState), #[prost(message, tag = "12")] Hll(super::HyperLogLogState), - #[prost(message, tag = "13")] - Kll(super::KllState), #[prost(message, tag = "14")] Ddsketch(super::DdSketchState), #[prost(message, tag = "15")] diff --git a/src/sketches/kll.rs b/src/sketches/kll.rs index a19a3d30..59a764c5 100644 --- a/src/sketches/kll.rs +++ b/src/sketches/kll.rs @@ -329,66 +329,6 @@ impl KLL { s } - /// Reconstruct a KLL directly from portable wire state — the retained - /// `items` in level order plus the `levels` boundary array (proto contract: - /// `levels[0] == 0`, `levels[num_levels] == items.len()`). Avoids replaying - /// every item through `update()`: it places the items straight into the - /// internal buffer and fixes up the level boundaries, so the result is - /// bit-identical to the source sketch (identical quantiles) at a fraction - /// of the cost. Errors if the supplied state is inconsistent. - pub fn from_portable_state( - k: usize, - items: &[T], - levels: &[usize], - num_levels: usize, - ) -> Result { - let mut s = Self::init_kll(k as i32); - if items.is_empty() { - return Ok(s); - } - if num_levels == 0 || num_levels >= s.levels.len() { - return Err(format!( - "from_portable_state: invalid num_levels {num_levels}" - )); - } - if levels.len() != num_levels + 1 { - return Err(format!( - "from_portable_state: levels.len()={} != num_levels+1={}", - levels.len(), - num_levels + 1 - )); - } - if levels[0] != 0 || levels[num_levels] != items.len() { - return Err(format!( - "from_portable_state: level bounds [{}..{}] inconsistent with items.len()={}", - levels[0], - levels[num_levels], - items.len() - )); - } - if items.len() > s.max_capacity { - return Err(format!( - "from_portable_state: {} items exceed max_capacity {} for k={}", - items.len(), - s.max_capacity, - k - )); - } - // Live data occupies the high end of the buffer; free space at the front. - let offset = s.max_capacity - items.len(); - let max_cap = s.max_capacity; - s.items[offset..].clone_from_slice(items); - for (dst, &src) in s.levels.iter_mut().zip(levels[..=num_levels].iter()) { - *dst = src + offset; - } - for dst in s.levels.iter_mut().skip(num_levels + 1) { - *dst = max_cap; - } - s.num_levels = num_levels; - s.rebuild_capacity_cache(); // recomputes top_height + level0_capacity for num_levels - Ok(s) - } - /// Hot-path insert: decrement `levels[0]`, write item, check capacity. #[inline] fn push_value(&mut self, value: T) { @@ -785,12 +725,8 @@ impl KLL { self.num_levels as u32 } - /// Returns the compaction-coin state in `sketchlib-go::CoinState` - /// shape: `(state, bit_cache, remaining_bits)`. Wire-format wrappers - /// pack this directly into the `KLLState.coin` proto field; without - /// it the coin state would have to be poked from private internals - /// or reconstructed by a serde round-trip, neither of which is - /// stable across `asap_sketchlib` releases. + /// Returns the compaction-coin state as `(state, bit_cache, + /// remaining_bits)`, the ASAPv1 payload's `coin` array. #[inline] pub fn wire_coin(&self) -> (u64, u64, u32) { ( @@ -804,11 +740,9 @@ impl KLL { /// length `num_levels + 1`, starting at 0 and ending at the total /// number of retained items. The chunk /// `wire_items()[wire_levels()[i] .. wire_levels()[i+1]]` is the - /// **top-most-first** run for the proto's `KLLState.levels`/`items` - /// fields, matching `sketchlib-go::KLLSketch.SerializePortable`. - /// See the `KLLState` docstring in `proto/kll/kll.proto`: index `i` - /// in `levels` maps to compactor level `num_levels - 1 - i` in the - /// in-memory representation. + /// **top-most-first** run of the ASAPv1 payload's `levels`/`items`: + /// index `i` in `levels` maps to compactor level `num_levels - 1 - i` + /// in the in-memory representation. pub fn wire_levels(&self) -> Vec { // Walk from top compactor-level downward, accumulating sizes. let n = self.num_levels; @@ -1184,39 +1118,6 @@ mod tests { ); } - // Direct reconstruction from portable wire state must be BIT-EXACT: - // identical quantiles to the source sketch (unlike a lossy - // replay-through-`update()` reconstruction). - #[test] - fn from_portable_state_reproduces_source_exactly() { - let k = 200usize; - let mut src = KLL::::init_kll(k as i32); - for i in 0..200_000u32 { - src.update(&((i as f64) * 0.0007 + 3.0)); - } - // Extract the compacted portable form (proto contract: levels[0]==0, - // levels[num_levels]==items.len()). - let off = src.levels[0]; - let items: Vec = src.items[off..src.max_capacity].to_vec(); - let levels: Vec = (0..=src.num_levels).map(|h| src.levels[h] - off).collect(); - let n = src.num_levels; - - // (A) bit-exact: direct reconstruction reproduces the source exactly. - let rebuilt = KLL::::from_portable_state(k, &items, &levels, n).unwrap(); - for &q in &[0.0, 0.01, 0.1, 0.25, 0.5, 0.75, 0.9, 0.99, 1.0] { - assert_eq!( - rebuilt.quantile(q), - src.quantile(q), - "direct reconstruction quantile mismatch at q={q}" - ); - } - - // An empty sketch round-trips to an empty sketch. - let empty = KLL::::from_portable_state(k, &[], &[], 0).unwrap(); - assert_eq!(empty.num_levels, 1); - assert_eq!(empty.levels[0], empty.max_capacity); - } - // Ensure each 64-bit chunk is consumed bit-by-bit before refilling. #[test] fn coin_bit_cache_behavior() { diff --git a/src/sketches/kll/wire.rs b/src/sketches/kll/wire.rs index 63e4a3f9..30deba8a 100644 --- a/src/sketches/kll/wire.rs +++ b/src/sketches/kll/wire.rs @@ -20,9 +20,8 @@ //! ## Payload item order (cross-language contract) //! //! The payload's `levels` / `items` use the **top-most-level-first** layout, -//! byte-for-byte matching `sketchlib-go`'s `KLLState` (index `i` in `levels` -//! maps to compactor level `num_levels - 1 - i`; level 0's run is in input -//! order). For the compact KLL this is exactly what [`KLL::wire_levels`] / +//! byte-for-byte matching `sketchlib-go` (index `i` in `levels` maps to +//! compactor level `num_levels - 1 - i`; level 0's run is in input order). For the compact KLL this is exactly what [`KLL::wire_levels`] / //! [`KLL::wire_items`] emit (they reverse the leftward-grown L0 buffer back to //! input order); decode inverts that mapping. @@ -229,7 +228,7 @@ where { /// Serializes the sketch into an ASAPv1 MessagePack envelope /// (kind_id `0x06 0x00`). The retained samples use the top-most-level-first - /// layout that matches `sketchlib-go`'s `KLLState`. + /// layout that matches `sketchlib-go`. pub fn serialize_to_bytes(&self) -> Result, RmpEncodeError> { let metadata = rmp_serde::to_vec_named(&kll_metadata::(self.k as u32, self.m as u32, self.seed))?; diff --git a/tests/PBT_COVERAGE.md b/tests/PBT_COVERAGE.md index cddee44f..07df409c 100644 --- a/tests/PBT_COVERAGE.md +++ b/tests/PBT_COVERAGE.md @@ -296,7 +296,7 @@ Compaction is randomized, so no law names retained items. Compared against: the - same seed, same stream: same items - `clear` re-seeds from the stored seed: replay == fresh, items and levels - wire - - msgpack keeps `k`, count, quantiles + - ASAPv1 keeps `k`, count, quantiles - ASAPv1 keeps count and quantiles under NaN, ±inf, ±0, f64 extremes ## kll_dynamic.rs — 14 diff --git a/tests/asapv1_golden.rs b/tests/asapv1_golden.rs index 30b41651..186d0993 100644 --- a/tests/asapv1_golden.rs +++ b/tests/asapv1_golden.rs @@ -245,9 +245,7 @@ fn cs_i32_regular_2x4_matches_golden() { // --------------------------------------------------------------------------- // KLL: build known state (k=200, seed 42, integers 1..=50 — below the level-0 // capacity, so no compaction fires and the retained set is deterministic) -> -// serialize == golden, and golden round-trips. Matches the deterministic -// scenario the proto parity test uses (sketchlib-go's KLLSketch over the same -// input), so the coin state (42) lines up cross-language. +// serialize == golden, and golden round-trips. // --------------------------------------------------------------------------- /// A k=200 KLL over `1..=50` with a fixed compaction seed: fully deterministic. diff --git a/tests/e2e/composition.rs b/tests/e2e/composition.rs index 23e42c41..1aebd99e 100644 --- a/tests/e2e/composition.rs +++ b/tests/e2e/composition.rs @@ -17,7 +17,7 @@ use common::{FreqTruth, NumericTruth, uniform_u64, zipf_u64}; use asap_sketchlib::{ Classic, Count, CountMin, CountMinSketchWithHeap, CountSketchWithHeap, DataInput, - EnsembleSketch, ErtlMLE, FastPath, HashSketchEnsemble, HyperLogLog, HyperLogLogHIP, KllSketch, + EnsembleSketch, ErtlMLE, FastPath, HashSketchEnsemble, HyperLogLog, HyperLogLogHIP, KLL, MessagePackCodec, UnivMonQ, UnivMonQConfig, Vector2D, }; @@ -697,16 +697,15 @@ fn portable_count_sketch_with_heap_satisfies_the_l2_bound_through_merge_and_wire ); } -/// The portable KLL facade under the DataSketches maximum-rank-error -/// characterization, seeded so a failure reproduces. `KllSketch::new` seeds -/// from the wall clock; `with_seed` is what an accuracy test must use. +/// KLL under the DataSketches maximum-rank-error characterization through a +/// two-shard merge and an ASAPv1 round trip, seeded so a failure reproduces. /// /// Trial unit is one sketch, scored on its worst rank error over the `q` grid, /// with an independent compaction seed per trial. The post-wire sketch is *not* /// a separate trial — a round trip that preserves every answer bit for bit, as /// asserted below, gives literally the same numbers. #[test] -fn portable_kll_sketch_satisfies_the_rank_error_characterization_through_merge_and_wire() { +fn kll_satisfies_the_rank_error_characterization_through_merge_and_wire() { const K: u16 = 200; const TRIALS: u64 = 12; let values: Vec = uniform_u64(N, 1_000_000, STREAM_SEED) @@ -724,29 +723,29 @@ fn portable_kll_sketch_satisfies_the_rank_error_characterization_through_merge_a let mut tally = Tally::default(); for t in 0..TRIALS { let seed = 0x5EED_0400u64.wrapping_add(t.wrapping_mul(0x9E37_79B9_7F4A_7C15)); - let mut single = KllSketch::with_seed(K, seed); - let mut left = KllSketch::with_seed(K, seed ^ 0xAAAA); - let mut right = KllSketch::with_seed(K, seed ^ 0x5555); + let mut single = KLL::::init_kll_with_seed(K as i32, seed); + let mut left = KLL::::init_kll_with_seed(K as i32, seed ^ 0xAAAA); + let mut right = KLL::::init_kll_with_seed(K as i32, seed ^ 0x5555); for (i, v) in values.iter().enumerate() { - single.update(*v); + single.update(v); if i % 2 == 0 { - left.update(*v); + left.update(v); } else { - right.update(*v); + right.update(v); } } - left.merge(&right).expect("same-k merge"); + left.merge(&right); spec.record_trial( &mut tally, - &format!("portable KllSketch single pass seed={seed:#x}"), + &format!("KLL single pass seed={seed:#x}"), truth.sorted(), &qs, |q| single.quantile(q), ); spec.record_trial( &mut tally, - &format!("portable KllSketch two-shard merge seed={seed:#x}"), + &format!("KLL two-shard merge seed={seed:#x}"), truth.sorted(), &qs, |q| left.quantile(q), @@ -755,9 +754,9 @@ fn portable_kll_sketch_satisfies_the_rank_error_characterization_through_merge_a // The wire round trip must preserve every answer bit for bit, which is // an equality rather than a band — and is why the decoded sketch does // not enter the rank battery as a second trial. - let bytes = single.to_msgpack().expect("encode"); - let decoded = KllSketch::from_msgpack(&bytes).expect("decode"); - assert_eq!(decoded.k(), K, "k must survive the wire"); + let bytes = single.serialize_to_bytes().expect("encode"); + let decoded = KLL::::deserialize_from_bytes(&bytes).expect("decode"); + assert_eq!(decoded.k(), K as usize, "k must survive the wire"); assert_eq!( decoded.count(), single.count(), @@ -768,21 +767,11 @@ fn portable_kll_sketch_satisfies_the_rank_error_characterization_through_merge_a let (a, b) = (single.quantile(q), decoded.quantile(q)); wire_tally.record(a == b, || format!("q={q}: before {a} after {b}")); } - wire_tally.assert_none( - &format!("portable KllSketch wire round trip (seed={seed:#x})"), - &context, - ); - - // A mismatched `k` must be refused rather than silently merged. - let other_k = KllSketch::with_seed(K * 2, seed); - assert!( - single.merge(&other_k).is_err(), - "merging sketches with different k must fail" - ); + wire_tally.assert_none(&format!("KLL wire round trip (seed={seed:#x})"), &context); } tally.assert_independent_binomial( - "portable KllSketch / maximum normalized rank error per compaction seed", + "KLL / maximum normalized rank error per compaction seed", spec.trial_failure_probability, &format!("{context}; single pass and two-shard merge, q grid {qs:?}"), ); diff --git a/tests/msgpack_compat.rs b/tests/msgpack_compat.rs index 267a51c6..7e56ccbd 100644 --- a/tests/msgpack_compat.rs +++ b/tests/msgpack_compat.rs @@ -29,10 +29,9 @@ use asap_sketchlib::message_pack_format::portable::countminsketch_topk::{ }; use asap_sketchlib::message_pack_format::portable::delta_set_aggregator::DeltaResult; use asap_sketchlib::message_pack_format::portable::hydra_kll::HydraKllSketchWire; -use asap_sketchlib::message_pack_format::portable::kll::KllSketchData; use asap_sketchlib::{ CountMinSketch, CountMinSketchWithHeap, CountSketch, DdSketch, HllSketch, HllVariant, - HydraKllSketch, KllSketch, SetAggregator, + HydraKllSketch, SetAggregator, }; // ===== round-trip: every wire-format-aligned type ===== @@ -95,18 +94,6 @@ fn hll_sketch_round_trip() { assert_eq!(restored.registers.len(), s.registers.len()); } -#[test] -fn kll_sketch_round_trip() { - let mut s = KllSketch::with_seed(200, 0x5EED_0800); - for i in 0..100 { - s.update(i as f64); - } - let bytes = s.to_msgpack().expect("encode"); - let restored = KllSketch::from_msgpack(&bytes).expect("decode"); - assert_eq!(restored.k, 200); - assert_eq!(restored.count(), 100); -} - #[test] fn hydra_kll_sketch_round_trip() { let mut s = HydraKllSketch::with_seed(2, 4, 200, 0x5EED_0900); @@ -186,21 +173,7 @@ fn count_min_with_heap_wire_shape() { #[test] fn hydra_kll_wire_shape() { - let wire = HydraKllSketchWire { - rows: 2, - cols: 3, - sketches: vec![ - vec![ - KllSketchData { - k: 200, - sketch_bytes: vec![], - }; - 3 - ]; - 2 - ], - }; - let bytes = rmp_serde::to_vec(&wire).unwrap(); + let bytes = HydraKllSketch::new(2, 3, 200).to_msgpack().unwrap(); let restored: HydraKllSketchWire = rmp_serde::from_slice(&bytes).unwrap(); assert_eq!(restored.rows, 2); assert_eq!(restored.cols, 3); @@ -210,7 +183,7 @@ fn hydra_kll_wire_shape() { // ===== golden-bytes placeholders: an ignored, uncovered gap ===== // -// The five tests below are **empty and ignored**. They verify nothing today. +// The four tests below are **empty and ignored**. They verify nothing today. // // What they need is a msgpack payload produced by `sketchlib-go` and checked in // under `tests/fixtures/msgpack/`. That fixture cannot be generated here: this @@ -245,10 +218,6 @@ fn dd_sketch_decodes_go_bytes() {} #[test] fn hll_sketch_decodes_go_bytes() {} -#[ignore = "gap: needs a sketchlib-go-produced msgpack fixture; none can be generated in this repo"] -#[test] -fn kll_sketch_decodes_go_bytes() {} - #[ignore = "gap: needs a sketchlib-go-produced msgpack fixture; none can be generated in this repo"] #[test] fn hydra_kll_decodes_go_bytes() {} diff --git a/tests/pbt/kll.rs b/tests/pbt/kll.rs index f5c8d5ff..c540a143 100644 --- a/tests/pbt/kll.rs +++ b/tests/pbt/kll.rs @@ -1,5 +1,4 @@ -//! Property tests for KLL: the native sketch and the portable `KllSketch` -//! that carries it over the wire. +//! Property tests for the compact KLL sketch. //! //! Karnin, Lang, Liberty, FOCS '16. Compaction is randomized, so a law may //! not name the items it keeps. What survives: a quantile is monotone in q, @@ -12,8 +11,7 @@ //! `kll_the_ends_are_exact_until_the_first_compaction`. use crate::support::{QUANTILES, close, extreme_values}; -use asap_sketchlib::message_pack_format::MessagePackCodec; -use asap_sketchlib::{KLL, KllSketch}; +use asap_sketchlib::KLL; use proptest::prelude::*; const QUANTILE_PROBES: [f64; 7] = [0.0, 0.01, 0.25, 0.5, 0.75, 0.99, 1.0]; @@ -89,12 +87,8 @@ fn top_weight(s: &KLL) -> f64 { (1u64 << (s.wire_num_levels() - 1)) as f64 } -fn kll_of(k: u16, values: &[f64]) -> KllSketch { - let mut s = KllSketch::with_seed(k, 0xC0FFEE); - for v in values { - s.update(*v); - } - s +fn kll_of(k: u16, values: &[f64]) -> KLL { + native_kll(k as i32, 0xC0FFEE, values) } proptest! { @@ -106,7 +100,7 @@ proptest! { values in prop::collection::vec(-1e5f64..1e5, 0..8), ) { let s = kll_of(k, &values); - prop_assert_eq!(s.count(), values.len() as u64); + prop_assert_eq!(s.count(), values.len()); } #[test] @@ -116,7 +110,7 @@ proptest! { b in prop::collection::vec(-1e5f64..1e5, 0..400), ) { let mut merged = kll_of(k, &a); - merged.merge(&kll_of(k, &b)).expect("merge"); + merged.merge(&kll_of(k, &b)); let n = (a.len() + b.len()) as f64; let got = merged.count() as f64; @@ -164,18 +158,15 @@ proptest! { k in 8u16..256, values in prop::collection::vec(-1e6f64..1e6, 0..300), ) { - let mut s = KllSketch::with_seed(k, 0x5EED); - for v in &values { - s.update(*v); - } + let s = native_kll(k as i32, 0x5EED, &values); - let bytes = s.to_msgpack().expect("encode"); - let restored = KllSketch::from_msgpack(&bytes).expect("decode"); + let bytes = s.serialize_to_bytes().expect("encode"); + let restored = KLL::::deserialize_from_bytes(&bytes).expect("decode"); prop_assert_eq!(restored.k(), s.k()); prop_assert_eq!(restored.count(), s.count()); for q in QUANTILE_PROBES { - prop_assert_eq!(restored.quantile(q), s.quantile(q), "q={}", q); + prop_assert_eq!(restored.quantile(q).to_bits(), s.quantile(q).to_bits(), "q={}", q); } } diff --git a/tests/xtest_consumer.rs b/tests/xtest_consumer.rs index 0113ff45..dd12bcc6 100644 --- a/tests/xtest_consumer.rs +++ b/tests/xtest_consumer.rs @@ -6,7 +6,6 @@ //! //! Files consumed (from $XTEST_DIR/): //! countmin.pb — CountMinState with float64 counters -//! kll.pb — KLLState with quantile items and coin RNG state //! ddsketch.pb — DDSketchState with alpha + bucket array //! hll.pb — HyperLogLogState (ErtlMLE variant) //! countsketch.pb — CountSketchState with float64 signed counters @@ -117,48 +116,6 @@ fn cross_language_proto() { println!("[CountMin] PASS"); } - // ----------------------------------------------------------------------- - // KLL - // ----------------------------------------------------------------------- - println!(); - println!("[KLL] Step 1/3 — Read kll.pb"); - let bytes = read_file(in_dir.join("kll.pb")); - let env = SketchEnvelope::decode(bytes.as_slice()).expect("decode kll envelope"); - - println!( - "[KLL] Step 2/3 — Validate envelope (format_version={}, producer={})", - env.format_version, - env.producer.as_ref().map_or("?", |p| &p.library) - ); - - let kll_state = match env.sketch_state { - Some(sketch_envelope::SketchState::Kll(ref s)) => s.clone(), - other => panic!("expected KLL sketch_state, got {other:?}"), - }; - - println!( - "[KLL] k={} m={} num_levels={} items={} levels={}", - kll_state.k, - kll_state.m, - kll_state.num_levels, - kll_state.items.len(), - kll_state.levels.len() - ); - - let kll = KllFromProto::from_state(&kll_state); - let p50 = kll.quantile(0.50); - let p99 = kll.quantile(0.99); - println!("[KLL] Step 3/3 — p50 ≈ {p50:.1} (expect ~5000) p99 ≈ {p99:.1} (expect ~9900)"); - - let p50_ok = (p50 - 5000.0).abs() / 5000.0 < 0.05; - let p99_ok = (p99 - 9900.0).abs() / 9900.0 < 0.05; - if p50_ok && p99_ok { - println!("[KLL] PASS"); - } else { - eprintln!("[KLL] FAIL: p50={p50:.1} p99={p99:.1}"); - all_ok = false; - } - // ----------------------------------------------------------------------- // DDSketch // ----------------------------------------------------------------------- @@ -1097,74 +1054,6 @@ fn cm_query_min(cm: &CountMinState, hash: u64) -> f64 { if min_val == f64::MAX { 0.0 } else { min_val } } -// --------------------------------------------------------------------------- -// Minimal KLL deserialization + CDF query -// --------------------------------------------------------------------------- - -struct KllFromProto { - items: Vec, - levels: Vec, - num_levels: usize, -} - -impl KllFromProto { - fn from_state(s: &KllState) -> Self { - // Dual-read: prefer the value-offset fixed-point representation when - // residuals are present, else fall back to raw f64 items[] (v1). - let items: Vec = if !s.residuals.is_empty() { - asap_sketchlib::message_pack_format::portable::kll::decode_value_offset( - s.offset, - s.value_scale, - &s.residuals, - ) - } else { - s.items.clone() - }; - let levels: Vec = s.levels.iter().map(|&v| v as usize).collect(); - let num_levels = s.num_levels as usize; - Self { - items, - levels, - num_levels, - } - } - - fn weighted_samples(&self) -> Vec<(f64, u64)> { - let mut out = Vec::with_capacity(self.items.len()); - for h in 0..self.num_levels { - let weight: u64 = 1 << h; - let idx = self.num_levels - 1 - h; - if idx + 1 >= self.levels.len() { - continue; - } - let start = self.levels[idx]; - let end = self.levels[idx + 1]; - for &v in &self.items[start..end] { - out.push((v, weight)); - } - } - out - } - - fn quantile(&self, q: f64) -> f64 { - let mut samples = self.weighted_samples(); - if samples.is_empty() { - return 0.0; - } - samples.sort_by(|a, b| a.0.partial_cmp(&b.0).unwrap()); - let total: u64 = samples.iter().map(|(_, w)| w).sum(); - let target = (q * total as f64).ceil() as u64; - let mut acc = 0u64; - for (v, w) in &samples { - acc += w; - if acc >= target { - return *v; - } - } - samples.last().unwrap().0 - } -} - // --------------------------------------------------------------------------- // Minimal DDSketch deserialization + quantile query // --------------------------------------------------------------------------- From 747ed0934516d7d854a45c057750931eeed17427 Mon Sep 17 00:00:00 2001 From: GordonYuanyc Date: Fri, 2 Oct 2026 00:40:42 -0600 Subject: [PATCH 02/36] refactor(countsketch)!: drop the proto and portable formats Count Sketch serializes as ASAPv1 only. Removes the portable CountSketch, CountSketchDelta and COUNT_SKETCH_TOPK_CAPACITY, the native MessagePackCodec shim for Count, the proto CountSketchDelta and CountSketchCell messages, and SketchEnvelope's count_sketch field (11, now reserved). CountSketchState stays: UnivMon and Hydra embed it. Tests that exercised Count Sketch through the removed formats are dropped (proto parity probe, msgpack round trip, xtest consumer case, portable e2e bound) or ported to the native Count (the four portable property laws). Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 15 + docs/message_pack_format.md | 4 +- docs/tests.md | 33 +- proto/countminsketch/countminsketch.proto | 10 +- proto/countsketch/countsketch.proto | 53 +- proto/sketchlib.proto | 3 +- src/common/hashspec.rs | 8 +- src/lib.rs | 5 +- src/message_pack_format/native/countsketch.rs | 32 - src/message_pack_format/native/mod.rs | 1 - .../portable/countminsketch.rs | 24 +- .../portable/countsketch.rs | 974 ------------------ src/message_pack_format/portable/mod.rs | 3 +- src/proto/generated/sketchlib.v1.rs | 78 +- tests/PBT_COVERAGE.md | 6 +- tests/cs_envelope_golden.hex | 1 - tests/e2e/frequency.rs | 25 +- tests/msgpack_compat.rs | 15 +- tests/pbt/count_sketch.rs | 115 +-- tests/sketches_go_parity_probe.rs | 66 +- tests/xtest_consumer.rs | 71 -- 21 files changed, 115 insertions(+), 1427 deletions(-) delete mode 100644 src/message_pack_format/native/countsketch.rs delete mode 100644 src/message_pack_format/portable/countsketch.rs delete mode 100644 tests/cs_envelope_golden.hex diff --git a/CHANGELOG.md b/CHANGELOG.md index 80d85f98..85cd72de 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -88,6 +88,21 @@ signals a backwards-compatible change. as an empty window like any other, and that window takes a place in `max_windows`. +### Removed + +- **Count Sketch's proto and portable formats.** Count Sketch (`Count`) + serializes as ASAPv1 only (`serialize_to_bytes` / `deserialize_from_bytes`, + kind `0x04 0x00`). Removed: the portable `CountSketch`, `CountSketchDelta` + and `COUNT_SKETCH_TOPK_CAPACITY` (module + `message_pack_format::portable::countsketch` and their crate-root + re-exports), including `CountSketch`'s proto and MessagePack delta codecs + (`compute_delta`, `apply_delta_bytes`, `compute_delta_msgpack`, + `apply_delta_msgpack_bytes`); the `MessagePackCodec` impl for `Count` + (module `message_pack_format::native::countsketch`); the proto messages + `CountSketchDelta` and `CountSketchCell`; and `SketchEnvelope`'s + `count_sketch` field (11, now reserved). `CountSketchState` stays, as + UnivMon and Hydra embed it. + ### Fixed - `KLLDynamic` panicked on every quantile query once a NaN had entered the diff --git a/docs/message_pack_format.md b/docs/message_pack_format.md index 12a2b91c..a229a8d9 100644 --- a/docs/message_pack_format.md +++ b/docs/message_pack_format.md @@ -150,8 +150,8 @@ being **phased out** in favor of the per-sketch `wire.rs` + the shared `envelope.rs`. - `portable/` holds per-sketch wire types that predate the envelope. They - carry their own byte-parity goldens against `sketchlib-go` — HLL, Count-Min, - Count Sketch and KLL (both directions) — separate from `asapv1_golden/`. + carry their own byte-parity goldens against `sketchlib-go` — HLL, Count-Min + and KLL (both directions) — separate from `asapv1_golden/`. - `native/` is a set of thin `MessagePackCodec` shims over the sketches' `serialize_to_bytes` / `deserialize_from_bytes`. Every type it wraps emits the ASAPv1 envelope, so each shim is a pass-through. diff --git a/docs/tests.md b/docs/tests.md index ccb1571c..47732b4d 100644 --- a/docs/tests.md +++ b/docs/tests.md @@ -1254,31 +1254,6 @@ Test file: [`src/message_pack_format/portable/countminsketch_topk.rs`](../src/me | `test_aggregate_topk` | `aggregate_topk` builds a decodable composite honouring its heap bound. | For `aggregate_topk(4, 100, 2, ["a", "b", "a", "c"], [1.0, 2.0, 3.0, 0.5])`, verifies the decode reports `heap_size` `2` and at most 2 heap items; the heap contents themselves are not checked. | | `test_aggregate_topk_empty` | An empty stream produces nothing. | Verifies `aggregate_topk(4, 100, 10, &[], &[])` returns `None`. | -### Portable CountSketch - -Test file: [`src/message_pack_format/portable/countsketch.rs`](../src/message_pack_format/portable/countsketch.rs) - -| test_name | test_description | what_is_tested | -| --- | --- | --- | -| `msgpack_delta_against_empty_round_trips` | A msgpack delta against an empty snapshot carries the signed cells exactly. | For a `5x1024` sketch fed 200 updates of `"hot"` at weight `1.0`, verifies the delta against a fresh sketch at threshold `1.0`, applied to another fresh sketch, reproduces `sketch()` cell for cell. | -| `msgpack_delta_wire_layout_is_parallel_arrays` | The delta is a five-element array of scalars plus parallel index arrays. | For a `3x8` sketch fed `"x"` at weight `5.0`, verifies the delta bytes equal `rmp_serde::to_vec(&(rows as u64, cols as u64, row_idx: Vec, col_idx: Vec, d_count: Vec))` recomputed from the non-zero cells, and that decoding reports `rows` `3` and `cols` `8`. | -| `test_new_empty` | A fresh sketch has the requested shape and zeroed cells. | For `CountSketch::new(2, 3)`, verifies `rows` is `2`, `cols` is `3`, and `sketch()` is `[[0.0, 0.0, 0.0], [0.0, 0.0, 0.0]]`. | -| `test_from_legacy_matrix` | A legacy matrix is adopted unchanged, signs included. | Verifies `from_legacy_matrix([[1.0, -2.0, 3.0], [-4.0, 5.0, -6.0]], 2, 3)` reports exactly that matrix from `sketch()`. | -| `test_merge_element_wise` | Merge sums signed cells element-wise. | Merges the `2x2` matrix `[[-1.0, -2.0], [-3.0, -4.0]]` into `[[1.0, 2.0], [3.0, 4.0]]` and verifies every merged cell is `0.0`. | -| `test_merge_dimension_mismatch` | Mismatched row counts are refused. | Verifies merging a `3x3` sketch into a `2x3` one fails. | -| `test_merge_refs` | `merge_refs` sums every input. | Merges three `1x2` sketches holding `[1.0, 2.0]`, `[3.0, 4.0]`, and `[5.0, 6.0]` and verifies the result is `[[9.0, 12.0]]`. | -| `test_apply_delta_additive` | Delta cells add into the matrix, negative weights included. | For a `2x3` sketch holding `[[1.0, -2.0, 3.0], [-4.0, 5.0, -6.0]]`, applies cells `(0,0,10)`, `(0,2,-3)`, and `(1,1,-15)` and verifies the matrix becomes `[[11.0, -2.0, 0.0], [-4.0, -10.0, -6.0]]`. | -| `test_apply_delta_matches_full_merge` | Applying a delta lands where merging the same addition lands. | For a `2x2` base `[[1.0, 2.0], [3.0, 4.0]]`, verifies merging the addition `[[10.0, 0.0], [0.0, 20.0]]` and applying the delta cells `(0,0,10)` and `(1,1,20)` give the same matrix. | -| `test_apply_delta_out_of_range` | A cell outside the declared geometry is refused. | Verifies a delta carrying cell `(2, 0, 1)` against a two-row `2x3` sketch fails to apply. | -| `test_apply_delta_rebuilds_topk_from_hh_keys` | The `hh_keys` channel rebuilds Top-K by re-querying the merged matrix. | For a `3x16` sketch fed `"alpha"` at `5.0` and `"beta"` at `3.0`, applies a delta with no cells and `hh_keys` `["alpha", "beta"]` and verifies `topk` holds both keys and that `"alpha"` reads above `"beta"`. | -| `test_apply_delta_hh_keys_topk_capacity` | The rebuilt Top-K is bounded by `COUNT_SKETCH_TOPK_CAPACITY`. | For a `3x1024` sketch fed 105 keys `k0000`..`k0104` at weights `1.0`..`105.0`, applies a delta whose `hh_keys` carries all 105 and verifies `topk` holds exactly `COUNT_SKETCH_TOPK_CAPACITY` (`100`) entries; which entries survive is not checked. | -| `test_msgpack_round_trip` | The msgpack round trip preserves the signed matrix and its shape. | Round-trips a `2x2` sketch holding `[[1.5, -2.5], [3.5, -4.5]]` and verifies the matrix, `rows`, and `cols` are unchanged. | -| `test_update_then_envelope_matches_sketchlib_go_bytes` | The proto envelope for a populated matrix is byte-identical to the golden. | For a `3x512` sketch fed 25 updates over the five keys `k-a`..`k-e` (each five times) at weight `1.0`, builds a `CountSketchState` with `counter_type` `INT64`, row-major `counts_int`, per-row `l2` of `125.0`, and no `topk`, wraps it in a `SketchEnvelope` with `format_version` `1`, and verifies the prost encoding equals a 1,577-byte inline golden. | -| `test_hh_keys_matches_go_golden_bytes` | Heavy-hitter keys at tag 6 reach the wire in the golden's shape. | For the same `3x512` sketch over `k-a`..`k-e`, decodes the packed delta against an empty sketch as a proto `CountSketchDelta`, sets `hh_keys` to `["flow-0", "flow-3", "flow-7"]`, and verifies the re-encoding equals a 117-byte inline golden. | -| `test_compute_delta_against_empty_round_trips` | A proto delta against an empty snapshot carries the whole signed matrix. | For a `3x512` sketch fed 200 updates of `k-{i % 23}` at weight `1.0`, verifies the delta against an empty sketch at threshold `1.0`, applied to a fresh `3x512` sketch, reproduces `sketch()` cell for cell. | -| `test_compute_delta_then_apply_matches_current` | A delta between two non-empty snapshots reconstructs the later one. | For a `3x64` base fed 40 updates of `x{i % 8}` and a current fed 30 more, verifies the delta applied to the base reproduces the current matrix. | -| `test_compute_delta_matches_go_golden_bytes` | The packed delta bytes and per-row L2 are pinned by a golden. | For a `3x512` sketch fed the 25 `k-a`..`k-e` updates, verifies the delta against an empty sketch at threshold `1.0` equals a 93-byte inline golden carrying the packed cell arrays and a per-row L2 of `125.0`. | - ### Portable CountSketchWithHeap Test file: [`src/message_pack_format/portable/countsketch_topk.rs`](../src/message_pack_format/portable/countsketch_topk.rs) @@ -1606,13 +1581,13 @@ Test file: [`tests/e2e/frequency.rs`](../tests/e2e/frequency.rs) | `fold_cms_foldcs_counts_and_hierarchical_merge` | Weighted counts are exact on sparse dimensions and survive a same-level merge, a signed update, and a hierarchical merge. | On `FoldCMS::::new(3, 2048, 0, 32)` sketches: 100 inserts of `Str("alpha")` at weight 2 into each of two sketches plus 100 of `Str("beta")` at weight 1 into the second, verifying `query` reads `200`, `200` and `100`; after `merge_same_level`, `400` for alpha and `100` for beta. On `FoldCS::::new(3, 2048, 0, 32)`, `Str("gamma")` at `+60` then `-20` is verified to query `40`. Finally `FoldCMS::hierarchical_merge` of two sketches built by `build_fold_cms` (30 and 45 inserts at weight 3) is verified to query `90` and `135`. | | `folded_sketches_keep_their_own_bounds_through_a_sixteen_way_merge` | Folded sketches keep their own theorems, evaluated at the folded width, through a sixteen-way hierarchical merge. | Splits `zipf_u64(240_000, 10_000, 1.1, 1009)` into 16 windows of 15,000, giving each window a `FoldCMS` and a `FoldCS` of `3x4096` at `fold_level = 4` and `top_k = 20`, then `hierarchical_merge`s each family. Asserts `CountMinSpec::new(3, 256)` over `FoldCMS::query` and `CountSketchSpec::new(3, 256)` over `FoldCS::query` - the bounds are evaluated at the folded width `4096 >> 4 = 256`, not the unfolded 4096 - with the specs' usual rules: zero violations for one-sidedness and both simultaneous bounds at `delta = 1e-3`, and the marginal bounds as rate pins at `e^-3` and `P[Bin(3, 1/3) >= 2]`. | -### Frequency: Portable Count-Min and Count Sketch +### Frequency: Portable Count-Min Test file: [`tests/e2e/frequency.rs`](../tests/e2e/frequency.rs) | test_name | test_description | what_is_tested | | --- | --- | --- | -| `portable_cms_and_cs_string_keys_satisfy_their_own_bounds` | The portable wire twins satisfy their own theorems over string keys with no extra slack. | Feeds `zipf_u64(50_000, 2048, 1.1, 1006)` as string keys `"k{k}"` through `update(&key, 1.0)` into `portable::countminsketch::CountMinSketch::new(3, 4096)` and `portable::countsketch::CountSketch::new(5, 4096)`, then asserts `CountMinSpec::new(3, 4096).assert_contract` over the portable Count-Min's `estimate` and `CountSketchSpec::new(5, 4096).assert_contract` over the portable Count Sketch's `estimate`, against a `FreqTruth` keyed on the underlying `i64` identities. | +| `portable_cms_string_keys_satisfy_their_own_bound` | The portable Count-Min satisfies its own theorem over string keys with no extra slack. | Feeds `zipf_u64(50_000, 2048, 1.1, 1006)` as string keys `"k{k}"` through `update(&key, 1.0)` into `portable::countminsketch::CountMinSketch::new(3, 4096)`, then asserts `CountMinSpec::new(3, 4096).assert_contract` over its `estimate`, against a `FreqTruth` keyed on the underlying `i64` identities. | ### Frequency: Documented Input Matrix @@ -2143,7 +2118,6 @@ Test file: [`tests/msgpack_compat.rs`](../tests/msgpack_compat.rs) | `count_min_sketch_with_heap_round_trip` | A portable `CountMinSketchWithHeap` survives a msgpack round trip with its geometry and heap size intact. | Updates `CountMinSketchWithHeap::new(3, 64, 8)` with `hot` `+100.0` and `warm` `+10.0`, then verifies the decode reports `rows` `3`, `cols` `64` and `heap_size` `8`; the heap's contents are not checked. | | `count_min_wire_shape` | The `CountMinSketchWire` DTO round-trips through `rmp_serde` with its field shape intact. | Serializes `CountMinSketchWire { sketch: vec![vec![1.0, 2.0]; 3], rows: 3, cols: 2 }` with `rmp_serde::to_vec` and reads it back with `rmp_serde::from_slice`, verifying `rows` `3`, `cols` `2` and a `sketch` of length `3`; the cell values are not checked. | | `count_min_with_heap_wire_shape` | The nested `CountMinSketchWithHeapWire` DTO round-trips through `rmp_serde` with its nesting intact. | Serializes a wire struct holding a `CountMinSketchInnerWire` of `2x4` zeros, one `CmsHeapItem { key: "hot", value: 42.0 }` and `heap_size: 8`, then verifies the `rmp_serde` decode reports `heap_size` `8`, one heap entry and the key `hot`; the item's value and the inner matrix are not re-checked. | -| `count_sketch_round_trip` | A portable `CountSketch` survives a msgpack round trip with its geometry intact. | Updates `CountSketch::new(3, 64)` with `k1` `+1.0` and `k2` `+2.0`, then verifies the decode reports `rows` `3` and `cols` `64`; the signed counters and the estimates are not checked. | | `dd_sketch_decodes_go_bytes` | The test asserts nothing. | The test asserts nothing: the body is empty and the test is marked `#[ignore]` with the reason that no cross-language msgpack fixture exists in the repository. | | `dd_sketch_round_trip` | A portable `DdSketch` survives a msgpack round trip carrying its full count in the bucket array. | Updates `DdSketch::new(0.01)` with `1.0`, `10.0` and `100.0`, then verifies the decode's `total_count()` — recovered by summing the buckets, since the count is not on the wire — is `3`; `alpha`, the bucket offset and the quantiles are not checked. | | `delta_result_round_trip` | A `DeltaResult` survives a msgpack round trip with both of its sets. | Serializes a `DeltaResult` whose `added` holds `a` and whose `removed` holds `b`, then verifies the decode's `added` contains `a` and its `removed` contains `b`. | @@ -2191,7 +2165,6 @@ Test file: [`tests/sketches_go_parity_probe.rs`](../tests/sketches_go_parity_pro | test_name | test_description | what_is_tested | | --- | --- | --- | -| `sketches_count_fastpath_matches_go_count_sketch_envelope` | A `sketches::Count` fast-path matrix encodes to the checked-in Count-Sketch envelope bytes. | Fills `Count, FastPath, DefaultXxHasher>` at `3x512` with 25 `insert_many` calls over the five keys `k-a`..`k-e`, packs the row-major cells and per-row L2 sums into a `CountSketchState` with `counter_type` `INT64` inside a `SketchEnvelope { format_version: 1 }` with no producer or hash_spec, and verifies the prost encoding equals the 1,577-byte fixture `tests/cs_envelope_golden.hex`. | | `sketches_countmin_fastpath_matches_go_count_min_envelope` | A `sketches::CountMin` fast-path matrix encodes to the checked-in Count-Min envelope bytes. | Fills `CountMin, FastPath, DefaultXxHasher>` at `4x2048` with 50 `insert_many` calls over the ten keys `flow-0`..`flow-9`, packs the cells cast to `i64` plus per-row L1 and L2 sums into a `CountMinState` with `counter_type` `INT64`, and verifies the prost encoding matches the 8,275-byte fixture `src/sketches/testdata/cms_envelope_golden.hex` in length and then byte for byte. | | `sketches_ddsketch_matches_go_envelope` | A `sketches::DDSketch` encodes to the DDSketch envelope hex constant held in the test file. | Fills `DDSketch::new(0.01)` with `1..=50` as `f64`, encodes a `DdSketchState` carrying `store_counts`, `store_offset` and an alpha round-tripped through gamma (`gamma = (1 + a) / (1 - a)`, `alpha_wire = (gamma - 1) / (gamma + 1)`), and verifies the envelope matches the 403-byte inline hex constant in length and then byte for byte; the comment on that constant records it as this crate's own output rather than a fixture from another implementation, so no cross-language parity is pinned. | | `sketches_hll_classic_matches_go_envelope` | An ErtlMLE P14 register set encodes to the checked-in HyperLogLog envelope bytes. | Feeds `HyperLogLogImpl` the `hash64_seeded(CANONICAL_HASH_SEED, ...)` digests of the IEEE-754 little-endian bytes of `1.0..=50.0` through `insert_with_hash`, encodes a `HyperLogLogState` with `variant` `2`, `precision` `14`, the dense registers, zeroed HIP scalars and no sparse form, and verifies the envelope matches the 16,398-byte fixture `src/sketches/testdata/hll_envelope_golden.hex` in length and then byte for byte; despite the name, the variant under test is `ErtlMLE`, not `Classic`. | @@ -2202,7 +2175,7 @@ Test file: [`tests/xtest_consumer.rs`](../tests/xtest_consumer.rs) | test_name | test_description | what_is_tested | | --- | --- | --- | -| `cross_language_proto` | Protobuf envelopes read from the cross-language fixture directory answer their point queries in band. | Asserts nothing unless fixtures are available: with no `XTEST_DIR` and no discoverable producer directory holding `tests/cross_language/xtest_producer_test.go`, it prints a skip line and returns. With fixtures it decodes nine `SketchEnvelope` files and records a pass or failure for each — `countmin.pb` min-frequency for `item:42` at least `101`, `kll.pb` p50 within 5% of `5000` and p99 within 5% of `9900`, `ddsketch.pb` the same two quantiles within 2%, `hll.pb` an ErtlMLE cardinality inside `[40_000, 65_000]`, `countsketch.pb` `cs:hot` at least `200`, `coco.pb` `coco:hot` at least `500`, `elastic.pb` `elephant` at least `900`, `univmon.pb` a cardinality inside `[1_000, 15_000]`, `hydra.pb` `hydra:42` at least `51` — using the hash seeds `0xcafe3553`, `0x6a09e667` and `0xbb67ae85`, and, when `countmin_sampled.pb` and `hll_sampled.pb` exist, requires `effective_sample_p` of `0.1` and a `rescale_count` by `1/p` within 5% of `100_000` and 6% of `200_000`. A single `assert!` at the end fails if any check failed; a missing or mistyped envelope panics on the spot. | +| `cross_language_proto` | Protobuf envelopes read from the cross-language fixture directory answer their point queries in band. | Asserts nothing unless fixtures are available: with no `XTEST_DIR` and no discoverable producer directory holding `tests/cross_language/xtest_producer_test.go`, it prints a skip line and returns. With fixtures it decodes eight `SketchEnvelope` files and records a pass or failure for each — `countmin.pb` min-frequency for `item:42` at least `101`, `kll.pb` p50 within 5% of `5000` and p99 within 5% of `9900`, `ddsketch.pb` the same two quantiles within 2%, `hll.pb` an ErtlMLE cardinality inside `[40_000, 65_000]`, `coco.pb` `coco:hot` at least `500`, `elastic.pb` `elephant` at least `900`, `univmon.pb` a cardinality inside `[1_000, 15_000]`, `hydra.pb` `hydra:42` at least `51` — using the hash seeds `0xcafe3553`, `0x6a09e667` and `0xbb67ae85`, and, when `countmin_sampled.pb` and `hll_sampled.pb` exist, requires `effective_sample_p` of `0.1` and a `rescale_count` by `1/p` within 5% of `100_000` and 6% of `200_000`. A single `assert!` at the end fails if any check failed; a missing or mistyped envelope panics on the spot. | ### HLL Custom Precision diff --git a/proto/countminsketch/countminsketch.proto b/proto/countminsketch/countminsketch.proto index 27e09d3b..18a9dd8f 100644 --- a/proto/countminsketch/countminsketch.proto +++ b/proto/countminsketch/countminsketch.proto @@ -77,10 +77,9 @@ message CountMinState { // byte-identical delta frames for identical window state (cross-language byte // parity). // -// CountMinDelta is structurally identical to CountSketchDelta: rows, cols, the -// packed cell encoding, per-row norm deltas, and an optional repeated hh_keys -// at the SAME tag number (6). Both sketches can track heavy hitters; whether -// hh_keys is populated is a control-plane decision (it is empty/omitted when +// CountMinDelta carries rows, cols, the packed cell encoding, per-row norm +// deltas, and an optional repeated hh_keys (tag 6). Whether hh_keys is +// populated is a control-plane decision (it is empty/omitted when // heavy-hitter tracking is not enabled for this sketch). message CountMinDelta { uint32 rows = 1; @@ -92,8 +91,7 @@ message CountMinDelta { repeated double l1 = 4 [packed = true]; // Per-row L2 norm deltas, length = rows. repeated double l2 = 5 [packed = true]; - // Heavy-hitter candidate keys from an upstream tracker, mirroring - // CountSketchDelta.hh_keys (same tag number 6, same wire shape). Downstream + // Heavy-hitter candidate keys from an upstream tracker. Downstream // queries the merged Count-Min matrix for each key to (re)build its Top-K // with globally-merged estimates. Empty/omitted when heavy-hitter tracking // is not enabled. diff --git a/proto/countsketch/countsketch.proto b/proto/countsketch/countsketch.proto index ae921100..71e05513 100644 --- a/proto/countsketch/countsketch.proto +++ b/proto/countsketch/countsketch.proto @@ -1,5 +1,5 @@ -// countsketch.proto — Portable wire format for Count Sketch state. -// Also contains TopKState and HeapEntry used by CountSketch and UnivMon. +// countsketch.proto — CountSketchState, TopKState and HeapEntry, the +// messages UnivMon and Hydra embed. syntax = "proto3"; @@ -72,52 +72,3 @@ message HeapEntry { reserved 3; // Reserved for bytes_key alternative. } - -// ============================================================================ -// Count Sketch Delta -// ============================================================================ - -// CountSketchDelta carries only the matrix cells that changed by at least the -// threshold between two consecutive snapshots, plus the full per-row L2 norm -// deltas and any heavy-hitter candidate keys. -// -// Cells apply additively on the receiver: matrix[row][col] += d_count. Count -// Sketch counters are signed (±1 increments), so d_count is a signed integer -// (sint64). Heavy-hitter candidate keys (hh_keys) are re-queried against the -// merged matrix downstream to rebuild the receiver's Top-K with accurate, -// globally-merged estimates. -// -// The packed cell encoding (cell_rows / cell_cols / d_counts, tags 9-11) plus -// l2 (tag 12) is the canonical form; the legacy per-cell message (tag 3), -// l2_legacy (tag 4) and topk (tag 5) are retained only so payloads from older -// producers still decode. This message is byte-identical to the Go reference -// implementation's CountSketchDelta so the two runtimes emit byte-identical -// delta frames for identical window state (cross-language byte parity). -message CountSketchDelta { - uint32 rows = 1; - uint32 cols = 2; - // Deprecated: use cell_rows/cell_cols/d_counts (tags 9-11). Retained for - // backward-compatible decode of payloads from older producers. - repeated CountSketchCell cells_legacy = 3; - // Deprecated: use l2 (tag 12) instead. - repeated double l2_legacy = 4 [packed = true]; - // Deprecated: use hh_keys (tag 6) instead. - TopKState topk = 5; - // Heavy-hitter candidate keys from the upstream tracker. Downstream queries - // the merged Count Sketch matrix for each key to build its Top-K. - repeated string hh_keys = 6; - // Packed cell encoding (canonical form): - repeated uint32 cell_rows = 9 [packed = true]; // row index of each changed cell - repeated uint32 cell_cols = 10 [packed = true]; // col index of each changed cell - repeated sint64 d_counts = 11 [packed = true]; // signed integer count delta - repeated double l2 = 12 [packed = true]; // per-row L2 norm deltas -} - -// CountSketchCell is the deprecated per-cell delta record. Producers emit the -// packed cell arrays on CountSketchDelta instead; this message exists only to -// decode payloads from older producers. -message CountSketchCell { - uint32 row = 1; - uint32 col = 2; - double d_count = 3; // deprecated: use CountSketchDelta.d_counts -} diff --git a/proto/sketchlib.proto b/proto/sketchlib.proto index 0a6d8193..e96d83de 100644 --- a/proto/sketchlib.proto +++ b/proto/sketchlib.proto @@ -67,7 +67,6 @@ message SketchEnvelope { // The sketch payload. Exactly one field must be set. oneof sketch_state { CountMinState count_min = 10; - CountSketchState count_sketch = 11; HyperLogLogState hll = 12; KLLState kll = 13; DDSketchState ddsketch = 14; @@ -77,6 +76,8 @@ message SketchEnvelope { ElasticState elastic = 18; } + reserved 11; + reserved "count_sketch"; reserved 19 to 31; // Reserved for future sketch types. // Geometric / NitroSketch sampling probability for the producing sketch. diff --git a/src/common/hashspec.rs b/src/common/hashspec.rs index 13c44c19..0c653029 100644 --- a/src/common/hashspec.rs +++ b/src/common/hashspec.rs @@ -13,8 +13,7 @@ //! ±1 signs. //! //! This module re-exposes that pipeline as a small, byte-key API so -//! sketches that consume keys as raw bytes (e.g. the wire-format -//! `CountSketch::update`) can match Go's emitted matrix +//! sketches that consume keys as raw bytes can match Go's emitted matrix //! cell-for-cell. CountMinSketch's pending byte-parity fix will reuse //! the same primitives — the seed table, derive_index, and derive_sign //! are agnostic to whether ±1 signing is applied. @@ -75,10 +74,7 @@ pub const CANONICAL_HASH_SEED: usize = 5; /// derivation strategy so consumers can validate hash compatibility /// before merging. /// -/// `asap_sketchlib`'s wire-format CountSketch constructs a -/// `HashSpec::default()` and feeds it to [`derive_index`] / -/// [`derive_sign`] on the hot path. The default matches Go's -/// `portableHashSpec()`: +/// The default matches Go's `portableHashSpec()`: /// /// - `seed_list = CANONICAL_HASH_SEED_TABLE` /// - `canonical_seed_index = CANONICAL_HASH_SEED` (= 5) diff --git a/src/lib.rs b/src/lib.rs index b062b8b9..47af9d8e 100644 --- a/src/lib.rs +++ b/src/lib.rs @@ -10,7 +10,7 @@ //! UnivMon, tumbling windows, and batch/parallel execution helpers. //! - [`message_pack_format`]: MessagePack/proto wire format shared with //! `sketchlib-go`. Owns the wire-DTO sketch types consumed by the ASAP -//! query engine (`CountMinSketch`, `CountSketch`, `KllSketch`, `HllSketch`, +//! query engine (`CountMinSketch`, `KllSketch`, `HllSketch`, //! `DdSketch`, `HydraKllSketch`, `CountMinSketchWithHeap`, //! `SetAggregator`, `DeltaResult`). //! @@ -43,9 +43,6 @@ pub use common::*; pub use message_pack_format::MessagePackCodec; pub use message_pack_format::portable::countminsketch::{CountMinSketch, CountMinSketchDelta}; pub use message_pack_format::portable::countminsketch_topk::{CmsHeapItem, CountMinSketchWithHeap}; -pub use message_pack_format::portable::countsketch::{ - COUNT_SKETCH_TOPK_CAPACITY, CountSketch, CountSketchDelta, -}; pub use message_pack_format::portable::countsketch_topk::{CountSketchWithHeap, CsHeapItem}; pub use message_pack_format::portable::ddsketch::{DDSKETCH_GROW_CHUNK, DdSketch, DdSketchDelta}; pub use message_pack_format::portable::delta_set_aggregator::DeltaResult; diff --git a/src/message_pack_format/native/countsketch.rs b/src/message_pack_format/native/countsketch.rs deleted file mode 100644 index 89375065..00000000 --- a/src/message_pack_format/native/countsketch.rs +++ /dev/null @@ -1,32 +0,0 @@ -//! Native MessagePack codec impl for [`crate::sketches::countsketch::Count`]. -//! -//! Only the canonical wire configs — `i32` or `i64` counters (`CsWireCounter`) -//! with a fast/regular mode (`CsWireMode`) — are serializable. There is no `f64` -//! counterpart to Count-Min's, because `CountSketchCounter` is implemented only -//! for `i32`, `i64` and `i128`; `i128` and non-`Vector2D` storage must be -//! converted first. - -use serde::{Deserialize, Serialize}; - -use crate::message_pack_format::{Error, MessagePackCodec}; -use crate::sketches::countsketch::{Count, CountSketchCounter, CsWireCounter, CsWireMode}; -use crate::{HashProfile, SketchHasher, Vector2D}; - -impl MessagePackCodec for Count, Mode, H> -where - T: CsWireCounter - + CountSketchCounter - + std::ops::AddAssign - + Serialize - + for<'de> Deserialize<'de>, - Mode: CsWireMode, - H: SketchHasher + HashProfile, -{ - fn to_msgpack(&self) -> Result, Error> { - Ok(self.serialize_to_bytes()?) - } - - fn from_msgpack(bytes: &[u8]) -> Result { - Ok(Self::deserialize_from_bytes(bytes)?) - } -} diff --git a/src/message_pack_format/native/mod.rs b/src/message_pack_format/native/mod.rs index 0a83ef96..2d01b364 100644 --- a/src/message_pack_format/native/mod.rs +++ b/src/message_pack_format/native/mod.rs @@ -8,7 +8,6 @@ //! specifies the format `sketchlib-go` mirrors. pub mod countminsketch; -pub mod countsketch; pub mod countsketch_topk; pub mod ddsketch; pub mod hll; diff --git a/src/message_pack_format/portable/countminsketch.rs b/src/message_pack_format/portable/countminsketch.rs index 16087bdb..e6b000a1 100644 --- a/src/message_pack_format/portable/countminsketch.rs +++ b/src/message_pack_format/portable/countminsketch.rs @@ -90,9 +90,8 @@ pub fn sketchlib_cms_query(inner: &SketchlibCms, key: &str) -> f64 { /// L1 and L2 norm deltas are carried for downstream error-accounting /// but are not consumed by `apply_delta` itself. /// -/// Structurally identical to [`super::countsketch::CountSketchDelta`]: the -/// optional `hh_keys` channel carries heavy-hitter candidate keys forwarded by -/// an upstream tracker. Count-Min can track heavy hitters, but whether +/// The optional `hh_keys` channel carries heavy-hitter candidate keys +/// forwarded by an upstream tracker. Count-Min can track heavy hitters, but whether /// `hh_keys` is populated is a control-plane decision — it is empty when /// heavy-hitter tracking is not enabled. The receiver re-queries the merged /// matrix for each key to (re)build its Top-K. Mirrors the Go reference @@ -266,9 +265,8 @@ impl CountMinSketch { /// Go producer maintains incrementally. Heavy-hitter candidate keys /// (`hh_keys`) are sourced from an upstream tracker that this minimal /// wrapper does not maintain, so the field is left empty (an empty repeated - /// field encodes to nothing on the wire), exactly as CountSketch's - /// `compute_delta` does. The returned bytes are a `prost`-encoded - /// [`crate::proto::sketchlib::CountMinDelta`], byte-identical to the Go + /// field encodes to nothing on the wire). The returned bytes are a + /// `prost`-encoded [`crate::proto::sketchlib::CountMinDelta`], byte-identical to the Go /// `proto.Marshal(CountMinDelta)` output for the same inputs when no /// heavy-hitter keys are forwarded (cross-language byte parity). /// @@ -334,7 +332,7 @@ impl CountMinSketch { l1, l2, // Heavy-hitter keys are control-plane-gated; this wrapper has no - // tracker, so the field is empty (mirrors CountSketch::compute_delta). + // tracker, so the field is empty. hh_keys: Vec::new(), cell_rows, cell_cols, @@ -348,9 +346,9 @@ impl CountMinSketch { /// Go reference implementation's `DeserializeDelta` + `ApplyDelta`. /// Reads the packed cell arrays (`cell_rows`/`cell_cols`/`d_counts`) /// and falls back to the legacy per-cell records for payloads from - /// older producers. Heavy-hitter keys (`hh_keys`) are read symmetrically - /// (mirroring CountSketch) and carried onto the decoded delta; a plain CMS - /// keeps no Top-K, so they are not otherwise consumed here. + /// older producers. Heavy-hitter keys (`hh_keys`) are read and carried + /// onto the decoded delta; a plain CMS keeps no Top-K, so they are not + /// otherwise consumed here. /// /// Returns `Err` if `bytes` is not a valid `CountMinDelta` proto or a /// cell is out of range for this sketch's dimensions. @@ -383,10 +381,8 @@ impl CountMinSketch { cells, l1: proto.l1, l2: proto.l2, - // Heavy-hitter keys are carried through symmetrically (mirrors - // CountSketch). This wrapper keeps no Top-K, so they are not - // otherwise consumed; a control-plane sink would re-query the - // merged matrix for each key. Empty for a plain CMS. + // This wrapper keeps no Top-K, so heavy-hitter keys are carried + // through but not otherwise consumed. Empty for a plain CMS. hh_keys: proto.hh_keys, }; self.apply_delta(&delta) diff --git a/src/message_pack_format/portable/countsketch.rs b/src/message_pack_format/portable/countsketch.rs deleted file mode 100644 index a94ad981..00000000 --- a/src/message_pack_format/portable/countsketch.rs +++ /dev/null @@ -1,974 +0,0 @@ -//! Wire-format-aligned Count Sketch types. The wire DTO and runtime -//! ops live together here. - -use serde::{Deserialize, Serialize}; - -use crate::message_pack_format::{Error as MsgPackError, MessagePackCodec}; - -// ===================================================================== -// ASAP runtime wire-format-aligned variant . -// -// `CountSketch` and `CountSketchDelta` below are the public-field, -// proto-decode-friendly types consumed by the ASAP query engine -// accumulators. The high-throughput in-process variant is -// `crate::sketches::countsketch::Count`, which keeps its own design. -// ===================================================================== - -// Count Sketch (a.k.a. Count-Min-style signed-counter sketch) — -// element-wise mergeable frequency estimator. -// -// Parallel to `count_min::CountMinSketch` but with **signed** counters, -// matching the `asap_sketchlib::proto::sketchlib::CountSketchState` wire -// format that DataCollector's `countsketchprocessor` emits via the -// modified OTLP `Metric.data = CountSketch{…}` variant. -// -// The surface is: construct from a decoded proto state, merge -// element-wise with another sketch, emit the matrix for queries and -// serialization. The richer query semantics of Count Sketch (median- -// of-estimators heavy-hitter tracking, `TopKState` integration) are not -// implemented here — the wire format carries the matrix losslessly, so -// the merge/store round-trip works with just a matrix. - -/// Default Top-K capacity. Mirrors sketchlib-go `TOPK_SIZE = 100`. -pub const COUNT_SKETCH_TOPK_CAPACITY: usize = 100; - -/// Sparse delta between two consecutive CountSketch snapshots — -/// the input shape for [`CountSketch::apply_delta`]. Mirrors the -/// `CountSketchDelta` proto in -/// `sketchlib-go/proto/countsketch/countsketch.proto` and the native -/// Go `Delta` in `sketchlib-go/sketches/CountSketch/delta.go`. -/// -/// Cells apply additively: `matrix[row][col] += d_count` for each -/// `(row, col, d_count)` triple. Per-row L2 norm deltas apply -/// additively. Heavy-hitter candidate keys (`hh_keys`) are queried -/// against the post-merge matrix and used to rebuild the receiver's -/// Top-K heap. -#[derive(Debug, Clone, Default)] -pub struct CountSketchDelta { - pub rows: u32, - pub cols: u32, - /// `(row, col, d_count)` cell updates, additive on the CS matrix. - pub cells: Vec<(u32, u32, i64)>, - /// Per-row L2 norm deltas. Additive, one scalar per row of the - /// base sketch. Kept on the delta surface for downstream - /// error-accounting; `apply_delta` itself ignores L2. - pub l2: Vec, - /// Heavy-hitter candidate keys forwarded by the upstream - /// Space-Saving tracker. The receiver re-queries the merged CS - /// matrix for each key and updates its Top-K heap with the - /// resulting estimate. Mirrors Go's `Delta.HHKeys`. - pub hh_keys: Vec, -} - -/// Minimal Count Sketch state — a flat `rows × cols` matrix of signed -/// counts. Element-wise mergeable (sum over aligned cells). Mirrors -/// sketchlib-go's `CountSketch.Count`/`TopK` pair (the on-the-wire -/// `L2` field is a derived value and is recomputed on load). -#[derive(Debug, Clone, Serialize, Deserialize)] -pub struct CountSketch { - #[serde(rename = "row_num")] - pub rows: usize, - #[serde(rename = "col_num")] - pub cols: usize, - /// Row-major matrix of signed counts. `matrix[r][c]` is the value of - /// hash row `r`, column `c`. - pub matrix: Vec>, - /// Top-K heavy hitters as `(key, count)` pairs, capped at - /// [`COUNT_SKETCH_TOPK_CAPACITY`]. Order is not guaranteed (heap - /// shape is not preserved on the wire). Mirrors Go's - /// `CountSketch.TopK` slot. Defaults to empty on legacy payloads. - #[serde(default)] - pub topk: Vec<(String, f64)>, -} - -impl CountSketch { - /// Construct an all-zero sketch with the given dimensions. - pub fn new(rows: usize, cols: usize) -> Self { - Self { - rows, - cols, - matrix: vec![vec![0.0; cols]; rows], - topk: Vec::new(), - } - } - - /// Construct from a pre-built matrix (used by the modified-OTLP - /// proto-decode path). `topk` is zero-initialised; callers that - /// need non-zero auxiliary state should use the msgpack/proto path. - pub fn from_legacy_matrix(matrix: Vec>, rows: usize, cols: usize) -> Self { - debug_assert_eq!(matrix.len(), rows, "row count mismatch"); - debug_assert!( - matrix.iter().all(|r| r.len() == cols), - "column count mismatch in at least one row" - ); - Self { - rows, - cols, - matrix, - topk: Vec::new(), - } - } - - /// Borrow the inner matrix. - pub fn sketch(&self) -> &Vec> { - &self.matrix - } - - /// Update the in-memory Top-K heap with `(key, count)`. Keeps the - /// heap bounded by [`COUNT_SKETCH_TOPK_CAPACITY`]; on overflow, - /// drops the smallest-count entry. If `key` is already present, - /// the new count replaces the old (max semantics). Used by - /// `apply_delta` to rebuild Top-K from `hh_keys`. - fn topk_update(&mut self, key: &str, count: f64) { - if let Some(slot) = self.topk.iter_mut().find(|(k, _)| k == key) { - if count > slot.1 { - slot.1 = count; - } - return; - } - if self.topk.len() < COUNT_SKETCH_TOPK_CAPACITY { - self.topk.push((key.to_owned(), count)); - return; - } - // Capacity hit: replace the minimum if `count` exceeds it. - if let Some((min_idx, min_count)) = self - .topk - .iter() - .enumerate() - .min_by(|a, b| { - a.1.1 - .partial_cmp(&b.1.1) - .unwrap_or(std::cmp::Ordering::Equal) - }) - .map(|(i, e)| (i, e.1)) - { - if count > min_count { - self.topk[min_idx] = (key.to_owned(), count); - } - } - } - - /// Insert a single weighted observation. Uses the shared - /// [`crate::MatrixHashType::Packed64`] fast-path hash so the matrix-cell - /// layout matches `sketchlib-go::CountSketch.UpdateString` - /// bit-for-bit. Locked in by - /// `tests/sketches_go_parity_probe.rs`. - /// - /// Constraints inherited from Go's Packed64 mode: - /// - `cols` must be a power of two; the column mask is `cols - 1`. - /// - `rows * (log2(cols) + 1) ≤ 64` so the packed hash holds all - /// row indices and sign bits. - pub fn update(&mut self, key: &str, value: f64) { - if self.rows == 0 || self.cols == 0 { - return; - } - let input = crate::common::DataInput::String(key.to_owned()); - let hashed = crate::common::MatrixHashType::Packed64( - ::hash64_seeded(0, &input), - ); - for r in 0..self.rows { - let col = ::col_for_row( - &hashed, r, self.cols, - ); - let sign = - ::sign_for_row( - &hashed, r, - ) as f64; - self.matrix[r][col] += sign * value; - } - } - - /// Estimate the frequency of `key` via the standard median-of-rows - /// CountSketch query. Returns 0 for an empty sketch. Inverts the - /// signed-counter projection that [`Self::update`] applied. - pub fn estimate(&self, key: &str) -> f64 { - if self.rows == 0 || self.cols == 0 { - return 0.0; - } - let input = crate::common::DataInput::String(key.to_owned()); - let hashed = crate::common::MatrixHashType::Packed64( - ::hash64_seeded(0, &input), - ); - let mut estimates: Vec = Vec::with_capacity(self.rows); - for r in 0..self.rows { - let col = ::col_for_row( - &hashed, r, self.cols, - ); - let sign = - ::sign_for_row( - &hashed, r, - ) as f64; - estimates.push(sign * self.matrix[r][col]); - } - estimates.sort_by(|a, b| a.partial_cmp(b).unwrap_or(std::cmp::Ordering::Equal)); - let mid = estimates.len() / 2; - if estimates.len() % 2 == 1 { - estimates[mid] - } else { - (estimates[mid - 1] + estimates[mid]) / 2.0 - } - } - - /// Merge one other sketch into self via element-wise addition. Both - /// operands must have identical dimensions. - pub fn merge( - &mut self, - other: &CountSketch, - ) -> Result<(), Box> { - if self.rows != other.rows || self.cols != other.cols { - return Err(format!( - "CountSketch dimension mismatch: self={}x{}, other={}x{}", - self.rows, self.cols, other.rows, other.cols - ) - .into()); - } - for r in 0..self.rows { - for c in 0..self.cols { - self.matrix[r][c] += other.matrix[r][c]; - } - } - Ok(()) - } - - /// Apply a sparse delta in place. Matches the `ApplyDelta` - /// semantics in `sketchlib-go/sketches/CountSketch/delta.go`: - /// * each `(row, col, d_count)` triple updates the count matrix - /// additively (`matrix[r][c] += d_count`); - /// * each `hh_key` is re-queried against the post-update matrix - /// and pushed into the receiver's Top-K with the merged-estimate - /// count (mirrors Go's `Delta.HHKeys` heavy-hitter rebuild). - /// - /// Returns `Err` if any `(row, col)` is out of range — indicating - /// a dimension mismatch between the snapshot this sketch was - /// built from and the delta sender. - pub fn apply_delta( - &mut self, - delta: &CountSketchDelta, - ) -> Result<(), Box> { - // 1. Cell additions. - for (row, col, d_count) in &delta.cells { - let r = *row as usize; - let c = *col as usize; - if r >= self.rows || c >= self.cols { - return Err(format!( - "CountSketchDelta cell ({r},{c}) out of range (matrix={}x{})", - self.rows, self.cols - ) - .into()); - } - // `d_count` is signed on the wire; CS counts are signed - // too (can go negative under adversarial keys). - self.matrix[r][c] += *d_count as f64; - } - // 2. Heavy-hitter rebuild from `hh_keys`. Re-estimate against - // the freshly-updated matrix and push into Top-K with the - // merged count. Mirrors sketchlib-go's `Delta.HHKeys` path. - for key in &delta.hh_keys { - let est = self.estimate(key); - self.topk_update(key, est); - } - Ok(()) - } - - /// Compute a sparse, proto-marshalled `CountSketchDelta` of `self` - /// against a `snapshot`. A cell is included when its signed count - /// delta `Δcount` (self − snapshot) is non-zero and its magnitude is - /// `>= threshold` (the threshold is truncated to an integer, matching - /// the Go reference). The full per-row L2 norm deltas are always - /// carried (one entry per row). - /// - /// This is the Rust twin of the Go reference implementation's - /// `ComputeDelta` + `SerializeDelta`: it iterates the matrix in - /// row-major order, subtracts the snapshot's value for each cell, and - /// emits the surviving signed cell deltas using the packed-array proto - /// encoding (`cell_rows`/`cell_cols`/`d_counts`). L2 row deltas are - /// derived from the matrices — `l2[r] = Σ_c count[r][c]^2` — matching - /// the Go producer's incrementally-maintained norm. Heavy-hitter - /// candidate keys (`hh_keys`) are sourced from an upstream Space-Saving - /// tracker that this minimal wrapper does not maintain, so the field is - /// left empty (an empty repeated field encodes to nothing on the wire). - /// The returned bytes are a `prost`-encoded - /// [`crate::proto::sketchlib::CountSketchDelta`], byte-identical to the - /// Go `proto.Marshal(CountSketchDelta)` output for the same inputs when - /// no heavy-hitter keys are forwarded (cross-language byte parity). - /// - /// Delta-against-empty: when `snapshot` is the all-zero sketch, every - /// surviving cell delta equals this window's own (signed) cell count, - /// so the result is this window's full state encoded as a delta (no - /// cross-window subtraction). CountSketch deltas carry only - /// sketch-internal cells/norms — there are no DataPoint-level metric - /// scalars to drop. - pub fn compute_delta( - &self, - snapshot: &CountSketch, - threshold: f64, - ) -> Result, Box> { - use crate::proto::sketchlib::CountSketchDelta as ProtoDelta; - use prost::Message; - - if self.rows != snapshot.rows || self.cols != snapshot.cols { - return Err(format!( - "CountSketch dimension mismatch: self={}x{}, snapshot={}x{}", - self.rows, self.cols, snapshot.rows, snapshot.cols - ) - .into()); - } - - // Threshold is compared against |Δ| after truncation to an - // integer, mirroring the Go reference's `int64(threshold)` cast. - let thresh = threshold as i64; - - let mut cell_rows: Vec = Vec::new(); - let mut cell_cols: Vec = Vec::new(); - let mut d_counts: Vec = Vec::new(); - let mut l2: Vec = Vec::with_capacity(self.rows); - - for r in 0..self.rows { - let mut cur_l2 = 0.0f64; - let mut snap_l2 = 0.0f64; - for c in 0..self.cols { - let cv = self.matrix[r][c]; - let sv = snapshot.matrix[r][c]; - cur_l2 += cv * cv; - snap_l2 += sv * sv; - let dc = (cv - sv) as i64; - if dc != 0 && (dc <= -thresh || dc >= thresh) { - cell_rows.push(r as u32); - cell_cols.push(c as u32); - d_counts.push(dc); - } - } - l2.push(cur_l2 - snap_l2); - } - - let delta = ProtoDelta { - rows: self.rows as u32, - cols: self.cols as u32, - cells_legacy: Vec::new(), - l2_legacy: Vec::new(), - topk: None, - hh_keys: Vec::new(), - cell_rows, - cell_cols, - d_counts, - l2, - }; - Ok(delta.encode_to_vec()) - } - - /// Apply a `prost`-encoded [`crate::proto::sketchlib::CountSketchDelta`] - /// to this sketch in place (additive signed-cell merge + heavy-hitter - /// rebuild). The Rust twin of the Go reference implementation's - /// `DeserializeDelta` + `ApplyDelta`. Reads the packed cell arrays - /// (`cell_rows`/`cell_cols`/`d_counts`) and the heavy-hitter keys - /// (`hh_keys`), falling back to the legacy per-cell records and TopK - /// entries for payloads from older producers. - /// - /// Returns `Err` if `bytes` is not a valid `CountSketchDelta` proto or - /// a cell is out of range for this sketch's dimensions. - pub fn apply_delta_bytes( - &mut self, - bytes: &[u8], - ) -> Result<(), Box> { - use crate::proto::sketchlib::CountSketchDelta as ProtoDelta; - use prost::Message; - - let proto = ProtoDelta::decode(bytes)?; - let cells: Vec<(u32, u32, i64)> = if !proto.cell_rows.is_empty() { - proto - .cell_rows - .iter() - .zip(proto.cell_cols.iter()) - .zip(proto.d_counts.iter()) - .map(|((&r, &c), &d)| (r, c, d)) - .collect() - } else { - proto - .cells_legacy - .iter() - .map(|c| (c.row, c.col, c.d_count as i64)) - .collect() - }; - let l2 = if !proto.l2.is_empty() { - proto.l2 - } else { - proto.l2_legacy - }; - let hh_keys = if !proto.hh_keys.is_empty() { - proto.hh_keys - } else if let Some(topk) = proto.topk { - topk.entries.into_iter().map(|e| e.key).collect() - } else { - Vec::new() - }; - let delta = CountSketchDelta { - rows: proto.rows, - cols: proto.cols, - cells, - l2, - hh_keys, - }; - self.apply_delta(&delta) - } - - /// MessagePack twin of [`Self::compute_delta`]. Computes the same - /// sparse signed-cell delta of `self` against `snapshot` (a cell is - /// carried when `Δcount` is non-zero and `|Δcount| >= threshold`, the - /// threshold truncated to an integer), but serializes it as the - /// cells-only parallel-array MessagePack layout via - /// [`CountSketchDelta`]'s [`MessagePackCodec`] instead of proto. The - /// per-row `l2` norms and `hh_keys` the proto delta carries are dropped - /// — the plain-sketch estimate path does not consume them. - /// Delta-against-empty carries this window's own signed cell counts. - pub fn compute_delta_msgpack( - &self, - snapshot: &CountSketch, - threshold: f64, - ) -> Result, Box> { - if self.rows != snapshot.rows || self.cols != snapshot.cols { - return Err(format!( - "CountSketch dimension mismatch: self={}x{}, snapshot={}x{}", - self.rows, self.cols, snapshot.rows, snapshot.cols - ) - .into()); - } - let thresh = threshold as i64; - let mut delta = CountSketchDelta { - rows: self.rows as u32, - cols: self.cols as u32, - ..CountSketchDelta::default() - }; - for r in 0..self.rows { - for c in 0..self.cols { - let dc = (self.matrix[r][c] - snapshot.matrix[r][c]) as i64; - if dc != 0 && (dc <= -thresh || dc >= thresh) { - delta.cells.push((r as u32, c as u32, dc)); - } - } - } - Ok(delta.to_msgpack()?) - } - - /// MessagePack twin of [`Self::apply_delta_bytes`]. Decodes the - /// cells-only parallel-array MessagePack [`CountSketchDelta`] and - /// applies it in place (additive signed-cell merge). - /// - /// Returns `Err` if `bytes` is not a valid MessagePack - /// `CountSketchDelta` or a cell is out of range for this sketch's - /// dimensions. - pub fn apply_delta_msgpack_bytes( - &mut self, - bytes: &[u8], - ) -> Result<(), Box> { - let delta = CountSketchDelta::from_msgpack(bytes)?; - self.apply_delta(&delta) - } - - /// Merge a slice of references into a single new sketch. All inputs - /// must share the same dimensions; returns `Err` on mismatch or an - /// empty input. - pub fn merge_refs( - inputs: &[&CountSketch], - ) -> Result> { - let first = inputs - .first() - .ok_or("CountSketch::merge_refs called with empty input")?; - let mut merged = CountSketch::new(first.rows, first.cols); - for cs in inputs { - merged.merge(cs)?; - } - Ok(merged) - } -} - -impl MessagePackCodec for CountSketch { - fn to_msgpack(&self) -> Result, MsgPackError> { - Ok(rmp_serde::to_vec(self)?) - } - - fn from_msgpack(bytes: &[u8]) -> Result { - Ok(rmp_serde::from_slice(bytes)?) - } -} - -impl MessagePackCodec for CountSketchDelta { - /// Encodes the sparse signed-cell delta as the 5-element MessagePack - /// array - /// `[ rows:u64, cols:u64, row_idx:[]u32, col_idx:[]u32, d_count:[]i64 ]` - /// — byte-identical to Rust - /// `rmp_serde::to_vec(&(u64, u64, Vec, Vec, Vec))` - /// (compact mode), the layout the Go `count_sketch_delta_sparse.go` - /// established (with signed `d_count` in place of the F2-broadcast's - /// `f64` vals). Only the matrix cells cross the wire; the per-row `l2` - /// norms and `hh_keys` heavy-hitter surface are not carried. - fn to_msgpack(&self) -> Result, MsgPackError> { - let mut row_idx: Vec = Vec::with_capacity(self.cells.len()); - let mut col_idx: Vec = Vec::with_capacity(self.cells.len()); - let mut d_count: Vec = Vec::with_capacity(self.cells.len()); - for (r, c, d) in &self.cells { - row_idx.push(*r); - col_idx.push(*c); - d_count.push(*d); - } - Ok(rmp_serde::to_vec(&( - self.rows as u64, - self.cols as u64, - row_idx, - col_idx, - d_count, - ))?) - } - - fn from_msgpack(bytes: &[u8]) -> Result { - let (rows, cols, row_idx, col_idx, d_count): (u64, u64, Vec, Vec, Vec) = - rmp_serde::from_slice(bytes)?; - let cells = row_idx - .into_iter() - .zip(col_idx) - .zip(d_count) - .map(|((r, c), d)| (r, c, d)) - .collect(); - Ok(CountSketchDelta { - rows: rows as u32, - cols: cols as u32, - cells, - ..CountSketchDelta::default() - }) - } -} - -#[cfg(test)] -mod tests { - use super::*; - - #[test] - fn msgpack_delta_against_empty_round_trips() { - let (rows, cols) = (5, 1024); - let mut w = CountSketch::new(rows, cols); - for _ in 0..200 { - w.update("hot", 1.0); - } - let empty = CountSketch::new(rows, cols); - let bytes = w.compute_delta_msgpack(&empty, 1.0).unwrap(); - let mut recon = CountSketch::new(rows, cols); - recon.apply_delta_msgpack_bytes(&bytes).unwrap(); - // CountSketch round-trips its cells exactly under delta-against-empty. - assert_eq!(recon.sketch(), w.sketch()); - } - - #[test] - fn msgpack_delta_wire_layout_is_parallel_arrays() { - use crate::message_pack_format::MessagePackCodec; - let (rows, cols) = (3, 8); - let mut w = CountSketch::new(rows, cols); - w.update("x", 5.0); - let empty = CountSketch::new(rows, cols); - let bytes = w.compute_delta_msgpack(&empty, 1.0).unwrap(); - let cur = w.sketch(); - let (mut ri, mut ci, mut dc): (Vec, Vec, Vec) = - (Vec::new(), Vec::new(), Vec::new()); - for (r, row) in cur.iter().enumerate() { - for (c, &cell) in row.iter().enumerate() { - let d = cell as i64; - if d != 0 { - ri.push(r as u32); - ci.push(c as u32); - dc.push(d); - } - } - } - let expected = rmp_serde::to_vec(&(rows as u64, cols as u64, ri, ci, dc)).unwrap(); - assert_eq!(bytes, expected); - let d = CountSketchDelta::from_msgpack(&bytes).unwrap(); - assert_eq!(d.rows, rows as u32); - assert_eq!(d.cols, cols as u32); - } - - #[test] - fn test_new_empty() { - let cs = CountSketch::new(2, 3); - assert_eq!(cs.rows, 2); - assert_eq!(cs.cols, 3); - assert_eq!(cs.sketch(), &vec![vec![0.0, 0.0, 0.0], vec![0.0, 0.0, 0.0]]); - } - - #[test] - fn test_from_legacy_matrix() { - let m = vec![vec![1.0, -2.0, 3.0], vec![-4.0, 5.0, -6.0]]; - let cs = CountSketch::from_legacy_matrix(m.clone(), 2, 3); - assert_eq!(cs.sketch(), &m); - } - - #[test] - fn test_merge_element_wise() { - let mut a = CountSketch::from_legacy_matrix(vec![vec![1.0, 2.0], vec![3.0, 4.0]], 2, 2); - let b = CountSketch::from_legacy_matrix(vec![vec![-1.0, -2.0], vec![-3.0, -4.0]], 2, 2); - a.merge(&b).unwrap(); - assert_eq!(a.sketch(), &vec![vec![0.0, 0.0], vec![0.0, 0.0]]); - } - - #[test] - fn test_merge_dimension_mismatch() { - let mut a = CountSketch::new(2, 3); - let b = CountSketch::new(3, 3); - assert!(a.merge(&b).is_err()); - } - - #[test] - fn test_merge_refs() { - let a = CountSketch::from_legacy_matrix(vec![vec![1.0, 2.0]], 1, 2); - let b = CountSketch::from_legacy_matrix(vec![vec![3.0, 4.0]], 1, 2); - let c = CountSketch::from_legacy_matrix(vec![vec![5.0, 6.0]], 1, 2); - let merged = CountSketch::merge_refs(&[&a, &b, &c]).unwrap(); - assert_eq!(merged.sketch(), &vec![vec![9.0, 12.0]]); - } - - #[test] - fn test_apply_delta_additive() { - let mut cs = CountSketch::from_legacy_matrix( - vec![vec![1.0, -2.0, 3.0], vec![-4.0, 5.0, -6.0]], - 2, - 3, - ); - let delta = CountSketchDelta { - rows: 2, - cols: 3, - cells: vec![ - (0, 0, 10), // 1 + 10 = 11 - (0, 2, -3), // 3 - 3 = 0 - (1, 1, -15), // 5 - 15 = -10 - ], - l2: vec![], - hh_keys: vec![], - }; - cs.apply_delta(&delta).unwrap(); - assert_eq!( - cs.sketch(), - &vec![vec![11.0, -2.0, 0.0], vec![-4.0, -10.0, -6.0]] - ); - } - - #[test] - fn test_apply_delta_matches_full_merge() { - let base = CountSketch::from_legacy_matrix(vec![vec![1.0, 2.0], vec![3.0, 4.0]], 2, 2); - let addition = - CountSketch::from_legacy_matrix(vec![vec![10.0, 0.0], vec![0.0, 20.0]], 2, 2); - let mut via_merge = base.clone(); - via_merge.merge(&addition).unwrap(); - - let delta = CountSketchDelta { - rows: 2, - cols: 2, - cells: vec![(0, 0, 10), (1, 1, 20)], - l2: vec![], - hh_keys: vec![], - }; - let mut via_delta = base; - via_delta.apply_delta(&delta).unwrap(); - assert_eq!(via_delta.sketch(), via_merge.sketch()); - } - - #[test] - fn test_apply_delta_out_of_range() { - let mut cs = CountSketch::new(2, 3); - let delta = CountSketchDelta { - rows: 2, - cols: 3, - cells: vec![(2, 0, 1)], // row 2 out of range for 2-row matrix - l2: vec![], - hh_keys: vec![], - }; - assert!(cs.apply_delta(&delta).is_err()); - } - - #[test] - fn test_apply_delta_rebuilds_topk_from_hh_keys() { - // Build a sketch with two known keys via the in-process - // `update` path so the matrix has a coherent shape, then - // send a delta that only carries `hh_keys` entries. The - // receiver should re-query the merged matrix and populate - // `topk` with the resulting estimates. Mirrors sketchlib-go's - // `Delta.HHKeys` heavy-hitter rebuild path. - let mut cs = CountSketch::new(3, 16); - cs.update("alpha", 5.0); - cs.update("beta", 3.0); - let delta = CountSketchDelta { - rows: 3, - cols: 16, - cells: vec![], - l2: vec![], - hh_keys: vec!["alpha".to_string(), "beta".to_string()], - }; - cs.apply_delta(&delta).unwrap(); - assert_eq!(cs.topk.len(), 2); - let alpha_count = cs - .topk - .iter() - .find(|(k, _)| k == "alpha") - .map(|(_, v)| *v) - .expect("alpha should be in topk"); - let beta_count = cs - .topk - .iter() - .find(|(k, _)| k == "beta") - .map(|(_, v)| *v) - .expect("beta should be in topk"); - // Alpha was inserted with weight 5; the median estimate - // should exceed beta's (weight 3) modulo signed-counter - // cancellation in this small 3x16 matrix. - assert!( - alpha_count > beta_count, - "alpha={alpha_count} beta={beta_count}" - ); - } - - #[test] - fn test_apply_delta_hh_keys_topk_capacity() { - // Verify the Top-K heap is bounded by COUNT_SKETCH_TOPK_CAPACITY - // and that on overflow, the smallest-count entry is evicted in - // favor of a larger-count newcomer. - let mut cs = CountSketch::new(3, 1024); - let n = COUNT_SKETCH_TOPK_CAPACITY + 5; - let keys: Vec = (0..n).map(|i| format!("k{i:04}")).collect(); - // Fill all keys into the matrix so estimates are non-zero. - for (i, k) in keys.iter().enumerate() { - cs.update(k, (i + 1) as f64); - } - let delta = CountSketchDelta { - rows: 3, - cols: 1024, - cells: vec![], - l2: vec![], - hh_keys: keys.clone(), - }; - cs.apply_delta(&delta).unwrap(); - assert_eq!(cs.topk.len(), COUNT_SKETCH_TOPK_CAPACITY); - } - - #[test] - fn test_msgpack_round_trip() { - let original = - CountSketch::from_legacy_matrix(vec![vec![1.5, -2.5], vec![3.5, -4.5]], 2, 2); - let bytes = original.to_msgpack().unwrap(); - let decoded = CountSketch::from_msgpack(&bytes).unwrap(); - assert_eq!(decoded.sketch(), original.sketch()); - assert_eq!(decoded.rows, original.rows); - assert_eq!(decoded.cols, original.cols); - } - - /// Cross-language byte-parity guard against `sketchlib-go`'s - /// `CountSketch.SerializeProtoBytes` output for the deterministic - /// input `goldenCsKeys()` (25 keys "k-a"..."k-e", each repeated 5×) - /// at dimensions `(rows=3, cols=512)`. The hex blob below was - /// captured from a `proto.Marshal` of the Go envelope with - /// `Producer` and `HashSpec` cleared (matching the - /// `integration/parity/golden_test.go::TestGenerateGoldenFixtures` - /// recipe used for DDSketch and KLL). - /// - /// Any drift in [`CountSketch::update`]'s hash → (col, sign) - /// derivation breaks this test cell-for-cell; that is the contract - /// `cross_language_parity::countsketch_byte_parity_with_go` in - /// ASAPCollector relies on. - #[test] - fn test_update_then_envelope_matches_sketchlib_go_bytes() { - use crate::proto::sketchlib::{ - CountSketchState, CounterType, SketchEnvelope, sketch_envelope::SketchState, - }; - use prost::Message; - - let rows = 3usize; - let cols = 512usize; - let mut sk = CountSketch::new(rows, cols); - for i in 0..25 { - let key = format!("k-{}", (b'a' + (i % 5) as u8) as char); - sk.update(&key, 1.0); - } - - // Build envelope mirroring sketchlib-go's CountSketch.SerializePortable: - // counter_type = INT64, counts packed sint64 in row-major - // order, l2 = per-row sum of squared cells, no TopK. - let mut counts_int: Vec = Vec::with_capacity(rows * cols); - let mut l2: Vec = Vec::with_capacity(rows); - for row in sk.matrix.iter().take(rows) { - let mut row_l2 = 0.0f64; - for &cell in row.iter().take(cols) { - counts_int.push(cell as i64); - row_l2 += cell * cell; - } - l2.push(row_l2); - } - - let state = CountSketchState { - rows: rows as u32, - cols: cols as u32, - counter_type: CounterType::Int64 as i32, - counts_int, - counts_float: Vec::new(), - l2, - topk: None, - }; - let envelope = SketchEnvelope { - format_version: 1, - producer: None, - hash_spec: None, - sample_p: 0.0, - sketch_state: Some(SketchState::CountSketch(state)), - }; - let mut got = Vec::with_capacity(envelope.encoded_len()); - envelope.encode(&mut got).expect("prost encode"); - - // Byte string captured from sketchlib-go for the same - // (3,512) × `goldenCsKeys()` input — see - // `integration/parity/golden_test.go` and - // `cross_language_parity::countsketch_byte_parity_with_go` - // in ASAPCollector. 1577 bytes total: a `SketchEnvelope` proto - // wrapping a `CountSketchState` whose `counts_int` is a packed - // sint64 (zigzag) row-major matrix and `l2` is `[125.0;3]` - // (each row's 5 hot cells hold ±5 → l2 = 5·25 = 125). - const GOLDEN_HEX: &str = "08015aa40c0803108004180222800c000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000900000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000a0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000090000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000009000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000900000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000a000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000900000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000a0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000090000000000000a000000000000000900000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000900000000000000000000000000000000000000000000000000000000000000000a0000000000000000000000090000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000009000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000032180000000000405f400000000000405f400000000000405f40"; - let want = decode_hex(GOLDEN_HEX); - assert_eq!( - got, - want, - "CountSketch envelope bytes diverge from sketchlib-go golden \ - ({} bytes got vs {} bytes want)", - got.len(), - want.len(), - ); - } - - /// Cross-language byte-parity guard for CountSketchDelta hh_keys: encoding - /// the canonical packed delta with heavy-hitter keys at tag 6 must be - /// byte-identical to the Go reference's `SerializeDelta` output for the same - /// state and keys. The empty-`hh_keys` path is already covered by - /// `test_compute_delta_matches_go_golden_bytes`. - #[test] - fn test_hh_keys_matches_go_golden_bytes() { - use crate::proto::sketchlib::CountSketchDelta as ProtoDelta; - use prost::Message; - - let rows = 3usize; - let cols = 512usize; - let mut current = CountSketch::new(rows, cols); - for i in 0..25 { - let key = format!("k-{}", (b'a' + (i % 5) as u8) as char); - current.update(&key, 1.0); - } - let empty = CountSketch::new(rows, cols); - - // Inject heavy-hitter keys onto the canonical packed delta at tag 6. - let no_hh = current.compute_delta(&empty, 1.0).unwrap(); - let mut proto = ProtoDelta::decode(no_hh.as_slice()).unwrap(); - proto.hh_keys = vec!["flow-0".into(), "flow-3".into(), "flow-7".into()]; - let got = proto.encode_to_vec(); - - // Captured from the Go reference implementation's SerializeDelta for the - // same input with HHKeys = ["flow-0","flow-3","flow-7"]. - const GOLDEN_HEX: &str = "08031080043206666c6f772d303206666c6f772d333206666c6f772d374a0f000000000001010101010202020202521a3d7fd9019402d3034eb9019602f703fe0306a501c601d201ba025a0f090a0909090a090a090a09090a090962180000000000405f400000000000405f400000000000405f40"; - let want = decode_hex(GOLDEN_HEX); - assert_eq!( - got, - want, - "CountSketch delta hh_keys bytes diverge from the Go reference golden \ - ({} bytes got vs {} bytes want)", - got.len(), - want.len(), - ); - } - - fn decode_hex(s: &str) -> Vec { - s.as_bytes() - .chunks(2) - .map(|pair| { - let high = hex_nibble(pair[0]); - let low = hex_nibble(pair[1]); - (high << 4) | low - }) - .collect() - } - - fn hex_nibble(c: u8) -> u8 { - match c { - b'0'..=b'9' => c - b'0', - b'a'..=b'f' => c - b'a' + 10, - b'A'..=b'F' => c - b'A' + 10, - _ => panic!("non-hex byte {}", c as char), - } - } - - /// `compute_delta` against an EMPTY snapshot reconstructs the window's - /// full (signed) state when its bytes are applied to a fresh empty - /// sketch (round-trip). With `threshold = 1.0` every changed cell - /// survives. - #[test] - fn test_compute_delta_against_empty_round_trips() { - let rows = 3usize; - let cols = 512usize; - let mut window = CountSketch::new(rows, cols); - for i in 0..200u64 { - window.update(&format!("k-{}", i % 23), 1.0); - } - let empty = CountSketch::new(rows, cols); - - let delta_bytes = window.compute_delta(&empty, 1.0).unwrap(); - - let mut reconstructed = CountSketch::new(rows, cols); - reconstructed.apply_delta_bytes(&delta_bytes).unwrap(); - - assert_eq!(reconstructed.sketch(), window.sketch()); - } - - /// A delta computed between two non-empty snapshots reconstructs the - /// current sketch when applied to the base. - #[test] - fn test_compute_delta_then_apply_matches_current() { - let mut base = CountSketch::new(3, 64); - for i in 0..40u64 { - base.update(&format!("x{}", i % 8), 1.0); - } - let mut current = base.clone(); - for i in 0..30u64 { - current.update(&format!("x{}", i % 8), 1.0); - } - - let delta_bytes = current.compute_delta(&base, 1.0).unwrap(); - let mut reconstructed = base.clone(); - reconstructed.apply_delta_bytes(&delta_bytes).unwrap(); - assert_eq!(reconstructed.sketch(), current.sketch()); - } - - /// Cross-language byte-parity guard: `compute_delta` against an empty - /// snapshot must emit bytes identical to the Go reference - /// implementation's `SerializeDelta(ComputeDelta(empty, current, 1.0))` - /// for the same `(rows=3, cols=512)` × "k-a".."k-e" (each 5×, 25 - /// unweighted updates) input. The golden hex was captured from a - /// `proto.Marshal` of the Go reference's `CountSketchDelta` - /// (packed-array encoding). No heavy-hitter keys are forwarded by this - /// minimal wrapper, so `hh_keys` is empty in both producers; the golden - /// pins the packed cell arrays and the per-row L2 (=125) norm deltas. - #[test] - fn test_compute_delta_matches_go_golden_bytes() { - let rows = 3usize; - let cols = 512usize; - let mut current = CountSketch::new(rows, cols); - for i in 0..25 { - let key = format!("k-{}", (b'a' + (i % 5) as u8) as char); - current.update(&key, 1.0); - } - let empty = CountSketch::new(rows, cols); - let got = current.compute_delta(&empty, 1.0).unwrap(); - - // Captured from the Go reference implementation's - // SerializeDelta(ComputeDelta(empty, current, 1.0)) for the same input. - const GOLDEN_HEX: &str = "08031080044a0f000000000001010101010202020202521a3d7fd9019402d3034eb9019602f703fe0306a501c601d201ba025a0f090a0909090a090a090a09090a090962180000000000405f400000000000405f400000000000405f40"; - let want = decode_hex(GOLDEN_HEX); - assert_eq!( - got, - want, - "CountSketch delta bytes diverge from the Go reference golden \ - ({} bytes got vs {} bytes want)", - got.len(), - want.len(), - ); - } -} diff --git a/src/message_pack_format/portable/mod.rs b/src/message_pack_format/portable/mod.rs index 97e0d372..7b0dfc7d 100644 --- a/src/message_pack_format/portable/mod.rs +++ b/src/message_pack_format/portable/mod.rs @@ -2,14 +2,13 @@ //! //! Each per-algorithm submodule holds one sketch's pre-envelope wire type. //! These carry a cross-language contract of their own, on goldens apart -//! from ASAPv1's `asapv1_golden/`: HLL, Count-Min, Count Sketch and KLL pin +//! from ASAPv1's `asapv1_golden/`: HLL, Count-Min and KLL pin //! byte parity against Go's serializers, KLL in both directions; DDSketch's //! golden is a Rust self-pin, not yet reconciled. New code uses ASAPv1: //! prefer a sketch's own `serialize_to_bytes`; do not add callers here. pub mod countminsketch; pub mod countminsketch_topk; -pub mod countsketch; pub mod countsketch_topk; pub mod ddsketch; pub mod delta_set_aggregator; diff --git a/src/proto/generated/sketchlib.v1.rs b/src/proto/generated/sketchlib.v1.rs index 2c098d3f..f2b55c37 100644 --- a/src/proto/generated/sketchlib.v1.rs +++ b/src/proto/generated/sketchlib.v1.rs @@ -213,10 +213,9 @@ pub struct CountMinState { /// byte-identical delta frames for identical window state (cross-language byte /// parity). /// -/// CountMinDelta is structurally identical to CountSketchDelta: rows, cols, the -/// packed cell encoding, per-row norm deltas, and an optional repeated hh_keys -/// at the SAME tag number (6). Both sketches can track heavy hitters; whether -/// hh_keys is populated is a control-plane decision (it is empty/omitted when +/// CountMinDelta carries rows, cols, the packed cell encoding, per-row norm +/// deltas, and an optional repeated hh_keys (tag 6). Whether hh_keys is +/// populated is a control-plane decision (it is empty/omitted when /// heavy-hitter tracking is not enabled for this sketch). #[derive(Clone, PartialEq, ::prost::Message)] pub struct CountMinDelta { @@ -234,8 +233,7 @@ pub struct CountMinDelta { /// Per-row L2 norm deltas, length = rows. #[prost(double, repeated, tag = "5")] pub l2: ::prost::alloc::vec::Vec, - /// Heavy-hitter candidate keys from an upstream tracker, mirroring - /// CountSketchDelta.hh_keys (same tag number 6, same wire shape). Downstream + /// Heavy-hitter candidate keys from an upstream tracker. Downstream /// queries the merged Count-Min matrix for each key to (re)build its Top-K /// with globally-merged estimates. Empty/omitted when heavy-hitter tracking /// is not enabled. @@ -328,70 +326,6 @@ pub struct HeapEntry { #[prost(double, tag = "2")] pub count: f64, } -/// CountSketchDelta carries only the matrix cells that changed by at least the -/// threshold between two consecutive snapshots, plus the full per-row L2 norm -/// deltas and any heavy-hitter candidate keys. -/// -/// Cells apply additively on the receiver: matrix[row][col] += d_count. Count -/// Sketch counters are signed (±1 increments), so d_count is a signed integer -/// (sint64). Heavy-hitter candidate keys (hh_keys) are re-queried against the -/// merged matrix downstream to rebuild the receiver's Top-K with accurate, -/// globally-merged estimates. -/// -/// The packed cell encoding (cell_rows / cell_cols / d_counts, tags 9-11) plus -/// l2 (tag 12) is the canonical form; the legacy per-cell message (tag 3), -/// l2_legacy (tag 4) and topk (tag 5) are retained only so payloads from older -/// producers still decode. This message is byte-identical to the Go reference -/// implementation's CountSketchDelta so the two runtimes emit byte-identical -/// delta frames for identical window state (cross-language byte parity). -#[derive(Clone, PartialEq, ::prost::Message)] -pub struct CountSketchDelta { - #[prost(uint32, tag = "1")] - pub rows: u32, - #[prost(uint32, tag = "2")] - pub cols: u32, - /// Deprecated: use cell_rows/cell_cols/d_counts (tags 9-11). Retained for - /// backward-compatible decode of payloads from older producers. - #[prost(message, repeated, tag = "3")] - pub cells_legacy: ::prost::alloc::vec::Vec, - /// Deprecated: use l2 (tag 12) instead. - #[prost(double, repeated, tag = "4")] - pub l2_legacy: ::prost::alloc::vec::Vec, - /// Deprecated: use hh_keys (tag 6) instead. - #[prost(message, optional, tag = "5")] - pub topk: ::core::option::Option, - /// Heavy-hitter candidate keys from the upstream tracker. Downstream queries - /// the merged Count Sketch matrix for each key to build its Top-K. - #[prost(string, repeated, tag = "6")] - pub hh_keys: ::prost::alloc::vec::Vec<::prost::alloc::string::String>, - /// Packed cell encoding (canonical form): - /// - /// row index of each changed cell - #[prost(uint32, repeated, tag = "9")] - pub cell_rows: ::prost::alloc::vec::Vec, - /// col index of each changed cell - #[prost(uint32, repeated, tag = "10")] - pub cell_cols: ::prost::alloc::vec::Vec, - /// signed integer count delta - #[prost(sint64, repeated, tag = "11")] - pub d_counts: ::prost::alloc::vec::Vec, - /// per-row L2 norm deltas - #[prost(double, repeated, tag = "12")] - pub l2: ::prost::alloc::vec::Vec, -} -/// CountSketchCell is the deprecated per-cell delta record. Producers emit the -/// packed cell arrays on CountSketchDelta instead; this message exists only to -/// decode payloads from older producers. -#[derive(Clone, Copy, PartialEq, ::prost::Message)] -pub struct CountSketchCell { - #[prost(uint32, tag = "1")] - pub row: u32, - #[prost(uint32, tag = "2")] - pub col: u32, - /// deprecated: use CountSketchDelta.d_counts - #[prost(double, tag = "3")] - pub d_count: f64, -} /// HyperLogLogState is the portable state of a HyperLogLog cardinality sketch. /// /// Precision p determines the register count: num_registers = 2^precision. @@ -892,7 +826,7 @@ pub struct SketchEnvelope { /// The sketch payload. Exactly one field must be set. #[prost( oneof = "sketch_envelope::SketchState", - tags = "10, 11, 12, 13, 14, 15, 16, 17, 18" + tags = "10, 12, 13, 14, 15, 16, 17, 18" )] pub sketch_state: ::core::option::Option, } @@ -903,8 +837,6 @@ pub mod sketch_envelope { pub enum SketchState { #[prost(message, tag = "10")] CountMin(super::CountMinState), - #[prost(message, tag = "11")] - CountSketch(super::CountSketchState), #[prost(message, tag = "12")] Hll(super::HyperLogLogState), #[prost(message, tag = "13")] diff --git a/tests/PBT_COVERAGE.md b/tests/PBT_COVERAGE.md index cddee44f..07b7741a 100644 --- a/tests/PBT_COVERAGE.md +++ b/tests/PBT_COVERAGE.md @@ -97,12 +97,12 @@ Compared against: a key's `(col, sign)` per row, read off a one-key sketch; the - linearity - insert `w` then `-w` for every `w`: cells back to start; signed stream over a background - `insert(k, -1)` undoes `insert(k)`, cell for cell - - `cells(S(xs)) + cells(S(ys)) == cells(S(xs ++ ys))`; integral weights, power-of-two widths + - `cells(S(xs)) + cells(S(ys)) == cells(S(xs ++ ys))`; signed weights - negate every weight: every cell negated - merge - - portable `CountSketch`: `merge(S(xs), S(ys)) == S(xs ++ ys)` to `REL_TOL` + - `cells(merge(S(xs), S(ys))) == cells(S(xs ++ ys))`; signed weights - wire - - msgpack keeps cells and estimates + - ASAPv1 keeps every cell and estimate; signed weights - ASAPv1 keeps 64 estimates; `edge_rows` x `edge_dimension` ## ddsketch.rs — 14 diff --git a/tests/cs_envelope_golden.hex b/tests/cs_envelope_golden.hex deleted file mode 100644 index e1c665c2..00000000 --- a/tests/cs_envelope_golden.hex +++ /dev/null @@ -1 +0,0 @@ -08015aa40c0803108004180222800c000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000900000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000a0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000090000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000009000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000900000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000a000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000900000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000a0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000090000000000000a000000000000000900000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000900000000000000000000000000000000000000000000000000000000000000000a0000000000000000000000090000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000009000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000032180000000000405f400000000000405f400000000000405f40 diff --git a/tests/e2e/frequency.rs b/tests/e2e/frequency.rs index a913f20f..279fc928 100644 --- a/tests/e2e/frequency.rs +++ b/tests/e2e/frequency.rs @@ -25,7 +25,6 @@ use common::{FreqTruth, uniform_u64, zipf_u64}; use std::collections::HashMap; use asap_sketchlib::message_pack_format::portable::countminsketch::CountMinSketch; -use asap_sketchlib::message_pack_format::portable::countsketch::CountSketch; use asap_sketchlib::{ Count, CountL2HH, CountMin, DataInput, DefaultXxHasher, FastPath, FoldCMS, FoldCS, RegularPath, Vector2D, @@ -693,30 +692,26 @@ fn folded_sketches_keep_their_own_bounds_through_a_sixteen_way_merge() { ); } -// ------------------------------------------------------ Portable wire twins +// ------------------------------------------------------ Portable wire twin -/// The portable wire twins answer over string keys and must satisfy the same -/// two theorems as their core counterparts — Count-Min additive, Count Sketch -/// L2 — with no extra slack for being on the wire side. +/// The portable Count-Min answers over string keys and must satisfy the same +/// additive theorem as its core counterpart, with no extra slack for being on +/// the wire side. #[test] -fn portable_cms_and_cs_string_keys_satisfy_their_own_bounds() { +fn portable_cms_string_keys_satisfy_their_own_bound() { const CM_ROWS: usize = 3; - const CS_ROWS: usize = 5; const COLS: usize = 4096; const STREAM_SEED: u64 = 1006; let stream = zipf_u64(50_000, 2048, 1.1, STREAM_SEED); - // The portable types key on strings; `FreqTruth` keys on i64. The stream's + // The portable type keys on strings; `FreqTruth` keys on i64. The stream's // u64 values are the identity behind both, so `k -> "k{k}"` is injective // and the two truths agree key for key. let mut truth = FreqTruth::default(); let mut pcs = CountMinSketch::new(CM_ROWS, COLS); - let mut pcss = CountSketch::new(CS_ROWS, COLS); for k in &stream { truth.observe(*k as i64); - let key = format!("k{k}"); - pcs.update(&key, 1.0); - pcss.update(&key, 1.0); + pcs.update(&format!("k{k}"), 1.0); } let context = format!("zipf(1.1) domain=2048 n=50000 string keys, stream_seed={STREAM_SEED}"); @@ -726,12 +721,6 @@ fn portable_cms_and_cs_string_keys_satisfy_their_own_bounds() { |k| pcs.estimate(&format!("k{k}")), &context, ); - CountSketchSpec::new(CS_ROWS, COLS).assert_contract( - "portable CountSketch", - &truth, - |k| pcss.estimate(&format!("k{k}")), - &context, - ); } // --------------------------------------------------------------------------- diff --git a/tests/msgpack_compat.rs b/tests/msgpack_compat.rs index 267a51c6..387230fc 100644 --- a/tests/msgpack_compat.rs +++ b/tests/msgpack_compat.rs @@ -31,8 +31,8 @@ use asap_sketchlib::message_pack_format::portable::delta_set_aggregator::DeltaRe use asap_sketchlib::message_pack_format::portable::hydra_kll::HydraKllSketchWire; use asap_sketchlib::message_pack_format::portable::kll::KllSketchData; use asap_sketchlib::{ - CountMinSketch, CountMinSketchWithHeap, CountSketch, DdSketch, HllSketch, HllVariant, - HydraKllSketch, KllSketch, SetAggregator, + CountMinSketch, CountMinSketchWithHeap, DdSketch, HllSketch, HllVariant, HydraKllSketch, + KllSketch, SetAggregator, }; // ===== round-trip: every wire-format-aligned type ===== @@ -61,17 +61,6 @@ fn count_min_sketch_with_heap_round_trip() { assert_eq!(restored.heap_size, 8); } -#[test] -fn count_sketch_round_trip() { - let mut s = CountSketch::new(3, 64); - s.update("k1", 1.0); - s.update("k2", 2.0); - let bytes = s.to_msgpack().expect("encode"); - let restored = CountSketch::from_msgpack(&bytes).expect("decode"); - assert_eq!(restored.rows, 3); - assert_eq!(restored.cols, 64); -} - #[test] fn dd_sketch_round_trip() { let mut s = DdSketch::new(0.01); diff --git a/tests/pbt/count_sketch.rs b/tests/pbt/count_sketch.rs index 54710227..d3f82b01 100644 --- a/tests/pbt/count_sketch.rs +++ b/tests/pbt/count_sketch.rs @@ -1,13 +1,9 @@ -//! Property tests for Count Sketch: the native matrix sketch and the portable -//! `CountSketch` that carries it over the wire. +//! Property tests for Count Sketch. //! //! Charikar, Chen, Farach-Colton, ICALP '02. -use crate::support::{ - PROBE_KEYS, edge_dimension, edge_rows, grid, keyed_updates, keys, matrices_close, -}; -use asap_sketchlib::message_pack_format::MessagePackCodec; -use asap_sketchlib::{Count, CountSketch, DataInput, RegularPath, Vector2D}; +use crate::support::{PROBE_KEYS, edge_dimension, edge_rows, grid, keys}; +use asap_sketchlib::{Count, DataInput, RegularPath, Vector2D}; use proptest::prelude::*; type Cs = Count, RegularPath>; @@ -28,30 +24,25 @@ fn weighted(max: usize) -> impl Strategy> { prop::collection::vec((0u64..512, 1i32..8), 0..max) } -fn cs_of(rows: usize, cols: usize, updates: &[(String, f64)]) -> CountSketch { - let mut s = CountSketch::new(rows, cols); - for (k, v) in updates { - s.update(k, *v); +fn cs_of(rows: usize, cols: usize, updates: &[(u64, i32)]) -> Cs { + let mut s = Cs::with_dimensions(rows, cols); + for (k, w) in updates { + s.insert_many(&DataInput::U64(*k), *w); } s } +fn wire_cells(sketch: &CsWire) -> Vec { + grid(sketch.rows(), sketch.cols(), |r, c| { + sketch.as_storage().query_one_counter(r, c) + }) +} + /// Signed weights small enough that no `i32` cell can overflow. fn signed_weights(max: usize) -> impl Strategy> { prop::collection::vec((0u64..512, -1_000i32..1_000), 0..max) } -/// Integral weights, so every portable cell stays exact in `f64`. -fn integral_updates(max: usize) -> impl Strategy> { - prop::collection::vec(("[a-z]{1,4}", -1_000i32..1_000), 0..max) - .prop_map(|v| v.into_iter().map(|(k, w)| (k, f64::from(w))).collect()) -} - -/// Power-of-two widths the packed-64 hash carries at these row counts. -fn packed_cols() -> impl Strategy { - prop_oneof![Just(2usize), Just(4), Just(8), Just(16), Just(32)] -} - /// The cell and sign a key holds in each row, read off a sketch holding that /// key alone. fn signature(rows: usize, cols: usize, key: u64) -> Vec<(usize, i32)> { @@ -133,22 +124,22 @@ proptest! { prop_assert_eq!(cs_cells(&by_weight), cs_cells(&by_repeat)); } - // ===== Merge algebra, on the portable representation ===== + // ===== Merge algebra ===== #[test] fn count_sketch_merge_matches_streaming_the_concatenation( rows in 1usize..6, - cols in 4usize..64, - a in keyed_updates(30), - b in keyed_updates(30), + cols in 1usize..64, + a in signed_weights(30), + b in signed_weights(30), ) { let mut merged = cs_of(rows, cols, &a); - merged.merge(&cs_of(rows, cols, &b)).expect("merge"); + merged.merge(&cs_of(rows, cols, &b)); let concatenated: Vec<_> = a.iter().chain(b.iter()).cloned().collect(); let streamed = cs_of(rows, cols, &concatenated); - prop_assert!(matrices_close(merged.sketch(), streamed.sketch())); + prop_assert_eq!(cs_cells(&merged), cs_cells(&streamed)); } // ===== Wire ===== @@ -157,16 +148,22 @@ proptest! { fn count_sketch_round_trip_preserves_every_cell( rows in 1usize..8, cols in 1usize..128, - updates in keyed_updates(40), + updates in signed_weights(40), ) { - let s = cs_of(rows, cols, &updates); + let mut s = CsWire::with_dimensions(rows, cols); + for (k, w) in &updates { + s.insert_many(&DataInput::U64(*k), i64::from(*w)); + } - let bytes = s.to_msgpack().expect("encode"); - let restored = CountSketch::from_msgpack(&bytes).expect("decode"); + let bytes = s.serialize_to_bytes().expect("encode"); + let restored = CsWire::deserialize_from_bytes(&bytes).expect("decode"); - prop_assert_eq!(restored.sketch(), s.sketch()); + prop_assert_eq!(wire_cells(&restored), wire_cells(&s)); for (k, _) in &updates { - prop_assert_eq!(restored.estimate(k), s.estimate(k), "key {}", k); + prop_assert_eq!( + restored.estimate(&DataInput::U64(*k)), s.estimate(&DataInput::U64(*k)), + "key {}", k + ); } } @@ -337,42 +334,38 @@ proptest! { #[test] fn cells_of_two_streams_add_to_the_cells_of_their_concatenation( rows in 1usize..6, - cols in packed_cols(), - a in integral_updates(30), - b in integral_updates(30), + cols in 1usize..64, + a in signed_weights(30), + b in signed_weights(30), ) { - let left = cs_of(rows, cols, &a); - let right = cs_of(rows, cols, &b); + let left = cs_cells(&cs_of(rows, cols, &a)); + let right = cs_cells(&cs_of(rows, cols, &b)); let concatenated: Vec<_> = a.iter().chain(b.iter()).cloned().collect(); - let both = cs_of(rows, cols, &concatenated); - - for r in 0..rows { - for c in 0..cols { - prop_assert_eq!( - left.sketch()[r][c] + right.sketch()[r][c], both.sketch()[r][c], - "cell ({}, {})", r, c - ); - } + let both = cs_cells(&cs_of(rows, cols, &concatenated)); + + for (i, cell) in both.iter().enumerate() { + prop_assert_eq!( + left[i] + right[i], *cell, + "cell ({}, {})", i / cols, i % cols + ); } } #[test] fn negating_every_weight_negates_every_cell( rows in 1usize..6, - cols in packed_cols(), - updates in integral_updates(40), + cols in 1usize..64, + updates in signed_weights(40), ) { - let forward = cs_of(rows, cols, &updates); - let negated: Vec<_> = updates.iter().map(|(k, v)| (k.clone(), -v)).collect(); - let backward = cs_of(rows, cols, &negated); - - for r in 0..rows { - for c in 0..cols { - prop_assert_eq!( - backward.sketch()[r][c], -forward.sketch()[r][c], - "cell ({}, {})", r, c - ); - } + let forward = cs_cells(&cs_of(rows, cols, &updates)); + let negated: Vec<_> = updates.iter().map(|(k, w)| (*k, -*w)).collect(); + let backward = cs_cells(&cs_of(rows, cols, &negated)); + + for (i, cell) in backward.iter().enumerate() { + prop_assert_eq!( + *cell, -forward[i], + "cell ({}, {})", i / cols, i % cols + ); } } } diff --git a/tests/sketches_go_parity_probe.rs b/tests/sketches_go_parity_probe.rs index 2997e486..cfed6faf 100644 --- a/tests/sketches_go_parity_probe.rs +++ b/tests/sketches_go_parity_probe.rs @@ -4,7 +4,7 @@ //! //! Approach: reuse the exact same Go-golden envelopes that the //! `message_pack_format::portable` parity tests use, but build the -//! matrix via `sketches::Count` / `sketches::CountMin` with `FastPath` +//! matrix via `sketches::CountMin` with `FastPath` //! and `DefaultXxHasher`. Identical bytes confirm the shared FastPath //! math is sufficient — no hashspec bypass needed. @@ -12,22 +12,15 @@ use asap_sketchlib::common::DataInput; use asap_sketchlib::common::hash::CANONICAL_HASH_SEED; use asap_sketchlib::common::structures::HllBucketListP14; use asap_sketchlib::proto::sketchlib::{ - CountMinState, CountSketchState, CounterType, DdSketchState, HyperLogLogState, SketchEnvelope, + CountMinState, CounterType, DdSketchState, HyperLogLogState, SketchEnvelope, sketch_envelope::SketchState, }; use asap_sketchlib::sketches::countminsketch::CountMin; -use asap_sketchlib::sketches::countsketch::Count; use asap_sketchlib::sketches::ddsketch::DDSketch; use asap_sketchlib::sketches::hll::{ErtlMLE, HyperLogLogImpl}; use asap_sketchlib::{DefaultXxHasher, FastPath, Vector2D}; use prost::Message; -/// 1577-byte envelope captured from `sketchlib-go::CountSketch.SerializeProtoBytes` -/// for `(rows=3, cols=512)` × `goldenCsKeys()` (25 keys "k-a".."k-e", each -/// repeated 5×). Identical to the constant in -/// `src/message_pack_format/portable/countsketch.rs::test_update_then_envelope_matches_sketchlib_go_bytes`. -const COUNTSKETCH_GOLDEN_HEX: &str = include_str!("cs_envelope_golden.hex"); - /// 8275-byte CMS Frequency-Only golden, captured from /// `sketchlib-go::CountMinSketch.SerializeProtoBytesFO` for /// `(rows=4, cols=2048)` × `goldenCmsKeys()` (10 keys "flow-0".."flow-9", @@ -48,61 +41,6 @@ const HLL_GOLDEN_HEX: &str = include_str!("../src/sketches/testdata/hll_envelope /// cross-language byte parity. const DDSKETCH_GOLDEN_HEX: &str = "0801728e03096214ae47e17a843f128003000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000000000001000000000000000000000000000000000000000100000000000000000000000000000100000000000000000000010000000000000000010000000000000001000000000001000000000001000000000001000000010000000001000000010000010000000100000100000100000100000100010000010001000100010001000100010001000100010100010100010100010101000101010100010101010101010100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000187f"; -#[test] -fn sketches_count_fastpath_matches_go_count_sketch_envelope() { - let rows = 3usize; - let cols = 512usize; - - let mut sk: Count, FastPath, DefaultXxHasher> = - Count::with_dimensions(rows, cols); - for i in 0..25 { - let key = format!("k-{}", (b'a' + (i % 5) as u8) as char); - sk.insert_many(&DataInput::String(key), 1i64); - } - - let storage = sk.as_storage(); - let mut counts_int: Vec = Vec::with_capacity(rows * cols); - let mut l2: Vec = Vec::with_capacity(rows); - for r in 0..rows { - let mut row_l2 = 0.0f64; - for c in 0..cols { - let cell = *storage.get(r, c).expect("cell in range"); - counts_int.push(cell); - row_l2 += (cell as f64) * (cell as f64); - } - l2.push(row_l2); - } - - let state = CountSketchState { - rows: rows as u32, - cols: cols as u32, - counter_type: CounterType::Int64 as i32, - counts_int, - counts_float: Vec::new(), - l2, - topk: None, - }; - let envelope = SketchEnvelope { - format_version: 1, - producer: None, - hash_spec: None, - sample_p: 0.0, - sketch_state: Some(SketchState::CountSketch(state)), - }; - let mut got = Vec::with_capacity(envelope.encoded_len()); - envelope.encode(&mut got).expect("prost encode"); - - let want = decode_hex(COUNTSKETCH_GOLDEN_HEX); - assert_eq!( - got, - want, - "sketches::Count, FastPath> envelope diverges from Go golden \ - ({} bytes got vs {} bytes want)", - got.len(), - want.len(), - ); -} - #[test] fn sketches_countmin_fastpath_matches_go_count_min_envelope() { let rows = 4usize; diff --git a/tests/xtest_consumer.rs b/tests/xtest_consumer.rs index 0113ff45..3c7dd943 100644 --- a/tests/xtest_consumer.rs +++ b/tests/xtest_consumer.rs @@ -9,7 +9,6 @@ //! kll.pb — KLLState with quantile items and coin RNG state //! ddsketch.pb — DDSketchState with alpha + bucket array //! hll.pb — HyperLogLogState (ErtlMLE variant) -//! countsketch.pb — CountSketchState with float64 signed counters //! coco.pb — CocoSketchState (hash+val+hasKey buckets) //! elastic.pb — ElasticState (heavy buckets + light CountMin) //! univmon.pb — UnivMonState (layered CountSketch + TopK heaps) @@ -245,44 +244,6 @@ fn cross_language_proto() { all_ok = false; } - // ----------------------------------------------------------------------- - // CountSketch - // ----------------------------------------------------------------------- - println!(); - println!("[CountSketch] Step 1/3 — Read countsketch.pb"); - let bytes = read_file(in_dir.join("countsketch.pb")); - let env = SketchEnvelope::decode(bytes.as_slice()).expect("decode countsketch envelope"); - - println!( - "[CountSketch] Step 2/3 — Validate envelope (format_version={}, producer={})", - env.format_version, - env.producer.as_ref().map_or("?", |p| &p.library) - ); - - let cs_state = match env.sketch_state { - Some(sketch_envelope::SketchState::CountSketch(ref s)) => s.clone(), - other => panic!("expected CountSketch sketch_state, got {other:?}"), - }; - - println!( - "[CountSketch] rows={} cols={} counter_type={:?}", - cs_state.rows, - cs_state.cols, - CounterType::try_from(cs_state.counter_type) - ); - - // Query "cs:hot" — inserted 200 extra times. - // Go hash: common.Hash64([]byte("cs:hot")) = xxh3_64_seeded(seedList[0], b"cs:hot") - let cs_hot_hash = xxh3_64_seeded(SEED_0, b"cs:hot"); - let cs_est = count_sketch_query_float(&cs_state, cs_hot_hash); - println!("[CountSketch] Step 3/3 — 'cs:hot' est = {cs_est:.0} (expect ≥ 200)"); - if cs_est >= 200.0 { - println!("[CountSketch] PASS"); - } else { - eprintln!("[CountSketch] FAIL: estimate {cs_est:.0} < 200"); - all_ok = false; - } - // ----------------------------------------------------------------------- // CocoSketch // ----------------------------------------------------------------------- @@ -790,38 +751,6 @@ fn hll_ertl_mle_estimate(state: &HyperLogLogState) -> u64 { (0.5 / std::f64::consts::LN_2 * m * m / z).round() as u64 } -// --------------------------------------------------------------------------- -// CountSketch median frequency query (float64 counters, packed hash) -// --------------------------------------------------------------------------- -// Mirrors Go's CountSketch.fastPacked64PosAndSign + ComputeMedianInlineF64. -// hash = xxh3_64_seeded(SEED_0, key) — Go's Hash64 - -fn count_sketch_query_float(state: &CountSketchState, hash: u64) -> f64 { - let rows = state.rows as usize; - let cols = state.cols as usize; - let bits_per_row = col_bits(cols); - let mask = (cols as u64) - 1; - // Read whichever counter field the producer populated (int vs float), - // discriminated by counter_type — same as CountMin. - let counts: Vec = if !state.counts_float.is_empty() { - state.counts_float.clone() - } else { - state.counts_int.iter().map(|&v| v as f64).collect() - }; - let counts = &counts; - - let mut estimates: Vec = Vec::with_capacity(rows); - for r in 0..rows { - let shift = (r as u64) * bits_per_row; - let col = ((hash >> shift) & mask) as usize; - // sign: bit (63-r) of hash → 1 means +1, 0 means −1 - let sign_bit = (hash >> (63 - r)) & 1; - let sign = if sign_bit == 1 { 1.0f64 } else { -1.0f64 }; - estimates.push(counts[r * cols + col] * sign); - } - median_f64(&mut estimates) -} - fn median_f64(v: &mut [f64]) -> f64 { if v.is_empty() { return 0.0; From d65e9b3a586c34d0450534f25aa3023e6e3a55c4 Mon Sep 17 00:00:00 2001 From: GordonYuanyc Date: Fri, 2 Oct 2026 00:42:27 -0600 Subject: [PATCH 03/36] refactor(cms)!: drop the proto and portable Count-Min formats CountMin serializes to ASAPv1 only. Removes the CountMinDelta and CountMinCell proto messages and SketchEnvelope's count_min field (10, now reserved), portable::countminsketch (CountMinSketch, CountMinSketchDelta, CountMinSketchWire, SketchlibCms and the sketchlib_cms_* helpers), the native MessagePackCodec shim for CountMin, and the cms_envelope_golden.hex proto golden. CountMinState stays as the nested Count-Min state of HydraCell and ElasticState. The Count-Min property laws that ran through the portable type now run on CountMin, FastPath> over string keys, with the ASAPv1 round trip and CountMin::apply_delta in the sequence law. Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 17 + docs/asapv1_wire_format.md | 5 +- docs/features.md | 2 +- docs/library_map.md | 2 +- docs/message_pack_format.md | 4 +- docs/tests.md | 35 +- examples/accuracy_probe.rs | 26 - proto/countminsketch/countminsketch.proto | 57 - proto/sketchlib.proto | 4 +- src/common/hashspec.rs | 7 +- src/lib.rs | 7 +- .../native/countminsketch.rs | 27 - src/message_pack_format/native/mod.rs | 1 - .../portable/countminsketch.rs | 1000 ----------------- .../portable/countminsketch_topk.rs | 5 +- .../portable/countsketch.rs | 8 +- src/message_pack_format/portable/mod.rs | 7 +- src/proto/generated/sketchlib.v1.rs | 78 +- src/sketches/testdata/cms_envelope_golden.hex | 1 - tests/PBT_COVERAGE.md | 6 +- tests/e2e/frequency.rs | 25 +- tests/msgpack_compat.rs | 39 +- tests/pbt/count_min.rs | 152 +-- tests/sketches_go_parity_probe.rs | 76 +- tests/xtest_consumer.rs | 127 --- 25 files changed, 139 insertions(+), 1579 deletions(-) delete mode 100644 src/message_pack_format/native/countminsketch.rs delete mode 100644 src/message_pack_format/portable/countminsketch.rs delete mode 100644 src/sketches/testdata/cms_envelope_golden.hex diff --git a/CHANGELOG.md b/CHANGELOG.md index 80d85f98..8934690b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -88,6 +88,23 @@ signals a backwards-compatible change. as an empty window like any other, and that window takes a place in `max_windows`. +### Removed + +- **Count-Min's proto and portable formats.** `CountMin` serializes to ASAPv1 + only (`serialize_to_bytes` / `deserialize_from_bytes`, kind `0x02 0x00`). + Removed: the `CountMinDelta` and `CountMinCell` proto messages and the + `count_min` field (10) of `SketchEnvelope`, now reserved; + `message_pack_format::portable::countminsketch` (`CountMinSketch`, + `CountMinSketchDelta`, `CountMinSketchWire`, `SketchlibCms`, + `new_sketchlib_cms`, `sketchlib_cms_from_matrix`, + `matrix_from_sketchlib_cms`, `sketchlib_cms_update`, `sketchlib_cms_query`) + and the crate-root `CountMinSketch` / `CountMinSketchDelta` re-exports; the + `MessagePackCodec` impl for `CountMin` in + `message_pack_format::native::countminsketch`; and the + `src/sketches/testdata/cms_envelope_golden.hex` proto golden. The + `CountMinState` proto message stays as the nested Count-Min state of + `HydraCell` and `ElasticState`. + ### Fixed - `KLLDynamic` panicked on every quantile query once a NaN had entered the diff --git a/docs/asapv1_wire_format.md b/docs/asapv1_wire_format.md index fd14e70a..2e4bec1a 100644 --- a/docs/asapv1_wire_format.md +++ b/docs/asapv1_wire_format.md @@ -371,9 +371,6 @@ Per sketch. **Raw state only**, a **positional msgpack array** in the order its - No field derivable from another (no HLL `precision`; no CMS `l1`/`l2`, which are `sum(count)` / `sum(count^2)` and are recomputed on decode). The rule is "derivable", not "named `l2`": CountL2HH and the UnivMon family carry a payload field spelled exactly `l2` (§3.19, §3.15, §3.16) and are right to, because that accumulator is **not** `sum(counts[row]^2)` — an insert path exists that moves counters without touching it, and it clamps. Nothing derivable rides the wire; nothing that only looks derivable is dropped. - msgpack array (positional), never a keyed map. The exact msgpack types are in "Wire encoding rules". -> Note: derived summaries like CMS `l1`/`l2` and `sum_counts`/`sum2_counts` live in the **delta / error-accounting** format (proto `CountMinState`), a separate wire format. -> They do not belong in the self-contained sketch payload. - ### 3.1: HLL payload (`0x01 0x01` / `0x01 0x02` / `0x01 0x03`) The variant is in `kind_id`, precision is in the metadata (and equals `log2(register count)`), so the only real state is the register bytes (plus three running scalars for HIP). @@ -1316,7 +1313,7 @@ Fail **closed** on any mismatch: The library provides no free wire serialization for exotic counters; only the owner knows if the mapping is lossless. Convert to a canonical counter type, then serialize. -Doable **today** with existing public API (the pattern `SketchlibCms` already uses): +Doable **today** with existing public API: ```rust // e.g. a u64-counter FastPath CMS to the i64 wire form diff --git a/docs/features.md b/docs/features.md index 64f296a3..a6d77cda 100644 --- a/docs/features.md +++ b/docs/features.md @@ -112,7 +112,7 @@ MessagePack (`rmp-serde`) support. **serde support** means the type derives `Ser | EHSketchList | Yes | Yes | | ExponentialHistogram | No | Yes | -Protobuf (prost): `.proto` definitions exist for CountMin, Count, HLL, DDSketch, KLL, Elastic, Coco, Hydra, and UnivMon. Rust conversion code is in progress. +Protobuf (prost): `.proto` definitions exist for Count, HLL, DDSketch, KLL, Elastic, Coco, Hydra, and UnivMon. Rust conversion code is in progress. ### API Stability diff --git a/docs/library_map.md b/docs/library_map.md index a8e614b3..8e34dd99 100644 --- a/docs/library_map.md +++ b/docs/library_map.md @@ -26,7 +26,7 @@ - **`src/message_pack_format/`** - Serialization plumbing ([message_pack_format.md](./message_pack_format.md)). The current format is **ASAPv1**, specified in [asapv1_wire_format.md](./asapv1_wire_format.md) - `envelope.rs` — the shared, sketch-agnostic ASAPv1 framing (magic/version/`kind_id` + length prefixes, `encode`/`split`); every `wire.rs` under `src/sketches/` and `src/sketch_framework/` calls into it - `codec.rs` — the `MessagePackCodec` trait; `error.rs` — the unified `Error` - - `portable/` — **deprecated**, being retired. The older per-sketch wire types (`CountMinSketch`, `HllSketch`, …); ASAPv1 (per-sketch `wire.rs`) is now what `sketchlib-go` mirrors, not these + - `portable/` — **deprecated**, being retired. The older per-sketch wire types (`HllSketch`, `CountSketch`, …); ASAPv1 (per-sketch `wire.rs`) is now what `sketchlib-go` mirrors, not these - `native/` — **deprecated**, being retired. Older `MessagePackCodec` shims over `src/sketches/` byte serialization ## Documentation diff --git a/docs/message_pack_format.md b/docs/message_pack_format.md index 12a2b91c..9c3f655d 100644 --- a/docs/message_pack_format.md +++ b/docs/message_pack_format.md @@ -150,8 +150,8 @@ being **phased out** in favor of the per-sketch `wire.rs` + the shared `envelope.rs`. - `portable/` holds per-sketch wire types that predate the envelope. They - carry their own byte-parity goldens against `sketchlib-go` — HLL, Count-Min, - Count Sketch and KLL (both directions) — separate from `asapv1_golden/`. + carry their own byte-parity goldens against `sketchlib-go` — HLL, Count + Sketch and KLL (both directions) — separate from `asapv1_golden/`. - `native/` is a set of thin `MessagePackCodec` shims over the sketches' `serialize_to_bytes` / `deserialize_from_bytes`. Every type it wraps emits the ASAPv1 envelope, so each shim is a pass-through. diff --git a/docs/tests.md b/docs/tests.md index ccb1571c..da587df6 100644 --- a/docs/tests.md +++ b/docs/tests.md @@ -1215,31 +1215,6 @@ Test file: [`src/sketch_framework/tumbling.rs`](../src/sketch_framework/tumbling ## MessagePack Portable Wire Types -### Portable CountMinSketch - -Test file: [`src/message_pack_format/portable/countminsketch.rs`](../src/message_pack_format/portable/countminsketch.rs) - -| test_name | test_description | what_is_tested | -| --- | --- | --- | -| `msgpack_delta_against_empty_round_trips` | A msgpack delta against an empty snapshot carries the window. | For a `5x1024` sketch fed 200 updates of `"hot"` at weight `1.0`, verifies the msgpack delta computed against a fresh `5x1024` sketch at threshold `1.0`, applied to another fresh sketch, reproduces the `"hot"` estimate to within 10%. | -| `msgpack_delta_wire_layout_is_parallel_arrays` | The delta is a five-element array of scalars plus parallel index arrays. | For a `3x8` sketch fed `"x"` at weight `5.0`, verifies the delta bytes equal `rmp_serde::to_vec(&(rows as u64, cols as u64, row_idx: Vec, col_idx: Vec, d_count: Vec))` recomputed from the non-zero cells of `sketch()`, and that decoding reports `rows` `3` and `cols` `8`. | -| `test_count_min_sketch_creation` | A fresh sketch has the requested shape and zeroed cells. | For `CountMinSketch::new(4, 1000)`, verifies `rows` is `4`, `cols` is `1000`, `sketch()` is 4 rows of 1,000, and every cell is `0.0`. | -| `test_count_min_sketch_update` | An insert never reads below its weight. | For a `2x10` sketch fed `"key1"` at weight `1.0`, verifies `estimate("key1")` is at or above `1.0`. | -| `test_count_min_sketch_query_empty` | An empty sketch answers zero. | Verifies a `2x10` sketch reports `estimate("anything")` of exactly `0.0`. | -| `test_count_min_sketch_merge` | Merge sums cells element-wise. | Merges two `2x3` legacy matrices, the left holding `5.0` at `(0,0)` and `10.0` at `(1,2)` and the right `3.0` at `(0,0)` and `7.0` at `(0,1)`, and verifies the merged cells read `8.0`, `7.0`, and `10.0`. | -| `test_count_min_sketch_merge_dimension_mismatch` | Mismatched row counts are refused. | Verifies merging a `3x3` sketch into a `2x3` one fails. | -| `test_count_min_sketch_msgpack_round_trip` | The msgpack round trip preserves shape and estimates. | Round-trips a `4x256` sketch fed `"apple"` at `5.0` then `2.0` and `"banana"` at `3.0`, and verifies the decode reports `rows` `4`, `cols` `256`, and estimates at or above `7.0` for `"apple"` and `3.0` for `"banana"`. | -| `test_aggregate_count` | `aggregate_count` builds a decodable sketch from parallel key and weight arrays. | For `aggregate_count(4, 100, ["a", "b", "a"], [1.0, 2.0, 3.0])`, verifies the returned bytes decode and report estimates at or above `4.0` for `"a"` and `2.0` for `"b"`. | -| `test_aggregate_count_empty` | An empty stream produces nothing. | Verifies `aggregate_count(4, 100, &[], &[])` returns `None`. | -| `test_apply_delta_additive` | Delta cells add into the matrix. | For a `2x3` sketch holding `[[1, 2, 3], [4, 5, 6]]`, applies cells `(0,0,10)` and `(1,2,100)` and verifies the matrix becomes `[[11, 2, 3], [4, 5, 106]]`. | -| `test_apply_delta_matches_full_merge` | Applying a delta lands where merging the same addition lands. | For a `2x2` base `[[1, 2], [3, 4]]`, verifies merging the addition `[[10, 0], [0, 20]]` and applying the delta cells `(0,0,10)` and `(1,1,20)` give the same matrix. | -| `test_apply_delta_out_of_range` | A cell outside the declared geometry is refused. | Verifies a delta carrying cell `(5, 0, 1)` against a `2x3` sketch fails to apply. | -| `test_update_then_envelope_matches_sketchlib_go_bytes` | The proto envelope for a populated matrix is byte-identical to the checked-in golden. | For a `4x2048` sketch fed 50 updates of `flow-{i % 10}` at weight `1.0`, builds a `CountMinState` with `counter_type` `INT64`, row-major `counts_int`, and per-row `l1`/`l2`, wraps it in a `SketchEnvelope` with `format_version` `1` and `sample_p` `0.0`, and verifies the prost encoding matches the 8,275-byte [`cms_envelope_golden.hex`](../src/sketches/testdata/cms_envelope_golden.hex) on both length and content. | -| `test_hh_keys_matches_go_golden_bytes` | Heavy-hitter keys at tag 6 reach the wire in the golden's shape. | For the same `4x2048` and 50-update `flow-{i % 10}` stream, decodes the packed delta as a proto `CountMinDelta`, sets `hh_keys` to `["flow-0", "flow-3", "flow-7"]`, and verifies the re-encoding equals a 263-byte inline golden, then that those bytes apply to a fresh `4x2048` sketch. | -| `test_compute_delta_against_empty_round_trips` | A proto delta against an empty snapshot carries the whole matrix. | For a `4x2048` sketch fed 200 updates of `flow-{i % 37}` at weight `1.0`, verifies the delta computed against an empty sketch at threshold `1.0`, applied to a fresh `4x2048` sketch, reproduces `sketch()` cell for cell. | -| `test_compute_delta_then_apply_matches_current` | A delta between two non-empty snapshots reconstructs the later one. | For a `2x64` base fed 40 updates of `k{i % 8}` and a current fed 30 more, verifies the delta applied to the base reproduces the current matrix. | -| `test_compute_delta_matches_go_golden_bytes` | The packed delta bytes are pinned by a golden. | For a `4x2048` sketch fed 50 updates of `flow-{i % 10}` at weight `1.0`, verifies the delta against an empty sketch at threshold `1.0` equals a 239-byte inline golden. | - ### Portable CountMinSketchWithHeap Test file: [`src/message_pack_format/portable/countminsketch_topk.rs`](../src/message_pack_format/portable/countminsketch_topk.rs) @@ -1606,13 +1581,13 @@ Test file: [`tests/e2e/frequency.rs`](../tests/e2e/frequency.rs) | `fold_cms_foldcs_counts_and_hierarchical_merge` | Weighted counts are exact on sparse dimensions and survive a same-level merge, a signed update, and a hierarchical merge. | On `FoldCMS::::new(3, 2048, 0, 32)` sketches: 100 inserts of `Str("alpha")` at weight 2 into each of two sketches plus 100 of `Str("beta")` at weight 1 into the second, verifying `query` reads `200`, `200` and `100`; after `merge_same_level`, `400` for alpha and `100` for beta. On `FoldCS::::new(3, 2048, 0, 32)`, `Str("gamma")` at `+60` then `-20` is verified to query `40`. Finally `FoldCMS::hierarchical_merge` of two sketches built by `build_fold_cms` (30 and 45 inserts at weight 3) is verified to query `90` and `135`. | | `folded_sketches_keep_their_own_bounds_through_a_sixteen_way_merge` | Folded sketches keep their own theorems, evaluated at the folded width, through a sixteen-way hierarchical merge. | Splits `zipf_u64(240_000, 10_000, 1.1, 1009)` into 16 windows of 15,000, giving each window a `FoldCMS` and a `FoldCS` of `3x4096` at `fold_level = 4` and `top_k = 20`, then `hierarchical_merge`s each family. Asserts `CountMinSpec::new(3, 256)` over `FoldCMS::query` and `CountSketchSpec::new(3, 256)` over `FoldCS::query` - the bounds are evaluated at the folded width `4096 >> 4 = 256`, not the unfolded 4096 - with the specs' usual rules: zero violations for one-sidedness and both simultaneous bounds at `delta = 1e-3`, and the marginal bounds as rate pins at `e^-3` and `P[Bin(3, 1/3) >= 2]`. | -### Frequency: Portable Count-Min and Count Sketch +### Frequency: Portable Count Sketch Test file: [`tests/e2e/frequency.rs`](../tests/e2e/frequency.rs) | test_name | test_description | what_is_tested | | --- | --- | --- | -| `portable_cms_and_cs_string_keys_satisfy_their_own_bounds` | The portable wire twins satisfy their own theorems over string keys with no extra slack. | Feeds `zipf_u64(50_000, 2048, 1.1, 1006)` as string keys `"k{k}"` through `update(&key, 1.0)` into `portable::countminsketch::CountMinSketch::new(3, 4096)` and `portable::countsketch::CountSketch::new(5, 4096)`, then asserts `CountMinSpec::new(3, 4096).assert_contract` over the portable Count-Min's `estimate` and `CountSketchSpec::new(5, 4096).assert_contract` over the portable Count Sketch's `estimate`, against a `FreqTruth` keyed on the underlying `i64` identities. | +| `portable_cs_string_keys_satisfy_its_own_bound` | The portable Count Sketch satisfies its own theorem over string keys with no extra slack. | Feeds `zipf_u64(50_000, 2048, 1.1, 1006)` as string keys `"k{k}"` through `update(&key, 1.0)` into `portable::countsketch::CountSketch::new(5, 4096)`, then asserts `CountSketchSpec::new(5, 4096).assert_contract` over its `estimate`, against a `FreqTruth` keyed on the underlying `i64` identities. | ### Frequency: Documented Input Matrix @@ -2138,10 +2113,7 @@ Test file: [`tests/msgpack_compat.rs`](../tests/msgpack_compat.rs) | test_name | test_description | what_is_tested | | --- | --- | --- | -| `count_min_decodes_go_bytes` | The test asserts nothing. | The test asserts nothing: the body is empty apart from three commented-out lines sketching an `include_bytes!` of `fixtures/msgpack/count_min.msgpack` and a `CountMinSketch::from_msgpack` check, and it is marked `#[ignore]` with the reason that no cross-language msgpack fixture exists in the repository. | -| `count_min_sketch_round_trip` | A portable `CountMinSketch` survives a msgpack round trip with its geometry intact. | Updates `CountMinSketch::new(3, 64)` with `alpha` `+1.0`, `beta` `+2.0` and `alpha` `+4.0`, then verifies `from_msgpack(to_msgpack())` reports `rows` `3` and `cols` `64`; the counter matrix and the per-key estimates are not checked. | | `count_min_sketch_with_heap_round_trip` | A portable `CountMinSketchWithHeap` survives a msgpack round trip with its geometry and heap size intact. | Updates `CountMinSketchWithHeap::new(3, 64, 8)` with `hot` `+100.0` and `warm` `+10.0`, then verifies the decode reports `rows` `3`, `cols` `64` and `heap_size` `8`; the heap's contents are not checked. | -| `count_min_wire_shape` | The `CountMinSketchWire` DTO round-trips through `rmp_serde` with its field shape intact. | Serializes `CountMinSketchWire { sketch: vec![vec![1.0, 2.0]; 3], rows: 3, cols: 2 }` with `rmp_serde::to_vec` and reads it back with `rmp_serde::from_slice`, verifying `rows` `3`, `cols` `2` and a `sketch` of length `3`; the cell values are not checked. | | `count_min_with_heap_wire_shape` | The nested `CountMinSketchWithHeapWire` DTO round-trips through `rmp_serde` with its nesting intact. | Serializes a wire struct holding a `CountMinSketchInnerWire` of `2x4` zeros, one `CmsHeapItem { key: "hot", value: 42.0 }` and `heap_size: 8`, then verifies the `rmp_serde` decode reports `heap_size` `8`, one heap entry and the key `hot`; the item's value and the inner matrix are not re-checked. | | `count_sketch_round_trip` | A portable `CountSketch` survives a msgpack round trip with its geometry intact. | Updates `CountSketch::new(3, 64)` with `k1` `+1.0` and `k2` `+2.0`, then verifies the decode reports `rows` `3` and `cols` `64`; the signed counters and the estimates are not checked. | | `dd_sketch_decodes_go_bytes` | The test asserts nothing. | The test asserts nothing: the body is empty and the test is marked `#[ignore]` with the reason that no cross-language msgpack fixture exists in the repository. | @@ -2192,7 +2164,6 @@ Test file: [`tests/sketches_go_parity_probe.rs`](../tests/sketches_go_parity_pro | test_name | test_description | what_is_tested | | --- | --- | --- | | `sketches_count_fastpath_matches_go_count_sketch_envelope` | A `sketches::Count` fast-path matrix encodes to the checked-in Count-Sketch envelope bytes. | Fills `Count, FastPath, DefaultXxHasher>` at `3x512` with 25 `insert_many` calls over the five keys `k-a`..`k-e`, packs the row-major cells and per-row L2 sums into a `CountSketchState` with `counter_type` `INT64` inside a `SketchEnvelope { format_version: 1 }` with no producer or hash_spec, and verifies the prost encoding equals the 1,577-byte fixture `tests/cs_envelope_golden.hex`. | -| `sketches_countmin_fastpath_matches_go_count_min_envelope` | A `sketches::CountMin` fast-path matrix encodes to the checked-in Count-Min envelope bytes. | Fills `CountMin, FastPath, DefaultXxHasher>` at `4x2048` with 50 `insert_many` calls over the ten keys `flow-0`..`flow-9`, packs the cells cast to `i64` plus per-row L1 and L2 sums into a `CountMinState` with `counter_type` `INT64`, and verifies the prost encoding matches the 8,275-byte fixture `src/sketches/testdata/cms_envelope_golden.hex` in length and then byte for byte. | | `sketches_ddsketch_matches_go_envelope` | A `sketches::DDSketch` encodes to the DDSketch envelope hex constant held in the test file. | Fills `DDSketch::new(0.01)` with `1..=50` as `f64`, encodes a `DdSketchState` carrying `store_counts`, `store_offset` and an alpha round-tripped through gamma (`gamma = (1 + a) / (1 - a)`, `alpha_wire = (gamma - 1) / (gamma + 1)`), and verifies the envelope matches the 403-byte inline hex constant in length and then byte for byte; the comment on that constant records it as this crate's own output rather than a fixture from another implementation, so no cross-language parity is pinned. | | `sketches_hll_classic_matches_go_envelope` | An ErtlMLE P14 register set encodes to the checked-in HyperLogLog envelope bytes. | Feeds `HyperLogLogImpl` the `hash64_seeded(CANONICAL_HASH_SEED, ...)` digests of the IEEE-754 little-endian bytes of `1.0..=50.0` through `insert_with_hash`, encodes a `HyperLogLogState` with `variant` `2`, `precision` `14`, the dense registers, zeroed HIP scalars and no sparse form, and verifies the envelope matches the 16,398-byte fixture `src/sketches/testdata/hll_envelope_golden.hex` in length and then byte for byte; despite the name, the variant under test is `ErtlMLE`, not `Classic`. | @@ -2202,7 +2173,7 @@ Test file: [`tests/xtest_consumer.rs`](../tests/xtest_consumer.rs) | test_name | test_description | what_is_tested | | --- | --- | --- | -| `cross_language_proto` | Protobuf envelopes read from the cross-language fixture directory answer their point queries in band. | Asserts nothing unless fixtures are available: with no `XTEST_DIR` and no discoverable producer directory holding `tests/cross_language/xtest_producer_test.go`, it prints a skip line and returns. With fixtures it decodes nine `SketchEnvelope` files and records a pass or failure for each — `countmin.pb` min-frequency for `item:42` at least `101`, `kll.pb` p50 within 5% of `5000` and p99 within 5% of `9900`, `ddsketch.pb` the same two quantiles within 2%, `hll.pb` an ErtlMLE cardinality inside `[40_000, 65_000]`, `countsketch.pb` `cs:hot` at least `200`, `coco.pb` `coco:hot` at least `500`, `elastic.pb` `elephant` at least `900`, `univmon.pb` a cardinality inside `[1_000, 15_000]`, `hydra.pb` `hydra:42` at least `51` — using the hash seeds `0xcafe3553`, `0x6a09e667` and `0xbb67ae85`, and, when `countmin_sampled.pb` and `hll_sampled.pb` exist, requires `effective_sample_p` of `0.1` and a `rescale_count` by `1/p` within 5% of `100_000` and 6% of `200_000`. A single `assert!` at the end fails if any check failed; a missing or mistyped envelope panics on the spot. | +| `cross_language_proto` | Protobuf envelopes read from the cross-language fixture directory answer their point queries in band. | Asserts nothing unless fixtures are available: with no `XTEST_DIR` and no discoverable producer directory holding `tests/cross_language/xtest_producer_test.go`, it prints a skip line and returns. With fixtures it decodes eight `SketchEnvelope` files and records a pass or failure for each — `kll.pb` p50 within 5% of `5000` and p99 within 5% of `9900`, `ddsketch.pb` the same two quantiles within 2%, `hll.pb` an ErtlMLE cardinality inside `[40_000, 65_000]`, `countsketch.pb` `cs:hot` at least `200`, `coco.pb` `coco:hot` at least `500`, `elastic.pb` `elephant` at least `900`, `univmon.pb` a cardinality inside `[1_000, 15_000]`, `hydra.pb` `hydra:42` at least `51` — using the hash seeds `0xcafe3553`, `0x6a09e667` and `0xbb67ae85`, and, when `hll_sampled.pb` exists, requires `effective_sample_p` of `0.1` and a `rescale_count` by `1/p` within 6% of `200_000`. A single `assert!` at the end fails if any check failed; a missing or mistyped envelope panics on the spot. | ### HLL Custom Precision diff --git a/examples/accuracy_probe.rs b/examples/accuracy_probe.rs index 643bfc3e..d7e2179b 100644 --- a/examples/accuracy_probe.rs +++ b/examples/accuracy_probe.rs @@ -7,7 +7,6 @@ //! printing EXPECTED vs ACTUAL and a theory-based verdict. use asap_sketchlib::common::input::{HydraCounter, HydraQuery}; -use asap_sketchlib::message_pack_format::portable::countminsketch::CountMinSketch; use asap_sketchlib::message_pack_format::portable::ddsketch::DdSketch as PortableDds; use asap_sketchlib::message_pack_format::portable::hll::{HllSketch, HllVariant}; use asap_sketchlib::message_pack_format::portable::hydra_kll::HydraKllSketch; @@ -1018,31 +1017,6 @@ fn probe_portable_wire_types() { let mut p = Probe::new(); println!("--- Portable wire types (Go-parity DTOs) ---"); - // Portable CountMin. - let mut pcs = CountMinSketch::new(3, 4096); - let mut truth: HashMap = HashMap::new(); - let mut z = Zipf::new(2048, 1.1, 54); - for _ in 0..50_000 { - let k = format!("k{}", z.sample_usize()); - *truth.entry(k.clone()).or_insert(0.0) += 1.0; - pcs.update(&k, 1.0); - } - let mut by_freq: Vec<(String, f64)> = truth.iter().map(|(k, v)| (k.clone(), *v)).collect(); - by_freq.sort_by(|a, b| b.1.partial_cmp(&a.1).unwrap()); - let eps = std::f64::consts::E / 4096.0; - let mut bad = 0usize; - for (k, v) in by_freq.iter().take(50) { - let est = pcs.estimate(k); - if est < *v || est > v + eps * 50_000.0 { - bad += 1; - } - } - p.check( - "portable CMS one-sided bound (top-50)", - format!("{bad} violations"), - bad == 0, - ); - // Portable HLL precision 12. let mut ph = HllSketch::new(HllVariant::Regular, 12); for i in 0..100_000u64 { diff --git a/proto/countminsketch/countminsketch.proto b/proto/countminsketch/countminsketch.proto index 27e09d3b..c3be0445 100644 --- a/proto/countminsketch/countminsketch.proto +++ b/proto/countminsketch/countminsketch.proto @@ -57,60 +57,3 @@ message CountMinState { reserved 10 to 15; } - -// ============================================================================ -// Count-Min Sketch Delta -// ============================================================================ - -// CountMinDelta carries only the matrix cells that changed by at least the -// threshold between two consecutive snapshots, plus the full per-row L1/L2 -// norm deltas (one entry per row, negligible size). -// -// Cells apply additively on the receiver: matrix[row][col] += d_count. CMS -// counters only ever grow, but the delta encodes a signed count (sint64) so -// the same wire form covers weighted/decay variants. -// -// The packed cell encoding (cell_rows / cell_cols / d_counts, tags 9-11) is -// the canonical form; the legacy per-cell message (tag 3) is retained only so -// payloads from older producers still decode. This message is byte-identical -// to the Go reference implementation's CountMinDelta so the two runtimes emit -// byte-identical delta frames for identical window state (cross-language byte -// parity). -// -// CountMinDelta is structurally identical to CountSketchDelta: rows, cols, the -// packed cell encoding, per-row norm deltas, and an optional repeated hh_keys -// at the SAME tag number (6). Both sketches can track heavy hitters; whether -// hh_keys is populated is a control-plane decision (it is empty/omitted when -// heavy-hitter tracking is not enabled for this sketch). -message CountMinDelta { - uint32 rows = 1; - uint32 cols = 2; - // Deprecated: use cell_rows/cell_cols/d_counts (tags 9-11). Retained for - // backward-compatible decode of payloads from older producers. - repeated CountMinCell cells_legacy = 3; - // Per-row L1 norm deltas, length = rows. - repeated double l1 = 4 [packed = true]; - // Per-row L2 norm deltas, length = rows. - repeated double l2 = 5 [packed = true]; - // Heavy-hitter candidate keys from an upstream tracker, mirroring - // CountSketchDelta.hh_keys (same tag number 6, same wire shape). Downstream - // queries the merged Count-Min matrix for each key to (re)build its Top-K - // with globally-merged estimates. Empty/omitted when heavy-hitter tracking - // is not enabled. - repeated string hh_keys = 6; - // Packed cell encoding (canonical form): - repeated uint32 cell_rows = 9 [packed = true]; // row index of each changed cell - repeated uint32 cell_cols = 10 [packed = true]; // col index of each changed cell - repeated sint64 d_counts = 11 [packed = true]; // signed integer count delta -} - -// CountMinCell is the deprecated per-cell delta record. Producers emit the -// packed cell arrays on CountMinDelta instead; this message exists only to -// decode payloads from older producers. -message CountMinCell { - uint32 row = 1; - uint32 col = 2; - double d_count = 3; // deprecated: use CountMinDelta.d_counts - double d_sum = 4; // deprecated: omitted in the packed encoding - double d_sum2 = 5; // deprecated: omitted in the packed encoding -} diff --git a/proto/sketchlib.proto b/proto/sketchlib.proto index 0a6d8193..2b5ad278 100644 --- a/proto/sketchlib.proto +++ b/proto/sketchlib.proto @@ -29,7 +29,6 @@ syntax = "proto3"; package sketchlib.v1; import "common/common.proto"; -import "countminsketch/countminsketch.proto"; import "countsketch/countsketch.proto"; import "hll/hll.proto"; import "kll/kll.proto"; @@ -66,7 +65,6 @@ message SketchEnvelope { // The sketch payload. Exactly one field must be set. oneof sketch_state { - CountMinState count_min = 10; CountSketchState count_sketch = 11; HyperLogLogState hll = 12; KLLState kll = 13; @@ -77,6 +75,8 @@ message SketchEnvelope { ElasticState elastic = 18; } + reserved 10; + reserved "count_min"; reserved 19 to 31; // Reserved for future sketch types. // Geometric / NitroSketch sampling probability for the producing sketch. diff --git a/src/common/hashspec.rs b/src/common/hashspec.rs index 13c44c19..68f1a3ff 100644 --- a/src/common/hashspec.rs +++ b/src/common/hashspec.rs @@ -1,6 +1,6 @@ //! Cross-language hash layer mirroring `sketchlib-go::common`'s //! `HashSpec` / `DeriveIndex` / `DeriveSign` decomposition for -//! matrix-backed sketches (CountSketch, CountMinSketch). +//! matrix-backed sketches (CountSketch, CountMin). //! //! `sketchlib-go` and `asap_sketchlib` share three building blocks for //! these sketches: @@ -15,9 +15,8 @@ //! This module re-exposes that pipeline as a small, byte-key API so //! sketches that consume keys as raw bytes (e.g. the wire-format //! `CountSketch::update`) can match Go's emitted matrix -//! cell-for-cell. CountMinSketch's pending byte-parity fix will reuse -//! the same primitives — the seed table, derive_index, and derive_sign -//! are agnostic to whether ±1 signing is applied. +//! cell-for-cell. The seed table, derive_index, and derive_sign are +//! agnostic to whether ±1 signing is applied. //! //! # Example //! diff --git a/src/lib.rs b/src/lib.rs index b062b8b9..1b1ee688 100644 --- a/src/lib.rs +++ b/src/lib.rs @@ -10,9 +10,9 @@ //! UnivMon, tumbling windows, and batch/parallel execution helpers. //! - [`message_pack_format`]: MessagePack/proto wire format shared with //! `sketchlib-go`. Owns the wire-DTO sketch types consumed by the ASAP -//! query engine (`CountMinSketch`, `CountSketch`, `KllSketch`, `HllSketch`, -//! `DdSketch`, `HydraKllSketch`, `CountMinSketchWithHeap`, -//! `SetAggregator`, `DeltaResult`). +//! query engine (`CountSketch`, `KllSketch`, `HllSketch`, `DdSketch`, +//! `HydraKllSketch`, `CountMinSketchWithHeap`, `SetAggregator`, +//! `DeltaResult`). //! //! Most users can start with the crate-root re-exports such as [`DataInput`], //! [`CountMin`], [`HyperLogLog`], [`KLL`], and [`DDSketch`]. Reach for the @@ -41,7 +41,6 @@ pub mod __private { pub use common::*; pub use message_pack_format::MessagePackCodec; -pub use message_pack_format::portable::countminsketch::{CountMinSketch, CountMinSketchDelta}; pub use message_pack_format::portable::countminsketch_topk::{CmsHeapItem, CountMinSketchWithHeap}; pub use message_pack_format::portable::countsketch::{ COUNT_SKETCH_TOPK_CAPACITY, CountSketch, CountSketchDelta, diff --git a/src/message_pack_format/native/countminsketch.rs b/src/message_pack_format/native/countminsketch.rs deleted file mode 100644 index c4cac9e1..00000000 --- a/src/message_pack_format/native/countminsketch.rs +++ /dev/null @@ -1,27 +0,0 @@ -//! Native MessagePack codec impl for [`crate::sketches::countminsketch::CountMin`]. -//! -//! Only the canonical wire configs — i32/i64/f64 counters (`CmsWireCounter`) -//! with a fast/regular mode (`CmsWireMode`) — are serializable. Exotic in-memory -//! counters (i128/…) must be converted to a wire type first. - -use serde::{Deserialize, Serialize}; - -use crate::message_pack_format::{Error, MessagePackCodec}; -use crate::sketches::countminsketch::{CmsWireCounter, CmsWireMode, CountMin}; -use crate::{HashProfile, SketchHasher, Vector2D}; - -impl MessagePackCodec for CountMin, Mode, H> -where - // `AddAssign` is required for `Vector2D: MatrixStorage`. - T: CmsWireCounter + std::ops::AddAssign + Serialize + for<'de> Deserialize<'de>, - Mode: CmsWireMode, - H: SketchHasher + HashProfile, -{ - fn to_msgpack(&self) -> Result, Error> { - Ok(self.serialize_to_bytes()?) - } - - fn from_msgpack(bytes: &[u8]) -> Result { - Ok(Self::deserialize_from_bytes(bytes)?) - } -} diff --git a/src/message_pack_format/native/mod.rs b/src/message_pack_format/native/mod.rs index 0a83ef96..941fd78e 100644 --- a/src/message_pack_format/native/mod.rs +++ b/src/message_pack_format/native/mod.rs @@ -7,7 +7,6 @@ //! to them. Call those methods directly; `docs/asapv1_wire_format.md` //! specifies the format `sketchlib-go` mirrors. -pub mod countminsketch; pub mod countsketch; pub mod countsketch_topk; pub mod ddsketch; diff --git a/src/message_pack_format/portable/countminsketch.rs b/src/message_pack_format/portable/countminsketch.rs deleted file mode 100644 index 16087bdb..00000000 --- a/src/message_pack_format/portable/countminsketch.rs +++ /dev/null @@ -1,1000 +0,0 @@ -//! Wire-format-aligned Count-Min sketch types. The wire DTO and -//! runtime ops live together here. - -use serde::{Deserialize, Serialize}; - -use crate::message_pack_format::{Error as MsgPackError, MessagePackCodec}; -use crate::sketches::countminsketch::CountMin; -use crate::{DataInput, FastPath, Vector2D}; - -// ===================================================================== -// asap_sketchlib wire-format-aligned variant. -// -// `CountMinSketch` and `CountMinDelta` below are the public-field, -// proto-decode-friendly types consumed by the ASAP query engine -// accumulators, backed by `asap_sketchlib`'s in-tree CountMin. The -// high-throughput in-process variant is -// `crate::sketches::countminsketch::CountMin`, which keeps its own design. -// ===================================================================== - -// ----- asap_sketchlib-backed Count-Min helpers ----- -// Used below by `CountMinSketch`. Lives in this file so the wire-format -// type and its backend share a single home. - -/// Concrete Count-Min type backing the wire-format `CountMinSketch`. -/// Uses f64 counters (`Vector2D`) for weighted updates without -/// integer rounding, and the `FastPath` packed-hash strategy so the -/// matrix-cell layout is byte-parity with `sketchlib-go`. Locked in by -/// `tests/sketches_go_parity_probe.rs`. -pub type SketchlibCms = CountMin, FastPath>; - -/// Creates a fresh sketchlib Count-Min sketch with the given dimensions. -pub fn new_sketchlib_cms(row_num: usize, col_num: usize) -> SketchlibCms { - SketchlibCms::with_dimensions(row_num, col_num) -} - -/// Builds a sketchlib Count-Min sketch from an existing `sketch` matrix. -pub fn sketchlib_cms_from_matrix( - row_num: usize, - col_num: usize, - sketch: &[Vec], -) -> SketchlibCms { - let matrix = Vector2D::from_fn(row_num, col_num, |r, c| { - sketch - .get(r) - .and_then(|row| row.get(c)) - .copied() - .unwrap_or(0.0) - }); - SketchlibCms::from_storage(matrix) -} - -/// Converts a sketchlib Count-Min sketch into a `Vec>` matrix. -pub fn matrix_from_sketchlib_cms(inner: &SketchlibCms) -> Vec> { - let storage: &Vector2D = inner.as_storage(); - let rows = storage.rows(); - let cols = storage.cols(); - let mut sketch = vec![vec![0.0; cols]; rows]; - - for (r, row) in sketch.iter_mut().enumerate().take(rows) { - for (c, cell) in row.iter_mut().enumerate().take(cols) { - if let Some(v) = storage.get(r, c) { - *cell = *v; - } - } - } - - sketch -} - -/// Helper to update a sketchlib Count-Min with a weighted key. -pub fn sketchlib_cms_update(inner: &mut SketchlibCms, key: &str, value: f64) { - if value <= 0.0 { - return; - } - inner.insert_many(&DataInput::String(key.to_owned()), value); -} - -/// Helper to query a sketchlib Count-Min for a key, returning f64. -pub fn sketchlib_cms_query(inner: &SketchlibCms, key: &str) -> f64 { - inner.estimate(&DataInput::String(key.to_owned())) -} - -/// Sparse delta between two consecutive CountMinSketch snapshots — -/// the input shape for [`CountMinSketch::apply_delta`]. Mirrors the -/// `CountMinSketchDelta` proto in -/// `sketchlib-go/proto/countminsketch/countminsketch.proto` (packed -/// encoding only). -/// -/// Cells apply additively: `matrix[row][col] += d_count`. Per-row -/// L1 and L2 norm deltas are carried for downstream error-accounting -/// but are not consumed by `apply_delta` itself. -/// -/// Structurally identical to [`super::countsketch::CountSketchDelta`]: the -/// optional `hh_keys` channel carries heavy-hitter candidate keys forwarded by -/// an upstream tracker. Count-Min can track heavy hitters, but whether -/// `hh_keys` is populated is a control-plane decision — it is empty when -/// heavy-hitter tracking is not enabled. The receiver re-queries the merged -/// matrix for each key to (re)build its Top-K. Mirrors the Go reference -/// implementation's `Delta.HHKeys`. -#[derive(Debug, Clone, Default)] -pub struct CountMinSketchDelta { - pub rows: u32, - pub cols: u32, - pub cells: Vec<(u32, u32, i64)>, - pub l1: Vec, - pub l2: Vec, - pub hh_keys: Vec, -} - -/// Provides approximate frequency counts with error bounds. -/// The msgpack wire format is the contract between sketch producers and -/// the query engine consumer. -#[derive(Debug, Clone)] -pub struct CountMinSketch { - pub rows: usize, - pub cols: usize, - pub(crate) backend: SketchlibCms, -} - -impl CountMinSketch { - pub fn new(rows: usize, cols: usize) -> Self { - Self { - rows, - cols, - backend: new_sketchlib_cms(rows, cols), - } - } - - /// Number of hash rows in the sketch matrix. - pub fn rows(&self) -> usize { - self.rows - } - - /// Number of columns (width) in the sketch matrix. - pub fn cols(&self) -> usize { - self.cols - } - - /// Returns the sketch matrix (for wire format, serialization, tests). - pub fn sketch(&self) -> Vec> { - matrix_from_sketchlib_cms(&self.backend) - } - - /// Construct from a `Vec>` matrix (used by deserialization and query engine). - pub fn from_legacy_matrix(sketch: Vec>, rows: usize, cols: usize) -> Self { - Self { - rows, - cols, - backend: sketchlib_cms_from_matrix(rows, cols, &sketch), - } - } - - /// Insert a single weighted observation. Delegates to the - /// `FastPath` backend so the matrix-cell layout matches - /// `sketchlib-go::CountMinSketch.InsertWithHash` bit-for-bit; the - /// parity is locked in by `tests/sketches_go_parity_probe.rs`. - /// - /// Negative or zero values are skipped, mirroring Go's - /// `UpdateWeight` behavior on `many == 0`. - pub fn update(&mut self, key: &str, value: f64) { - if value <= 0.0 || self.rows == 0 || self.cols == 0 { - return; - } - self.backend - .insert_many(&DataInput::String(key.to_owned()), value); - } - - /// Estimate the frequency of `key` (CountMin point query). - /// Delegates to the `FastPath` backend, which derives the same - /// per-row column index that [`Self::update`] used. - pub fn estimate(&self, key: &str) -> f64 { - if self.rows == 0 || self.cols == 0 { - return 0.0; - } - self.backend.estimate(&DataInput::String(key.to_owned())) - } - - /// Merge another CountMinSketch into self in place. Both operands - /// must have identical dimensions. - pub fn merge( - &mut self, - other: &CountMinSketch, - ) -> Result<(), Box> { - if self.rows != other.rows || self.cols != other.cols { - return Err(format!( - "CountMinSketch dimension mismatch: self={}x{}, other={}x{}", - self.rows, self.cols, other.rows, other.cols - ) - .into()); - } - self.backend.merge(&other.backend); - Ok(()) - } - - /// Merge from references, allocating only the output — no input clones. - pub fn merge_refs( - accumulators: &[&Self], - ) -> Result> { - if accumulators.is_empty() { - return Err("No accumulators to merge".into()); - } - - let rows = accumulators[0].rows; - let cols = accumulators[0].cols; - - for acc in accumulators { - if acc.rows != rows || acc.cols != cols { - return Err( - "Cannot merge CountMinSketch accumulators with different dimensions".into(), - ); - } - } - - let mut merged = CountMinSketch::new(rows, cols); - for acc in accumulators { - merged.backend.merge(&acc.backend); - } - Ok(merged) - } - - /// Apply a sparse delta in place. Matches the `ApplyDelta` - /// semantics in `sketchlib-go/sketches/CountMinSketch/delta.go`: - /// `matrix[row][col] += d_count` for each cell in the delta. - /// - /// The FFI handle is opaque, so we snapshot the matrix, apply - /// cell updates, and rebuild the backend. The rebuild is - /// O(rows × cols) per delta and is acceptable for ingest-side - /// reconstitution — no delta should fire more than once per - /// window (10s–300s in the paper's B3 / B4 configs). - pub fn apply_delta( - &mut self, - delta: &CountMinSketchDelta, - ) -> Result<(), Box> { - for (row, col, _) in &delta.cells { - let r = *row as usize; - let c = *col as usize; - if r >= self.rows || c >= self.cols { - return Err(format!( - "CountMinSketchDelta cell ({r},{c}) out of range (matrix={}x{})", - self.rows, self.cols - ) - .into()); - } - } - let mut matrix = self.sketch(); - for (row, col, d_count) in &delta.cells { - matrix[*row as usize][*col as usize] += *d_count as f64; - } - self.backend = sketchlib_cms_from_matrix(self.rows, self.cols, &matrix); - Ok(()) - } - - /// Compute a sparse, proto-marshalled `CountMinDelta` of `self` - /// against a `snapshot`. A cell is included when its absolute count - /// delta `|Δcount|` (self − snapshot) is `>= threshold` and non-zero. - /// The full per-row L1/L2 norm deltas are always carried (one entry - /// per row, negligible size). - /// - /// This is the Rust twin of the Go reference implementation's - /// `ComputeDelta` + `SerializeDelta`: it iterates the matrix in - /// row-major order, subtracts the snapshot's value for each cell, and - /// emits the surviving cell deltas using the packed-array proto - /// encoding (`cell_rows`/`cell_cols`/`d_counts`). L1/L2 row deltas are - /// derived from the matrices — `l1[r] = Σ_c count[r][c]` and - /// `l2[r] = Σ_c count[r][c]^2` — which telescopes to the same value the - /// Go producer maintains incrementally. Heavy-hitter candidate keys - /// (`hh_keys`) are sourced from an upstream tracker that this minimal - /// wrapper does not maintain, so the field is left empty (an empty repeated - /// field encodes to nothing on the wire), exactly as CountSketch's - /// `compute_delta` does. The returned bytes are a `prost`-encoded - /// [`crate::proto::sketchlib::CountMinDelta`], byte-identical to the Go - /// `proto.Marshal(CountMinDelta)` output for the same inputs when no - /// heavy-hitter keys are forwarded (cross-language byte parity). - /// - /// Delta-against-empty: when `snapshot` is the all-zero sketch, every - /// surviving cell delta equals this window's own cell count, so the - /// result is this window's full state encoded as a delta (no - /// cross-window subtraction). CMS deltas carry only sketch-internal - /// cells/norms — there are no DataPoint-level metric scalars to drop. - pub fn compute_delta( - &self, - snapshot: &CountMinSketch, - threshold: f64, - ) -> Result, Box> { - use crate::proto::sketchlib::CountMinDelta as ProtoDelta; - use prost::Message; - - if self.rows != snapshot.rows || self.cols != snapshot.cols { - return Err(format!( - "CountMinSketch dimension mismatch: self={}x{}, snapshot={}x{}", - self.rows, self.cols, snapshot.rows, snapshot.cols - ) - .into()); - } - - let cur = self.sketch(); - let snap = snapshot.sketch(); - - let mut cell_rows: Vec = Vec::new(); - let mut cell_cols: Vec = Vec::new(); - let mut d_counts: Vec = Vec::new(); - let mut l1: Vec = Vec::with_capacity(self.rows); - let mut l2: Vec = Vec::with_capacity(self.rows); - - for r in 0..self.rows { - let mut cur_l1 = 0.0f64; - let mut cur_l2 = 0.0f64; - let mut snap_l1 = 0.0f64; - let mut snap_l2 = 0.0f64; - for c in 0..self.cols { - let cv = cur[r][c]; - let sv = snap[r][c]; - cur_l1 += cv; - cur_l2 += cv * cv; - snap_l1 += sv; - snap_l2 += sv * sv; - // CMS counts are non-negative integers in the wire form; - // mirror the Go reference's signed Δ + |Δ| threshold test. - let dc = (cv - sv) as i64; - if dc != 0 && (dc.unsigned_abs() as f64) >= threshold { - cell_rows.push(r as u32); - cell_cols.push(c as u32); - d_counts.push(dc); - } - } - l1.push(cur_l1 - snap_l1); - l2.push(cur_l2 - snap_l2); - } - - let delta = ProtoDelta { - rows: self.rows as u32, - cols: self.cols as u32, - cells_legacy: Vec::new(), - l1, - l2, - // Heavy-hitter keys are control-plane-gated; this wrapper has no - // tracker, so the field is empty (mirrors CountSketch::compute_delta). - hh_keys: Vec::new(), - cell_rows, - cell_cols, - d_counts, - }; - Ok(delta.encode_to_vec()) - } - - /// Apply a `prost`-encoded [`crate::proto::sketchlib::CountMinDelta`] - /// to this sketch in place (additive cell merge). The Rust twin of the - /// Go reference implementation's `DeserializeDelta` + `ApplyDelta`. - /// Reads the packed cell arrays (`cell_rows`/`cell_cols`/`d_counts`) - /// and falls back to the legacy per-cell records for payloads from - /// older producers. Heavy-hitter keys (`hh_keys`) are read symmetrically - /// (mirroring CountSketch) and carried onto the decoded delta; a plain CMS - /// keeps no Top-K, so they are not otherwise consumed here. - /// - /// Returns `Err` if `bytes` is not a valid `CountMinDelta` proto or a - /// cell is out of range for this sketch's dimensions. - pub fn apply_delta_bytes( - &mut self, - bytes: &[u8], - ) -> Result<(), Box> { - use crate::proto::sketchlib::CountMinDelta as ProtoDelta; - use prost::Message; - - let proto = ProtoDelta::decode(bytes)?; - let cells: Vec<(u32, u32, i64)> = if !proto.cell_rows.is_empty() { - proto - .cell_rows - .iter() - .zip(proto.cell_cols.iter()) - .zip(proto.d_counts.iter()) - .map(|((&r, &c), &d)| (r, c, d)) - .collect() - } else { - proto - .cells_legacy - .iter() - .map(|c| (c.row, c.col, c.d_count as i64)) - .collect() - }; - let delta = CountMinSketchDelta { - rows: proto.rows, - cols: proto.cols, - cells, - l1: proto.l1, - l2: proto.l2, - // Heavy-hitter keys are carried through symmetrically (mirrors - // CountSketch). This wrapper keeps no Top-K, so they are not - // otherwise consumed; a control-plane sink would re-query the - // merged matrix for each key. Empty for a plain CMS. - hh_keys: proto.hh_keys, - }; - self.apply_delta(&delta) - } - - /// MessagePack twin of [`Self::compute_delta`]. Computes the same - /// sparse cell delta of `self` against `snapshot` (a cell is carried - /// when `|Δcount|` is non-zero and `>= threshold`), but serializes it - /// as the cells-only parallel-array MessagePack layout via - /// [`CountMinSketchDelta`]'s [`MessagePackCodec`] instead of proto. The - /// per-row `l1`/`l2` norms and `hh_keys` the proto delta carries are - /// dropped — the plain-sketch estimate path does not consume them. - /// Delta-against-empty carries this window's own cell counts. - pub fn compute_delta_msgpack( - &self, - snapshot: &CountMinSketch, - threshold: f64, - ) -> Result, Box> { - if self.rows != snapshot.rows || self.cols != snapshot.cols { - return Err(format!( - "CountMinSketch dimension mismatch: self={}x{}, snapshot={}x{}", - self.rows, self.cols, snapshot.rows, snapshot.cols - ) - .into()); - } - let cur = self.sketch(); - let snap = snapshot.sketch(); - let mut delta = CountMinSketchDelta { - rows: self.rows as u32, - cols: self.cols as u32, - ..CountMinSketchDelta::default() - }; - for r in 0..self.rows { - for c in 0..self.cols { - let dc = (cur[r][c] - snap[r][c]) as i64; - if dc != 0 && (dc.unsigned_abs() as f64) >= threshold { - delta.cells.push((r as u32, c as u32, dc)); - } - } - } - Ok(delta.to_msgpack()?) - } - - /// MessagePack twin of [`Self::apply_delta_bytes`]. Decodes the - /// cells-only parallel-array MessagePack [`CountMinSketchDelta`] and - /// applies it in place (additive cell merge). - /// - /// Returns `Err` if `bytes` is not a valid MessagePack - /// `CountMinSketchDelta` or a cell is out of range for this sketch's - /// dimensions. - pub fn apply_delta_msgpack_bytes( - &mut self, - bytes: &[u8], - ) -> Result<(), Box> { - let delta = CountMinSketchDelta::from_msgpack(bytes)?; - self.apply_delta(&delta) - } - - /// One-shot aggregation: build a sketch from parallel key/value slices - /// and return the msgpack bytes. - pub fn aggregate_count( - depth: usize, - width: usize, - keys: &[&str], - values: &[f64], - ) -> Option> { - if keys.is_empty() { - return None; - } - let mut sketch = Self::new(depth, width); - for (key, &value) in keys.iter().zip(values.iter()) { - sketch.update(key, value); - } - sketch.to_msgpack().ok() - } - - /// Same as aggregate_count — CMS accumulates sums by construction. - pub fn aggregate_sum( - depth: usize, - width: usize, - keys: &[&str], - values: &[f64], - ) -> Option> { - Self::aggregate_count(depth, width, keys, values) - } -} - -// ----- Wire format ----- - -/// Wire DTO for [`CountMinSketch`]. -#[derive(Debug, Clone, Serialize, Deserialize)] -pub struct CountMinSketchWire { - pub sketch: Vec>, - #[serde(rename = "row_num")] - pub rows: usize, - #[serde(rename = "col_num")] - pub cols: usize, -} - -impl MessagePackCodec for CountMinSketch { - fn to_msgpack(&self) -> Result, MsgPackError> { - let wire = CountMinSketchWire { - sketch: self.sketch(), - rows: self.rows, - cols: self.cols, - }; - Ok(rmp_serde::to_vec(&wire)?) - } - - fn from_msgpack(bytes: &[u8]) -> Result { - let wire: CountMinSketchWire = rmp_serde::from_slice(bytes)?; - let backend = sketchlib_cms_from_matrix(wire.rows, wire.cols, &wire.sketch); - Ok(Self { - rows: wire.rows, - cols: wire.cols, - backend, - }) - } -} - -impl MessagePackCodec for CountMinSketchDelta { - /// Encodes the sparse cell delta as the 5-element MessagePack array - /// `[ rows:u64, cols:u64, row_idx:[]u32, col_idx:[]u32, d_count:[]i64 ]` - /// — byte-identical to Rust - /// `rmp_serde::to_vec(&(u64, u64, Vec, Vec, Vec))` - /// (compact mode), the same struct-of-parallel-arrays shape the Go - /// `count_sketch_delta_sparse.go` established. Only the matrix cells - /// cross the wire; the per-row `l1`/`l2` norms and the `hh_keys` - /// heavy-hitter surface are not carried. - fn to_msgpack(&self) -> Result, MsgPackError> { - let mut row_idx: Vec = Vec::with_capacity(self.cells.len()); - let mut col_idx: Vec = Vec::with_capacity(self.cells.len()); - let mut d_count: Vec = Vec::with_capacity(self.cells.len()); - for (r, c, d) in &self.cells { - row_idx.push(*r); - col_idx.push(*c); - d_count.push(*d); - } - Ok(rmp_serde::to_vec(&( - self.rows as u64, - self.cols as u64, - row_idx, - col_idx, - d_count, - ))?) - } - - fn from_msgpack(bytes: &[u8]) -> Result { - let (rows, cols, row_idx, col_idx, d_count): (u64, u64, Vec, Vec, Vec) = - rmp_serde::from_slice(bytes)?; - let cells = row_idx - .into_iter() - .zip(col_idx) - .zip(d_count) - .map(|((r, c), d)| (r, c, d)) - .collect(); - Ok(CountMinSketchDelta { - rows: rows as u32, - cols: cols as u32, - cells, - ..CountMinSketchDelta::default() - }) - } -} - -#[cfg(test)] -mod tests { - use super::*; - - #[test] - fn msgpack_delta_against_empty_round_trips() { - let (rows, cols) = (5, 1024); - let mut w = CountMinSketch::new(rows, cols); - for _ in 0..200 { - w.update("hot", 1.0); - } - let empty = CountMinSketch::new(rows, cols); - let bytes = w.compute_delta_msgpack(&empty, 1.0).unwrap(); - let mut recon = CountMinSketch::new(rows, cols); - recon.apply_delta_msgpack_bytes(&bytes).unwrap(); - let (got, want) = (recon.estimate("hot"), w.estimate("hot")); - assert!((got - want).abs() <= want * 0.1, "got={got} want={want}"); - } - - #[test] - fn msgpack_delta_wire_layout_is_parallel_arrays() { - use crate::message_pack_format::MessagePackCodec; - let (rows, cols) = (3, 8); - let mut w = CountMinSketch::new(rows, cols); - w.update("x", 5.0); - let empty = CountMinSketch::new(rows, cols); - let bytes = w.compute_delta_msgpack(&empty, 1.0).unwrap(); - let cur = w.sketch(); - let (mut ri, mut ci, mut dc): (Vec, Vec, Vec) = - (Vec::new(), Vec::new(), Vec::new()); - for (r, row) in cur.iter().enumerate() { - for (c, &cell) in row.iter().enumerate() { - let d = cell as i64; - if d != 0 { - ri.push(r as u32); - ci.push(c as u32); - dc.push(d); - } - } - } - let expected = rmp_serde::to_vec(&(rows as u64, cols as u64, ri, ci, dc)).unwrap(); - assert_eq!(bytes, expected); - let d = CountMinSketchDelta::from_msgpack(&bytes).unwrap(); - assert_eq!(d.rows, rows as u32); - assert_eq!(d.cols, cols as u32); - } - - #[test] - fn test_count_min_sketch_creation() { - let cms = CountMinSketch::new(4, 1000); - assert_eq!(cms.rows, 4); - assert_eq!(cms.cols, 1000); - let sketch = cms.sketch(); - assert_eq!(sketch.len(), 4); - assert_eq!(sketch[0].len(), 1000); - - // Check all values are initialized to 0 - for row in &sketch { - for &value in row { - assert_eq!(value, 0.0); - } - } - } - - #[test] - fn test_count_min_sketch_update() { - let mut cms = CountMinSketch::new(2, 10); - cms.update("key1", 1.0); - // Query should return at least the updated value - let result = cms.estimate("key1"); - assert!(result >= 1.0); - } - - #[test] - fn test_count_min_sketch_query_empty() { - let cms = CountMinSketch::new(2, 10); - assert_eq!(cms.estimate("anything"), 0.0); - } - - #[test] - fn test_count_min_sketch_merge() { - // Use from_legacy_matrix so the test works regardless of sketchlib/legacy config - let mut sketch1 = vec![vec![0.0; 3]; 2]; - sketch1[0][0] = 5.0; - sketch1[1][2] = 10.0; - let mut cms1 = CountMinSketch::from_legacy_matrix(sketch1, 2, 3); - - let mut sketch2 = vec![vec![0.0; 3]; 2]; - sketch2[0][0] = 3.0; - sketch2[0][1] = 7.0; - let cms2 = CountMinSketch::from_legacy_matrix(sketch2, 2, 3); - - cms1.merge(&cms2).unwrap(); - let merged_sketch = cms1.sketch(); - - assert_eq!(merged_sketch[0][0], 8.0); // 5 + 3 - assert_eq!(merged_sketch[0][1], 7.0); // 0 + 7 - assert_eq!(merged_sketch[1][2], 10.0); // 10 + 0 - } - - #[test] - fn test_count_min_sketch_merge_dimension_mismatch() { - let mut cms1 = CountMinSketch::new(2, 3); - let cms2 = CountMinSketch::new(3, 3); - assert!(cms1.merge(&cms2).is_err()); - } - - #[test] - fn test_count_min_sketch_msgpack_round_trip() { - let mut cms = CountMinSketch::new(4, 256); - cms.update("apple", 5.0); - cms.update("banana", 3.0); - cms.update("apple", 2.0); // total "apple" = 7 - - let bytes = cms.to_msgpack().unwrap(); - let deserialized = CountMinSketch::from_msgpack(&bytes).unwrap(); - - assert_eq!(deserialized.rows, 4); - assert_eq!(deserialized.cols, 256); - assert!(deserialized.estimate("apple") >= 7.0); - assert!(deserialized.estimate("banana") >= 3.0); - } - - #[test] - fn test_aggregate_count() { - let keys = ["a", "b", "a"]; - let values = [1.0, 2.0, 3.0]; - let bytes = CountMinSketch::aggregate_count(4, 100, &keys, &values).unwrap(); - let cms = CountMinSketch::from_msgpack(&bytes).unwrap(); - // "a" was updated twice (1.0 + 3.0 = 4.0), "b" once (2.0) - assert!(cms.estimate("a") >= 4.0); - assert!(cms.estimate("b") >= 2.0); - } - - #[test] - fn test_aggregate_count_empty() { - assert!(CountMinSketch::aggregate_count(4, 100, &[], &[]).is_none()); - } - - #[test] - fn test_apply_delta_additive() { - let mut cms = CountMinSketch::from_legacy_matrix( - vec![vec![1.0, 2.0, 3.0], vec![4.0, 5.0, 6.0]], - 2, - 3, - ); - let delta = CountMinSketchDelta { - rows: 2, - cols: 3, - cells: vec![(0, 0, 10), (1, 2, 100)], - l1: vec![], - l2: vec![], - hh_keys: vec![], - }; - cms.apply_delta(&delta).unwrap(); - assert_eq!( - cms.sketch(), - vec![vec![11.0, 2.0, 3.0], vec![4.0, 5.0, 106.0]] - ); - } - - #[test] - fn test_apply_delta_matches_full_merge() { - let base = CountMinSketch::from_legacy_matrix(vec![vec![1.0, 2.0], vec![3.0, 4.0]], 2, 2); - let addition = - CountMinSketch::from_legacy_matrix(vec![vec![10.0, 0.0], vec![0.0, 20.0]], 2, 2); - let mut via_merge = base.clone(); - via_merge.merge(&addition).unwrap(); - - let delta = CountMinSketchDelta { - rows: 2, - cols: 2, - cells: vec![(0, 0, 10), (1, 1, 20)], - l1: vec![], - l2: vec![], - hh_keys: vec![], - }; - let mut via_delta = base; - via_delta.apply_delta(&delta).unwrap(); - assert_eq!(via_delta.sketch(), via_merge.sketch()); - } - - #[test] - fn test_apply_delta_out_of_range() { - let mut cms = CountMinSketch::new(2, 3); - let delta = CountMinSketchDelta { - rows: 2, - cols: 3, - cells: vec![(5, 0, 1)], - l1: vec![], - l2: vec![], - hh_keys: vec![], - }; - assert!(cms.apply_delta(&delta).is_err()); - } - - /// Cross-language byte-parity guard against `sketchlib-go`'s - /// `CountMinSketch.SerializeProtoBytesFO` output for the - /// deterministic input `goldenCmsKeys()` (10 keys "flow-0"..."flow-9", - /// each repeated 5×, 50 unweighted updates total) at dimensions - /// `(rows=4, cols=2048)` — the - /// `asap-precompute-rs::CMSWrapper::new(4, 2048)` default. The hex - /// blob below was captured from `proto.Marshal` of the Go envelope - /// with `Producer` and `HashSpec` cleared, matching the - /// `integration/parity/golden_test.go::TestGenerateGoldenFixtures` - /// recipe (and the byte-payload that the Rust wrapper's - /// `CMSWrapper::encode_envelope` emits to satisfy - /// `cross_language_parity::cms_byte_parity_with_go`). - /// - /// Frequency-Only payload structure (matches Go's - /// `SerializePortableFO`): - /// - `rows = 4`, `cols = 2048`, `counter_type = INT64` - /// - `counts_int` = packed sint64 row-major, `4*2048 = 8192` cells - /// - `sum_counts` and `sum2_counts` deliberately omitted - /// - `l1[r] = 50` (each row sees 50 unweighted inserts) - /// - `l2[r] = 250` (10 unique cells per row each holding count 5, - /// so `Σ count^2 = 10 * 25 = 250`) - /// - /// Any drift in [`CountMinSketch::update`]'s hash → column-index - /// derivation breaks this test cell-for-cell; that is the contract - /// `cross_language_parity::cms_byte_parity_with_go` in ASAPCollector - /// relies on. - #[test] - fn test_update_then_envelope_matches_sketchlib_go_bytes() { - use crate::proto::sketchlib::{ - CountMinState, CounterType, SketchEnvelope, sketch_envelope::SketchState, - }; - use prost::Message; - - let rows = 4usize; - let cols = 2048usize; - let mut sk = CountMinSketch::new(rows, cols); - for i in 0..50u64 { - let key = format!("flow-{}", i % 10); - sk.update(&key, 1.0); - } - - // Build the Frequency-Only envelope mirroring sketchlib-go's - // `CountMinSketch.SerializePortableFO`: - // - counter_type = INT64 - // - counts_int = packed sint64 row-major, len = rows * cols - // - sum_counts / sum2_counts deliberately omitted - // - l1[r] = Σ_c count[r][c] (Go maintains this incrementally - // in InsertWithHash; equals `weight * Σ inserts in row r`, - // which collapses to 50 for an unweighted 50-insert stream) - // - l2[r] = Σ_c count[r][c]^2 (Go maintains this as - // `L2[r] += curr*curr - prev*prev`, telescoping to the same - // sum-of-squares for unweighted streams) - let matrix = sk.sketch(); - let mut counts_int: Vec = Vec::with_capacity(rows * cols); - let mut l1: Vec = Vec::with_capacity(rows); - let mut l2: Vec = Vec::with_capacity(rows); - for row in matrix.iter().take(rows) { - let mut row_l1 = 0.0f64; - let mut row_l2 = 0.0f64; - for &cell in row.iter().take(cols) { - counts_int.push(cell as i64); - row_l1 += cell; - row_l2 += cell * cell; - } - l1.push(row_l1); - l2.push(row_l2); - } - - let state = CountMinState { - rows: rows as u32, - cols: cols as u32, - counter_type: CounterType::Int64 as i32, - counts_int, - counts_float: Vec::new(), - sum_counts: Vec::new(), - sum2_counts: Vec::new(), - l1, - l2, - }; - let envelope = SketchEnvelope { - format_version: 1, - producer: None, - hash_spec: None, - sample_p: 0.0, - sketch_state: Some(SketchState::CountMin(state)), - }; - let mut got = Vec::with_capacity(envelope.encoded_len()); - envelope.encode(&mut got).expect("prost encode"); - - // 8275-byte hex blob captured from - // `sketchlib-go::CountMinSketch.SerializeProtoBytesFO` for the - // same `(4, 2048) × goldenCmsKeys()` input — see - // `integration/parity/golden_test.go` and - // `cross_language_parity::cms_byte_parity_with_go` in - // ASAPCollector. - const GOLDEN_HEX: &str = include_str!("../../sketches/testdata/cms_envelope_golden.hex"); - let want = decode_hex_cms(GOLDEN_HEX); - assert_eq!( - got.len(), - want.len(), - "CMS envelope length differs: got {} bytes, want {} bytes", - got.len(), - want.len(), - ); - assert_eq!( - got, want, - "CMS envelope bytes diverge from sketchlib-go golden" - ); - } - - /// Cross-language byte-parity guard for CountMinDelta hh_keys: encoding the - /// canonical packed delta with heavy-hitter keys at tag 6 must be - /// byte-identical to the Go reference's `SerializeDelta` output for the same - /// state and keys. The empty-`hh_keys` path is already covered by - /// `test_compute_delta_matches_go_golden_bytes`. - #[test] - fn test_hh_keys_matches_go_golden_bytes() { - use crate::proto::sketchlib::CountMinDelta as ProtoDelta; - use prost::Message; - - let rows = 4usize; - let cols = 2048usize; - let mut current = CountMinSketch::new(rows, cols); - for i in 0..50u64 { - current.update(&format!("flow-{}", i % 10), 1.0); - } - let empty = CountMinSketch::new(rows, cols); - - // Inject heavy-hitter keys onto the canonical packed delta at tag 6. - let no_hh = current.compute_delta(&empty, 1.0).unwrap(); - let mut proto = ProtoDelta::decode(no_hh.as_slice()).unwrap(); - proto.hh_keys = vec!["flow-0".into(), "flow-3".into(), "flow-7".into()]; - let got = proto.encode_to_vec(); - - // Captured from the Go reference implementation's SerializeDelta for the - // same input with HHKeys = ["flow-0","flow-3","flow-7"]. - const GOLDEN_HEX: &str = "0804108010222000000000000049400000000000004940000000000000494000000000000049402a200000000000406f400000000000406f400000000000406f400000000000406f403206666c6f772d303206666c6f772d333206666c6f772d374a28000000000000000000000101010101010101010102020202020202020202030303030303030303035250c101b202b608b7088809bc09c20dc60d910fbb0f9f018d04a604c004d6058507a909920b900ec70ead038304bc058007c207cc0a820bd50cbe0fcd0fe2019e02cd02e702a707b809a00ac20cee0dfe0f5a280a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a"; - let want = decode_hex_cms(GOLDEN_HEX); - assert_eq!( - got, - want, - "CountMin delta hh_keys bytes diverge from the Go reference golden \ - ({} bytes got vs {} bytes want)", - got.len(), - want.len(), - ); - - // Symmetric decode: apply_delta_bytes round-trips hh_keys without error. - let mut sink = CountMinSketch::new(rows, cols); - sink.apply_delta_bytes(&got).unwrap(); - } - - fn decode_hex_cms(s: &str) -> Vec { - let s = s.trim(); - s.as_bytes() - .chunks(2) - .map(|pair| { - let high = hex_nibble_cms(pair[0]); - let low = hex_nibble_cms(pair[1]); - (high << 4) | low - }) - .collect() - } - - fn hex_nibble_cms(c: u8) -> u8 { - match c { - b'0'..=b'9' => c - b'0', - b'a'..=b'f' => c - b'a' + 10, - b'A'..=b'F' => c - b'A' + 10, - _ => panic!("non-hex byte {}", c as char), - } - } - - /// `compute_delta` against an EMPTY snapshot reconstructs the window's - /// full state when its bytes are applied to a fresh empty sketch - /// (round-trip). With `threshold = 1.0` every changed cell survives, so - /// applying the delta to a zero base yields the same matrix as the - /// original window. - #[test] - fn test_compute_delta_against_empty_round_trips() { - let rows = 4usize; - let cols = 2048usize; - let mut window = CountMinSketch::new(rows, cols); - for i in 0..200u64 { - window.update(&format!("flow-{}", i % 37), 1.0); - } - let empty = CountMinSketch::new(rows, cols); - - let delta_bytes = window.compute_delta(&empty, 1.0).unwrap(); - - let mut reconstructed = CountMinSketch::new(rows, cols); - reconstructed.apply_delta_bytes(&delta_bytes).unwrap(); - - assert_eq!(reconstructed.sketch(), window.sketch()); - } - - /// A delta computed between two non-empty snapshots reconstructs the - /// current sketch when applied to the base — matching a direct merge of - /// the cell-wise difference. - #[test] - fn test_compute_delta_then_apply_matches_current() { - let mut base = CountMinSketch::new(2, 64); - for i in 0..40u64 { - base.update(&format!("k{}", i % 8), 1.0); - } - let mut current = base.clone(); - for i in 0..30u64 { - current.update(&format!("k{}", i % 8), 1.0); - } - - let delta_bytes = current.compute_delta(&base, 1.0).unwrap(); - let mut reconstructed = base.clone(); - reconstructed.apply_delta_bytes(&delta_bytes).unwrap(); - assert_eq!(reconstructed.sketch(), current.sketch()); - } - - /// Cross-language byte-parity guard: `compute_delta` against an empty - /// snapshot must emit bytes identical to the Go reference - /// implementation's `SerializeDelta(ComputeDelta(empty, current, 1.0))` - /// for the same `(rows=4, cols=2048)` × "flow-0".."flow-9" (each 5×, - /// 50 unweighted updates) input. The golden hex was captured from a - /// `proto.Marshal` of the Go reference's `CountMinDelta` (packed-array - /// encoding) for that input. A delta-against-empty carries the window's - /// full state, so this also pins the per-row L1 (=50) / L2 (=250) norm - /// deltas and the packed cell arrays. - #[test] - fn test_compute_delta_matches_go_golden_bytes() { - let rows = 4usize; - let cols = 2048usize; - let mut current = CountMinSketch::new(rows, cols); - for i in 0..50u64 { - current.update(&format!("flow-{}", i % 10), 1.0); - } - let empty = CountMinSketch::new(rows, cols); - let got = current.compute_delta(&empty, 1.0).unwrap(); - - // Captured from the Go reference implementation's - // SerializeDelta(ComputeDelta(empty, current, 1.0)) for the same input. - const GOLDEN_HEX: &str = "0804108010222000000000000049400000000000004940000000000000494000000000000049402a200000000000406f400000000000406f400000000000406f400000000000406f404a28000000000000000000000101010101010101010102020202020202020202030303030303030303035250c101b202b608b7088809bc09c20dc60d910fbb0f9f018d04a604c004d6058507a909920b900ec70ead038304bc058007c207cc0a820bd50cbe0fcd0fe2019e02cd02e702a707b809a00ac20cee0dfe0f5a280a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a"; - let want = decode_hex_cms(GOLDEN_HEX); - assert_eq!( - got, - want, - "CountMin delta bytes diverge from the Go reference golden \ - ({} bytes got vs {} bytes want)", - got.len(), - want.len(), - ); - } -} diff --git a/src/message_pack_format/portable/countminsketch_topk.rs b/src/message_pack_format/portable/countminsketch_topk.rs index 0227824a..d927eeb2 100644 --- a/src/message_pack_format/portable/countminsketch_topk.rs +++ b/src/message_pack_format/portable/countminsketch_topk.rs @@ -1,8 +1,7 @@ //! Wire-format-aligned Count-Min sketch + top-k heap composite. //! -//! No `sketches::*` equivalent — this combines `sketches::CMSHeap` -//! with the wire-format `CountMinSketch` shape and exposes it as a -//! single Go-interop type. +//! No `sketches::*` equivalent — this wraps `sketches::CMSHeap` in a +//! row/column matrix shape and exposes it as a single Go-interop type. use serde::{Deserialize, Serialize}; diff --git a/src/message_pack_format/portable/countsketch.rs b/src/message_pack_format/portable/countsketch.rs index a94ad981..f9f1d89f 100644 --- a/src/message_pack_format/portable/countsketch.rs +++ b/src/message_pack_format/portable/countsketch.rs @@ -17,10 +17,10 @@ use crate::message_pack_format::{Error as MsgPackError, MessagePackCodec}; // Count Sketch (a.k.a. Count-Min-style signed-counter sketch) — // element-wise mergeable frequency estimator. // -// Parallel to `count_min::CountMinSketch` but with **signed** counters, -// matching the `asap_sketchlib::proto::sketchlib::CountSketchState` wire -// format that DataCollector's `countsketchprocessor` emits via the -// modified OTLP `Metric.data = CountSketch{…}` variant. +// Signed counters, matching the +// `asap_sketchlib::proto::sketchlib::CountSketchState` wire format that +// DataCollector's `countsketchprocessor` emits via the modified OTLP +// `Metric.data = CountSketch{…}` variant. // // The surface is: construct from a decoded proto state, merge // element-wise with another sketch, emit the matrix for queries and diff --git a/src/message_pack_format/portable/mod.rs b/src/message_pack_format/portable/mod.rs index 97e0d372..224b7d1c 100644 --- a/src/message_pack_format/portable/mod.rs +++ b/src/message_pack_format/portable/mod.rs @@ -2,12 +2,11 @@ //! //! Each per-algorithm submodule holds one sketch's pre-envelope wire type. //! These carry a cross-language contract of their own, on goldens apart -//! from ASAPv1's `asapv1_golden/`: HLL, Count-Min, Count Sketch and KLL pin -//! byte parity against Go's serializers, KLL in both directions; DDSketch's -//! golden is a Rust self-pin, not yet reconciled. New code uses ASAPv1: +//! from ASAPv1's `asapv1_golden/`: HLL, Count Sketch and KLL pin byte parity +//! against Go's serializers, KLL in both directions; DDSketch's golden is a +//! Rust self-pin, not yet reconciled. New code uses ASAPv1: //! prefer a sketch's own `serialize_to_bytes`; do not add callers here. -pub mod countminsketch; pub mod countminsketch_topk; pub mod countsketch; pub mod countsketch_topk; diff --git a/src/proto/generated/sketchlib.v1.rs b/src/proto/generated/sketchlib.v1.rs index 2c098d3f..c10445dc 100644 --- a/src/proto/generated/sketchlib.v1.rs +++ b/src/proto/generated/sketchlib.v1.rs @@ -198,80 +198,6 @@ pub struct CountMinState { #[prost(double, repeated, tag = "9")] pub l2: ::prost::alloc::vec::Vec, } -/// CountMinDelta carries only the matrix cells that changed by at least the -/// threshold between two consecutive snapshots, plus the full per-row L1/L2 -/// norm deltas (one entry per row, negligible size). -/// -/// Cells apply additively on the receiver: matrix[row][col] += d_count. CMS -/// counters only ever grow, but the delta encodes a signed count (sint64) so -/// the same wire form covers weighted/decay variants. -/// -/// The packed cell encoding (cell_rows / cell_cols / d_counts, tags 9-11) is -/// the canonical form; the legacy per-cell message (tag 3) is retained only so -/// payloads from older producers still decode. This message is byte-identical -/// to the Go reference implementation's CountMinDelta so the two runtimes emit -/// byte-identical delta frames for identical window state (cross-language byte -/// parity). -/// -/// CountMinDelta is structurally identical to CountSketchDelta: rows, cols, the -/// packed cell encoding, per-row norm deltas, and an optional repeated hh_keys -/// at the SAME tag number (6). Both sketches can track heavy hitters; whether -/// hh_keys is populated is a control-plane decision (it is empty/omitted when -/// heavy-hitter tracking is not enabled for this sketch). -#[derive(Clone, PartialEq, ::prost::Message)] -pub struct CountMinDelta { - #[prost(uint32, tag = "1")] - pub rows: u32, - #[prost(uint32, tag = "2")] - pub cols: u32, - /// Deprecated: use cell_rows/cell_cols/d_counts (tags 9-11). Retained for - /// backward-compatible decode of payloads from older producers. - #[prost(message, repeated, tag = "3")] - pub cells_legacy: ::prost::alloc::vec::Vec, - /// Per-row L1 norm deltas, length = rows. - #[prost(double, repeated, tag = "4")] - pub l1: ::prost::alloc::vec::Vec, - /// Per-row L2 norm deltas, length = rows. - #[prost(double, repeated, tag = "5")] - pub l2: ::prost::alloc::vec::Vec, - /// Heavy-hitter candidate keys from an upstream tracker, mirroring - /// CountSketchDelta.hh_keys (same tag number 6, same wire shape). Downstream - /// queries the merged Count-Min matrix for each key to (re)build its Top-K - /// with globally-merged estimates. Empty/omitted when heavy-hitter tracking - /// is not enabled. - #[prost(string, repeated, tag = "6")] - pub hh_keys: ::prost::alloc::vec::Vec<::prost::alloc::string::String>, - /// Packed cell encoding (canonical form): - /// - /// row index of each changed cell - #[prost(uint32, repeated, tag = "9")] - pub cell_rows: ::prost::alloc::vec::Vec, - /// col index of each changed cell - #[prost(uint32, repeated, tag = "10")] - pub cell_cols: ::prost::alloc::vec::Vec, - /// signed integer count delta - #[prost(sint64, repeated, tag = "11")] - pub d_counts: ::prost::alloc::vec::Vec, -} -/// CountMinCell is the deprecated per-cell delta record. Producers emit the -/// packed cell arrays on CountMinDelta instead; this message exists only to -/// decode payloads from older producers. -#[derive(Clone, Copy, PartialEq, ::prost::Message)] -pub struct CountMinCell { - #[prost(uint32, tag = "1")] - pub row: u32, - #[prost(uint32, tag = "2")] - pub col: u32, - /// deprecated: use CountMinDelta.d_counts - #[prost(double, tag = "3")] - pub d_count: f64, - /// deprecated: omitted in the packed encoding - #[prost(double, tag = "4")] - pub d_sum: f64, - /// deprecated: omitted in the packed encoding - #[prost(double, tag = "5")] - pub d_sum2: f64, -} /// CountSketchState is the portable state of a Count (±1) Sketch. /// /// The counter matrix is signed because Count Sketch uses ±1 increments. @@ -892,7 +818,7 @@ pub struct SketchEnvelope { /// The sketch payload. Exactly one field must be set. #[prost( oneof = "sketch_envelope::SketchState", - tags = "10, 11, 12, 13, 14, 15, 16, 17, 18" + tags = "11, 12, 13, 14, 15, 16, 17, 18" )] pub sketch_state: ::core::option::Option, } @@ -901,8 +827,6 @@ pub mod sketch_envelope { /// The sketch payload. Exactly one field must be set. #[derive(Clone, PartialEq, ::prost::Oneof)] pub enum SketchState { - #[prost(message, tag = "10")] - CountMin(super::CountMinState), #[prost(message, tag = "11")] CountSketch(super::CountSketchState), #[prost(message, tag = "12")] diff --git a/src/sketches/testdata/cms_envelope_golden.hex b/src/sketches/testdata/cms_envelope_golden.hex deleted file mode 100644 index c859ecc5..00000000 --- a/src/sketches/testdata/cms_envelope_golden.hex +++ /dev/null @@ -1 +0,0 @@ -080152ce4008041080101802228040000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000a000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000a0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000a0a00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000a0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000a000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000a0000000a000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000a00000000000000000000000000000000000000000000000000000000000000000000000000000000000a00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000a00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000a0000000000000000000000000000000000000000000000000a000000000000000000000000000000000000000000000000000a00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000a0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000a0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000a000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000a0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000a0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000a000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000a000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000a000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000a0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000a00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000a0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000a00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000a0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000a0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000a00000000000000000000000000000a0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000a00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000a000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000a000000000000000000000000000000000000000000000000000a00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000a00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000a000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000a000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000a0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000a000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000a00422000000000000049400000000000004940000000000000494000000000000049404a200000000000406f400000000000406f400000000000406f400000000000406f40 \ No newline at end of file diff --git a/tests/PBT_COVERAGE.md b/tests/PBT_COVERAGE.md index cddee44f..be0fc74b 100644 --- a/tests/PBT_COVERAGE.md +++ b/tests/PBT_COVERAGE.md @@ -68,7 +68,7 @@ Compared against: an exact `HashMap` count, columns via the public `hash_for_mat - estimator - `estimate(k)` never drops on a non-negative insert; all 32 keys probed after every insert - `truth(k) <= estimate(k) <= truth(k) + min over rows of colliding mass`; 1-8 cols x 32 keys, every row collides - - portable `CountMinSketch`: `estimate(k) >= truth(k)` + - `f64` counters over string keys: `estimate(k) >= truth(k)` - merge - commutative, associative, empty is identity - `merge(S(xs), S(ys)) == S(xs ++ ys)` @@ -78,9 +78,9 @@ Compared against: an exact `HashMap` count, columns via the public `hash_for_mat - `merge_max(s, s) == s` - shared keys: `max(a, b) <= cell <= a + b`, and the same for every estimate - model across a rebuilt backend - - after any sequence of insert, merge, `apply_delta`, msgpack round trip, mismatched merge: geometry unchanged; every cell `>= 0`; every row sums to total mass; `truth(k) <= estimate(k) <= band`. `apply_delta == merge`; a mismatched merge is refused and changes nothing + - after any sequence of insert, merge, `apply_delta`, ASAPv1 round trip, mismatched merge: geometry unchanged; every cell `>= 0`; every row sums to total mass; `truth(k) <= estimate(k) <= band`. `apply_delta == merge`; a mismatched merge is refused and changes nothing - wire - - msgpack keeps geometry, cells, estimates + - ASAPv1 keeps geometry, cells, estimates of `f64` counters - ASAPv1 keeps 64 estimates; `edge_rows` x `edge_dimension` ## count_sketch.rs — 12 diff --git a/tests/e2e/frequency.rs b/tests/e2e/frequency.rs index a913f20f..6bd4a2bc 100644 --- a/tests/e2e/frequency.rs +++ b/tests/e2e/frequency.rs @@ -24,7 +24,6 @@ use common::{ use common::{FreqTruth, uniform_u64, zipf_u64}; use std::collections::HashMap; -use asap_sketchlib::message_pack_format::portable::countminsketch::CountMinSketch; use asap_sketchlib::message_pack_format::portable::countsketch::CountSketch; use asap_sketchlib::{ Count, CountL2HH, CountMin, DataInput, DefaultXxHasher, FastPath, FoldCMS, FoldCS, RegularPath, @@ -693,39 +692,29 @@ fn folded_sketches_keep_their_own_bounds_through_a_sixteen_way_merge() { ); } -// ------------------------------------------------------ Portable wire twins +// ------------------------------------------------------- Portable wire twin -/// The portable wire twins answer over string keys and must satisfy the same -/// two theorems as their core counterparts — Count-Min additive, Count Sketch -/// L2 — with no extra slack for being on the wire side. +/// The portable Count Sketch answers over string keys and must satisfy the +/// same L2 theorem as its core counterpart, with no extra slack for being on +/// the wire side. #[test] -fn portable_cms_and_cs_string_keys_satisfy_their_own_bounds() { - const CM_ROWS: usize = 3; +fn portable_cs_string_keys_satisfy_its_own_bound() { const CS_ROWS: usize = 5; const COLS: usize = 4096; const STREAM_SEED: u64 = 1006; let stream = zipf_u64(50_000, 2048, 1.1, STREAM_SEED); - // The portable types key on strings; `FreqTruth` keys on i64. The stream's + // The portable type keys on strings; `FreqTruth` keys on i64. The stream's // u64 values are the identity behind both, so `k -> "k{k}"` is injective // and the two truths agree key for key. let mut truth = FreqTruth::default(); - let mut pcs = CountMinSketch::new(CM_ROWS, COLS); let mut pcss = CountSketch::new(CS_ROWS, COLS); for k in &stream { truth.observe(*k as i64); - let key = format!("k{k}"); - pcs.update(&key, 1.0); - pcss.update(&key, 1.0); + pcss.update(&format!("k{k}"), 1.0); } let context = format!("zipf(1.1) domain=2048 n=50000 string keys, stream_seed={STREAM_SEED}"); - CountMinSpec::new(CM_ROWS, COLS).assert_contract( - "portable CountMinSketch", - &truth, - |k| pcs.estimate(&format!("k{k}")), - &context, - ); CountSketchSpec::new(CS_ROWS, COLS).assert_contract( "portable CountSketch", &truth, diff --git a/tests/msgpack_compat.rs b/tests/msgpack_compat.rs index 267a51c6..45ec3c6a 100644 --- a/tests/msgpack_compat.rs +++ b/tests/msgpack_compat.rs @@ -23,7 +23,6 @@ use std::collections::HashSet; use asap_sketchlib::CmsHeapItem; use asap_sketchlib::message_pack_format::MessagePackCodec; -use asap_sketchlib::message_pack_format::portable::countminsketch::CountMinSketchWire; use asap_sketchlib::message_pack_format::portable::countminsketch_topk::{ CountMinSketchInnerWire, CountMinSketchWithHeapWire, }; @@ -31,24 +30,12 @@ use asap_sketchlib::message_pack_format::portable::delta_set_aggregator::DeltaRe use asap_sketchlib::message_pack_format::portable::hydra_kll::HydraKllSketchWire; use asap_sketchlib::message_pack_format::portable::kll::KllSketchData; use asap_sketchlib::{ - CountMinSketch, CountMinSketchWithHeap, CountSketch, DdSketch, HllSketch, HllVariant, - HydraKllSketch, KllSketch, SetAggregator, + CountMinSketchWithHeap, CountSketch, DdSketch, HllSketch, HllVariant, HydraKllSketch, + KllSketch, SetAggregator, }; // ===== round-trip: every wire-format-aligned type ===== -#[test] -fn count_min_sketch_round_trip() { - let mut s = CountMinSketch::new(3, 64); - s.update("alpha", 1.0); - s.update("beta", 2.0); - s.update("alpha", 4.0); - let bytes = s.to_msgpack().expect("encode"); - let restored = CountMinSketch::from_msgpack(&bytes).expect("decode"); - assert_eq!(restored.rows, 3); - assert_eq!(restored.cols, 64); -} - #[test] fn count_min_sketch_with_heap_round_trip() { let mut s = CountMinSketchWithHeap::new(3, 64, 8); @@ -149,20 +136,6 @@ fn delta_result_round_trip() { // expects (map keys / nesting). A producer that drops a field would // trip these. -#[test] -fn count_min_wire_shape() { - let wire = CountMinSketchWire { - sketch: vec![vec![1.0, 2.0]; 3], - rows: 3, - cols: 2, - }; - let bytes = rmp_serde::to_vec(&wire).unwrap(); - let restored: CountMinSketchWire = rmp_serde::from_slice(&bytes).unwrap(); - assert_eq!(restored.rows, 3); - assert_eq!(restored.cols, 2); - assert_eq!(restored.sketch.len(), 3); -} - #[test] fn count_min_with_heap_wire_shape() { let wire = CountMinSketchWithHeapWire { @@ -229,14 +202,6 @@ fn hydra_kll_wire_shape() { // 2. ::from_msgpack(bytes) succeeds // 3. assert specific field values match the Go producer -#[ignore = "gap: needs a sketchlib-go-produced msgpack fixture; none can be generated in this repo"] -#[test] -fn count_min_decodes_go_bytes() { - // let bytes = include_bytes!("fixtures/msgpack/count_min.msgpack"); - // let s = CountMinSketch::from_msgpack(bytes).unwrap(); - // assert_eq!(s.rows, EXPECTED_ROWS); -} - #[ignore = "gap: needs a sketchlib-go-produced msgpack fixture; none can be generated in this repo"] #[test] fn dd_sketch_decodes_go_bytes() {} diff --git a/tests/pbt/count_min.rs b/tests/pbt/count_min.rs index ef917665..1ab1c894 100644 --- a/tests/pbt/count_min.rs +++ b/tests/pbt/count_min.rs @@ -1,5 +1,4 @@ -//! Property tests for Count-Min: the native matrix sketch and the portable -//! `CountMinSketch` that carries it over the wire. +//! Property tests for Count-Min. //! //! Cormode and Muthukrishnan, J. Algorithms '05. //! @@ -12,17 +11,18 @@ use crate::support::{ PROBE_KEYS, close, edge_dimension, edge_rows, grid, keyed_updates, keys, matrices_close, }; -use asap_sketchlib::message_pack_format::MessagePackCodec; use asap_sketchlib::{ - CountMin, CountMinSketch, CountMinSketchDelta, DataInput, DefaultMatrixI32, FastPath, - MatrixFastHash, MatrixStorage, QuickMatrixI64, RegularPath, Vector2D, hash_for_matrix, + CmDelta, CountMin, DataInput, DefaultMatrixI32, FastPath, MatrixFastHash, MatrixStorage, + QuickMatrixI64, RegularPath, Vector2D, hash_for_matrix, }; use proptest::prelude::*; use std::collections::HashMap; +use std::panic::{AssertUnwindSafe, catch_unwind}; type Cm = CountMin, RegularPath>; type CmFast = CountMin, FastPath>; type CmWire = CountMin, FastPath>; +type CmF64 = CountMin, FastPath>; type CmFixedI32 = CountMin; type CmFixedI64 = CountMin; @@ -70,18 +70,33 @@ fn columns_of(rows: usize, cols: usize, k: u64) -> Vec { (0..rows).map(|r| hashed.col_for_row(r, cols)).collect() } -fn cms_of(rows: usize, cols: usize, updates: &[(String, f64)]) -> CountMinSketch { - let mut s = CountMinSketch::new(rows, cols); +fn cms_of(rows: usize, cols: usize, updates: &[(String, f64)]) -> CmF64 { + let mut s = CmF64::with_dimensions(rows, cols); for (k, v) in updates { - s.update(k, *v); + s.insert_many(&DataInput::Str(k), *v); } s } +/// The counter matrix of `sketch`, one `Vec` per row. +fn matrix_of(sketch: &CmF64) -> Vec> { + (0..sketch.rows()) + .map(|r| { + (0..sketch.cols()) + .map(|c| sketch.as_storage().query_one_counter(r, c)) + .collect() + }) + .collect() +} + +fn str_estimate(sketch: &CmF64, key: &str) -> f64 { + sketch.estimate(&DataInput::Str(key)) +} + // ===== Sequence-law scaffolding ===== /// A transition the sequence law draws from. Every variant is a public entry -/// point of `CountMinSketch`. +/// point of `CountMin`. #[derive(Debug, Clone)] enum Op { Insert(u64, i64), @@ -114,40 +129,34 @@ fn seq_op() -> impl Strategy { ] } -/// `"0"` through `"31"`, the string keys the portable sketch takes. +/// `"0"` through `"31"`, the string keys the sequence law inserts. fn key_domain() -> Vec { (0..DOMAIN).map(|k| k.to_string()).collect() } -fn cms_stream_of( - rows: usize, - cols: usize, - keys: &[String], - stream: &[(u64, i64)], -) -> CountMinSketch { - let mut sketch = CountMinSketch::new(rows, cols); +fn cms_stream_of(rows: usize, cols: usize, keys: &[String], stream: &[(u64, i64)]) -> CmF64 { + let mut sketch = CmF64::with_dimensions(rows, cols); for (k, w) in stream { - sketch.update(&keys[*k as usize], *w as f64); + sketch.insert_many(&DataInput::Str(&keys[*k as usize]), *w as f64); } sketch } -/// Every non-zero cell of `sketch`, as the delta that adds it to a peer. -fn delta_of(sketch: &CountMinSketch) -> CountMinSketchDelta { - let mut cells = Vec::new(); - for (r, row) in sketch.sketch().iter().enumerate() { +/// Every non-zero cell of `sketch`, as the deltas that add it to a peer. +fn deltas_of(sketch: &CmF64) -> Vec { + let mut deltas = Vec::new(); + for (r, row) in matrix_of(sketch).iter().enumerate() { for (c, count) in row.iter().enumerate() { if *count != 0.0 { - cells.push((r as u32, c as u32, *count as i64)); + deltas.push(CmDelta { + row: r as u32, + col: c as u32, + value: *count as u32, + }); } } } - CountMinSketchDelta { - rows: sketch.rows() as u32, - cols: sketch.cols() as u32, - cells, - ..CountMinSketchDelta::default() - } + deltas } /// The column a string key occupies on each row, from the public hash entry @@ -184,7 +193,7 @@ fn cm_fast_of(rows: usize, cols: usize, stream: &[(u64, i32)]) -> CmFast { /// built with, non-negative cells, every row carrying the model's whole mass, /// and each key's estimate between its truth and its lightest row's mass. fn holds_model( - sketch: &CountMinSketch, + sketch: &CmF64, model: &[i64], keys: &[String], columns: &[Vec], @@ -195,7 +204,7 @@ fn holds_model( prop_assert_eq!(sketch.rows(), rows, "{}: rows", what); prop_assert_eq!(sketch.cols(), cols, "{}: cols", what); - let cells = sketch.sketch(); + let cells = matrix_of(sketch); prop_assert_eq!(cells.len(), rows, "{}: rows of the matrix", what); let total: i64 = model.iter().sum(); @@ -228,7 +237,7 @@ fn holds_model( } for (k, own) in model.iter().enumerate() { - let est = sketch.estimate(&keys[k]).round() as i64; + let est = str_estimate(sketch, &keys[k]).round() as i64; let ceiling = (0..rows) .map(|r| column_mass[r * cols + columns[k][r]]) .min() @@ -318,7 +327,7 @@ proptest! { prop_assert_eq!(cm_fast_cells(&by_hashes), cm_fast_cells(&by_loop)); } - // ===== Merge algebra, on the portable representation ===== + // ===== Merge algebra, on f64 counters over string keys ===== #[test] fn count_min_merge_is_commutative( @@ -328,11 +337,11 @@ proptest! { b in keyed_updates(30), ) { let mut ab = cms_of(rows, cols, &a); - ab.merge(&cms_of(rows, cols, &b)).expect("merge"); + ab.merge(&cms_of(rows, cols, &b)); let mut ba = cms_of(rows, cols, &b); - ba.merge(&cms_of(rows, cols, &a)).expect("merge"); + ba.merge(&cms_of(rows, cols, &a)); - prop_assert_eq!(ab.sketch(), ba.sketch()); + prop_assert_eq!(matrix_of(&ab), matrix_of(&ba)); } #[test] @@ -344,15 +353,15 @@ proptest! { c in keyed_updates(20), ) { let mut left = cms_of(rows, cols, &a); - left.merge(&cms_of(rows, cols, &b)).expect("merge"); - left.merge(&cms_of(rows, cols, &c)).expect("merge"); + left.merge(&cms_of(rows, cols, &b)); + left.merge(&cms_of(rows, cols, &c)); let mut right = cms_of(rows, cols, &b); - right.merge(&cms_of(rows, cols, &c)).expect("merge"); + right.merge(&cms_of(rows, cols, &c)); let mut right_full = cms_of(rows, cols, &a); - right_full.merge(&right).expect("merge"); + right_full.merge(&right); - prop_assert!(matrices_close(&left.sketch(), &right_full.sketch())); + prop_assert!(matrices_close(&matrix_of(&left), &matrix_of(&right_full))); } #[test] @@ -363,9 +372,9 @@ proptest! { ) { let base = cms_of(rows, cols, &a); let mut merged = base.clone(); - merged.merge(&CountMinSketch::new(rows, cols)).expect("merge"); + merged.merge(&CmF64::with_dimensions(rows, cols)); - prop_assert_eq!(merged.sketch(), base.sketch()); + prop_assert_eq!(matrix_of(&merged), matrix_of(&base)); } #[test] @@ -376,12 +385,12 @@ proptest! { b in keyed_updates(30), ) { let mut merged = cms_of(rows, cols, &a); - merged.merge(&cms_of(rows, cols, &b)).expect("merge"); + merged.merge(&cms_of(rows, cols, &b)); let concatenated: Vec<_> = a.iter().chain(b.iter()).cloned().collect(); let streamed = cms_of(rows, cols, &concatenated); - prop_assert!(matrices_close(&merged.sketch(), &streamed.sketch())); + prop_assert!(matrices_close(&matrix_of(&merged), &matrix_of(&streamed))); } #[test] @@ -393,7 +402,7 @@ proptest! { let s = cms_of(rows, cols, &updates); for (k, _) in &updates { let truth: f64 = updates.iter().filter(|(x, _)| x == k).map(|(_, v)| v).sum(); - let est = s.estimate(k); + let est = str_estimate(&s, k); prop_assert!( est >= truth || close(est, truth), "key {}: estimate {} < truth {}", k, est, truth @@ -411,14 +420,14 @@ proptest! { ) { let s = cms_of(rows, cols, &updates); - let bytes = s.to_msgpack().expect("encode"); - let restored = CountMinSketch::from_msgpack(&bytes).expect("decode"); + let bytes = s.serialize_to_bytes().expect("encode"); + let restored = CmF64::deserialize_from_bytes(&bytes).expect("decode"); prop_assert_eq!(restored.rows(), s.rows()); prop_assert_eq!(restored.cols(), s.cols()); - prop_assert_eq!(restored.sketch(), s.sketch()); + prop_assert_eq!(matrix_of(&restored), matrix_of(&s)); for (k, _) in &updates { - prop_assert_eq!(restored.estimate(k), s.estimate(k), "key {}", k); + prop_assert_eq!(str_estimate(&restored, k), str_estimate(&s, k), "key {}", k); } } @@ -527,7 +536,7 @@ proptest! { let left = cms_of(rows, cols, &a); let right = cms_of(rows, cols, &b); let mut merged = left.clone(); - merged.merge(&right).expect("merge"); + merged.merge(&right); let probes = a .iter() @@ -535,14 +544,15 @@ proptest! { .map(|(k, _)| k.as_str()) .chain(["0", "!"]); for k in probes { - let after = merged.estimate(k); + let after = str_estimate(&merged, k); + let (mine, theirs) = (str_estimate(&left, k), str_estimate(&right, k)); prop_assert!( - after >= left.estimate(k), - "key {}: merged {} below the left estimate {}", k, after, left.estimate(k) + after >= mine, + "key {}: merged {} below the left estimate {}", k, after, mine ); prop_assert!( - after >= right.estimate(k), - "key {}: merged {} below the right estimate {}", k, after, right.estimate(k) + after >= theirs, + "key {}: merged {} below the right estimate {}", k, after, theirs ); } } @@ -600,7 +610,7 @@ proptest! { // ===== A state that survives being rebuilt ===== // - // Merge, apply_delta and a msgpack round trip each replace the backend + // Merge, apply_delta and an ASAPv1 round trip each rewrite the matrix // mid-stream. The model is the exact ledger of everything inserted so far, // and every invariant is re-checked after every transition. @@ -616,18 +626,18 @@ proptest! { .map(|k| str_columns_of(rows, cols, k)) .collect(); let mut model = vec![0i64; keys.len()]; - let mut sketch = CountMinSketch::new(rows, cols); + let mut sketch = CmF64::with_dimensions(rows, cols); holds_model(&sketch, &model, &keys, &columns, (rows, cols), "before the first op")?; for (step, op) in ops.iter().enumerate() { match op { Op::Insert(k, w) => { - sketch.update(&keys[*k as usize], *w as f64); + sketch.insert_many(&DataInput::Str(&keys[*k as usize]), *w as f64); model[*k as usize] += *w; } Op::Merge(stream) => { let other = cms_stream_of(rows, cols, &keys, stream); - sketch.merge(&other).expect("a peer of the same geometry merges"); + sketch.merge(&other); for (k, w) in stream { model[*k as usize] += *w; } @@ -635,37 +645,39 @@ proptest! { Op::ApplyDelta(stream) => { let other = cms_stream_of(rows, cols, &keys, stream); let mut merged = sketch.clone(); - merged.merge(&other).expect("a peer of the same geometry merges"); + merged.merge(&other); - sketch.apply_delta(&delta_of(&other)).expect("an in-range delta applies"); + for delta in deltas_of(&other) { + sketch.apply_delta(delta); + } for (k, w) in stream { model[*k as usize] += *w; } prop_assert_eq!( - sketch.sketch(), merged.sketch(), + matrix_of(&sketch), matrix_of(&merged), "step {}: apply_delta and the merge it stands for left different matrices", step ); } Op::RoundTrip => { - let bytes = sketch.to_msgpack().expect("encode"); - let restored = CountMinSketch::from_msgpack(&bytes).expect("decode"); + let bytes = sketch.serialize_to_bytes().expect("encode"); + let restored = CmF64::deserialize_from_bytes(&bytes).expect("decode"); prop_assert_eq!( - restored.sketch(), sketch.sketch(), + matrix_of(&restored), matrix_of(&sketch), "step {}: the decoded matrix differs", step ); sketch = restored; } Op::MergeMismatched(over_rows, over_cols) => { - let before = sketch.sketch(); - let peer = CountMinSketch::new(rows + over_rows, cols + over_cols); + let before = matrix_of(&sketch); + let peer = CmF64::with_dimensions(rows + over_rows, cols + over_cols); prop_assert!( - sketch.merge(&peer).is_err(), + catch_unwind(AssertUnwindSafe(|| sketch.merge(&peer))).is_err(), "step {}: merging a {}x{} peer into a {}x{} sketch was accepted", step, peer.rows(), peer.cols(), rows, cols ); prop_assert_eq!( - sketch.sketch(), before, + matrix_of(&sketch), before, "step {}: a refused merge still changed the matrix", step ); } diff --git a/tests/sketches_go_parity_probe.rs b/tests/sketches_go_parity_probe.rs index 2997e486..98f78683 100644 --- a/tests/sketches_go_parity_probe.rs +++ b/tests/sketches_go_parity_probe.rs @@ -4,18 +4,16 @@ //! //! Approach: reuse the exact same Go-golden envelopes that the //! `message_pack_format::portable` parity tests use, but build the -//! matrix via `sketches::Count` / `sketches::CountMin` with `FastPath` -//! and `DefaultXxHasher`. Identical bytes confirm the shared FastPath +//! matrix via `sketches::Count` with `FastPath` and `DefaultXxHasher`. Identical bytes confirm the shared FastPath //! math is sufficient — no hashspec bypass needed. use asap_sketchlib::common::DataInput; use asap_sketchlib::common::hash::CANONICAL_HASH_SEED; use asap_sketchlib::common::structures::HllBucketListP14; use asap_sketchlib::proto::sketchlib::{ - CountMinState, CountSketchState, CounterType, DdSketchState, HyperLogLogState, SketchEnvelope, + CountSketchState, CounterType, DdSketchState, HyperLogLogState, SketchEnvelope, sketch_envelope::SketchState, }; -use asap_sketchlib::sketches::countminsketch::CountMin; use asap_sketchlib::sketches::countsketch::Count; use asap_sketchlib::sketches::ddsketch::DDSketch; use asap_sketchlib::sketches::hll::{ErtlMLE, HyperLogLogImpl}; @@ -28,12 +26,6 @@ use prost::Message; /// `src/message_pack_format/portable/countsketch.rs::test_update_then_envelope_matches_sketchlib_go_bytes`. const COUNTSKETCH_GOLDEN_HEX: &str = include_str!("cs_envelope_golden.hex"); -/// 8275-byte CMS Frequency-Only golden, captured from -/// `sketchlib-go::CountMinSketch.SerializeProtoBytesFO` for -/// `(rows=4, cols=2048)` × `goldenCmsKeys()` (10 keys "flow-0".."flow-9", -/// each repeated 5×). Same file the wrapper test already uses. -const COUNTMIN_GOLDEN_HEX: &str = include_str!("../src/sketches/testdata/cms_envelope_golden.hex"); - /// 16 398-byte envelope captured from `sketchlib-go::HyperLogLog.SerializePortable` /// for precision=14, `(1..=50)` IEEE-754-LE byte-key input. Same file /// the wrapper test already uses. @@ -103,70 +95,6 @@ fn sketches_count_fastpath_matches_go_count_sketch_envelope() { ); } -#[test] -fn sketches_countmin_fastpath_matches_go_count_min_envelope() { - let rows = 4usize; - let cols = 2048usize; - - let mut sk: CountMin, FastPath, DefaultXxHasher> = - CountMin::with_dimensions(rows, cols); - for i in 0..50u64 { - let key = format!("flow-{}", i % 10); - sk.insert_many(&DataInput::String(key), 1.0f64); - } - - let storage = sk.as_storage(); - let mut counts_int: Vec = Vec::with_capacity(rows * cols); - let mut l1: Vec = Vec::with_capacity(rows); - let mut l2: Vec = Vec::with_capacity(rows); - for r in 0..rows { - let mut row_l1 = 0.0f64; - let mut row_l2 = 0.0f64; - for c in 0..cols { - let cell = *storage.get(r, c).expect("cell in range"); - counts_int.push(cell as i64); - row_l1 += cell; - row_l2 += cell * cell; - } - l1.push(row_l1); - l2.push(row_l2); - } - - let state = CountMinState { - rows: rows as u32, - cols: cols as u32, - counter_type: CounterType::Int64 as i32, - counts_int, - counts_float: Vec::new(), - sum_counts: Vec::new(), - sum2_counts: Vec::new(), - l1, - l2, - }; - let envelope = SketchEnvelope { - format_version: 1, - producer: None, - hash_spec: None, - sample_p: 0.0, - sketch_state: Some(SketchState::CountMin(state)), - }; - let mut got = Vec::with_capacity(envelope.encoded_len()); - envelope.encode(&mut got).expect("prost encode"); - - let want = decode_hex(COUNTMIN_GOLDEN_HEX); - assert_eq!( - got.len(), - want.len(), - "CountMin envelope length: got {} want {}", - got.len(), - want.len(), - ); - assert_eq!( - got, want, - "sketches::CountMin, FastPath> envelope diverges from Go golden", - ); -} - #[test] fn sketches_hll_classic_matches_go_envelope() { // Go's `HyperLogLog.SerializePortable` emits variant=DATAFUSION=2 diff --git a/tests/xtest_consumer.rs b/tests/xtest_consumer.rs index 0113ff45..d7909ea5 100644 --- a/tests/xtest_consumer.rs +++ b/tests/xtest_consumer.rs @@ -5,7 +5,6 @@ //! to confirm the data survived the language boundary intact. //! //! Files consumed (from $XTEST_DIR/): -//! countmin.pb — CountMinState with float64 counters //! kll.pb — KLLState with quantile items and coin RNG state //! ddsketch.pb — DDSketchState with alpha + bucket array //! hll.pb — HyperLogLogState (ErtlMLE variant) @@ -46,77 +45,6 @@ fn cross_language_proto() { let mut all_ok = true; - // ----------------------------------------------------------------------- - // CountMin - // ----------------------------------------------------------------------- - println!(); - println!("[CountMin] Step 1/3 — Read countmin.pb"); - let bytes = read_file(in_dir.join("countmin.pb")); - let env = SketchEnvelope::decode(bytes.as_slice()).expect("decode countmin envelope"); - - println!( - "[CountMin] Step 2/3 — Validate envelope (format_version={}, producer={})", - env.format_version, - env.producer.as_ref().map_or("?", |p| &p.library) - ); - - let cm_state = match env.sketch_state { - Some(sketch_envelope::SketchState::CountMin(ref s)) => s.clone(), - other => panic!("expected CountMin sketch_state, got {other:?}"), - }; - - println!( - "[CountMin] rows={} cols={} counter_type={:?}", - cm_state.rows, - cm_state.cols, - CounterType::try_from(cm_state.counter_type) - ); - - // Reconstruct the row-major matrix as f64. The producer populates EITHER - // counts_int (counter_type ∈ {INT32,INT64}) OR counts_float (FLOAT64), - // discriminated by counter_type — read whichever is present. - let rows = cm_state.rows as usize; - let cols = cm_state.cols as usize; - let counts: Vec = if !cm_state.counts_float.is_empty() { - cm_state.counts_float.clone() - } else { - cm_state.counts_int.iter().map(|&v| v as f64).collect() - }; - let counts = &counts; - if counts.len() != rows * cols { - eprintln!( - "[CountMin] FAIL: counts length {} != rows*cols {}", - counts.len(), - rows * cols - ); - all_ok = false; - } - - // Point query for hot key "item:42" using the same hash formula as Go. - let hot_key = b"item:42" as &[u8]; - let hot_hash = xxh3_64_seeded(SEED_0, hot_key); - - println!("[CountMin] Step 3/3 — Point query 'item:42' (hash=0x{hot_hash:016x})"); - - let bits_per_row = col_bits(cols); - let mask = (cols as u64) - 1; - let mut min_freq = f64::MAX; - for r in 0..rows { - let shift = (r as u64) * bits_per_row; - let c = ((hot_hash >> shift) & mask) as usize; - let v = counts[r * cols + c]; - if v < min_freq { - min_freq = v; - } - } - println!("[CountMin] min frequency estimate = {min_freq:.0} (expect ≥ 101)"); - if min_freq < 101.0 { - eprintln!("[CountMin] FAIL: frequency {min_freq:.0} < 101"); - all_ok = false; - } else { - println!("[CountMin] PASS"); - } - // ----------------------------------------------------------------------- // KLL // ----------------------------------------------------------------------- @@ -436,61 +364,6 @@ fn cross_language_proto() { all_ok = false; } - // ----------------------------------------------------------------------- - // Sampled CountMin (geometric skip-sampling, p=0.1) — exercises the - // sample_p envelope field + the backend ×1/p rescale at query. - // ----------------------------------------------------------------------- - let sampled_cms_path = in_dir.join("countmin_sampled.pb"); - if sampled_cms_path.exists() { - println!(); - println!("[CountMin/sampled] Step 1/3 — Read countmin_sampled.pb"); - let bytes = read_file(&sampled_cms_path); - let env = SketchEnvelope::decode(bytes.as_slice()).expect("decode sampled countmin"); - - // Dual-read: 0.0/unset means 1.0; a sampled producer stamps the real p. - let p = effective_sample_p(&env); - println!("[CountMin/sampled] Step 2/3 — envelope sample_p = {p}"); - if (p - 0.1).abs() > 1e-9 { - eprintln!("[CountMin/sampled] FAIL: sample_p {p} != 0.1"); - all_ok = false; - } - - let cm_state = match env.sketch_state { - Some(sketch_envelope::SketchState::CountMin(ref s)) => s.clone(), - other => panic!("expected CountMin sketch_state, got {other:?}"), - }; - let rows = cm_state.rows as usize; - let cols = cm_state.cols as usize; - let counts: Vec = if !cm_state.counts_float.is_empty() { - cm_state.counts_float.clone() - } else { - cm_state.counts_int.iter().map(|&v| v as f64).collect() - }; - - // Raw min-frequency point query for "item:hot". - let hot_hash = xxh3_64_seeded(SEED_0, b"item:hot"); - let bits_per_row = col_bits(cols); - let mask = (cols as u64) - 1; - let mut raw = f64::MAX; - for r in 0..rows { - let shift = (r as u64) * bits_per_row; - let c = ((hot_hash >> shift) & mask) as usize; - raw = raw.min(counts[r * cols + c]); - } - // Backend rescale: ×1/p recovers the full-stream frequency. - let rescaled = rescale_count(raw, p); - let rel_err = (rescaled - 100_000.0).abs() / 100_000.0; - println!( - "[CountMin/sampled] Step 3/3 — raw={raw:.0} rescaled(×1/p)={rescaled:.0} truth=100000 relErr={rel_err:.4}" - ); - if rel_err <= 0.05 { - println!("[CountMin/sampled] PASS"); - } else { - eprintln!("[CountMin/sampled] FAIL: rescaled rel err {rel_err:.4} > 0.05"); - all_ok = false; - } - } - // ----------------------------------------------------------------------- // Sampled HLL (hash-threshold sampling, p=0.1) — cardinality ×1/p rescale. // ----------------------------------------------------------------------- From 8684e42ad9d70137598f48b6ab65bbf97ea8871b Mon Sep 17 00:00:00 2001 From: GordonYuanyc Date: Fri, 2 Oct 2026 00:44:33 -0600 Subject: [PATCH 04/36] refactor(hll)!: drop the proto and portable formats HyperLogLog serializes only as ASAPv1. Remove the portable HllSketch, HllSketchDelta and HllVariant with the sparse-register helpers, the native MessagePackCodec shims, the HLLDelta proto message and the SketchEnvelope hll field (12, now reserved). HyperLogLogState stays for HydraCell. Add HyperLogLogImpl::from_storage and HyperLogLogHIPImpl::from_storage, and check the three HLL ASAPv1 goldens through the native types. Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 19 +- docs/api/api_hyperloglog.md | 2 + docs/asapv1_wire_format.md | 2 +- docs/features.md | 2 +- docs/library_map.md | 2 +- docs/message_pack_format.md | 2 +- docs/tests.md | 45 +- examples/accuracy_probe.rs | 16 - proto/hll/hll.proto | 37 +- proto/sketchlib.proto | 5 +- src/lib.rs | 3 +- src/message_pack_format/native/hll.rs | 27 - src/message_pack_format/native/mod.rs | 1 - src/message_pack_format/portable/hll.rs | 1056 ----------------- src/message_pack_format/portable/mod.rs | 3 +- src/message_pack_format/portable/sampling.rs | 11 +- src/proto/generated/sketchlib.v1.rs | 36 +- src/sketches/hll.rs | 23 +- src/sketches/hll/wire.rs | 66 +- src/sketches/testdata/hll_envelope_golden.hex | 1 - tests/PBT_COVERAGE.md | 12 +- tests/asapv1_golden.rs | 66 +- tests/conformance_kit.rs | 21 - tests/e2e/cardinality.rs | 72 -- tests/msgpack_compat.rs | 19 +- tests/pbt/hll.rs | 128 +- tests/proptest-regressions/hll.txt | 7 + tests/sketches_go_parity_probe.rs | 61 +- tests/xtest_consumer.rs | 150 --- 29 files changed, 134 insertions(+), 1761 deletions(-) delete mode 100644 src/message_pack_format/native/hll.rs delete mode 100644 src/message_pack_format/portable/hll.rs delete mode 100644 src/sketches/testdata/hll_envelope_golden.hex create mode 100644 tests/proptest-regressions/hll.txt diff --git a/CHANGELOG.md b/CHANGELOG.md index 80d85f98..5266204e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -49,7 +49,7 @@ signals a backwards-compatible change. law that sketch claims — merge algebra, path equivalence, the paper's own model, and the wire / ASAPv1 round trip. Shared generators, the `grid` reader and the `round_trip!` macro live in `tests/pbt/support.rs`; four - modules sit behind `experimental`. 435 tests under `--all-features`, 367 + modules sit behind `experimental`. 433 tests under `--all-features`, 365 under the default feature set. Every law was mutation-checked: the implementation was broken deliberately, the law confirmed red, and confirmed not to take unrelated laws down with it; a law no semantically real mutation @@ -66,6 +66,9 @@ signals a backwards-compatible change. worker's seed and the aggregator's from one seed. The unseeded constructors draw from the thread generator. The seed is not serialized, and the ASAPv1 payload is unchanged. +- **HyperLogLog `from_storage` constructors.** `HyperLogLogImpl::from_storage(registers)` + and `HyperLogLogHIPImpl::from_storage(registers, kxq0, kxq1, est)` build a + sketch from known register storage (and HIP running state). ### Changed @@ -88,6 +91,20 @@ signals a backwards-compatible change. as an empty window like any other, and that window takes a place in `max_windows`. +### Removed + +- **HyperLogLog's protobuf and portable formats.** HyperLogLog serializes only + as ASAPv1 (`serialize_to_bytes` / `deserialize_from_bytes`). Removed: the + portable `HllSketch`, `HllSketchDelta` and `HllVariant` (module + `message_pack_format::portable::hll` and their crate-root re-exports), with + `registers_from_state`, `decode_sparse_registers` and + `encode_sparse_registers`; the `MessagePackCodec` impls for + `HyperLogLogImpl` and `HyperLogLogHIPImpl` (`message_pack_format::native::hll`); + the `HLLDelta` proto message (`proto::sketchlib::HllDelta`); and the `hll` + field (12) of `SketchEnvelope`, now reserved. `HyperLogLogState`, + `HLLSparseRegisters` and `HLLVariant` remain only as the `HydraCell` HLL + state. + ### Fixed - `KLLDynamic` panicked on every quantile query once a NaN had entered the diff --git a/docs/api/api_hyperloglog.md b/docs/api/api_hyperloglog.md index 4e2752bf..aaf3d482 100644 --- a/docs/api/api_hyperloglog.md +++ b/docs/api/api_hyperloglog.md @@ -40,6 +40,8 @@ but improves accuracy. ```rust fn new() -> Self fn default() -> Self +fn from_storage(registers: Registers) -> Self // HyperLogLogImpl +fn from_storage(registers: Registers, kxq0: f64, kxq1: f64, est: f64) -> Self // HyperLogLogHIPImpl ``` ## Insert / Update diff --git a/docs/asapv1_wire_format.md b/docs/asapv1_wire_format.md index fd14e70a..e4540fbe 100644 --- a/docs/asapv1_wire_format.md +++ b/docs/asapv1_wire_format.md @@ -1348,7 +1348,7 @@ Rust derives the hash spec from a generic `HashProfile` bound on the hasher type On the Go side the profile is simply **written into** the metadata on encode and **read from** it on decode. Go MUST validate the profile it reads (same fail-closed intent as Rust): a sketch is only mergeable/queryable if its `hash_profile_id` + seeds match the profile Go is prepared to reproduce. -Sequencing: (2) covers six `kind_id`s, and the one `native bytes == portable bytes` test is HLL-only, since no other `portable` type emits ASAPv1 bytes. +Sequencing: (2) covers six `kind_id`s. Retire `portable` once the fixtures cover the rest. --- diff --git a/docs/features.md b/docs/features.md index 64f296a3..9f0a5b65 100644 --- a/docs/features.md +++ b/docs/features.md @@ -112,7 +112,7 @@ MessagePack (`rmp-serde`) support. **serde support** means the type derives `Ser | EHSketchList | Yes | Yes | | ExponentialHistogram | No | Yes | -Protobuf (prost): `.proto` definitions exist for CountMin, Count, HLL, DDSketch, KLL, Elastic, Coco, Hydra, and UnivMon. Rust conversion code is in progress. +Protobuf (prost): `.proto` definitions exist for CountMin, Count, DDSketch, KLL, Elastic, Coco, Hydra, and UnivMon. Rust conversion code is in progress. ### API Stability diff --git a/docs/library_map.md b/docs/library_map.md index a8e614b3..48dc23e6 100644 --- a/docs/library_map.md +++ b/docs/library_map.md @@ -26,7 +26,7 @@ - **`src/message_pack_format/`** - Serialization plumbing ([message_pack_format.md](./message_pack_format.md)). The current format is **ASAPv1**, specified in [asapv1_wire_format.md](./asapv1_wire_format.md) - `envelope.rs` — the shared, sketch-agnostic ASAPv1 framing (magic/version/`kind_id` + length prefixes, `encode`/`split`); every `wire.rs` under `src/sketches/` and `src/sketch_framework/` calls into it - `codec.rs` — the `MessagePackCodec` trait; `error.rs` — the unified `Error` - - `portable/` — **deprecated**, being retired. The older per-sketch wire types (`CountMinSketch`, `HllSketch`, …); ASAPv1 (per-sketch `wire.rs`) is now what `sketchlib-go` mirrors, not these + - `portable/` — **deprecated**, being retired. The older per-sketch wire types (`CountMinSketch`, `KllSketch`, …); ASAPv1 (per-sketch `wire.rs`) is now what `sketchlib-go` mirrors, not these - `native/` — **deprecated**, being retired. Older `MessagePackCodec` shims over `src/sketches/` byte serialization ## Documentation diff --git a/docs/message_pack_format.md b/docs/message_pack_format.md index 12a2b91c..3cecb28a 100644 --- a/docs/message_pack_format.md +++ b/docs/message_pack_format.md @@ -150,7 +150,7 @@ being **phased out** in favor of the per-sketch `wire.rs` + the shared `envelope.rs`. - `portable/` holds per-sketch wire types that predate the envelope. They - carry their own byte-parity goldens against `sketchlib-go` — HLL, Count-Min, + carry their own byte-parity goldens against `sketchlib-go` — Count-Min, Count Sketch and KLL (both directions) — separate from `asapv1_golden/`. - `native/` is a set of thin `MessagePackCodec` shims over the sketches' `serialize_to_bytes` / `deserialize_from_bytes`. Every type it wraps emits the diff --git a/docs/tests.md b/docs/tests.md index ccb1571c..66513dab 100644 --- a/docs/tests.md +++ b/docs/tests.md @@ -110,7 +110,6 @@ Wire tests: [`src/sketches/hll/wire.rs`](../src/sketches/hll/wire.rs) | `hll_correctness_test` | Register update logic matches expected bucket/index behavior for all HLL variants. | Runs fixed hashed inserts against Classic, ErtlMLE, and HIP variants; asserts exact expected register values at specific bucket indices and confirms an untouched bucket remains zero. | | `hll_envelope_structure_and_kind_id_guard` | The envelope frames an Ertl-MLE sketch under kind_id `0x01 0x02`, and a Classic decoder refuses it. | For 1,000 inserts into `HyperLogLog`, verifies the bytes open with the ASAPv1 magic, `envelope::VERSION`, a `kind_id_len` of `2`, and `0x01 0x02`, that the decoded registers match the source, and that a `HyperLogLog` decode fails. | | `hll_hip_round_trip_preserves_state` | The HIP running scalars travel beside the registers. | For 1,000 inserts into `HyperLogLogHIP`, verifies the bytes carry kind_id `0x01 0x03` and that the decoded sketch matches the source on the registers and on `kxq0`, `kxq1`, and `est`. | -| `native_and_portable_hll_bytes_match` | The native and portable encoders emit the same envelope. | For 1,000 inserts each, verifies an `ErtlMLE` sketch's bytes equal the portable `HllSketch` `Datafusion` encoding of its registers, and a `HyperLogLogHIP` sketch's bytes equal the portable `Hip` encoding of its registers plus `kxq0`, `kxq1`, and `est`. | | `hll_custom_hasher_profile_round_trips_and_is_self_describing` | The metadata describes the hasher that built the sketch rather than a hardcoded profile. | For a P14 `ErtlMLE` sketch over a hasher declaring its own `HashProfile`, verifies the registers round-trip, the bytes differ from the standard-profile sketch's over the same 1,000 inserts, and a standard-profile decode rejects them. | | `hll_metadata_rejects_unknown_keys` | An unexpected metadata key fails closed. | Encodes the eight `HllMetadata` fields plus a `bogus_field` as a named map and verifies it does not decode as `HllMetadata`. | | `hll_precision_cross_rejection` | The metadata `precision` pins the register storage the bytes belong to. | Verifies a populated `HyperLogLogP12` envelope fails to decode as the P14 `HyperLogLog`. | @@ -1327,36 +1326,6 @@ Test file: [`src/message_pack_format/portable/delta_set_aggregator.rs`](../src/m | `test_msgpack_round_trip` | Both key sets survive the round trip. | Round-trips a `DeltaResult` with `added` `{"web", "api"}` and `removed` `{"db"}` and verifies the decode holds 2 added keys including both names and 1 removed key. | | `test_empty_sets` | A `DeltaResult` holding nothing round-trips. | Verifies a `DeltaResult` with both sets empty encodes and decodes back with both sets still empty. | -### Portable HllSketch - -Test file: [`src/message_pack_format/portable/hll.rs`](../src/message_pack_format/portable/hll.rs) - -| test_name | test_description | what_is_tested | -| --- | --- | --- | -| `estimate_holds_past_the_32_bit_range` | The estimator follows the HyperLogLog formula rather than saturating. | For each rank in `12`, `16`, `18`, and `20` filled across all 16,384 registers of a precision-14 sketch, verifies `estimate()` is within a relative `1e-9` of `alpha_m * m * 2^rank` with `alpha_m = 0.7213 / (1 + 1.079 / m)` and `m = 16384`. | -| `msgpack_delta_against_empty_round_trips` | A msgpack register delta against an empty snapshot carries every register. | For a precision-12 sketch fed the little-endian bytes of `0..2000u64`, verifies the delta against an empty sketch at threshold `1`, applied to a fresh sketch, reproduces the register array exactly and the estimate to within 0.1%. | -| `msgpack_delta_wire_layout_is_parallel_arrays` | The register delta is two parallel arrays of index and value. | For a precision-4 sketch fed `b"a"` and `b"b"`, verifies the delta bytes equal `rmp_serde::to_vec(&(reg_idx: Vec, reg_val: Vec))` recomputed from the non-zero registers, and that decoding yields one update per non-zero register. | -| `test_new_empty` | A fresh sketch is `2^precision` zeroed registers. | For `HllSketch::new(Regular, 4)`, verifies the register array is 16 long and every register is `0`. | -| `test_merge_register_wise_max` | Merge takes the register-wise maximum. | Merges the precision-2 registers `[4, 2, 6, 0]` into `[1, 5, 3, 7]` and verifies the result is `[4, 5, 6, 7]`. | -| `test_apply_delta_max_semantics` | A delta raises registers to the maximum, never lowers them. | Applies the updates `(0,4)`, `(1,2)`, `(2,6)`, `(3,0)` to the precision-2 registers `[1, 5, 3, 7]` and verifies the result is `[4, 5, 6, 7]`. | -| `test_apply_delta_out_of_range` | A register index past the array is refused. | Verifies a delta carrying update `(7, 3)` against a precision-2 (four-register) sketch fails to apply. | -| `test_apply_delta_matches_full_merge` | Applying a delta lands where merging the same addition lands. | For the precision-2 base `[1, 5, 3, 7]`, verifies merging the addition `[4, 0, 6, 0]` and applying the delta updates `(0,4)` and `(2,6)` give the same register array. | -| `test_merge_variant_mismatch` | Two estimator variants are not mergeable. | Verifies merging a `Datafusion` precision-4 sketch into a `Regular` precision-4 one fails. | -| `test_merge_precision_mismatch` | Two precisions are not mergeable. | Verifies merging a precision-5 `Regular` sketch into a precision-4 one fails. | -| `test_merge_refs` | `merge_refs` takes the maximum across every input. | Merges three precision-1 sketches holding `[1, 0]`, `[0, 3]`, and `[2, 2]` and verifies the result is `[2, 3]`. | -| `test_update_then_estimate_within_2pct` | The estimate lands within 2% on a 50,000-key stream. | Feeds `key-0`..`key-49999` into a precision-12 `Regular` sketch and verifies the relative error of `estimate()` against `50_000` is below `0.02`. | -| `test_estimate_empty_is_zero` | An empty sketch estimates zero. | Verifies a precision-4 `Regular` sketch reports an `estimate()` of exactly `0.0`. | -| `test_msgpack_round_trip` | The encoding is an ASAPv1 envelope and the HIP state survives it. | Round-trips a `Hip` precision-3 sketch holding registers `[0, 1, 2, 3, 4, 5, 6, 7]` with `hip_kxq0` `1.0`, `hip_kxq1` `2.0`, and `hip_est` `3.0`, and verifies the bytes open with the `ASAPv1` magic and that the decode matches on registers, precision, and a `hip_kxq0` of `1.0`. | -| `test_update_then_envelope_matches_sketchlib_go_bytes` | The proto envelope for a populated register array is byte-identical to the checked-in golden. | For a `Datafusion` precision-14 sketch fed the little-endian `f64` bytes of `1..=50`, builds a `HyperLogLogState` from `wire_proto_variant()`, the precision, and the registers with `registers_sparse` unset, wraps it in a `SketchEnvelope` with `format_version` `1` and `sample_p` `0.0`, and verifies the prost encoding matches the 16,398-byte [`hll_envelope_golden.hex`](../src/sketches/testdata/hll_envelope_golden.hex) on both length and content. | -| `sparse_round_trip_exact` | The sparse register encoding is exact. | For a 16,384-register array holding `1` at index `0`, `7` at `1`, `51` at `300`, `12` at `8191`, and `3` at `16383`, verifies `encode_sparse_registers` reports `num_registers` `16384` and that the decode returns the array unchanged. | -| `registers_from_state_reads_both` | `registers_from_state` reads the sparse and the dense form alike. | For the same 16,384-register array at `variant` `2` and `precision` `14`, verifies the registers come back identical whether the state carries `registers_sparse` with `registers` empty or `registers` with `registers_sparse` unset. | -| `sparse_empty_is_all_zero` | An all-zero register array encodes to nothing. | For 16,384 zeroed registers, verifies the sparse `packed` field is empty and the decode returns the zeros. | -| `sparse_rejects_out_of_range_index` | A sparse index past `num_registers` is refused. | Verifies `HllSparseRegisters { num_registers: 4, packed: [5, 3] }`, whose first uvarint delta of `5` lands outside a four-register array, fails to decode. | -| `test_compute_delta_against_empty_round_trips` | A proto register delta against an empty snapshot carries every register. | For a `Datafusion` precision-14 sketch fed the little-endian bytes of `0..5000u64`, verifies the delta against an empty sketch at threshold `0`, applied to a fresh sketch, reproduces the register array. | -| `test_compute_delta_then_apply_matches_current` | A delta between two non-empty snapshots reconstructs the later one. | For a `Datafusion` precision-12 base fed `0..2000u64` and a current fed `2000..5000u64` on top, verifies the delta applied to the base reproduces the current register array. | -| `test_compute_delta_matches_go_golden_bytes` | The packed register delta bytes are pinned by a golden. | For a `Datafusion` precision-14 sketch fed the little-endian `f64` bytes of `1..=50`, verifies the delta against an empty sketch at threshold `0` equals a 132-byte inline golden in the packed `(index_delta, value)` form. | -| `test_compute_subwindow_delta_matches_go_golden_bytes` | A sub-window delta's bytes are pinned by a golden and still reconstruct the window. | For a `Datafusion` precision-14 base fed `0..2000` as `f64` little-endian bytes and a current fed `2000..5000` on top, verifies the delta carries only the registers that grew and equals a 5,097-byte inline golden, then that applying it to the base reproduces the current register array. | - ### Portable HydraKllSketch Test file: [`src/message_pack_format/portable/hydra_kll.rs`](../src/message_pack_format/portable/hydra_kll.rs) @@ -1632,7 +1601,6 @@ Test file: [`tests/e2e/cardinality.rs`](../tests/e2e/cardinality.rs) | --- | --- | --- | | `hll_{classic,ertl_mle,hip}_p{12,14,16}_satisfies_..._error_model` | Each core HLL alias tracks the true cardinality inside its own estimator's `z = 4` band, ignores replayed identities, and reproduces a single pass from disjoint shards. | One `hll_battery!` body per invocation, over `CHECKPOINTS = [10, 100, 1_000, 10_000, 100_000, 1_000_000, 10_000_000]`. A fresh sketch per checkpoint is fed `DataInput::U64(base + k)` from its own namespace `base = (1 << 40) * (i + 1)`, and each `estimate()` goes through `CardinalityConfidenceSpec::check` at `z = 4.0`: relative band `4 * 1.04/sqrt(m)` for the `hll` model, `4 * sqrt(ln 2 / m)` for `hll_hip`. Then re-inserting the first `min(target, 200_000)` already-seen identities must leave `estimate()` bit-identical (`assert_eq!`). Then over a fresh `200_000`-identity namespace, an even/odd shard merge must give `registers_as_slice()` and `estimate()` equal to the single pass. Invocations: `hll_classic_p12_satisfies_its_register_error_model` (`HyperLogLogP12`, `hll`, mergeable), `hll_classic_p14_satisfies_its_register_error_model` (`HyperLogLogP14`), `hll_classic_p16_satisfies_its_register_error_model` (`HyperLogLogP16`), `hll_ertl_mle_p12_satisfies_the_cramer_rao_error_model` (`HyperLogLogP12`), `hll_ertl_mle_p14_satisfies_the_cramer_rao_error_model` (`HyperLogLogP14`), `hll_ertl_mle_p16_satisfies_the_cramer_rao_error_model` (`HyperLogLogP16`), `hll_hip_p12_satisfies_the_hip_error_model` (`HyperLogLogHIPP12`, `hll_hip`, not_mergeable), `hll_hip_p14_satisfies_the_hip_error_model` (`HyperLogLogHIPP14`), `hll_hip_p16_satisfies_the_hip_error_model` (`HyperLogLogHIPP16`). The three `not_mergeable` HIP invocations still build and fill the three `200_000`-identity shard sketches but assert nothing about them. | | `hll_{classic,ertl_mle,hip}_custom_p{10..18}_satisfies_..._error_model` | Every custom register geometry from p10 to p18 holds its estimator's `z = 4` band over its own checkpoint grid, is unmoved by a replay, and merges shard-for-shard. | The same `hll_battery!` body as the core aliases (fresh sketch per checkpoint on namespace `(1 << 40) * (i + 1)`, `CardinalityConfidenceSpec::check` at `z = 4.0`, replay of `min(target, 200_000)` identities asserted bit-identical, `200_000`-identity even/odd merge asserted equal to the single pass register for register), but over `HyperLogLogImpl` (`hll`, mergeable), `HyperLogLogImpl` (`hll`, mergeable) and `HyperLogLogHIPImpl` (`hll_hip`, not_mergeable, so its merge block asserts nothing), each with the precision's own checkpoints: p10 `[100, 1_000, 20_000, 200_000]` for `hll_classic_custom_p10_satisfies_its_register_error_model`, `hll_ertl_mle_custom_p10_satisfies_the_cramer_rao_error_model`, `hll_hip_custom_p10_satisfies_the_hip_error_model`; p11 `[200, 2_000, 40_000, 200_000]` for `hll_classic_custom_p11_satisfies_its_register_error_model`, `hll_ertl_mle_custom_p11_satisfies_the_cramer_rao_error_model`, `hll_hip_custom_p11_satisfies_the_hip_error_model`; p12 `[400, 4_000, 50_000, 300_000]` for `hll_classic_custom_p12_satisfies_its_register_error_model`, `hll_ertl_mle_custom_p12_satisfies_the_cramer_rao_error_model`, `hll_hip_custom_p12_satisfies_the_hip_error_model`; p13 `[1_000, 10_000, 100_000, 500_000]` for `hll_classic_custom_p13_satisfies_its_register_error_model`, `hll_ertl_mle_custom_p13_satisfies_the_cramer_rao_error_model`, `hll_hip_custom_p13_satisfies_the_hip_error_model`; p14 `[2_000, 16_000, 200_000, 800_000]` for `hll_classic_custom_p14_satisfies_its_register_error_model`, `hll_ertl_mle_custom_p14_satisfies_the_cramer_rao_error_model`, `hll_hip_custom_p14_satisfies_the_hip_error_model`; p15 `[4_000, 32_000, 300_000, 1_000_000]` for `hll_classic_custom_p15_satisfies_its_register_error_model`, `hll_ertl_mle_custom_p15_satisfies_the_cramer_rao_error_model`, `hll_hip_custom_p15_satisfies_the_hip_error_model`; p16 `[5_000, 60_000, 500_000, 1_200_000]` for `hll_classic_custom_p16_satisfies_its_register_error_model`, `hll_ertl_mle_custom_p16_satisfies_the_cramer_rao_error_model`, `hll_hip_custom_p16_satisfies_the_hip_error_model`; p17 `[10_000, 100_000, 800_000, 1_500_000]` for `hll_classic_custom_p17_satisfies_its_register_error_model`, `hll_ertl_mle_custom_p17_satisfies_the_cramer_rao_error_model`, `hll_hip_custom_p17_satisfies_the_hip_error_model`; p18 `[10_000, 100_000, 1_500_000]` for `hll_classic_custom_p18_satisfies_its_register_error_model`, `hll_ertl_mle_custom_p18_satisfies_the_cramer_rao_error_model`, `hll_hip_custom_p18_satisfies_the_hip_error_model`. | -| `portable_hll_variants_and_precisions_satisfy_the_register_error_model` | Every portable wire variant and precision holds the register error model and merges by register-wise maximum. | For each of `HllVariant::Regular`, `HllVariant::Datafusion`, `HllVariant::Hip` crossed with precisions `12`, `14`, `16`, draws its own stream `uniform_u64(200_000, u64::MAX / 4, seed)` with `seed = 2001 + (v * 3 + p)`, feeds `HllSketch::update` with `to_be_bytes()`, and checks `HllSketch::estimate()` against the `HashSet` distinct count via `CardinalityConfidenceSpec::hll(precision, 4.0)` — all three tags are held to `4 * 1.04/sqrt(m)`, since `estimate()` runs the same register formula for each. Then `merge` of a second sketch built over the same distinct identities must leave `estimate()` unchanged, and an even/odd shard merge of the same stream must give exactly that same `estimate()`. | | `hll_accuracy_improves_with_precision_as_the_error_model_predicts` | A larger precision actually buys accuracy, and the measured RSE stays near the predicted `1.04/sqrt(m)`. | Measures the RMS relative error of `HyperLogLogP12`, `HyperLogLogP14` and `HyperLogLogP16` over `BLOCKS = 6` disjoint blocks of `N = 1_000_000` identities (`DataInput::U64(b * N + i)`), asserts each measured RSE is `<= 2.0 *` `CardinalityConfidenceSpec::hll(p, 4.0).sigma_rel()`, and asserts the p12 RSE is at least `2.0x` the p16 RSE; the model's predicted 4x improvement is not asserted, only 2x. | | `hll_classic_switchover_band_stays_within_the_documented_empirical_band` | The Classic estimator's error at the linear-counting switchover stays inside a 1.5x-to-10x empirical band around the asymptotic RSE. | For `HyperLogLogP12/P14/P16` at `n = 2.5m` (`10_240`, `40_960`, `163_840`), computes the RMS relative error over `BLOCKS = 6` disjoint blocks (identities `b * 20_000_000 + i`) and forms `ratio = rse / (1.04/sqrt(m))` from `CardinalityConfidenceSpec::hll(p, 4.0).sigma_rel()`; asserts `ratio <= 10.0` and `ratio >= 1.5`, so both a widened cliff and a vanished one fail. | | `set_aggregator_union_is_exact` | A `SetAggregator` union keeps every member exactly. | Feeds `uniform_u64(20_000, 500, 2002)` as `member-{k}` strings into `SetAggregator::update`, merges a second aggregator holding `extra-a` and `extra-b`, then asserts `agg.values.len()` equals the length of a parallel `HashSet` of the same members and that `agg.values` contains every one of them; the distinct count is compared against the in-test truth set rather than a pinned number. | @@ -2126,9 +2094,9 @@ Test file: [`tests/asapv1_golden.rs`](../tests/asapv1_golden.rs) | `cs_i32_regular_2x4_matches_golden` | The i32 counter type reaches the metadata, and an i32 golden is not an i64 sketch. | Verifies `Count::, RegularPath>::from_storage` over `[[0, 127, 128, 65536], [-1, -33, -32768, -2147483648]]` cast to `i32` serializes to `asapv1_golden/cs_i32_regular_2x4.hex`, that decoding it recovers the eight cells and re-serializes identically, that the i32 and i64 regular fixtures have equal length yet differ, and that the i32 fixture fails to decode as `Count, RegularPath>`. | | `cs_i64_fast_2x4_matches_golden` | The fast-path mode reaches the bytes, so the same matrix encodes differently per path. | Verifies `Count::, FastPath>::from_storage` over `[[0, 127, 128, 65536], [-1, -33, -32768, -2147483648]]` serializes to `asapv1_golden/cs_i64_fast_2x4.hex`, that decoding it recovers the eight cells and re-serializes identically, and that the fixture differs from `cs_i64_regular_2x4.hex` built from the same matrix. | | `cs_i64_regular_2x4_matches_golden` | A known signed Count-Sketch matrix serializes to its checked-in golden bytes. | Verifies `Count::, RegularPath>::from_storage` over `[[0, 127, 128, 65536], [-1, -33, -32768, -2147483648]]` — sweeping the msgpack integer widths in both directions — serializes to `asapv1_golden/cs_i64_regular_2x4.hex`, and that decoding the fixture reports `rows()` `2`, `cols()` `4`, the eight cells, and a byte-identical re-serialization. | -| `hll_classic_p12_matches_golden` | A known P12 Classic register pattern serializes to its golden bytes on both the portable and native paths. | Builds `HllSketch::from_raw(HllVariant::Regular, 12, regs, 0.0, 0.0, 0.0)` where `regs` is 4,096 zeros with `[0] = 1`, `[1] = 7`, `[100] = 42` and `[4095] = 3`, verifies `to_msgpack()` equals `asapv1_golden/hll_classic_p12.hex`, that `HyperLogLogP12::::deserialize_from_bytes` on the fixture recovers those registers and re-serializes identically, and that `HllSketch::from_msgpack` reports the same registers, `precision` `12` and `HllVariant::Regular`. | -| `hll_ertl_mle_p12_matches_golden` | The Datafusion variant tag reaches the bytes for the same P12 register pattern. | Builds `HllSketch::from_raw(HllVariant::Datafusion, 12, regs, 0.0, 0.0, 0.0)` over the shared P12 pattern (`[0] = 1`, `[1] = 7`, `[100] = 42`, `[4095] = 3` in 4,096 registers), verifies `to_msgpack()` equals `asapv1_golden/hll_ertl_mle_p12.hex`, that `HyperLogLogP12::::deserialize_from_bytes` recovers the registers and re-serializes identically, and that `HllSketch::from_msgpack` reports `HllVariant::Datafusion`. | -| `hll_hip_p12_matches_golden` | The HIP variant's three running scalars travel on the wire beside its registers. | Builds `HllSketch::from_raw(HllVariant::Hip, 12, regs, 1.5, 2.5, 3.0)` over the shared P12 pattern, verifies `to_msgpack()` equals `asapv1_golden/hll_hip_p12.hex`, that `HyperLogLogHIPP12::deserialize_from_bytes` re-serializes the fixture identically, and that `HllSketch::from_msgpack` reports the registers, `HllVariant::Hip`, `hip_kxq0` `1.5`, `hip_kxq1` `2.5` and `hip_est` `3.0`. | +| `hll_classic_p12_matches_golden` | A known P12 Classic register pattern serializes to its golden bytes, which round-trip unchanged. | Builds `HyperLogLogP12::::from_storage` over 4,096 registers that are zero except `[0] = 1`, `[1] = 7`, `[100] = 42` and `[4095] = 3`, verifies `serialize_to_bytes()` equals `asapv1_golden/hll_classic_p12.hex`, and that `deserialize_from_bytes` on the fixture recovers those registers and re-serializes identically. | +| `hll_ertl_mle_p12_matches_golden` | The ErtlMLE kind_id reaches the bytes for the same P12 register pattern. | Builds `HyperLogLogP12::::from_storage` over the shared P12 pattern, verifies `serialize_to_bytes()` equals `asapv1_golden/hll_ertl_mle_p12.hex`, and that `deserialize_from_bytes` recovers the registers and re-serializes identically. | +| `hll_hip_p12_matches_golden` | The HIP variant's three running scalars travel on the wire beside its registers. | Builds `HyperLogLogHIPP12::from_storage` over the shared P12 pattern with `kxq0` `1.5`, `kxq1` `2.5` and `est` `3.0`, verifies `serialize_to_bytes()` equals `asapv1_golden/hll_hip_p12.hex`, that the decode's `Debug` form (registers and all three scalars) equals the built sketch's, and that it re-serializes identically. | | `kll_f64_k200_matches_golden` | A deterministic f64 KLL serializes to its golden bytes, which round-trip unchanged. | Verifies `KLL::::init_kll_with_seed(200, 42)` fed `1..=50` (below the level-0 capacity, so no compaction fires) serializes to `asapv1_golden/kll_f64_k200.hex`, and that decoding the fixture re-serializes to the same bytes and reports `quantile(0.0)` `1.0` and `quantile(1.0)` `50.0`. | | `kll_i64_k200_matches_golden` | A deterministic i64 KLL serializes to its golden bytes, which round-trip unchanged. | Verifies `KLL::::init_kll_with_seed(200, 42)` fed `1..=50` serializes to `asapv1_golden/kll_i64_k200.hex`, and that decoding the fixture re-serializes to the same bytes and reports `quantile(0.0)` `1.0` and `quantile(1.0)` `50.0`. | @@ -2147,8 +2115,6 @@ Test file: [`tests/msgpack_compat.rs`](../tests/msgpack_compat.rs) | `dd_sketch_decodes_go_bytes` | The test asserts nothing. | The test asserts nothing: the body is empty and the test is marked `#[ignore]` with the reason that no cross-language msgpack fixture exists in the repository. | | `dd_sketch_round_trip` | A portable `DdSketch` survives a msgpack round trip carrying its full count in the bucket array. | Updates `DdSketch::new(0.01)` with `1.0`, `10.0` and `100.0`, then verifies the decode's `total_count()` — recovered by summing the buckets, since the count is not on the wire — is `3`; `alpha`, the bucket offset and the quantiles are not checked. | | `delta_result_round_trip` | A `DeltaResult` survives a msgpack round trip with both of its sets. | Serializes a `DeltaResult` whose `added` holds `a` and whose `removed` holds `b`, then verifies the decode's `added` contains `a` and its `removed` contains `b`. | -| `hll_sketch_decodes_go_bytes` | The test asserts nothing. | The test asserts nothing: the body is empty and the test is marked `#[ignore]` with the reason that no cross-language msgpack fixture exists in the repository. | -| `hll_sketch_round_trip` | A portable `HllSketch` survives a msgpack round trip with its register count intact. | Updates `HllSketch::new(HllVariant::Regular, 8)` with `b"a"`, `b"b"` and `b"c"`, then verifies only that the decode's `registers` has the same length as the source's; the register values, the precision and the variant are not checked. | | `hydra_kll_decodes_go_bytes` | The test asserts nothing. | The test asserts nothing: the body is empty and the test is marked `#[ignore]` with the reason that no cross-language msgpack fixture exists in the repository. | | `hydra_kll_sketch_round_trip` | A portable `HydraKllSketch` survives a msgpack round trip with its grid shape intact. | Updates `HydraKllSketch::with_seed(2, 4, 200, 0x5EED_0900)` with `("a", 1.0)`, `("a", 2.0)` and `("b", 3.0)`, then verifies the decode reports `rows` `2` and `cols` `4`; the nested KLL payloads are not checked. | | `hydra_kll_wire_shape` | The `HydraKllSketchWire` DTO round-trips through `rmp_serde` with its two levels of nesting intact. | Serializes a wire struct of `rows: 2`, `cols: 3` and a `2x3` grid of `KllSketchData { k: 200, sketch_bytes: vec![] }`, then verifies the `rmp_serde` decode reports `rows` `2`, `cols` `3`, an outer length of `2` and an inner length of `3`. | @@ -2168,7 +2134,7 @@ Test file: [`tests/conformance_kit.rs`](../tests/conformance_kit.rs) | `countsketch_passes_signed_frequency_conformance` | `Count` passes the two-sided frequency, turnstile and merge-equivalence batteries. | Drives `Count::, RegularPath>::with_dimensions(5, 4096)` over the 60,000-draw Zipf stream through `frequency_battery` two-sided at `rel_tol` `0.06` and `abs_tol` `25.0`, `turnstile_battery` on key `42` via `insert_many` (`+500 - 200` must read `300` within `1.0`, then cancel to within `1e-6`), and `merge_equivalence_battery` at the same spec. | | `ddsketch_passes_relative_quantile_conformance` | The portable `DdSketch` holds its relative-value-error promise under its own rank convention. | Runs `relative_quantile_battery` on `PortableDds::new(0.01)` over the positive samples of `normal_f64(40_000, 500.0, 80.0, 7001)`, requiring at q `0.1`, `0.25`, `0.5`, `0.75` and `0.9` a relative error of at most `0.01` plus a few-ULP numerical slack against the exact `floor(q*(n-1))` order statistic. | | `heap_backed_frequency_sketches_pass_frequency_and_merge_conformance` | Both heap-backed matrix sketches pass the frequency and merge-equivalence batteries. | Over the 60,000-draw Zipf stream (domain 2,048, exponent 1.1, seed `9001`), drives `CMSHeap::, FastPath>::new(4, 4_096, 64)` through `frequency_battery` and `merge_equivalence_battery` one-sided at `rel_tol` `0.01` and `abs_tol` `8.0`, and `CSHeap::, RegularPath>::new(5, 4_096, 64)` through both two-sided at `rel_tol` `0.06` and `abs_tol` `25.0`. | -| `hll_variants_pass_cardinality_conformance` | All three native HyperLogLog estimators and the portable one estimate within 3%. | Runs `cardinality_battery` at `rel_tol` `0.03` over 100,000 distinct keys `i.wrapping_mul(0x9E37_79B9_7F4A_7C15)` on `HyperLogLog`, `HyperLogLog`, `HyperLogLogHIP` and the portable `HllSketch::new(HllVariant::Regular, 14)`; each must estimate `100_000` within 3% and stay in band after the whole stream is replayed as duplicates. | +| `hll_variants_pass_cardinality_conformance` | All three HyperLogLog estimators estimate within 3%. | Runs `cardinality_battery` at `rel_tol` `0.03` over 100,000 distinct keys `i.wrapping_mul(0x9E37_79B9_7F4A_7C15)` on `HyperLogLog`, `HyperLogLog` and `HyperLogLogHIP`; each must estimate `100_000` within 3% and stay in band after the whole stream is replayed as duplicates. | | `kll_family_passes_quantile_conformance` | The three KLL adapters hold the shared rank-error band. | Runs `quantile_battery` at the default `rank_tol` of `0.03` over q `0.1`, `0.25`, `0.5`, `0.75` and `0.9` on the positive samples of `normal_f64(40_000, 500.0, 80.0, 7001)` for `KLL::init_kll_with_seed(200, 0x4017_0001)`, `KLLDynamic::::init_kll_with_seed(200, 0x4017_0001)`, and a `KLL` at the same `k` and seed queried through `quantile_cached`. | | `kmv_passes_cardinality_conformance` | `KMV` estimates a 100,000-key distinct count within 8%. | Compiled only under the `experimental` feature: runs `cardinality_battery` at `rel_tol` `0.08` on `KMV::new(4_096)` over 100,000 distinct keys `i.wrapping_mul(0x9E37_79B9_7F4A_7C15)`, requiring the estimate in band both after the first pass and after the duplicate replay. | | `uniform_sampling_at_full_rate_passes_quantile_conformance` | A sampler retaining everything answers quantiles exactly. | Compiled only under the `experimental` feature: runs `quantile_battery` at `rank_tol` `0.0` on `UniformSampling::with_seed(1.0, 7_704)` over 20,000 Zipf draws (domain 4,096, exponent 1.1, seed `7_703`), with each quantile read from the sorted retained samples at index `ceil(q * len)`. | @@ -2194,7 +2160,6 @@ Test file: [`tests/sketches_go_parity_probe.rs`](../tests/sketches_go_parity_pro | `sketches_count_fastpath_matches_go_count_sketch_envelope` | A `sketches::Count` fast-path matrix encodes to the checked-in Count-Sketch envelope bytes. | Fills `Count, FastPath, DefaultXxHasher>` at `3x512` with 25 `insert_many` calls over the five keys `k-a`..`k-e`, packs the row-major cells and per-row L2 sums into a `CountSketchState` with `counter_type` `INT64` inside a `SketchEnvelope { format_version: 1 }` with no producer or hash_spec, and verifies the prost encoding equals the 1,577-byte fixture `tests/cs_envelope_golden.hex`. | | `sketches_countmin_fastpath_matches_go_count_min_envelope` | A `sketches::CountMin` fast-path matrix encodes to the checked-in Count-Min envelope bytes. | Fills `CountMin, FastPath, DefaultXxHasher>` at `4x2048` with 50 `insert_many` calls over the ten keys `flow-0`..`flow-9`, packs the cells cast to `i64` plus per-row L1 and L2 sums into a `CountMinState` with `counter_type` `INT64`, and verifies the prost encoding matches the 8,275-byte fixture `src/sketches/testdata/cms_envelope_golden.hex` in length and then byte for byte. | | `sketches_ddsketch_matches_go_envelope` | A `sketches::DDSketch` encodes to the DDSketch envelope hex constant held in the test file. | Fills `DDSketch::new(0.01)` with `1..=50` as `f64`, encodes a `DdSketchState` carrying `store_counts`, `store_offset` and an alpha round-tripped through gamma (`gamma = (1 + a) / (1 - a)`, `alpha_wire = (gamma - 1) / (gamma + 1)`), and verifies the envelope matches the 403-byte inline hex constant in length and then byte for byte; the comment on that constant records it as this crate's own output rather than a fixture from another implementation, so no cross-language parity is pinned. | -| `sketches_hll_classic_matches_go_envelope` | An ErtlMLE P14 register set encodes to the checked-in HyperLogLog envelope bytes. | Feeds `HyperLogLogImpl` the `hash64_seeded(CANONICAL_HASH_SEED, ...)` digests of the IEEE-754 little-endian bytes of `1.0..=50.0` through `insert_with_hash`, encodes a `HyperLogLogState` with `variant` `2`, `precision` `14`, the dense registers, zeroed HIP scalars and no sparse form, and verifies the envelope matches the 16,398-byte fixture `src/sketches/testdata/hll_envelope_golden.hex` in length and then byte for byte; despite the name, the variant under test is `ErtlMLE`, not `Classic`. | ### Cross-Language Proto Consumer @@ -2202,7 +2167,7 @@ Test file: [`tests/xtest_consumer.rs`](../tests/xtest_consumer.rs) | test_name | test_description | what_is_tested | | --- | --- | --- | -| `cross_language_proto` | Protobuf envelopes read from the cross-language fixture directory answer their point queries in band. | Asserts nothing unless fixtures are available: with no `XTEST_DIR` and no discoverable producer directory holding `tests/cross_language/xtest_producer_test.go`, it prints a skip line and returns. With fixtures it decodes nine `SketchEnvelope` files and records a pass or failure for each — `countmin.pb` min-frequency for `item:42` at least `101`, `kll.pb` p50 within 5% of `5000` and p99 within 5% of `9900`, `ddsketch.pb` the same two quantiles within 2%, `hll.pb` an ErtlMLE cardinality inside `[40_000, 65_000]`, `countsketch.pb` `cs:hot` at least `200`, `coco.pb` `coco:hot` at least `500`, `elastic.pb` `elephant` at least `900`, `univmon.pb` a cardinality inside `[1_000, 15_000]`, `hydra.pb` `hydra:42` at least `51` — using the hash seeds `0xcafe3553`, `0x6a09e667` and `0xbb67ae85`, and, when `countmin_sampled.pb` and `hll_sampled.pb` exist, requires `effective_sample_p` of `0.1` and a `rescale_count` by `1/p` within 5% of `100_000` and 6% of `200_000`. A single `assert!` at the end fails if any check failed; a missing or mistyped envelope panics on the spot. | +| `cross_language_proto` | Protobuf envelopes read from the cross-language fixture directory answer their point queries in band. | Asserts nothing unless fixtures are available: with no `XTEST_DIR` and no discoverable producer directory holding `tests/cross_language/xtest_producer_test.go`, it prints a skip line and returns. With fixtures it decodes eight `SketchEnvelope` files and records a pass or failure for each — `countmin.pb` min-frequency for `item:42` at least `101`, `kll.pb` p50 within 5% of `5000` and p99 within 5% of `9900`, `ddsketch.pb` the same two quantiles within 2%, `countsketch.pb` `cs:hot` at least `200`, `coco.pb` `coco:hot` at least `500`, `elastic.pb` `elephant` at least `900`, `univmon.pb` a cardinality inside `[1_000, 15_000]`, `hydra.pb` `hydra:42` at least `51` — using the hash seeds `0xcafe3553`, `0x6a09e667` and `0xbb67ae85`, and, when `countmin_sampled.pb` exists, requires `effective_sample_p` of `0.1` and a `rescale_count` by `1/p` within 5% of `100_000`. A single `assert!` at the end fails if any check failed; a missing or mistyped envelope panics on the spot. | ### HLL Custom Precision diff --git a/examples/accuracy_probe.rs b/examples/accuracy_probe.rs index 643bfc3e..7265278f 100644 --- a/examples/accuracy_probe.rs +++ b/examples/accuracy_probe.rs @@ -9,7 +9,6 @@ use asap_sketchlib::common::input::{HydraCounter, HydraQuery}; use asap_sketchlib::message_pack_format::portable::countminsketch::CountMinSketch; use asap_sketchlib::message_pack_format::portable::ddsketch::DdSketch as PortableDds; -use asap_sketchlib::message_pack_format::portable::hll::{HllSketch, HllVariant}; use asap_sketchlib::message_pack_format::portable::hydra_kll::HydraKllSketch; use asap_sketchlib::message_pack_format::portable::kll::KllSketch as PortableKll; use asap_sketchlib::{ @@ -1043,21 +1042,6 @@ fn probe_portable_wire_types() { bad == 0, ); - // Portable HLL precision 12. - let mut ph = HllSketch::new(HllVariant::Regular, 12); - for i in 0..100_000u64 { - ph.update(i.to_le_bytes().as_slice()); - } - let est = ph.estimate(); - p.check( - "portable HLL p12 within 3%", - format!( - "expected 100000, got {est:.0} (rel {:.4})", - rel_err(est, 100_000.0) - ), - rel_err(est, 100_000.0) < 0.03, - ); - // Portable KLL. let mut pk = PortableKll::new(200); let mut rng = StdRng::seed_from_u64(55); diff --git a/proto/hll/hll.proto b/proto/hll/hll.proto index 1cdf6124..eba61943 100644 --- a/proto/hll/hll.proto +++ b/proto/hll/hll.proto @@ -1,4 +1,4 @@ -// hll.proto — Portable wire format for HyperLogLog sketch state. +// hll.proto — HyperLogLog cell state for HydraCell (hydra/hydra.proto). syntax = "proto3"; @@ -97,38 +97,3 @@ enum HLLVariant { // Requires hip_kxq0, hip_kxq1, hip_est to be populated. HLL_VARIANT_HIP = 3; } - -// ============================================================================ -// HyperLogLog Delta -// ============================================================================ - -// HLLDelta carries the registers that increased between two consecutive -// snapshots. HLL uses max semantics, so only increases are meaningful; -// at a fixed precision a register can never decrease. Each update is applied -// losslessly on the receiver via register[index] = max(register[index], value) -// — no register update is ever dropped (there is no threshold to apply). -// -// The increased registers are varint-packed exactly like -// HLLSparseRegisters.packed: sorted ascending by index, each register emitted -// as (index_delta, value) where index_delta = index - prev_index (prev_index -// starts at 0) and value is the new (larger) register value: -// -// for each increased register, in ascending index order: -// uvarint(index - prev_index) // prev_index starts at 0; deltas are >= 0 -// uvarint(value) // 1..=Q+1, always 1 byte in practice -// -// A single-emit delta against the all-zero snapshot is therefore the same size -// as the full sparse frame, and a sub-window delta is strictly smaller (it -// packs only the registers that actually grew). This replaces the previous -// per-register sub-message encoding, eliminating ~6–8 bytes of tag/length -// overhead per register. -// -// This message is byte-identical to the Go reference implementation's HLLDelta -// so the two runtimes emit byte-identical delta frames for identical window -// state (cross-language byte parity). -message HLLDelta { - // Varint-packed (index_delta, value) pairs for the registers that increased, - // in ascending index order. Same layout as HLLSparseRegisters.packed. Empty - // when no register changed. - bytes packed_updates = 1; -} diff --git a/proto/sketchlib.proto b/proto/sketchlib.proto index 0a6d8193..d6f253b9 100644 --- a/proto/sketchlib.proto +++ b/proto/sketchlib.proto @@ -6,7 +6,6 @@ // common/common.proto — HashSpec, ProducerInfo, CounterType, ... // countminsketch/countminsketch.proto — CountMinState // countsketch/countsketch.proto — CountSketchState, TopKState, HeapEntry -// hll/hll.proto — HyperLogLogState, HLLVariant // kll/kll.proto — KLLState, CoinState // ddsketch/ddsketch.proto — DDSketchState // univmon/univmon.proto — UnivMonState, UnivMonLayer @@ -31,7 +30,6 @@ package sketchlib.v1; import "common/common.proto"; import "countminsketch/countminsketch.proto"; import "countsketch/countsketch.proto"; -import "hll/hll.proto"; import "kll/kll.proto"; import "ddsketch/ddsketch.proto"; import "univmon/univmon.proto"; @@ -68,7 +66,6 @@ message SketchEnvelope { oneof sketch_state { CountMinState count_min = 10; CountSketchState count_sketch = 11; - HyperLogLogState hll = 12; KLLState kll = 13; DDSketchState ddsketch = 14; UnivMonState univmon = 15; @@ -77,6 +74,8 @@ message SketchEnvelope { ElasticState elastic = 18; } + reserved 12; + reserved "hll"; reserved 19 to 31; // Reserved for future sketch types. // Geometric / NitroSketch sampling probability for the producing sketch. diff --git a/src/lib.rs b/src/lib.rs index b062b8b9..3aca2dda 100644 --- a/src/lib.rs +++ b/src/lib.rs @@ -10,7 +10,7 @@ //! UnivMon, tumbling windows, and batch/parallel execution helpers. //! - [`message_pack_format`]: MessagePack/proto wire format shared with //! `sketchlib-go`. Owns the wire-DTO sketch types consumed by the ASAP -//! query engine (`CountMinSketch`, `CountSketch`, `KllSketch`, `HllSketch`, +//! query engine (`CountMinSketch`, `CountSketch`, `KllSketch`, //! `DdSketch`, `HydraKllSketch`, `CountMinSketchWithHeap`, //! `SetAggregator`, `DeltaResult`). //! @@ -49,7 +49,6 @@ pub use message_pack_format::portable::countsketch::{ pub use message_pack_format::portable::countsketch_topk::{CountSketchWithHeap, CsHeapItem}; pub use message_pack_format::portable::ddsketch::{DDSKETCH_GROW_CHUNK, DdSketch, DdSketchDelta}; pub use message_pack_format::portable::delta_set_aggregator::DeltaResult; -pub use message_pack_format::portable::hll::{HllSketch, HllSketchDelta, HllVariant}; pub use message_pack_format::portable::hydra_kll::HydraKllSketch; pub use message_pack_format::portable::kll::{KllSketch, KllSketchData}; pub use message_pack_format::portable::sampling::{ diff --git a/src/message_pack_format/native/hll.rs b/src/message_pack_format/native/hll.rs deleted file mode 100644 index b5de3118..00000000 --- a/src/message_pack_format/native/hll.rs +++ /dev/null @@ -1,27 +0,0 @@ -//! Native MessagePack codec impls for [`crate::sketches::hll`]. - -use crate::message_pack_format::{Error, MessagePackCodec}; -use crate::sketches::hll::{HllWireVariant, HyperLogLogHIPImpl, HyperLogLogImpl}; -use crate::{HashProfile, HllRegisterStorage, SketchHasher}; - -impl - MessagePackCodec for HyperLogLogImpl -{ - fn to_msgpack(&self) -> Result, Error> { - Ok(self.serialize_to_bytes()?) - } - - fn from_msgpack(bytes: &[u8]) -> Result { - Ok(Self::deserialize_from_bytes(bytes)?) - } -} - -impl MessagePackCodec for HyperLogLogHIPImpl { - fn to_msgpack(&self) -> Result, Error> { - Ok(self.serialize_to_bytes()?) - } - - fn from_msgpack(bytes: &[u8]) -> Result { - Ok(Self::deserialize_from_bytes(bytes)?) - } -} diff --git a/src/message_pack_format/native/mod.rs b/src/message_pack_format/native/mod.rs index 0a83ef96..91f3493b 100644 --- a/src/message_pack_format/native/mod.rs +++ b/src/message_pack_format/native/mod.rs @@ -11,7 +11,6 @@ pub mod countminsketch; pub mod countsketch; pub mod countsketch_topk; pub mod ddsketch; -pub mod hll; pub mod kll; pub mod kll_dynamic; #[cfg(feature = "experimental")] diff --git a/src/message_pack_format/portable/hll.rs b/src/message_pack_format/portable/hll.rs deleted file mode 100644 index b70dfe80..00000000 --- a/src/message_pack_format/portable/hll.rs +++ /dev/null @@ -1,1056 +0,0 @@ -//! Wire-format-aligned HyperLogLog types. -//! -//! Self-contained: maintains its own `Vec` register array. Parity -//! with Go's `HyperLogLog.SerializePortable` (and equivalence with -//! `sketches::HyperLogLogImpl<.., ErtlMLE, P14>`) is locked in by the -//! `test_update_then_envelope_matches_sketchlib_go_bytes` test below and -//! by `tests/sketches_go_parity_probe.rs`. - -use serde::{Deserialize, Serialize}; - -use crate::message_pack_format::{Error as MsgPackError, MessagePackCodec}; -use crate::{CANONICAL_HASH_SEED, DataInput, hash64_seeded}; - -// --------------------------------------------------------------------------- -// DEPRECATED — temporary compatibility shim, slated for removal. -// -// A few call sites still depend on this portable `HllSketch`. Its ASAPv1 codec -// now reuses the shared `envelope` framing plus the HLL metadata/payload types -// in `crate::sketches::hll`, so it stays byte-identical with the native -// `HyperLogLogImpl` / `HyperLogLogHIPImpl` serialization. Do NOT add new callers. -// -// Prefer instead the native path: -// `sketches::hll::HyperLogLogImpl::serialize_to_bytes` / -// `HyperLogLogHIPImpl::serialize_to_bytes` (+ `deserialize_from_bytes`). -// --------------------------------------------------------------------------- - -use crate::message_pack_format::envelope; -use crate::sketches::hll::{ - HLL_KIND_CLASSIC, HLL_KIND_ERTL_MLE, HLL_KIND_HIP, HllMetadata, HllPayloadHip, HllPayloadPlain, - standard_hll_metadata, -}; - -/// Retained only for `test_msgpack_round_trip`'s magic-prefix assertion. -#[cfg(test)] -const HLL_WRAPPER_MAGIC: &[u8; 6] = b"ASAPv1"; - -/// Map a portable [`HllVariant`] to its ASAPv1 wire `kind_id`. `Unspecified` is -/// a placeholder with no wire form. -fn wire_kind_id(variant: HllVariant) -> Result<&'static [u8], MsgPackError> { - match variant { - HllVariant::Regular => Ok(HLL_KIND_CLASSIC), - HllVariant::Datafusion => Ok(HLL_KIND_ERTL_MLE), - HllVariant::Hip => Ok(HLL_KIND_HIP), - HllVariant::Unspecified => Err(MsgPackError::Decode( - rmp_serde::decode::Error::Uncategorized( - "ASAPv1 HLL: the Unspecified variant is not serializable".to_string(), - ), - )), - } -} - -/// HLL estimator variant. Mirrors `asap_sketchlib::proto::sketchlib::HllVariant`. -#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] -pub enum HllVariant { - Unspecified, - Regular, - Datafusion, - Hip, -} - -/// Sparse delta between two consecutive HLL snapshots — input shape for -/// [`HllSketch::apply_delta`]. Mirrors the `HLLDelta` proto in -/// `sketchlib-go/proto/hll/hll.proto`. -#[derive(Debug, Clone, Default)] -pub struct HllSketchDelta { - pub updates: Vec<(u32, u8)>, -} - -/// Minimal HLL state — registers + variant + precision. Register-wise -/// mergeable (max over aligned cells). -#[derive(Debug, Clone, Serialize, Deserialize)] -pub struct HllSketch { - pub variant: HllVariant, - pub precision: u32, - pub registers: Vec, - pub hip_kxq0: f64, - pub hip_kxq1: f64, - pub hip_est: f64, -} - -impl HllSketch { - pub fn new(variant: HllVariant, precision: u32) -> Self { - let n = 1usize << precision; - Self { - variant, - precision, - registers: vec![0u8; n], - hip_kxq0: 0.0, - hip_kxq1: 0.0, - hip_est: 0.0, - } - } - - pub fn from_raw( - variant: HllVariant, - precision: u32, - registers: Vec, - hip_kxq0: f64, - hip_kxq1: f64, - hip_est: f64, - ) -> Self { - Self { - variant, - precision, - registers, - hip_kxq0, - hip_kxq1, - hip_est, - } - } - - pub fn merge( - &mut self, - other: &HllSketch, - ) -> Result<(), Box> { - if self.variant != other.variant { - return Err(format!( - "HllSketch variant mismatch: self={:?}, other={:?}", - self.variant, other.variant - ) - .into()); - } - if self.precision != other.precision { - return Err(format!( - "HllSketch precision mismatch: self={}, other={}", - self.precision, other.precision - ) - .into()); - } - if self.registers.len() != other.registers.len() { - return Err(format!( - "HllSketch register-length mismatch: self={}, other={}", - self.registers.len(), - other.registers.len() - ) - .into()); - } - for (s, o) in self.registers.iter_mut().zip(other.registers.iter()) { - if *o > *s { - *s = *o; - } - } - if self.variant == HllVariant::Hip { - self.hip_kxq0 += other.hip_kxq0; - self.hip_kxq1 += other.hip_kxq1; - self.hip_est += other.hip_est; - } - Ok(()) - } - - pub fn apply_delta( - &mut self, - delta: &HllSketchDelta, - ) -> Result<(), Box> { - let n = self.registers.len(); - for (idx, new_val) in &delta.updates { - let i = *idx as usize; - if i >= n { - return Err(format!( - "HllSketchDelta index {i} out of range (precision={} → {n} registers)", - self.precision - ) - .into()); - } - if *new_val > self.registers[i] { - self.registers[i] = *new_val; - } - } - Ok(()) - } - - /// Compute a sparse, proto-marshalled `HLLDelta` of `self` against a - /// `snapshot`. A register update is included when `self`'s register - /// value increased over the snapshot's (`self[i] > snapshot[i]`); its - /// carried value is the new (larger) register value. - /// - /// This is the Rust twin of the Go reference implementation's - /// `ComputeRegisterDelta` + `SerializeRegisterDelta`. HLL uses max - /// semantics, so only increases are meaningful and the delta is - /// **lossless** — every increased register is carried, regardless of - /// `threshold` (the parameter is accepted for a uniform delta API and - /// has no effect here; an HLL register update is never dropped). The - /// returned bytes are a `prost`-encoded - /// [`crate::proto::sketchlib::HllDelta`], byte-identical to the Go - /// `proto.Marshal(HLLDelta)` output for the same inputs (cross-language - /// byte parity). - /// - /// Delta-against-empty: when `snapshot` is the all-zero sketch, every - /// non-zero register of `self` is carried, so the result is this - /// window's full register state encoded as a delta (no cross-window - /// subtraction). HLL deltas carry only register max-updates — there are - /// no DataPoint-level metric scalars to drop. - pub fn compute_delta(&self, snapshot: &HllSketch, _threshold: u64) -> Vec { - use crate::proto::sketchlib::HllDelta as ProtoDelta; - use prost::Message; - - let cur = &self.registers; - let snap = &snapshot.registers; - let n = cur.len().min(snap.len()); - - // Varint-pack the increased registers as (index_delta, value) pairs in - // ascending index order — the same layout `encode_sparse_registers` - // uses for the full sparse state. Walking registers in index order - // yields a sorted sequence, so `prev` only ever advances. - let mut packed: Vec = Vec::new(); - let mut prev: usize = 0; - for i in 0..n { - if cur[i] > snap[i] { - write_uvarint(&mut packed, (i - prev) as u64); - write_uvarint(&mut packed, cur[i] as u64); - prev = i; - } - } - // Guard: if `self` has more registers than the snapshot (should not - // happen at a fixed precision), carry all non-zero extras. Matches - // the Go reference's trailing-register guard. - for (i, &v) in cur.iter().enumerate().take(cur.len()).skip(n) { - if v > 0 { - write_uvarint(&mut packed, (i - prev) as u64); - write_uvarint(&mut packed, v as u64); - prev = i; - } - } - - ProtoDelta { - packed_updates: packed, - } - .encode_to_vec() - } - - /// Apply a `prost`-encoded [`crate::proto::sketchlib::HllDelta`] to this - /// sketch in place (register max-merge). The Rust twin of the Go - /// reference implementation's `DeserializeRegisterDelta` + - /// `ApplyRegisterDelta`: each update sets - /// `register[index] = max(register[index], value)`. - /// - /// Returns `Err` if `bytes` is not a valid `HLLDelta` proto or a - /// register index is out of range for this sketch's precision. - pub fn apply_delta_bytes( - &mut self, - bytes: &[u8], - ) -> Result<(), Box> { - use crate::proto::sketchlib::HllDelta as ProtoDelta; - use prost::Message; - - let proto = ProtoDelta::decode(bytes)?; - - // Unpack the (index_delta, value) blob — inverse of `compute_delta`, - // same layout as `decode_sparse_registers` but bounded by the blob's - // own contents rather than a fixed register count. - let packed = &proto.packed_updates; - let mut updates: Vec<(u32, u8)> = Vec::new(); - let mut off = 0usize; - let mut prev: u32 = 0; - let mut first = true; - while off < packed.len() { - let (delta, used) = - read_uvarint(&packed[off..]).ok_or("hll: delta index-delta varint corrupt")?; - off += used; - let (val, used) = - read_uvarint(&packed[off..]).ok_or("hll: delta value varint corrupt")?; - off += used; - - let idx = prev + delta as u32; - // First delta is an absolute index (prev=0); later deltas must be - // > 0 so indices stay strictly increasing and unique. - if !first && delta == 0 { - return Err(format!("hll: delta non-increasing index at idx {idx}").into()); - } - first = false; - if val > 0xff { - return Err(format!("hll: delta value {val} exceeds u8 at idx {idx}").into()); - } - updates.push((idx, val as u8)); - prev = idx; - } - - let delta = HllSketchDelta { updates }; - self.apply_delta(&delta) - } - - /// MessagePack twin of [`Self::compute_delta`]. Carries every register - /// that increased over `snapshot` (`self[i] > snapshot[i]`) as an - /// `(index, new_value)` pair, serialized as the parallel-array - /// MessagePack layout via [`HllSketchDelta`]'s [`MessagePackCodec`] - /// instead of proto. HLL uses max semantics, so the delta is lossless - /// regardless of `_threshold` (accepted for a uniform delta API). - /// Delta-against-empty carries every non-zero register — this window's - /// full register state encoded as a delta. - pub fn compute_delta_msgpack(&self, snapshot: &HllSketch, _threshold: u64) -> Vec { - let cur = &self.registers; - let snap = &snapshot.registers; - let n = cur.len().min(snap.len()); - let mut updates: Vec<(u32, u8)> = Vec::new(); - for i in 0..n { - if cur[i] > snap[i] { - updates.push((i as u32, cur[i])); - } - } - // Guard: registers present in `self` beyond the snapshot length - // (should not happen at a fixed precision) carry their non-zero - // values. Matches the proto path's trailing-register guard. - for (i, &v) in cur.iter().enumerate().skip(n) { - if v > 0 { - updates.push((i as u32, v)); - } - } - HllSketchDelta { updates } - .to_msgpack() - .expect("HllSketchDelta msgpack encode is infallible for owned update arrays") - } - - /// MessagePack twin of [`Self::apply_delta_bytes`]. Decodes the - /// parallel-array MessagePack [`HllSketchDelta`] and applies it in - /// place (register max-merge). - /// - /// Returns `Err` if `bytes` is not a valid MessagePack `HllSketchDelta` - /// or a register index is out of range for this sketch's precision. - pub fn apply_delta_msgpack_bytes( - &mut self, - bytes: &[u8], - ) -> Result<(), Box> { - let delta = HllSketchDelta::from_msgpack(bytes)?; - self.apply_delta(&delta) - } - - pub fn merge_refs( - inputs: &[&HllSketch], - ) -> Result> { - let first = inputs - .first() - .ok_or("HllSketch::merge_refs called with empty input")?; - let mut merged = HllSketch::new(first.variant, first.precision); - for hll in inputs { - merged.merge(hll)?; - } - Ok(merged) - } - - /// Insert a value. Math mirrors `sketches::HyperLogLogImpl::insert_with_hash` - /// (CANONICAL_HASH_SEED, packed bucket/leading-zero); locked in by - /// `tests/sketches_go_parity_probe.rs`. - pub fn update(&mut self, value: &[u8]) { - let hashed_val = hash64_seeded(CANONICAL_HASH_SEED, &DataInput::Bytes(value)); - let p = self.precision as usize; - let register_bits = (u64::BITS as usize) - p; - let p_mask: u64 = (1u64 << p) - 1; - let bucket_num = ((hashed_val >> register_bits) & p_mask) as usize; - let leading_zero = ((hashed_val << p) + p_mask).leading_zeros() as u8 + 1; - if bucket_num < self.registers.len() && leading_zero > self.registers[bucket_num] { - self.registers[bucket_num] = leading_zero; - } - } - - /// Estimate cardinality (Classic HLL estimator with the small-range - /// correction). Ranks are drawn from a 64-bit hash, so the estimate needs - /// no large-range correction. - pub fn estimate(&self) -> f64 { - let m = self.registers.len() as f64; - if m == 0.0 { - return 0.0; - } - let mut z = 0.0_f64; - let mut zero_count = 0usize; - for ®_val in &self.registers { - if reg_val == 0 { - zero_count += 1; - } - z += 2f64.powi(-(reg_val as i32)); - } - let indicator = 1.0 / z; - - let alpha_m = 0.7213 / (1.0 + 1.079 / m); - let mut est = alpha_m * m * m * indicator; - - if est <= m * 5.0 / 2.0 && zero_count != 0 { - est = m * (m / zero_count as f64).ln(); - } - est - } - - /// Return the proto-enum value Go's `HyperLogLog.SerializePortable` - /// emits for this sketch's [`HllVariant`]. - #[inline] - pub fn wire_proto_variant(&self) -> i32 { - match self.variant { - HllVariant::Unspecified => 0, - HllVariant::Regular => 1, - HllVariant::Datafusion => 2, - HllVariant::Hip => 3, - } - } -} - -// --------------------------------------------------------------------------- -// Proto dual-read decode (DENSE tag 3 + SPARSE tag 7) -// --------------------------------------------------------------------------- - -use crate::proto::sketchlib::{HllSparseRegisters, HyperLogLogState}; - -/// Decode the full dense register array from a `HyperLogLogState`, accepting -/// BOTH wire encodings emitted by sketchlib-go: -/// - DENSE (`registers`, tag 3): raw 1-byte-per-register array, or -/// - SPARSE (`registers_sparse`, tag 7): HLL++ style varint-packed -/// (index_delta, value) pairs for non-zero registers only. -/// -/// The SPARSE field takes priority when present; otherwise the DENSE field is -/// used. Both reconstruct the identical `2^precision`-byte register array, so -/// downstream estimation / merge is encoding-agnostic. This is the Rust half of -/// the backend-decode-first rollout: it must ship before any producer emits -/// sparse. Mirrors the Go decoder in -/// `sketchlib-go/sketches/HLL/portable.go::registersFromState` + -/// `sparse.go::decodeSparseRegisters`. -pub fn registers_from_state( - state: &HyperLogLogState, -) -> Result, Box> { - if let Some(sp) = &state.registers_sparse { - return decode_sparse_registers(sp); - } - Ok(state.registers.clone()) -} - -/// Reconstruct the dense register array from a sparse message. Inverse of the -/// Go encoder `sparse.go::encodeSparseRegisters`. -pub fn decode_sparse_registers( - sp: &HllSparseRegisters, -) -> Result, Box> { - let n = sp.num_registers as usize; - let mut regs = vec![0u8; n]; - - let packed = &sp.packed; - let mut off = 0usize; - let mut prev: usize = 0; - let mut first = true; - while off < packed.len() { - let (delta, used) = - read_uvarint(&packed[off..]).ok_or("hll: sparse index-delta varint corrupt")?; - off += used; - let (val, used) = read_uvarint(&packed[off..]).ok_or("hll: sparse value varint corrupt")?; - off += used; - - let idx = prev + delta as usize; - // First delta is an absolute index (prev=0); later deltas must be > 0 - // so indices stay strictly increasing and unique. - if !first && delta == 0 { - return Err(format!("hll: sparse non-increasing index at idx {idx}").into()); - } - first = false; - if idx >= n { - return Err(format!("hll: sparse index {idx} out of range [0,{n})").into()); - } - if val > 0xff { - return Err(format!("hll: sparse value {val} exceeds u8 at idx {idx}").into()); - } - regs[idx] = val as u8; - prev = idx; - } - Ok(regs) -} - -/// Encode the non-zero registers of `regs` into an [`HllSparseRegisters`] -/// message, byte-identical to the Go encoder `sparse.go::encodeSparseRegisters`. -/// Provided so a Rust producer (or the reverse-direction golden test) can emit -/// the same sparse wire form Go does. -pub fn encode_sparse_registers(regs: &[u8]) -> HllSparseRegisters { - let mut packed: Vec = Vec::new(); - let mut prev: usize = 0; - for (i, &v) in regs.iter().enumerate() { - if v == 0 { - continue; - } - write_uvarint(&mut packed, (i - prev) as u64); - write_uvarint(&mut packed, v as u64); - prev = i; - } - HllSparseRegisters { - num_registers: regs.len() as u32, - packed, - } -} - -/// Append one unsigned LEB128 varint to `out`. Matches Go's -/// `encoding/binary.PutUvarint`. -fn write_uvarint(out: &mut Vec, mut v: u64) { - while v >= 0x80 { - out.push((v as u8) | 0x80); - v >>= 7; - } - out.push(v as u8); -} - -/// Read one unsigned LEB128 varint, returning (value, bytes_consumed). Matches -/// Go's `encoding/binary.Uvarint`. Returns None on truncation / overflow. -fn read_uvarint(buf: &[u8]) -> Option<(u64, usize)> { - let mut result: u64 = 0; - let mut shift = 0u32; - for (i, &b) in buf.iter().enumerate() { - if i >= 10 { - return None; // varint too long for u64 - } - if shift == 63 && b > 1 { - return None; // overflow - } - result |= ((b & 0x7f) as u64) << shift; - if b < 0x80 { - return Some((result, i + 1)); - } - shift += 7; - } - None -} - -impl MessagePackCodec for HllSketch { - fn to_msgpack(&self) -> Result, MsgPackError> { - let kind_id = wire_kind_id(self.variant)?; - let metadata = rmp_serde::to_vec_named(&standard_hll_metadata(self.precision))?; - let payload = if self.variant == HllVariant::Hip { - rmp_serde::to_vec(&HllPayloadHip { - registers: self.registers.clone(), - hip_kxq0: self.hip_kxq0, - hip_kxq1: self.hip_kxq1, - hip_est: self.hip_est, - })? - } else { - rmp_serde::to_vec(&HllPayloadPlain { - registers: self.registers.clone(), - })? - }; - Ok(envelope::encode(kind_id, &metadata, &payload)) - } - - fn from_msgpack(bytes: &[u8]) -> Result { - let (kind_id, metadata, payload) = envelope::split(bytes) - .map_err(|msg| MsgPackError::Decode(rmp_serde::decode::Error::Uncategorized(msg)))?; - - // Validate the hash spec is the standard profile (precision is read from - // the metadata, so compare against a standard block built with it). - let meta: HllMetadata = rmp_serde::from_slice(metadata)?; - if meta != standard_hll_metadata(meta.precision) { - return Err(MsgPackError::Decode( - rmp_serde::decode::Error::Uncategorized( - "ASAPv1 HLL envelope: hash metadata mismatch".to_string(), - ), - )); - } - - let variant = if kind_id == HLL_KIND_CLASSIC { - HllVariant::Regular - } else if kind_id == HLL_KIND_ERTL_MLE { - HllVariant::Datafusion - } else if kind_id == HLL_KIND_HIP { - HllVariant::Hip - } else { - return Err(MsgPackError::Decode( - rmp_serde::decode::Error::Uncategorized(format!( - "ASAPv1 HLL envelope: unsupported kind_id {kind_id:?}" - )), - )); - }; - - let (registers, hip_kxq0, hip_kxq1, hip_est) = if variant == HllVariant::Hip { - let p: HllPayloadHip = rmp_serde::from_slice(payload)?; - (p.registers, p.hip_kxq0, p.hip_kxq1, p.hip_est) - } else { - let p: HllPayloadPlain = rmp_serde::from_slice(payload)?; - (p.registers, 0.0, 0.0, 0.0) - }; - - // Fail closed if the register bin does not match `2^precision` - // (doc §2 validation rule 3). `checked_shl` avoids a shift-overflow - // panic on a crafted out-of-range `precision`. - let expected = 1usize.checked_shl(meta.precision).ok_or_else(|| { - MsgPackError::Decode(rmp_serde::decode::Error::Uncategorized(format!( - "ASAPv1 HLL envelope: precision {} out of range", - meta.precision - ))) - })?; - if registers.len() != expected { - return Err(MsgPackError::Decode( - rmp_serde::decode::Error::Uncategorized(format!( - "ASAPv1 HLL envelope: {} registers, expected 2^{} = {expected}", - registers.len(), - meta.precision - )), - )); - } - - Ok(HllSketch::from_raw( - variant, - meta.precision, - registers, - hip_kxq0, - hip_kxq1, - hip_est, - )) - } -} - -impl MessagePackCodec for HllSketchDelta { - /// Encodes the register delta as the 2-element MessagePack array - /// `[ reg_idx:[]u32, reg_val:[]u8 ]` — parallel arrays of register - /// index → new (max) value — byte-identical to Rust - /// `rmp_serde::to_vec(&(Vec, Vec))` (compact mode). The - /// `Vec` serializes as a msgpack array-of-int (not `bin`), matching - /// the Go `asapmsgpack` register-array convention. - fn to_msgpack(&self) -> Result, MsgPackError> { - let reg_idx: Vec = self.updates.iter().map(|(i, _)| *i).collect(); - let reg_val: Vec = self.updates.iter().map(|(_, v)| *v).collect(); - Ok(rmp_serde::to_vec(&(reg_idx, reg_val))?) - } - - fn from_msgpack(bytes: &[u8]) -> Result { - let (reg_idx, reg_val): (Vec, Vec) = rmp_serde::from_slice(bytes)?; - Ok(HllSketchDelta { - updates: reg_idx.into_iter().zip(reg_val).collect(), - }) - } -} - -#[cfg(test)] -mod tests { - use super::*; - - /// Ranks are drawn from a 64-bit hash, so a register state whose estimate - /// runs past the 32-bit range must still follow the HyperLogLog formula - /// rather than saturate or turn into a NaN. - #[test] - fn estimate_holds_past_the_32_bit_range() { - for rank in [12u8, 16, 18, 20] { - let mut sketch = HllSketch::new(HllVariant::Regular, 14); - sketch.registers.fill(rank); - let m = sketch.registers.len() as f64; - let alpha_m = 0.7213 / (1.0 + 1.079 / m); - let expected = alpha_m * m * 2f64.powi(rank as i32); - let estimate = sketch.estimate(); - let error = (estimate - expected).abs() / expected; - assert!( - error <= 1e-9, - "rank {rank} estimate {estimate} deviates from {expected} by {error:.6}" - ); - } - } - - #[test] - fn msgpack_delta_against_empty_round_trips() { - let mut w = HllSketch::new(HllVariant::Regular, 12); - for i in 0..2000u64 { - w.update(&i.to_le_bytes()); - } - let empty = HllSketch::new(HllVariant::Regular, 12); - let bytes = w.compute_delta_msgpack(&empty, 1); - let mut recon = HllSketch::new(HllVariant::Regular, 12); - recon.apply_delta_msgpack_bytes(&bytes).unwrap(); - // Register delta over an empty base carries every non-zero register, - // so reconstructed registers are identical. - assert_eq!(recon.registers, w.registers); - let (got, want) = (recon.estimate(), w.estimate()); - assert!((got - want).abs() <= want * 0.001, "got={got} want={want}"); - } - - #[test] - fn msgpack_delta_wire_layout_is_parallel_arrays() { - use crate::message_pack_format::MessagePackCodec; - let mut w = HllSketch::new(HllVariant::Regular, 4); - w.update(b"a"); - w.update(b"b"); - let empty = HllSketch::new(HllVariant::Regular, 4); - let bytes = w.compute_delta_msgpack(&empty, 1); - let mut reg_idx: Vec = Vec::new(); - let mut reg_val: Vec = Vec::new(); - for (i, &v) in w.registers.iter().enumerate() { - if v > 0 { - reg_idx.push(i as u32); - reg_val.push(v); - } - } - let expected = rmp_serde::to_vec(&(reg_idx, reg_val)).unwrap(); - assert_eq!(bytes, expected); - let d = HllSketchDelta::from_msgpack(&bytes).unwrap(); - assert_eq!( - d.updates.len(), - w.registers.iter().filter(|&&v| v > 0).count() - ); - } - - #[test] - fn test_new_empty() { - let h = HllSketch::new(HllVariant::Regular, 4); - assert_eq!(h.registers.len(), 16); - assert!(h.registers.iter().all(|&r| r == 0)); - } - - #[test] - fn test_merge_register_wise_max() { - let mut a = HllSketch::from_raw(HllVariant::Regular, 2, vec![1, 5, 3, 7], 0.0, 0.0, 0.0); - let b = HllSketch::from_raw(HllVariant::Regular, 2, vec![4, 2, 6, 0], 0.0, 0.0, 0.0); - a.merge(&b).unwrap(); - assert_eq!(a.registers, vec![4, 5, 6, 7]); - } - - #[test] - fn test_apply_delta_max_semantics() { - let mut h = HllSketch::from_raw(HllVariant::Regular, 2, vec![1, 5, 3, 7], 0.0, 0.0, 0.0); - let delta = HllSketchDelta { - updates: vec![(0, 4), (1, 2), (2, 6), (3, 0)], - }; - h.apply_delta(&delta).unwrap(); - assert_eq!(h.registers, vec![4, 5, 6, 7]); - } - - #[test] - fn test_apply_delta_out_of_range() { - let mut h = HllSketch::new(HllVariant::Regular, 2); - let delta = HllSketchDelta { - updates: vec![(7, 3)], - }; - assert!(h.apply_delta(&delta).is_err()); - } - - #[test] - fn test_apply_delta_matches_full_merge() { - let base = HllSketch::from_raw(HllVariant::Regular, 2, vec![1, 5, 3, 7], 0.0, 0.0, 0.0); - let addition = HllSketch::from_raw(HllVariant::Regular, 2, vec![4, 0, 6, 0], 0.0, 0.0, 0.0); - let mut via_merge = base.clone(); - via_merge.merge(&addition).unwrap(); - - let delta = HllSketchDelta { - updates: vec![(0, 4), (2, 6)], - }; - let mut via_delta = base; - via_delta.apply_delta(&delta).unwrap(); - assert_eq!(via_delta.registers, via_merge.registers); - } - - #[test] - fn test_merge_variant_mismatch() { - let mut a = HllSketch::new(HllVariant::Regular, 4); - let b = HllSketch::new(HllVariant::Datafusion, 4); - assert!(a.merge(&b).is_err()); - } - - #[test] - fn test_merge_precision_mismatch() { - let mut a = HllSketch::new(HllVariant::Regular, 4); - let b = HllSketch::new(HllVariant::Regular, 5); - assert!(a.merge(&b).is_err()); - } - - #[test] - fn test_merge_refs() { - let a = HllSketch::from_raw(HllVariant::Regular, 1, vec![1, 0], 0.0, 0.0, 0.0); - let b = HllSketch::from_raw(HllVariant::Regular, 1, vec![0, 3], 0.0, 0.0, 0.0); - let c = HllSketch::from_raw(HllVariant::Regular, 1, vec![2, 2], 0.0, 0.0, 0.0); - let merged = HllSketch::merge_refs(&[&a, &b, &c]).unwrap(); - assert_eq!(merged.registers, vec![2, 3]); - } - - #[test] - fn test_update_then_estimate_within_2pct() { - let n: usize = 50_000; - let mut h = HllSketch::new(HllVariant::Regular, 12); - for i in 0..n { - let key = format!("key-{i}"); - h.update(key.as_bytes()); - } - let est = h.estimate(); - let rel_err = (est - n as f64).abs() / n as f64; - assert!( - rel_err < 0.02, - "HLL estimate {est} not within 2% of {n} (rel_err {rel_err:.4})", - ); - } - - #[test] - fn test_estimate_empty_is_zero() { - let h = HllSketch::new(HllVariant::Regular, 4); - assert_eq!(h.estimate(), 0.0); - } - - #[test] - fn test_msgpack_round_trip() { - let original = HllSketch::from_raw( - HllVariant::Hip, - 3, - vec![0, 1, 2, 3, 4, 5, 6, 7], - 1.0, - 2.0, - 3.0, - ); - let bytes = original.to_msgpack().unwrap(); - assert!(bytes.starts_with(HLL_WRAPPER_MAGIC)); - let decoded = HllSketch::from_msgpack(&bytes).unwrap(); - assert_eq!(decoded.registers, original.registers); - assert_eq!(decoded.precision, original.precision); - assert_eq!(decoded.hip_kxq0, 1.0); - } - - /// Cross-language byte-parity guard against - /// `sketchlib-go::HyperLogLog.SerializePortable` for the - /// deterministic input stream `(1..=50i32).map(|i| (i as f64) - /// .to_le_bytes())`. - #[test] - fn test_update_then_envelope_matches_sketchlib_go_bytes() { - use crate::proto::sketchlib::{ - HyperLogLogState, SketchEnvelope, sketch_envelope::SketchState, - }; - use prost::Message; - - let mut sk = HllSketch::new(HllVariant::Datafusion, 14); - for i in 1..=50i32 { - let v = i as f64; - sk.update(&v.to_le_bytes()); - } - - let state = HyperLogLogState { - variant: sk.wire_proto_variant(), - precision: sk.precision, - registers: sk.registers.clone(), - hip_kxq0: sk.hip_kxq0, - hip_kxq1: sk.hip_kxq1, - hip_est: sk.hip_est, - // This parity golden predates the sparse field and intentionally - // pins the DENSE wire form, so leave the sparse field unset. - registers_sparse: None, - }; - let envelope = SketchEnvelope { - format_version: 1, - producer: None, - hash_spec: None, - // Golden pins the pre-sampling wire form; 0.0 is the proto3 default - // (dual-read as 1.0) so the encoded bytes are unchanged. - sample_p: 0.0, - sketch_state: Some(SketchState::Hll(state)), - }; - let mut got = Vec::with_capacity(envelope.encoded_len()); - envelope.encode(&mut got).expect("prost encode"); - - const GOLDEN_HEX: &str = include_str!("../../sketches/testdata/hll_envelope_golden.hex"); - let want = decode_hex(GOLDEN_HEX); - assert_eq!( - got.len(), - want.len(), - "HLL envelope length differs: got {} bytes, want {} bytes", - got.len(), - want.len(), - ); - assert_eq!( - got, want, - "HLL envelope bytes diverge from sketchlib-go golden" - ); - } - - fn decode_hex(s: &str) -> Vec { - let s = s.trim(); - s.as_bytes() - .chunks(2) - .map(|pair| { - let high = hex_nibble(pair[0]); - let low = hex_nibble(pair[1]); - (high << 4) | low - }) - .collect() - } - - fn hex_nibble(c: u8) -> u8 { - match c { - b'0'..=b'9' => c - b'0', - b'a'..=b'f' => c - b'a' + 10, - b'A'..=b'F' => c - b'A' + 10, - _ => panic!("non-hex byte {}", c as char), - } - } - - // ----- sparse full-state encoding ----- - - use crate::proto::sketchlib::HyperLogLogState; - - /// A handful of (index, value) pairs that exercise small and large index - /// deltas plus the maximum register value. - fn sample_regs(n: usize) -> Vec { - let mut regs = vec![0u8; n]; - regs[0] = 1; // first delta is absolute index 0 - regs[1] = 7; // adjacent index (delta 1) - regs[300] = 51; // max p=14 register value (Q+1) - regs[8191] = 12; // mid-range, multi-byte index delta - regs[n - 1] = 3; // last register - regs - } - - #[test] - fn sparse_round_trip_exact() { - let regs = sample_regs(16384); - let sp = encode_sparse_registers(®s); - assert_eq!(sp.num_registers, 16384); - let decoded = decode_sparse_registers(&sp).unwrap(); - assert_eq!(decoded, regs, "sparse round trip must be exact"); - } - - #[test] - fn registers_from_state_reads_both() { - let regs = sample_regs(16384); - - // SPARSE: registers_sparse set, registers empty. - let sparse_state = HyperLogLogState { - variant: 2, - precision: 14, - registers: Vec::new(), - hip_kxq0: 0.0, - hip_kxq1: 0.0, - hip_est: 0.0, - registers_sparse: Some(encode_sparse_registers(®s)), - }; - assert_eq!(registers_from_state(&sparse_state).unwrap(), regs); - - // DENSE: registers set, registers_sparse None. - let dense_state = HyperLogLogState { - variant: 2, - precision: 14, - registers: regs.clone(), - hip_kxq0: 0.0, - hip_kxq1: 0.0, - hip_est: 0.0, - registers_sparse: None, - }; - assert_eq!(registers_from_state(&dense_state).unwrap(), regs); - } - - #[test] - fn sparse_empty_is_all_zero() { - let regs = vec![0u8; 16384]; - let sp = encode_sparse_registers(®s); - assert!(sp.packed.is_empty()); - assert_eq!(decode_sparse_registers(&sp).unwrap(), regs); - } - - #[test] - fn sparse_rejects_out_of_range_index() { - // packed = uvarint(delta=5) , uvarint(value=3) but num_registers=4. - let sp = HllSparseRegisters { - num_registers: 4, - packed: vec![5, 3], - }; - assert!(decode_sparse_registers(&sp).is_err()); - } - - /// `compute_delta` against an EMPTY snapshot reconstructs the window's - /// full register state when its bytes are applied to a fresh empty - /// sketch (register max-merge round-trip). HLL deltas are lossless — - /// every increased register is carried. - #[test] - fn test_compute_delta_against_empty_round_trips() { - let mut window = HllSketch::new(HllVariant::Datafusion, 14); - for i in 0..5000u64 { - window.update(&i.to_le_bytes()); - } - let empty = HllSketch::new(HllVariant::Datafusion, 14); - - let delta_bytes = window.compute_delta(&empty, 0); - - let mut reconstructed = HllSketch::new(HllVariant::Datafusion, 14); - reconstructed.apply_delta_bytes(&delta_bytes).unwrap(); - - assert_eq!(reconstructed.registers, window.registers); - } - - /// A delta computed between two non-empty snapshots reconstructs the - /// current sketch when applied to the base (max-merge of the registers - /// that increased). - #[test] - fn test_compute_delta_then_apply_matches_current() { - let mut base = HllSketch::new(HllVariant::Datafusion, 12); - for i in 0..2000u64 { - base.update(&i.to_le_bytes()); - } - let mut current = base.clone(); - for i in 2000..5000u64 { - current.update(&i.to_le_bytes()); - } - - let delta_bytes = current.compute_delta(&base, 0); - let mut reconstructed = base.clone(); - reconstructed.apply_delta_bytes(&delta_bytes).unwrap(); - assert_eq!(reconstructed.registers, current.registers); - } - - /// Cross-language byte-parity guard: `compute_delta` against an empty - /// snapshot must emit bytes identical to the Go reference - /// implementation's `SerializeRegisterDelta(ComputeRegisterDelta(empty, - /// current))` for the same precision-14 sketch fed `(1..=50)` as f64 - /// little-endian byte values. A delta-against-empty carries every - /// non-zero register as an `(index, value)` update; the golden hex was - /// captured from a `proto.Marshal` of the Go reference's `HLLDelta`. - #[test] - fn test_compute_delta_matches_go_golden_bytes() { - let mut current = HllSketch::new(HllVariant::Datafusion, 14); - for i in 1..=50i32 { - let v = i as f64; - current.update(&v.to_le_bytes()); - } - let empty = HllSketch::new(HllVariant::Datafusion, 14); - let got = current.compute_delta(&empty, 0); - - // Captured from the Go reference implementation's - // SerializeRegisterDelta(ComputeRegisterDelta(empty, current)) for the - // same input, with the packed (index_delta, value) HLLDelta encoding. - const GOLDEN_HEX: &str = "0a81015801bb0402800c015001970401dd0301b10101b2080132016a01a901022f018402020902a10a029a03027501c7020428042b05ec0702a60102850205c2030258021101fe030128022002a60403cd0102e7060211060b0334020b069c0101ff06012a015e0581020170041201f80401a70404a60101bc03019e0103cc0201880203"; - let want = decode_hex(GOLDEN_HEX); - assert_eq!( - got, - want, - "HLL delta bytes diverge from the Go reference golden \ - ({} bytes got vs {} bytes want)", - got.len(), - want.len(), - ); - } - - /// Cross-language byte-parity guard for a SUB-WINDOW delta (not against - /// empty): a precision-14 sketch is fed `(0..2000)` as f64 little-endian - /// bytes (the base/snapshot), then `(2000..5000)`; the delta carries only - /// the registers that grew between the two, packed as (index_delta, value) - /// pairs. Must be byte-identical to the Go reference implementation's - /// `SerializeRegisterDelta(ComputeRegisterDelta(base, current))`. - #[test] - fn test_compute_subwindow_delta_matches_go_golden_bytes() { - let mut base = HllSketch::new(HllVariant::Datafusion, 14); - for i in 0..2000i32 { - base.update(&(i as f64).to_le_bytes()); - } - let mut current = base.clone(); - for i in 2000..5000i32 { - current.update(&(i as f64).to_le_bytes()); - } - let got = current.compute_delta(&base, 0); - - // Captured from the Go reference implementation's - // SerializeRegisterDelta(ComputeRegisterDelta(base, current)) for the - // same input, with the packed (index_delta, value) HLLDelta encoding. - const GOLDEN_HEX: &str = "0ae627000402030401020302030c0109020502040201011108100202021304030101010b010e02040102030b0407020201050405021404060105010301010101020101030306020201040102011101010407010106080104030101010103020c0204021105040108010101180201030f010b01010103010e01120201030101010303030301030107030a0108010a0108012106060309010302070203010202030101010e0102020a020102060101020a01040101011702020203021003060103030e02110206010102040604010d010a050c011104010104050a0103020f0203010305050209010301020401020805010101010201020104010201060501040601020203020601020304010401020101010601010419010e010503050107020302010102010d010501110105050503020104050201020215050401030101020d041a010601190209020301020404070e01090202010b01030107040a0103010403010117050602040601010301040101030203030105010401020108030b0517020c01010506010402030103040704010102020d010a011201060304020c010c020e010b020702070101040101010104021504070109040804060502021802030104021605010107020603010101010201100104010605020205010e010406040102020f0203020a02040201030201010303010401030302010a030201060514020d02050104010201050103020802040105020602080213020d02020105020301140204010a0509060d03070206020902070101080a0202010a010e0102020503080201010e0101010403080104020d0102010501040403060401050405011203010101040e010202040303020c010203090201030d010a0306020201030208020e0103010701030113020202030501020c010b0101040502010105010203060105020a0315050b030c0101010f060405040204030b03010502020101080302020c010b02060403010404030102010f020f0107030601040109060301040301010703040303020801010104010b011101070102011101010102030201070106010601030201020301030302020e030501190108030201040205020503030106030a01050112020102010107030b02060209050201010304010502030106060601050108020602080201010905080205010204010203020303060205010102070201010308160304040f02010105020605070407030b0104010701010205020c02060103010b040c0102030202060102010e030102070108030301030205010101040301030303010203020802190105020601120101020701010101010302080303010f0104020202060402020403010103010503060203030701010105030903030508010101020103020305060605030e0203020603010102060f010201040202030504030103020f01130204010201020104010102010401010403050201030502040306060d0302010a010a02080103020101020208030301030102010403030305011003030206010f01010609030302020102030402020102010103020408010502130306010303060102010c01030202030401030101020101070202010901150104010f010401080107010601010206020b0201010d040303070202011403090210010302090403010403050201010b05010401010301090102010c010b04040206010d0102010b05040301020702020302010d0204030102110106010803020108030101070104020104040201030605050102040101010302010b01020103030e031101030101020103070112010403010101010b011001060313030301070203030803050101030601050207030a0104030302020202030c010101080103010c0102020404020102030c01110102030901140307010602020303020401070102010b020402030707010101010201070705130102011b0201060501010101010201020109010601020205010b0103040504070202010501080108020703030104030401010201030a010201030206010101020104020d02040102010102040301040605100201020801030306020e010402110106030602080105011101060106020d0216010702050301020101040409010403010201040102020205010401070207010d0205040704050202021201030510040303020602020703090204060102070102010301020101010401200101030202040103021301010107050a010702030311010303020103020b010505100302060102010104020a010402070210010704030304050a02010409010101010106030901030110031002020103010d0301020202040101020806080105010401060126020201080109020e0102030503120106010901010301010b01040201020402020115020d0108030f010c0303020c0101010602080504030802050202030f030202050404010603030208010a0204010d02020202030a0209030e030304030101060d0104020702010209020a0406010101070204020b0201030d021201020106031a0504050202040102010101020103030601080102020f021a020201120112020d02030405020403090304020102010108010203080101010704060104010902010401020203040107010e02070102060601080105010101060204010e010c020d010601050108010801020508010102070302030201030101010b020b02060107010c020b01020104020302060203040c02020105010301030202020902090103030503070606010103040216020401050205020404020208010b0207010502050108010502030102040f0206021203050105010a010a010b04090101030401040c03010102040104020c04090203020301040202020101060205020d010303040202020b010c010f0205030d010301050202030a02030309020203110103010102090106020402010202010303020206010a0104010601030208020803080110020702030205050201130102022102060204021504040102010a040c0401010503030207010701050103010d0104040601100201010201020302010a010501110102080504010106030101060601040a01050202080203020509010c010102090203010d0101020102120713021104070204010201010208020502060408010309020101020d0104010206030102010201010202050202040101010102050102030201060209020b01020105040202060401020f010103090504030101030111020e031001110204020602010201011101050204010a01040402030501010202010602070503011002020406010401030103020f0403050e0104020201010101040b0105030103010106020803080201010101010202050404010103011c0306030201010105010202060206010102020101010703010502010c010303020206030c021601130121020f03020307030e0109010602040102011f01090506010b01050402020e01020303020a040201020202010301030204030a010201080404030304110104010403070203060301020107020c050502020109030901020102010801030203090d0104021502010104040504090105030a010503110102020a010203050303040e030502030101030105020108010103020102030103010209030202010205010301010202020804010207010d020d020a010605080306010c0202021101090202030f010101040109020f0301020501010103020501040109021202080601070f0204020105070119030101110208040a0203020c010201070503030b0108030201040204020402010203010c022603040505010e04010103010702010109050b020801020405020e01140204020201050406020a020301060404010c010102050106020207010102010801050202020105040103010108010113050d0109020703070205010f0703050b0112010f02010103070b010201050101030201010203010705090605020f0103040b010801020210020501060204040c010104070105020a020104040201010403040109010c0101050a0305010301090108010b030b0201010601030202030e020203030101050202010106010d010d010601020101010c0304010901010103010401030201010902010101050b01010107010105020105020c01030405011404020104030a01030105010801050202020a0107010c04150105030301050401020603020203040b01100101010301030103070101010215020302010101010101080203020a010a021d031302030104020201030305030b01030203030b03010212020302030202050d010703040202040501040101030203060405031802010206020402020109020202070305021c01010105031c010503050305030d010e01090105030401070104010302010103010201100103020b0107050301090104010203030205030103010105010504080104020302020111050601050404022202070102010201090705080b01030109012903090206030104040107010102070201040f020201030103030b020804020207012d0105010301080103020201030202050501070105010202010304020303020401030802020103020102010501010c0106021201050317010a0304010101020207090102030205010b0204010b010801080312010f01150102020802040102011102080101040a0102010c020403020203010603050203010301040111031505040202030e01010102050607050101010502020301010c0210010d0407010903020202010e01050106020401070402010c010602090311060202100206020202020316020103010101010a030301060101010203150201050e04030204021c020c020c030102030101010502060102010d02010411040c0208010101010101040201010205050101060214010901010407020206160101050407050104010101010205050501060103010101060411020302070211010201090203021a0602010f0106030a04010106010301090101070501020107060102010404020b0102010202040110010201050106020901090309020d0105020601080101010501010205030403180401030d0105040802240209010a02020305020502040102010301090303020802040108010405020201040d0403010501020102020503040215010d030902030116010f010d0107030204070204040a0109011002090119031e020502090104031b050605020201010c0207020503010104010c0102010a01060502010c020d020202050101011102050107040f0102010201010106020305020101010402010302010c0306051505010104020802010102070c010803010506011101050207020302040103010301080103080d030c0103040d01010209040501030101010402020301010f01020102010301010401010202050408011c050904080106040401040105010402120102020103030201010a030103020402021303020305020403030201030901010504010902010305010301030203020a0108010702050107020a01160205020a060201030106030e02030103031001070102030d0119020202070505021202030108010101060808010202050108010f0102010701030404020802020214030c0211010b020301080303020105080105010401100106011309040203010103010208030b020102010301010e0107021501080203010402010101010606030204010d0108010e0102010401010105030903020302030c0103010104080101010405030201020c020101070201020102010401010301080102041f01070110030101060107010201060108010d010a030a0114041003020104010302020115011a02070802010301010106020b0101010201050109020703020205010a05050402030b0108010901070201020101020105040401110101020101040103010301010107010201180101020a030201030211011a02110103010102050103040e02160106010b02050a0e0202010203030101010402120305010301030305010805070304010402050102020402040106040105060105010b01110104010c0202020b01020104020e0201020704070206010c02030105010601030106011303070103010c0601010e021a01010704020801020201030c0506010a010203010202030e0311010201050105010b010504040203040a0102010302030302010102090301010403010104010e010501050a04070201060304030b0203010e01030201011b0102020c010701040303020d0604011001130202021304020505020201010105010a010f0106020801030113010802020104040b020104080104010e03090205020905020205010e040306020304010204100106010101150103050501050101010c0102010401050201030203010315010f0111020e0403010702030203030c0101020d02030205030a01040605050a02020101011201060502020201020101010a030f010706040105010101080502020301010608040b050101010104010d0103032001030201010f01040103011003010102060401070107010f01030116030301030205030902020611030102010101010101010201020202050403011b0208010b02030202010e0108010b010e01010508040701010103050604040515020202030104020703060203010601060106020f0101020302040102010c0302010101030105010a03020311020301030201010d03040107020502010109030b0302090801070304010301010402020501010201010501050202040302030305010101040201020802020101020f0107010b020302070313010b06050201030c010b04010106050302030102030401090208010901050307020d030b0309030e01090106010302020202010501040708010e0102010401010101020304010303010505050104020201070107020101040102010e0104011a0105050a030d030f010a02130103030b0107050e0203040403010208030d0701030101080206010a010e0203020501070303011102030203010101050117010e040d040701180201010e0201010402090204012c031c010e0204020502080113010201"; - let want = decode_hex(GOLDEN_HEX); - assert_eq!( - got, - want, - "HLL sub-window delta bytes diverge from the Go reference golden \ - ({} bytes got vs {} bytes want)", - got.len(), - want.len(), - ); - - // Sanity: applying the packed sub-window delta to the base - // reconstructs the current register state (register max-merge). - let mut reconstructed = base.clone(); - reconstructed.apply_delta_bytes(&got).unwrap(); - assert_eq!(reconstructed.registers, current.registers); - } -} diff --git a/src/message_pack_format/portable/mod.rs b/src/message_pack_format/portable/mod.rs index 97e0d372..cc6d0e63 100644 --- a/src/message_pack_format/portable/mod.rs +++ b/src/message_pack_format/portable/mod.rs @@ -2,7 +2,7 @@ //! //! Each per-algorithm submodule holds one sketch's pre-envelope wire type. //! These carry a cross-language contract of their own, on goldens apart -//! from ASAPv1's `asapv1_golden/`: HLL, Count-Min, Count Sketch and KLL pin +//! from ASAPv1's `asapv1_golden/`: Count-Min, Count Sketch and KLL pin //! byte parity against Go's serializers, KLL in both directions; DDSketch's //! golden is a Rust self-pin, not yet reconciled. New code uses ASAPv1: //! prefer a sketch's own `serialize_to_bytes`; do not add callers here. @@ -13,7 +13,6 @@ pub mod countsketch; pub mod countsketch_topk; pub mod ddsketch; pub mod delta_set_aggregator; -pub mod hll; pub mod hydra_kll; pub mod kll; pub mod sampling; diff --git a/src/message_pack_format/portable/sampling.rs b/src/message_pack_format/portable/sampling.rs index 9f62b452..e6951d18 100644 --- a/src/message_pack_format/portable/sampling.rs +++ b/src/message_pack_format/portable/sampling.rs @@ -71,7 +71,6 @@ pub const fn is_quantile_scale_invariant() -> bool { #[cfg(test)] mod tests { use super::*; - use crate::proto::sketchlib::{HyperLogLogState, SketchEnvelope, sketch_envelope::SketchState}; fn env_with_p(p: f64) -> SketchEnvelope { SketchEnvelope { @@ -79,15 +78,7 @@ mod tests { producer: None, hash_spec: None, sample_p: p, - sketch_state: Some(SketchState::Hll(HyperLogLogState { - variant: 2, - precision: 14, - registers: Vec::new(), - hip_kxq0: 0.0, - hip_kxq1: 0.0, - hip_est: 0.0, - registers_sparse: None, - })), + sketch_state: None, } } diff --git a/src/proto/generated/sketchlib.v1.rs b/src/proto/generated/sketchlib.v1.rs index 2c098d3f..8b1303a4 100644 --- a/src/proto/generated/sketchlib.v1.rs +++ b/src/proto/generated/sketchlib.v1.rs @@ -461,38 +461,6 @@ pub struct HllSparseRegisters { #[prost(bytes = "vec", tag = "2")] pub packed: ::prost::alloc::vec::Vec, } -/// HLLDelta carries the registers that increased between two consecutive -/// snapshots. HLL uses max semantics, so only increases are meaningful; -/// at a fixed precision a register can never decrease. Each update is applied -/// losslessly on the receiver via register\[index\] = max(register\[index\], value) -/// — no register update is ever dropped (there is no threshold to apply). -/// -/// The increased registers are varint-packed exactly like -/// HLLSparseRegisters.packed: sorted ascending by index, each register emitted -/// as (index_delta, value) where index_delta = index - prev_index (prev_index -/// starts at 0) and value is the new (larger) register value: -/// -/// for each increased register, in ascending index order: -/// uvarint(index - prev_index) // prev_index starts at 0; deltas are >= 0 -/// uvarint(value) // 1..=Q+1, always 1 byte in practice -/// -/// A single-emit delta against the all-zero snapshot is therefore the same size -/// as the full sparse frame, and a sub-window delta is strictly smaller (it -/// packs only the registers that actually grew). This replaces the previous -/// per-register sub-message encoding, eliminating ~6–8 bytes of tag/length -/// overhead per register. -/// -/// This message is byte-identical to the Go reference implementation's HLLDelta -/// so the two runtimes emit byte-identical delta frames for identical window -/// state (cross-language byte parity). -#[derive(Clone, PartialEq, ::prost::Message)] -pub struct HllDelta { - /// Varint-packed (index_delta, value) pairs for the registers that increased, - /// in ascending index order. Same layout as HLLSparseRegisters.packed. Empty - /// when no register changed. - #[prost(bytes = "vec", tag = "1")] - pub packed_updates: ::prost::alloc::vec::Vec, -} /// HLLVariant identifies which HLL estimator algorithm the registers belong to. #[derive(Clone, Copy, Debug, PartialEq, Eq, Hash, PartialOrd, Ord, ::prost::Enumeration)] #[repr(i32)] @@ -892,7 +860,7 @@ pub struct SketchEnvelope { /// The sketch payload. Exactly one field must be set. #[prost( oneof = "sketch_envelope::SketchState", - tags = "10, 11, 12, 13, 14, 15, 16, 17, 18" + tags = "10, 11, 13, 14, 15, 16, 17, 18" )] pub sketch_state: ::core::option::Option, } @@ -905,8 +873,6 @@ pub mod sketch_envelope { CountMin(super::CountMinState), #[prost(message, tag = "11")] CountSketch(super::CountSketchState), - #[prost(message, tag = "12")] - Hll(super::HyperLogLogState), #[prost(message, tag = "13")] Kll(super::KllState), #[prost(message, tag = "14")] diff --git a/src/sketches/hll.rs b/src/sketches/hll.rs index 5464cd0a..08bb8c05 100644 --- a/src/sketches/hll.rs +++ b/src/sketches/hll.rs @@ -42,10 +42,6 @@ use std::marker::PhantomData; mod wire; pub use wire::HllWireVariant; -pub(crate) use wire::{ - HLL_KIND_CLASSIC, HLL_KIND_ERTL_MLE, HLL_KIND_HIP, HllMetadata, HllPayloadHip, HllPayloadPlain, - standard_hll_metadata, -}; /// Generic HyperLogLog sketch parameterized by estimation variant, register storage, and hasher. #[derive(Clone, Debug, Serialize, Deserialize)] @@ -140,8 +136,13 @@ impl HyperLogLogImpl { fn new_base() -> Self { + Self::from_storage(Registers::default()) + } + + /// Creates a sketch from an existing register storage instance. + pub fn from_storage(registers: Registers) -> Self { Self { - registers: Registers::default(), + registers, _marker: PhantomData, _hasher: PhantomData, } @@ -340,6 +341,18 @@ impl HyperLogLogHIPImpl { est: 0.0, } } + + /// Creates a sketch from register storage and the HIP running state + /// (`kxq0`, `kxq1`, `est`). + pub fn from_storage(registers: Registers, kxq0: f64, kxq1: f64, est: f64) -> Self { + Self { + registers, + kxq0, + kxq1, + est, + } + } + /// Inserts a pre-hashed value, updating both the register and the HIP running estimate. #[inline(always)] pub fn insert_with_hash(&mut self, hashed: u64) { diff --git a/src/sketches/hll/wire.rs b/src/sketches/hll/wire.rs index 56a0ff84..bb4e0f5e 100644 --- a/src/sketches/hll/wire.rs +++ b/src/sketches/hll/wire.rs @@ -91,12 +91,6 @@ pub(crate) fn hll_metadata(precision: u32) -> HllMetadata { } } -/// The standard ProjectASAP profile metadata (the [`DefaultXxHasher`] profile). -/// Used by the portable path, which only represents standard-profile sketches. -pub(crate) fn standard_hll_metadata(precision: u32) -> HllMetadata { - hll_metadata::(precision) -} - /// Validate the envelope for a known target and return the raw payload bytes. /// The metadata is checked against the profile of `H`, so bytes hashed under a /// different profile are rejected (fail closed). @@ -215,8 +209,9 @@ impl HyperLogLogHIPImpl { /// than bytes that would be refused on decode. pub fn serialize_to_bytes(&self) -> Result, RmpEncodeError> { let registers = registers_to_bytes(&self.registers)?; - let metadata = - rmp_serde::to_vec_named(&standard_hll_metadata(Registers::PRECISION as u32))?; + let metadata = rmp_serde::to_vec_named(&hll_metadata::( + Registers::PRECISION as u32, + ))?; let payload = rmp_serde::to_vec(&HllPayloadHip { registers, hip_kxq0: self.kxq0, @@ -247,9 +242,9 @@ impl HyperLogLogHIPImpl { #[cfg(test)] mod tests { use super::*; + use crate::DataInput; use crate::sketches::hll::{HyperLogLog, HyperLogLogHIP, HyperLogLogHIPP12, HyperLogLogP12}; use crate::structures::fixed_structure::{HllBucketListP12, HllBucketListP14}; - use crate::{DataInput, HllBucketList}; const ERROR_TOLERANCE: f64 = 0.02; const SERDE_SAMPLE: usize = 100_000; @@ -444,52 +439,6 @@ mod tests { assert_eq!(decoded.est, sketch.est); } - /// Drift guard: the native path and the (deprecated) portable path must emit - /// byte-identical ASAPv1 envelopes. Keep until golden byte-vectors exist. - #[test] - fn native_and_portable_hll_bytes_match() { - use crate::message_pack_format::MessagePackCodec; - use crate::message_pack_format::portable::hll::{HllSketch, HllVariant}; - - // Ertl-MLE / Datafusion: registers-only payload. - let mut native = HyperLogLog::::default(); - for v in 0..1000 { - native.insert(&DataInput::U64(v)); - } - let native_bytes = native.serialize_to_bytes().expect("native serialize"); - let portable = HllSketch::from_raw( - HllVariant::Datafusion, - HllBucketList::PRECISION as u32, - native.registers_as_slice().to_vec(), - 0.0, - 0.0, - 0.0, - ); - assert_eq!( - native_bytes, - portable.to_msgpack().expect("portable serialize") - ); - - // HIP: registers + three running scalars. - let mut hip = HyperLogLogHIP::default(); - for v in 0..1000 { - hip.insert(&DataInput::U64(v)); - } - let hip_bytes = hip.serialize_to_bytes().expect("native serialize"); - let portable_hip = HllSketch::from_raw( - HllVariant::Hip, - HllBucketList::PRECISION as u32, - hip.registers.as_slice().to_vec(), - hip.kxq0, - hip.kxq1, - hip.est, - ); - assert_eq!( - hip_bytes, - portable_hip.to_msgpack().expect("portable serialize") - ); - } - // A test-only custom hasher: it hashes exactly like `DefaultXxHasher` // (delegation) but declares a DIFFERENT `HashProfile` (distinct profile id // and seed list). Serialization is derived from the profile, so an @@ -590,7 +539,7 @@ mod tests { precision: u32, bogus_field: u8, // key not in HllMetadata } - let std = standard_hll_metadata(14); + let std = hll_metadata::(14); let extra = WithExtra { metadata_version: std.metadata_version, hash_profile_id: std.hash_profile_id.clone(), @@ -626,7 +575,8 @@ mod tests { /// Builds a well-framed envelope around a crafted register bin. fn crafted_plain(kind_id: &[u8], precision: u32, registers: Vec) -> Vec { - let metadata = rmp_serde::to_vec_named(&standard_hll_metadata(precision)).expect("meta"); + let metadata = + rmp_serde::to_vec_named(&hll_metadata::(precision)).expect("meta"); let payload = rmp_serde::to_vec(&HllPayloadPlain { registers }).expect("payload"); envelope::encode(kind_id, &metadata, &payload) } @@ -682,7 +632,7 @@ mod tests { ); // The HIP payload carries the same bin and is held to the same bound. - let metadata = rmp_serde::to_vec_named(&standard_hll_metadata(14)).expect("meta"); + let metadata = rmp_serde::to_vec_named(&hll_metadata::(14)).expect("meta"); let payload = rmp_serde::to_vec(&HllPayloadHip { registers: bin(p14, P14_MAX + 1), hip_kxq0: p14 as f64, diff --git a/src/sketches/testdata/hll_envelope_golden.hex b/src/sketches/testdata/hll_envelope_golden.hex deleted file mode 100644 index cd7760ec..00000000 --- a/src/sketches/testdata/hll_envelope_golden.hex +++ /dev/null @@ -1 +0,0 @@ -0801628880010802100e1a80800100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000200000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000002000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000020000000000000000020000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000002000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000200000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000400000000000000000000000000000000000000000000000000000000000000000000000000000004000000000000000000000000000000000000000000000000000000000000000000000000000000000000050000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000002000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000020000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000050000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000020000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000200000000000000000000000000000000010000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000000000000000000000000000000000200000000000000000000000000000000000000000000000000000000000000020000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000300000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000002000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000020000000000000000000000000000000006000000000000000000000300000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000002000000000000000000000600000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000001000000000000000000000000000000000000000000000000000000000000000000000000000000000001000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000050000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000001000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000040000000000000000000000000000000000010000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000001000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000400000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000001000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000001000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000300000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000030000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 \ No newline at end of file diff --git a/tests/PBT_COVERAGE.md b/tests/PBT_COVERAGE.md index cddee44f..a7d1d923 100644 --- a/tests/PBT_COVERAGE.md +++ b/tests/PBT_COVERAGE.md @@ -2,8 +2,8 @@ One `proptest` law per line. Each compares the sketch against an answer computed without it; what that answer is comes first in every section. One -file per sketch under `tests/pbt/`. `cargo test --test pbt` runs 367 laws; -with `--all-features`, 435, adding the four **experimental** modules. Every +file per sketch under `tests/pbt/`. `cargo test --test pbt` runs 365 laws; +with `--all-features`, 433, adding the four **experimental** modules. Every law was mutation-checked: break the implementation, the law goes red. Shared generators, `grid` and `round_trip!` live in `support.rs`. @@ -228,7 +228,7 @@ Compared against: the offers sorted and truncated. `HHHeap` is path dependent on - one offer per key: the `k` largest counts resident - `update` returns true iff the key was resident or there was room, i.e. nothing was displaced -## hll.rs — 101 +## hll.rs — 99 Models use only what the paper fixes; register indexing and rank mapping stay free. For `Classic` and `ErtlMLE`, `estimate` is a function of the registers, so laws relate estimates rather than restate the formula. Registers `<= 40`, so `sum(2^-M[j])` is exact in f64. @@ -252,10 +252,8 @@ Models use only what the paper fixes; register indexing and rank mapping stay fr - `estimate` non-decreasing; a repeated key adds 0 - order matters: a descending run into one register estimates exactly 1, an ascending run `> 1` - an arrival that raises a register adds `m / sum(2^-M[j])` over the prior registers; one that raises nothing adds 0; both sides exact in f64, compared after truncation to `usize` -- portable `HllSketch` (5) - - merge commutative, idempotent, empty identity, `== S(xs ++ ys)`; precisions 4-11 - - msgpack keeps variant, precision, registers, estimate; `Regular`, `Datafusion`, `Hip`; precisions 4-13 - - ASAPv1 on `HyperLogLog` keeps registers and estimate; empty included +- wire (3) + - ASAPv1 on `HyperLogLog` and `HyperLogLog` keeps registers and estimate, on `HyperLogLogHIP` keeps estimate; re-encode byte-identical; empty included ## hydra.rs — 7 diff --git a/tests/asapv1_golden.rs b/tests/asapv1_golden.rs index 30b41651..b820c978 100644 --- a/tests/asapv1_golden.rs +++ b/tests/asapv1_golden.rs @@ -9,8 +9,8 @@ //! See `asapv1_golden/README.md`. use asap_sketchlib::{ - Classic, Count, CountMin, ErtlMLE, FastPath, HllSketch, HllVariant, HyperLogLogHIPP12, - HyperLogLogP12, KLL, MessagePackCodec, RegularPath, Vector2D, + Classic, Count, CountMin, ErtlMLE, FastPath, HllBucketListP12, HllRegisterStorage, + HyperLogLogHIPP12, HyperLogLogP12, KLL, RegularPath, Vector2D, }; fn decode_hex(s: &str) -> Vec { @@ -43,6 +43,12 @@ fn p12_registers() -> Vec { r } +fn p12_storage() -> HllBucketListP12 { + let mut storage = HllBucketListP12::default(); + storage.as_mut_slice().copy_from_slice(&p12_registers()); + storage +} + const I64_VALS: [[i64; 3]; 2] = [[0, 1, 127], [128, 300, 65536]]; const F64_VALS: [[f64; 3]; 2] = [[0.0, 1.5, 2.25], [3.75, 4.125, 5.0625]]; @@ -59,65 +65,41 @@ const CS_VALS: [[i64; 4]; 2] = [[0, 127, 128, 65536], [-1, -33, -32768, -2147483 #[test] fn hll_classic_p12_matches_golden() { let want = decode_hex(GOLDEN_CLASSIC); - let regs = p12_registers(); - // Build known state and serialize. - let got = HllSketch::from_raw(HllVariant::Regular, 12, regs.clone(), 0.0, 0.0, 0.0) - .to_msgpack() - .expect("serialize"); + let sketch = HyperLogLogP12::::from_storage(p12_storage()); + let got = sketch.serialize_to_bytes().expect("serialize"); assert_eq!(got, want, "Classic P12 bytes diverge from golden"); - // Native wire path (src/sketches/hll.rs) round-trips the golden identically. - let native = HyperLogLogP12::::deserialize_from_bytes(&want).expect("native decode"); - assert_eq!(native.registers_as_slice(), regs.as_slice()); - assert_eq!(native.serialize_to_bytes().expect("re-serialize"), want); - - // Portable decode round-trips to the same known state. - let decoded = HllSketch::from_msgpack(&want).expect("decode"); - assert_eq!(decoded.registers, regs); - assert_eq!(decoded.precision, 12); - assert_eq!(decoded.variant, HllVariant::Regular); + let decoded = HyperLogLogP12::::deserialize_from_bytes(&want).expect("decode"); + assert_eq!(decoded.registers_as_slice(), p12_registers().as_slice()); + assert_eq!(decoded.serialize_to_bytes().expect("re-serialize"), want); } #[test] fn hll_ertl_mle_p12_matches_golden() { let want = decode_hex(GOLDEN_ERTL); - let regs = p12_registers(); - let got = HllSketch::from_raw(HllVariant::Datafusion, 12, regs.clone(), 0.0, 0.0, 0.0) - .to_msgpack() - .expect("serialize"); + let sketch = HyperLogLogP12::::from_storage(p12_storage()); + let got = sketch.serialize_to_bytes().expect("serialize"); assert_eq!(got, want, "Ertl-MLE P12 bytes diverge from golden"); - let native = HyperLogLogP12::::deserialize_from_bytes(&want).expect("native decode"); - assert_eq!(native.registers_as_slice(), regs.as_slice()); - assert_eq!(native.serialize_to_bytes().expect("re-serialize"), want); - - let decoded = HllSketch::from_msgpack(&want).expect("decode"); - assert_eq!(decoded.registers, regs); - assert_eq!(decoded.variant, HllVariant::Datafusion); + let decoded = HyperLogLogP12::::deserialize_from_bytes(&want).expect("decode"); + assert_eq!(decoded.registers_as_slice(), p12_registers().as_slice()); + assert_eq!(decoded.serialize_to_bytes().expect("re-serialize"), want); } #[test] fn hll_hip_p12_matches_golden() { let want = decode_hex(GOLDEN_HIP); - let regs = p12_registers(); - let got = HllSketch::from_raw(HllVariant::Hip, 12, regs.clone(), 1.5, 2.5, 3.0) - .to_msgpack() - .expect("serialize"); + let sketch = HyperLogLogHIPP12::from_storage(p12_storage(), 1.5, 2.5, 3.0); + let got = sketch.serialize_to_bytes().expect("serialize"); assert_eq!(got, want, "HIP P12 bytes diverge from golden"); - // Native HIP wire path round-trips the golden identically. - let native = HyperLogLogHIPP12::deserialize_from_bytes(&want).expect("native decode"); - assert_eq!(native.serialize_to_bytes().expect("re-serialize"), want); - - let decoded = HllSketch::from_msgpack(&want).expect("decode"); - assert_eq!(decoded.registers, regs); - assert_eq!(decoded.variant, HllVariant::Hip); - assert_eq!(decoded.hip_kxq0, 1.5); - assert_eq!(decoded.hip_kxq1, 2.5); - assert_eq!(decoded.hip_est, 3.0); + // `Debug` prints the registers and the three HIP scalars exactly. + let decoded = HyperLogLogHIPP12::deserialize_from_bytes(&want).expect("decode"); + assert_eq!(format!("{decoded:?}"), format!("{sketch:?}")); + assert_eq!(decoded.serialize_to_bytes().expect("re-serialize"), want); } // --------------------------------------------------------------------------- diff --git a/tests/conformance_kit.rs b/tests/conformance_kit.rs index dd57a219..682b158f 100644 --- a/tests/conformance_kit.rs +++ b/tests/conformance_kit.rs @@ -11,7 +11,6 @@ use common::conformance::{ use common::{FreqTruth, zipf_u64}; use asap_sketchlib::message_pack_format::portable::ddsketch::DdSketch as PortableDds; -use asap_sketchlib::message_pack_format::portable::hll::{HllSketch, HllVariant}; use asap_sketchlib::{ Bloom, CMSHeap, CSHeap, CountL2HH, CountMin, DDSketch, DataInput, FastPath, FoldCMS, FoldCS, HyperLogLog, HyperLogLogHIP, KLL, KLLDynamic, RegularPath, SpaceSaving, UnivMonQ, Vector2D, @@ -131,17 +130,6 @@ impl CardinalityOps for HllHipAdapter { } } -struct PortableHllAdapter(HllSketch); - -impl CardinalityOps for PortableHllAdapter { - fn ingest(&mut self, key: &[u8]) { - self.0.update(key); - } - fn estimate(&self) -> f64 { - self.0.estimate() - } -} - struct KllAdapter(KLL); impl QuantileOps for KllAdapter { @@ -312,15 +300,6 @@ fn hll_variants_pass_cardinality_conformance() { spec, ) .assert_ok(); - - conformance::cardinality_battery( - "portable HllSketch", - || PortableHllAdapter(HllSketch::new(HllVariant::Regular, 14)), - &unique, - 100_000, - spec, - ) - .assert_ok(); } /// Compaction-coin seed for the KLL adapters. The unseeded `init_kll` diff --git a/tests/e2e/cardinality.rs b/tests/e2e/cardinality.rs index bf0d9636..01111149 100644 --- a/tests/e2e/cardinality.rs +++ b/tests/e2e/cardinality.rs @@ -26,7 +26,6 @@ use common::{ }; use common::{uniform_u64, zipf_u64}; -use asap_sketchlib::message_pack_format::portable::hll::{HllSketch, HllVariant}; use asap_sketchlib::sketches::hll::{HyperLogLogHIPImpl, HyperLogLogImpl}; use asap_sketchlib::{ Classic, DataInput, ErtlMLE, HyperLogLogHIPP12, HyperLogLogHIPP14, HyperLogLogHIPP16, @@ -248,77 +247,6 @@ hll_battery!( not_mergeable ); -/// The portable wire type across every variant tag and precision. -/// -/// `HllVariant` selects the *wire tag*, not the estimator: `HllSketch::estimate` -/// runs the same register-based Classic formula for `Regular`, `Datafusion` and -/// `Hip` alike. So all three are held to the register model — checking the -/// `Hip` tag against HIP's tighter constant would be asserting a property this -/// type does not implement. -#[test] -fn portable_hll_variants_and_precisions_satisfy_the_register_error_model() { - const N: usize = 200_000; - - for (v, variant) in [HllVariant::Regular, HllVariant::Datafusion, HllVariant::Hip] - .into_iter() - .enumerate() - { - for (p, precision) in [12u32, 14, 16].into_iter().enumerate() { - let spec = CardinalityConfidenceSpec::hll(precision, Z); - // A distinct stream per (variant, precision). Reusing one stream - // across the three variants would feed three estimators the *same* - // registers, whose errors are then near-perfectly correlated — nine - // readings of at most three independent experiments. - let seed = 2001 + (v * 3 + p) as u64; - let stream = uniform_u64(N, u64::MAX / 4, seed); - let truth: std::collections::HashSet = stream.iter().copied().collect(); - - let mut hll = HllSketch::new(variant, precision); - for k in &stream { - hll.update(k.to_be_bytes().as_slice()); - } - if let Err(detail) = spec.check(hll.estimate(), truth.len()) { - panic!("{variant:?}/p{precision} stream_seed={seed}: {detail}"); - } - - // Merging a second sketch built over the SAME identities is a - // register-wise max with itself: the estimate must not move at all. - let mut other = HllSketch::new(variant, precision); - for k in &truth { - other.update((*k).to_be_bytes().as_slice()); - } - let before = hll.estimate(); - hll.merge(&other).expect("merge"); - assert_eq!( - hll.estimate(), - before, - "{variant:?} p{precision}: merging identical identities moved the estimate" - ); - - // Disjoint shards over the same stream merge by register-wise max, - // so the result is the *same* sketch as the single pass and its - // estimate is the *same number*. That is an equality, not a second - // confidence-band reading. - let mut left = HllSketch::new(variant, precision); - let mut right = HllSketch::new(variant, precision); - for (i, k) in stream.iter().enumerate() { - if i % 2 == 0 { - left.update(k.to_be_bytes().as_slice()); - } else { - right.update(k.to_be_bytes().as_slice()); - } - } - left.merge(&right).expect("shard merge"); - assert_eq!( - left.estimate(), - before, - "{variant:?} p{precision}: an even/odd shard merge must reproduce the \ - single pass exactly (registers combine by maximum)" - ); - } - } -} - /// A larger precision must actually buy accuracy. A fixed percentage band /// cannot see this — it passes identically at p12 and p16, so it would not /// notice a precision parameter that never reached the register array. diff --git a/tests/msgpack_compat.rs b/tests/msgpack_compat.rs index 267a51c6..1d377266 100644 --- a/tests/msgpack_compat.rs +++ b/tests/msgpack_compat.rs @@ -31,8 +31,8 @@ use asap_sketchlib::message_pack_format::portable::delta_set_aggregator::DeltaRe use asap_sketchlib::message_pack_format::portable::hydra_kll::HydraKllSketchWire; use asap_sketchlib::message_pack_format::portable::kll::KllSketchData; use asap_sketchlib::{ - CountMinSketch, CountMinSketchWithHeap, CountSketch, DdSketch, HllSketch, HllVariant, - HydraKllSketch, KllSketch, SetAggregator, + CountMinSketch, CountMinSketchWithHeap, CountSketch, DdSketch, HydraKllSketch, KllSketch, + SetAggregator, }; // ===== round-trip: every wire-format-aligned type ===== @@ -84,17 +84,6 @@ fn dd_sketch_round_trip() { assert_eq!(restored.total_count(), 3); } -#[test] -fn hll_sketch_round_trip() { - let mut s = HllSketch::new(HllVariant::Regular, 8); - s.update(b"a"); - s.update(b"b"); - s.update(b"c"); - let bytes = s.to_msgpack().expect("encode"); - let restored = HllSketch::from_msgpack(&bytes).expect("decode"); - assert_eq!(restored.registers.len(), s.registers.len()); -} - #[test] fn kll_sketch_round_trip() { let mut s = KllSketch::with_seed(200, 0x5EED_0800); @@ -241,10 +230,6 @@ fn count_min_decodes_go_bytes() { #[test] fn dd_sketch_decodes_go_bytes() {} -#[ignore = "gap: needs a sketchlib-go-produced msgpack fixture; none can be generated in this repo"] -#[test] -fn hll_sketch_decodes_go_bytes() {} - #[ignore = "gap: needs a sketchlib-go-produced msgpack fixture; none can be generated in this repo"] #[test] fn kll_sketch_decodes_go_bytes() {} diff --git a/tests/pbt/hll.rs b/tests/pbt/hll.rs index e362f129..7159350a 100644 --- a/tests/pbt/hll.rs +++ b/tests/pbt/hll.rs @@ -20,15 +20,11 @@ //! `impl` block, so the two variants share one //! monomorphization. Estimation expands over both. //! -//! The merge laws cover both representations: `HyperLogLogImpl` itself and -//! `HllSketch`, the portable one with a register array of its own. -//! //! Flajolet, Fusy, Gandouet, Meunier, AofA '07. use crate::support::keys; -use asap_sketchlib::message_pack_format::MessagePackCodec; use asap_sketchlib::sketches::hll::{HyperLogLogHIPImpl, HyperLogLogImpl}; -use asap_sketchlib::{Classic, DataInput, ErtlMLE, HllSketch, HllVariant, HyperLogLog}; +use asap_sketchlib::{Classic, DataInput, ErtlMLE, HyperLogLog, HyperLogLogHIP}; use proptest::prelude::*; asap_sketchlib::impl_hll_bucket_list!(BucketsP4, 4, 1_usize << 4); @@ -446,14 +442,20 @@ macro_rules! hip_laws { const M: usize = <$buckets>::NUM_REGISTERS; const P: u32 = <$buckets>::PRECISION as u32; - /// The registers, read back through the wire format. The struct + /// The registers, read back out of the ASAPv1 payload. The struct /// holds them privately behind the `kxq0`/`kxq1` accumulators, and /// those accumulators are what the increment law checks, so the /// envelope is the one view of the registers that stays independent /// of it. fn registers_of(sketch: &S) -> Vec { - let bytes = sketch.to_msgpack().expect("encode"); - HllSketch::from_msgpack(&bytes).expect("decode").registers + let bytes = sketch.serialize_to_bytes().expect("encode"); + let lengths_at = 8 + bytes[7] as usize; + let metadata_len = + u32::from_be_bytes(bytes[lengths_at..lengths_at + 4].try_into().unwrap()); + let payload = &bytes[lengths_at + 8 + metadata_len as usize..]; + let (registers, ..): (serde_bytes::ByteBuf, f64, f64, f64) = + rmp_serde::from_slice(payload).expect("decode"); + registers.into_vec() } #[test] @@ -565,102 +567,28 @@ hip_laws!(hip_p6, BucketsP6); hip_laws!(hip_p8, BucketsP8); hip_laws!(hip_p10, BucketsP10); -fn byte_items(max: usize) -> impl Strategy>> { - prop::collection::vec(prop::collection::vec(any::(), 1..16), 0..max) -} - -fn portable_hll_of(precision: u32, items: &[Vec]) -> HllSketch { - let mut s = HllSketch::new(HllVariant::Regular, precision); - for i in items { - s.update(i); - } - s -} - -proptest! { - // ===== Merge ===== - - #[test] - fn hll_register_merge_is_commutative_and_idempotent( - precision in 4u32..12, - a in byte_items(50), - b in byte_items(50), - ) { - let mut ab = portable_hll_of(precision, &a); - ab.merge(&portable_hll_of(precision, &b)).expect("merge"); - let mut ba = portable_hll_of(precision, &b); - ba.merge(&portable_hll_of(precision, &a)).expect("merge"); - - prop_assert_eq!(&ab.registers, &ba.registers); - - let before = ab.registers.clone(); - let again = ab.clone(); - ab.merge(&again).expect("merge"); - prop_assert_eq!(&ab.registers, &before); - } - - #[test] - fn hll_merge_equals_streaming_the_concatenation( - precision in 4u32..12, - a in byte_items(50), - b in byte_items(50), - ) { - let mut merged = portable_hll_of(precision, &a); - merged.merge(&portable_hll_of(precision, &b)).expect("merge"); - - let concatenated: Vec<_> = a.iter().chain(b.iter()).cloned().collect(); - let streamed = portable_hll_of(precision, &concatenated); - - prop_assert_eq!(&merged.registers, &streamed.registers); - } - - #[test] - fn hll_empty_is_a_merge_identity( - precision in 4u32..12, - a in byte_items(50), - ) { - let base = portable_hll_of(precision, &a); - let mut merged = base.clone(); - merged.merge(&HllSketch::new(HllVariant::Regular, precision)).expect("merge"); - - prop_assert_eq!(&merged.registers, &base.registers); - } -} - -fn hll_variant() -> impl Strategy { - prop_oneof![ - Just(HllVariant::Regular), - Just(HllVariant::Datafusion), - Just(HllVariant::Hip), - ] -} - proptest! { // ===== Wire ===== #[test] - fn hll_round_trip_preserves_registers( - variant in hll_variant(), - precision in 4u32..14, - items in byte_items(64), - ) { - let mut s = HllSketch::new(variant, precision); - for item in &items { - s.update(item); + fn hyperloglog_round_trips_including_the_empty_sketch(stream in keys(400)) { + type Hll = HyperLogLog; + let mut sketch = Hll::new(); + for k in &stream { + sketch.insert(&DataInput::U64(*k)); } - let bytes = s.to_msgpack().expect("encode"); - let restored = HllSketch::from_msgpack(&bytes).expect("decode"); - - prop_assert_eq!(restored.variant, s.variant); - prop_assert_eq!(restored.precision, s.precision); - prop_assert_eq!(&restored.registers, &s.registers); - prop_assert_eq!(restored.estimate(), s.estimate()); + round_trip!( + Hll, + sketch, + |s: &Hll| s.registers_as_slice().to_vec(), + |s: &Hll| s.estimate(), + ); } #[test] - fn hyperloglog_round_trips_including_the_empty_sketch(stream in keys(400)) { - type Hll = HyperLogLog; + fn classic_round_trips_including_the_empty_sketch(stream in keys(400)) { + type Hll = HyperLogLog; let mut sketch = Hll::new(); for k in &stream { sketch.insert(&DataInput::U64(*k)); @@ -673,4 +601,14 @@ proptest! { |s: &Hll| s.estimate(), ); } + + #[test] + fn hip_round_trips_including_the_empty_sketch(stream in keys(400)) { + let mut sketch = HyperLogLogHIP::new(); + for k in &stream { + sketch.insert(&DataInput::U64(*k)); + } + + round_trip!(HyperLogLogHIP, sketch, |s: &HyperLogLogHIP| s.estimate()); + } } diff --git a/tests/proptest-regressions/hll.txt b/tests/proptest-regressions/hll.txt new file mode 100644 index 00000000..cb8fb6e2 --- /dev/null +++ b/tests/proptest-regressions/hll.txt @@ -0,0 +1,7 @@ +# Seeds for failure cases proptest has generated in the past. It is +# automatically read and these particular cases re-run before any +# novel cases are generated. +# +# It is recommended to check this file in to source control so that +# everyone who runs the test benefits from these saved cases. +cc 115f567853acb20cbde319427f1d20dbe1769471bdccb8f153dab74ff382dc7a # shrinks to stream = [] diff --git a/tests/sketches_go_parity_probe.rs b/tests/sketches_go_parity_probe.rs index 2997e486..485bf01d 100644 --- a/tests/sketches_go_parity_probe.rs +++ b/tests/sketches_go_parity_probe.rs @@ -9,16 +9,13 @@ //! math is sufficient — no hashspec bypass needed. use asap_sketchlib::common::DataInput; -use asap_sketchlib::common::hash::CANONICAL_HASH_SEED; -use asap_sketchlib::common::structures::HllBucketListP14; use asap_sketchlib::proto::sketchlib::{ - CountMinState, CountSketchState, CounterType, DdSketchState, HyperLogLogState, SketchEnvelope, + CountMinState, CountSketchState, CounterType, DdSketchState, SketchEnvelope, sketch_envelope::SketchState, }; use asap_sketchlib::sketches::countminsketch::CountMin; use asap_sketchlib::sketches::countsketch::Count; use asap_sketchlib::sketches::ddsketch::DDSketch; -use asap_sketchlib::sketches::hll::{ErtlMLE, HyperLogLogImpl}; use asap_sketchlib::{DefaultXxHasher, FastPath, Vector2D}; use prost::Message; @@ -34,11 +31,6 @@ const COUNTSKETCH_GOLDEN_HEX: &str = include_str!("cs_envelope_golden.hex"); /// each repeated 5×). Same file the wrapper test already uses. const COUNTMIN_GOLDEN_HEX: &str = include_str!("../src/sketches/testdata/cms_envelope_golden.hex"); -/// 16 398-byte envelope captured from `sketchlib-go::HyperLogLog.SerializePortable` -/// for precision=14, `(1..=50)` IEEE-754-LE byte-key input. Same file -/// the wrapper test already uses. -const HLL_GOLDEN_HEX: &str = include_str!("../src/sketches/testdata/hll_envelope_golden.hex"); - /// 403-byte DDSketch envelope for `alpha=0.01`, `(1..=50)` integer-as-f64 /// input, without the DataPoint-level METRIC scalars (count/sum/min/max → /// proto tags 4-7 reserved). @@ -167,57 +159,6 @@ fn sketches_countmin_fastpath_matches_go_count_min_envelope() { ); } -#[test] -fn sketches_hll_classic_matches_go_envelope() { - // Go's `HyperLogLog.SerializePortable` emits variant=DATAFUSION=2 - // (ErtlMLE). Use the same variant + precision=14 + same input here. - let mut sk: HyperLogLogImpl = - >::new(); - for i in 1..=50i32 { - let v = i as f64; - let bytes = v.to_le_bytes(); - let hashed = ::hash64_seeded( - CANONICAL_HASH_SEED, - &DataInput::Bytes(&bytes), - ); - sk.insert_with_hash(hashed); - } - - let state = HyperLogLogState { - // 2 = DATAFUSION on the Go wire (matches ErtlMLE) - variant: 2, - precision: 14, - registers: sk.registers_as_slice().to_vec(), - hip_kxq0: 0.0, - hip_kxq1: 0.0, - hip_est: 0.0, - // This parity probe pins the DENSE wire form; leave sparse unset. - registers_sparse: None, - }; - let envelope = SketchEnvelope { - format_version: 1, - producer: None, - hash_spec: None, - sample_p: 0.0, - sketch_state: Some(SketchState::Hll(state)), - }; - let mut got = Vec::with_capacity(envelope.encoded_len()); - envelope.encode(&mut got).expect("prost encode"); - - let want = decode_hex(HLL_GOLDEN_HEX); - assert_eq!( - got.len(), - want.len(), - "HLL envelope length: got {} want {}", - got.len(), - want.len(), - ); - assert_eq!( - got, want, - "sketches::HyperLogLogImpl envelope diverges from Go golden", - ); -} - #[test] fn sketches_ddsketch_matches_go_envelope() { let mut sk = DDSketch::new(0.01); diff --git a/tests/xtest_consumer.rs b/tests/xtest_consumer.rs index 0113ff45..58dfddfe 100644 --- a/tests/xtest_consumer.rs +++ b/tests/xtest_consumer.rs @@ -8,7 +8,6 @@ //! countmin.pb — CountMinState with float64 counters //! kll.pb — KLLState with quantile items and coin RNG state //! ddsketch.pb — DDSketchState with alpha + bucket array -//! hll.pb — HyperLogLogState (ErtlMLE variant) //! countsketch.pb — CountSketchState with float64 signed counters //! coco.pb — CocoSketchState (hash+val+hasKey buckets) //! elastic.pb — ElasticState (heavy buckets + light CountMin) @@ -210,41 +209,6 @@ fn cross_language_proto() { all_ok = false; } - // ----------------------------------------------------------------------- - // HLL (ErtlMLE estimator) - // ----------------------------------------------------------------------- - println!(); - println!("[HLL] Step 1/3 — Read hll.pb"); - let bytes = read_file(in_dir.join("hll.pb")); - let env = SketchEnvelope::decode(bytes.as_slice()).expect("decode hll envelope"); - - println!( - "[HLL] Step 2/3 — Validate envelope (format_version={}, producer={})", - env.format_version, - env.producer.as_ref().map_or("?", |p| &p.library) - ); - - let hll_state = match env.sketch_state { - Some(sketch_envelope::SketchState::Hll(ref s)) => s.clone(), - other => panic!("expected HLL sketch_state, got {other:?}"), - }; - - println!( - "[HLL] variant={} precision={} registers={}", - hll_state.variant, - hll_state.precision, - hll_state.registers.len() - ); - - let hll_card = hll_ertl_mle_estimate(&hll_state); - println!("[HLL] Step 3/3 — cardinality ≈ {hll_card} (expect ~50000)"); - if (40_000..=65_000).contains(&hll_card) { - println!("[HLL] PASS"); - } else { - eprintln!("[HLL] FAIL: cardinality {hll_card} not in [40000, 65000]"); - all_ok = false; - } - // ----------------------------------------------------------------------- // CountSketch // ----------------------------------------------------------------------- @@ -491,48 +455,6 @@ fn cross_language_proto() { } } - // ----------------------------------------------------------------------- - // Sampled HLL (hash-threshold sampling, p=0.1) — cardinality ×1/p rescale. - // ----------------------------------------------------------------------- - let sampled_hll_path = in_dir.join("hll_sampled.pb"); - if sampled_hll_path.exists() { - println!(); - println!("[HLL/sampled] Step 1/3 — Read hll_sampled.pb"); - let bytes = read_file(&sampled_hll_path); - let env = SketchEnvelope::decode(bytes.as_slice()).expect("decode sampled hll"); - - let p = effective_sample_p(&env); - println!("[HLL/sampled] Step 2/3 — envelope sample_p = {p}"); - if (p - 0.1).abs() > 1e-9 { - eprintln!("[HLL/sampled] FAIL: sample_p {p} != 0.1"); - all_ok = false; - } - - let mut hll_state = match env.sketch_state { - Some(sketch_envelope::SketchState::Hll(ref s)) => s.clone(), - other => panic!("expected HLL sketch_state, got {other:?}"), - }; - // Dual-read the registers (sampling thins them → may be sparse-encoded). - let regs = - asap_sketchlib::message_pack_format::portable::hll::registers_from_state(&hll_state) - .expect("decode sampled hll registers"); - hll_state.registers = regs; - hll_state.registers_sparse = None; - - let raw = hll_ertl_mle_estimate(&hll_state) as f64; - let rescaled = rescale_count(raw, p); - let rel_err = (rescaled - 200_000.0).abs() / 200_000.0; - println!( - "[HLL/sampled] Step 3/3 — raw≈{raw:.0} rescaled(×1/p)≈{rescaled:.0} truth=200000 relErr={rel_err:.4}" - ); - if rel_err <= 0.06 { - println!("[HLL/sampled] PASS"); - } else { - eprintln!("[HLL/sampled] FAIL: rescaled rel err {rel_err:.4} > 0.06"); - all_ok = false; - } - } - // ----------------------------------------------------------------------- // Final summary // ----------------------------------------------------------------------- @@ -718,78 +640,6 @@ fn xxh3_64_seeded(seed: u64, data: &[u8]) -> u64 { XxHash3_64::oneshot_with_seed(seed, data) } -// --------------------------------------------------------------------------- -// HLL ErtlMLE estimator (Ertl 2017) -// --------------------------------------------------------------------------- -// Mirrors Go's HyperLogLog.Estimate() which uses HLLRegisterBits = 50, -// HLLPrecision = 14, HLLRegisterCount = 16384. - -fn hll_sigma(mut x: f64) -> f64 { - if x == 1.0 { - return f64::INFINITY; - } - let mut y = 1.0f64; - let mut z = x; - loop { - x *= x; - let z_prev = z; - z += x * y; - y += y; - if z_prev == z { - break; - } - } - z -} - -fn hll_tau(mut x: f64) -> f64 { - if x == 0.0 || x == 1.0 { - return 0.0; - } - let mut y = 1.0f64; - let mut z = 1.0 - x; - loop { - x = x.sqrt(); - let z_prev = z; - y *= 0.5; - z -= (1.0 - x).powi(2) * y; - if z_prev == z { - break; - } - } - z / 3.0 -} - -fn hll_ertl_mle_estimate(state: &HyperLogLogState) -> u64 { - let precision = state.precision as usize; - let register_bits = 64 - precision; // Q = 50 - let m = (1usize << precision) as f64; // 16384 - - // Build histogram C[v] for v in 0..=(register_bits+1) - let hist_len = register_bits + 2; - let mut hist = vec![0u32; hist_len]; - for &r in &state.registers { - let v = r as usize; - let capped = v.min(hist_len - 1); - hist[capped] += 1; - } - - // z = m * tau((m - C[register_bits+1]) / m) - let mut z = m * hll_tau((m - hist[register_bits + 1] as f64) / m); - - // for i from register_bits down to 1: z = (z + C[i]) * 0.5 - for i in (1..=register_bits).rev() { - z += hist[i] as f64; - z *= 0.5; - } - - // z += m * sigma(C[0] / m) - z += m * hll_sigma(hist[0] as f64 / m); - - // estimate = round(0.5 / ln(2) * m^2 / z) - (0.5 / std::f64::consts::LN_2 * m * m / z).round() as u64 -} - // --------------------------------------------------------------------------- // CountSketch median frequency query (float64 counters, packed hash) // --------------------------------------------------------------------------- From 489912c0ccd83745bf9cdd9f52e9b22443acf75a Mon Sep 17 00:00:00 2001 From: GordonYuanyc Date: Fri, 2 Oct 2026 00:49:29 -0600 Subject: [PATCH 05/36] test(countsketch): pin fast-path string-key cells against Go's placement Count, FastPath> fed the 25 k-a..k-e inserts must hold the 15 signed cells Go's CountSketch wrote for the same stream. Also rewords the CSHeap portable module doc and the features.md proto list. Co-Authored-By: Claude Opus 5.5 --- docs/features.md | 2 +- docs/tests.md | 1 + .../portable/countsketch_topk.rs | 2 +- src/sketches/countsketch.rs | 38 +++++++++++++++++++ 4 files changed, 41 insertions(+), 2 deletions(-) diff --git a/docs/features.md b/docs/features.md index 64f296a3..232b591c 100644 --- a/docs/features.md +++ b/docs/features.md @@ -112,7 +112,7 @@ MessagePack (`rmp-serde`) support. **serde support** means the type derives `Ser | EHSketchList | Yes | Yes | | ExponentialHistogram | No | Yes | -Protobuf (prost): `.proto` definitions exist for CountMin, Count, HLL, DDSketch, KLL, Elastic, Coco, Hydra, and UnivMon. Rust conversion code is in progress. +Protobuf (prost): `.proto` definitions exist for CountMin, HLL, DDSketch, KLL, Elastic, Coco, Hydra, and UnivMon; Count Sketch has only the `CountSketchState` that Hydra and UnivMon embed. Rust conversion code is in progress. ### API Stability diff --git a/docs/tests.md b/docs/tests.md index 47732b4d..b0256e86 100644 --- a/docs/tests.md +++ b/docs/tests.md @@ -71,6 +71,7 @@ Wire tests: [`src/sketches/countsketch/wire.rs`](../src/sketches/countsketch/wir | `zipf_stream_stays_within_twenty_percent_for_most_keys` | Zipf stream keeps relative error under 20% for most keys. | On Zipf stream (`rows=5`, `cols=8192`, `domain=8192`, `exponent=1.1`, `N=200_000`), computes per-key relative error and requires at least 70% of keys with error `< 0.20`. | | `cs_regular_path_correctness` | Regular-path counter/sign mapping and estimates are exact on deterministic inserts. | Recomputes expected signed counter updates for `I32(0..9)` using regular hashing/sign logic, asserts exact matrix match after one pass, doubled counters after second pass, and estimate `== 2.0` for each inserted key. | | `cs_fast_path_correctness` | Fast-path row-hash/sign mapping matches expected counters. | Recomputes expected fast-path updates for `I32(0..9)` using matrix hash row slices and row signs, then asserts exact full-matrix equality. | +| `fast_path_string_keys_land_in_go_cells` | Fast-path string keys land in the same cells with the same signs as `sketchlib-go`'s Count Sketch. | Fills `Count, FastPath, DefaultXxHasher>` at `3x512` with 25 `insert_many(DataInput::String(..), 1)` calls over the five keys `k-a`..`k-e` and verifies the matrix is zero except 15 cells, five per row, each holding `+5` or `-5`, at the positions Go's serializer wrote for the same inserts. | | `count_sketch_round_trip_serialization` | The envelope frames a sketch under kind_id `0x04 0x00`, and the state survives a round trip. | Serializes a populated `3x8` regular-path `Vector2D` sketch and verifies the bytes open with the ASAPv1 magic, a `kind_id_len` of `2`, and `0x04 0x00`, then that the decode preserves rows, cols, and the full counter array. | | `count_sketch_negative_counters_round_trip` | Signed cells reach the wire and come back unchanged. | Round-trips a `2x4` matrix holding alternating positive and negative counters and verifies the decoded slice equals the source and still holds a negative cell. | | `count_sketch_custom_hasher_profile_round_trips_and_is_self_describing` | The metadata describes the hasher that built the sketch rather than a hardcoded profile. | For a `3x8` sketch over a hasher declaring its own `HashProfile`, verifies the bytes round-trip, differ from the standard-profile sketch's bytes over the same inserts, and are rejected by a standard-profile decode. | diff --git a/src/message_pack_format/portable/countsketch_topk.rs b/src/message_pack_format/portable/countsketch_topk.rs index 82ec8958..a3d1a7ec 100644 --- a/src/message_pack_format/portable/countsketch_topk.rs +++ b/src/message_pack_format/portable/countsketch_topk.rs @@ -1,7 +1,7 @@ //! Wire-format-aligned Count Sketch + top-k heap composite. //! //! No `sketches::*` equivalent — this combines `sketches::CSHeap` with -//! the wire-format `CountSketch` shape and exposes it as a single +//! the signed `rows x cols` matrix shape and exposes it as a single //! Go-interop type. Mirrors [`crate::message_pack_format::portable::countminsketch_topk::CountMinSketchWithHeap`] //! structurally, but backed by `CSHeap` (median estimator, signed rows) //! instead of `CMSHeap` (min-over-rows estimator) — the two are diff --git a/src/sketches/countsketch.rs b/src/sketches/countsketch.rs index 6a3c7ef5..ab07610f 100644 --- a/src/sketches/countsketch.rs +++ b/src/sketches/countsketch.rs @@ -1012,4 +1012,42 @@ mod tests { hi / lo ); } + + /// Go's `CountSketch` places these 25 string inserts in these 15 cells + /// with these signs. + #[test] + fn fast_path_string_keys_land_in_go_cells() { + let (rows, cols) = (3usize, 512usize); + let mut sk: Count, FastPath, DefaultXxHasher> = + Count::with_dimensions(rows, cols); + for i in 0..25 { + let key = format!("k-{}", (b'a' + (i % 5) as u8) as char); + sk.insert_many(&DataInput::String(key), 1i64); + } + + let mut want = vec![0i64; rows * cols]; + for (r, c, v) in [ + (0, 61, -5), + (0, 127, 5), + (0, 217, -5), + (0, 276, -5), + (0, 467, -5), + (1, 78, 5), + (1, 185, -5), + (1, 278, 5), + (1, 503, -5), + (1, 510, 5), + (2, 6, -5), + (2, 165, -5), + (2, 198, 5), + (2, 210, -5), + (2, 314, -5), + ] { + want[r * cols + c] = v; + } + let got: Vec = (0..rows * cols) + .map(|i| sk.as_storage().query_one_counter(i / cols, i % cols)) + .collect(); + assert_eq!(got, want); + } } From 32a2ebe3c03448a617c1405ad9fdde33db368152 Mon Sep 17 00:00:00 2001 From: GordonYuanyc Date: Fri, 2 Oct 2026 00:50:12 -0600 Subject: [PATCH 06/36] refactor(csheap)!: drop the portable Count-Sketch-with-heap format CSHeap serializes through ASAPv1 only. Removes CountSketchWithHeap, CsHeapItem and the rest of message_pack_format::portable::countsketch_topk, with the e2e test and docs rows that covered them, and adds the CSHeap golden fixture csheap_i64_regular_2x4_strkeys (kind_id 0x0a 0x00). Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 16 + asapv1_golden | 2 +- docs/asapv1_wire_format.md | 4 +- docs/message_pack_format.md | 4 +- docs/tests.md | 17 +- src/lib.rs | 1 - .../portable/countsketch_topk.rs | 471 ------------------ src/message_pack_format/portable/mod.rs | 1 - tests/asapv1_golden.rs | 58 ++- tests/e2e/composition.rs | 67 +-- 10 files changed, 81 insertions(+), 560 deletions(-) delete mode 100644 src/message_pack_format/portable/countsketch_topk.rs diff --git a/CHANGELOG.md b/CHANGELOG.md index 80d85f98..12e00ce4 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -66,6 +66,9 @@ signals a backwards-compatible change. worker's seed and the aggregator's from one seed. The unseeded constructors draw from the thread generator. The seed is not serialized, and the ASAPv1 payload is unchanged. +- **An ASAPv1 golden fixture for CSHeap** (`0x0a 0x00`), + `csheap_i64_regular_2x4_strkeys`: the Count Sketch 2x4 signed matrix with a + string-keyed heap at `k = 5`, pinned in `tests/asapv1_golden.rs`. ### Changed @@ -88,6 +91,19 @@ signals a backwards-compatible change. as an empty window like any other, and that window takes a place in `max_windows`. +### Removed + +- **The portable Count-Sketch-with-heap type.** `CountSketchWithHeap` and + `CsHeapItem` (crate root) and the module + `message_pack_format::portable::countsketch_topk` with `WireHeapItem`, + `SketchlibCSHeap`, `new_sketchlib_cs_heap`, + `sketchlib_cs_heap_from_matrix_and_heap`, `matrix_from_sketchlib_cs_heap`, + `heap_to_wire`, `sketchlib_cs_heap_update`, `sketchlib_cs_heap_query`, + `CountSketchInnerWire` and `CountSketchWithHeapWire`. Its MessagePack + `{sketch, topk_heap, heap_size}` encoding is gone with it; `CSHeap` encodes + and decodes ASAPv1 only, through `serialize_to_bytes` / + `deserialize_from_bytes`. + ### Fixed - `KLLDynamic` panicked on every quantile query once a NaN had entered the diff --git a/asapv1_golden b/asapv1_golden index 8287a4ae..5760846e 160000 --- a/asapv1_golden +++ b/asapv1_golden @@ -1 +1 @@ -Subproject commit 8287a4ae04e6cfbfb3ed33dafbf1dac77d8f7fc1 +Subproject commit 5760846eb1485defbd93aa5a7e89f1f9ed2dd104 diff --git a/docs/asapv1_wire_format.md b/docs/asapv1_wire_format.md index fd14e70a..f58c9221 100644 --- a/docs/asapv1_wire_format.md +++ b/docs/asapv1_wire_format.md @@ -1341,14 +1341,14 @@ Good direction (more compact, higher fidelity, less Rust-internal duplication), 2. **Golden byte-vector fixtures** in one shared repo, [`sketchlib-golden-bytes`](https://github.com/ProjectASAP/sketchlib-golden-bytes), which each implementation mounts at `asapv1_golden/`; both languages decode and re-encode them byte-identically. These replace the `portable`-as-oracle round-trip test. 3. **This registry**, mirrored, never independently allocated. -Fixtures exist for six `kind_id`s — HLL's three estimators, Count-Min, Count Sketch and compact KLL — and `sketchlib-go` mirrors those. Every other kind here has a payload and **no fixture**, so this document is its only contract; `asapv1_golden/README.md` lists the gap. `portable` carries its own Go goldens. +Fixtures exist for seven `kind_id`s — HLL's three estimators, Count-Min, Count Sketch, compact KLL and CSHeap — and `sketchlib-go` mirrors all but CSHeap's. Every other kind here has a payload and **no fixture**, so this document is its only contract; `asapv1_golden/README.md` lists the gap. `portable` carries its own Go goldens. **Hash profile on the Go side.** Rust derives the hash spec from a generic `HashProfile` bound on the hasher type; Go has no generic hasher type, so there is nothing to derive from. On the Go side the profile is simply **written into** the metadata on encode and **read from** it on decode. Go MUST validate the profile it reads (same fail-closed intent as Rust): a sketch is only mergeable/queryable if its `hash_profile_id` + seeds match the profile Go is prepared to reproduce. -Sequencing: (2) covers six `kind_id`s, and the one `native bytes == portable bytes` test is HLL-only, since no other `portable` type emits ASAPv1 bytes. +Sequencing: (2) covers seven `kind_id`s, and the one `native bytes == portable bytes` test is HLL-only, since no other `portable` type emits ASAPv1 bytes. Retire `portable` once the fixtures cover the rest. --- diff --git a/docs/message_pack_format.md b/docs/message_pack_format.md index 12a2b91c..35c77252 100644 --- a/docs/message_pack_format.md +++ b/docs/message_pack_format.md @@ -169,8 +169,8 @@ decode → re-encode them byte-identically. The `kind_id` registry is mirrored verbatim with Go's `wire/asapmsgpack/magic_ids.go`, never independently allocated. -That proof covers the six `kind_id`s a fixture exists for — HLL's three -estimators, Count-Min, Count Sketch and compact KLL. Every other implemented +That proof covers the seven `kind_id`s a fixture exists for — HLL's three +estimators, Count-Min, Count Sketch, compact KLL and CSHeap. Every other implemented kind has **no ASAPv1 golden and therefore no ASAPv1 drift guard**; [its README](https://github.com/ProjectASAP/sketchlib-golden-bytes#coverage) lists what is covered. The older `portable` path is guarded separately, by the `sketchlib-go` goldens diff --git a/docs/tests.md b/docs/tests.md index ccb1571c..086738c4 100644 --- a/docs/tests.md +++ b/docs/tests.md @@ -1279,21 +1279,6 @@ Test file: [`src/message_pack_format/portable/countsketch.rs`](../src/message_pa | `test_compute_delta_then_apply_matches_current` | A delta between two non-empty snapshots reconstructs the later one. | For a `3x64` base fed 40 updates of `x{i % 8}` and a current fed 30 more, verifies the delta applied to the base reproduces the current matrix. | | `test_compute_delta_matches_go_golden_bytes` | The packed delta bytes and per-row L2 are pinned by a golden. | For a `3x512` sketch fed the 25 `k-a`..`k-e` updates, verifies the delta against an empty sketch at threshold `1.0` equals a 93-byte inline golden carrying the packed cell arrays and a per-row L2 of `125.0`. | -### Portable CountSketchWithHeap - -Test file: [`src/message_pack_format/portable/countsketch_topk.rs`](../src/message_pack_format/portable/countsketch_topk.rs) - -| test_name | test_description | what_is_tested | -| --- | --- | --- | -| `test_creation` | A fresh composite has the requested matrix and an empty heap. | For `CountSketchWithHeap::new(4, 1000, 20)`, verifies `rows` `4`, `cols` `1000`, `heap_size` `20`, a `sketch_matrix()` of 4 rows of 1,000, and no heap items. | -| `test_query_empty` | An empty composite answers zero. | Verifies a `2x10` composite with a 5-slot heap reports `estimate("anything")` of `0.0`. | -| `test_merge` | Merge sums the matrices and adopts the peer's heap bound. | Merges a `2x10` composite holding `5.0` at `(0,0)`, `15.0` at `(1,1)`, heap `{key3: 75.0, key1: 80.0}` and `heap_size` `3` into one holding `10.0` at `(0,0)`, `20.0` at `(1,1)`, heap `{key1: 100.0, key2: 50.0}` and `heap_size` `5`, and verifies the merged cells read `15.0` and `35.0`, `heap_size` becomes `3`, and the heap holds at most 3 items. | -| `test_merge_dimension_mismatch` | Mismatched row counts are refused. | Verifies merging a `3x10` composite into a `2x10` one fails. | -| `test_msgpack_round_trip` | The round trip carries the matrix, the heap bound, and the heap itself. | Round-trips a `4x128` composite with a 3-slot heap fed `"hot"` at `100.0` and `"cold"` at `1.0`, and verifies the decode reports `rows` `4`, `cols` `128`, `heap_size` `3`, a non-empty heap holding `"hot"` at or above `100.0`, and estimates at or above `100.0` and `1.0`. | -| `test_aggregate_topk` | `aggregate_topk` builds a decodable composite honouring its heap bound. | For `aggregate_topk(4, 100, 2, ["a", "b", "a", "c"], [1.0, 2.0, 3.0, 0.5])`, verifies the decode reports `heap_size` `2` and at most 2 heap items; the heap contents themselves are not checked. | -| `test_aggregate_topk_empty` | An empty stream produces nothing. | Verifies `aggregate_topk(4, 100, 10, &[], &[])` returns `None`. | -| `test_median_estimator_differs_from_cms_style_min` | The median-over-rows estimator tracks the true count rather than inflating. | For a `5x64` composite with a 10-slot heap fed `"heavy"` 50 times at weight `1.0` and 20 keys `light-0`..`light-19` once each, verifies `estimate("heavy")` falls in `40.0..=60.0`. | - ### Portable DdSketch Test file: [`src/message_pack_format/portable/ddsketch.rs`](../src/message_pack_format/portable/ddsketch.rs) @@ -1957,7 +1942,6 @@ Test file: [`tests/e2e/composition.rs`](../tests/e2e/composition.rs) | test_name | test_description | what_is_tested | | --- | --- | --- | | `portable_count_min_with_heap_satisfies_the_count_min_bound_through_merge_and_wire` | The portable Count-Min-plus-heap holds Count-Min's bound and an agreeing heap through a merge and a MessagePack round trip. | Feeds `40_000` Zipf(1.1) keys over a `2_048` domain (seed `0xC090_5101`) as `"k{key}"` strings into `CountMinSketchWithHeap::new(3, 4_096, 32)`, plus even/odd halves into two more. Runs `CountMinSpec::new(3, 4_096).assert_contract` on the single sketch, asserts every `topk_heap_items()` entry's `value` equals `estimate(&item.key)` with zero violations, re-runs the contract on `CountMinSketchWithHeap::merge_refs(&[&left, &right])` against the full truth, and after `to_msgpack`/`from_msgpack` asserts the decoded sketch's `estimate` is exactly equal to the original's for every observed key and still satisfies the contract. | -| `portable_count_sketch_with_heap_satisfies_the_l2_bound_through_merge_and_wire` | The portable Count-Sketch-plus-heap holds the L2 bound and an agreeing heap through a merge and a MessagePack round trip. | Feeds the same `40_000`-key Zipf(1.1) stream (domain `2_048`, seed `0xC090_5101`) as `"k{key}"` strings into `CountSketchWithHeap::new(5, 4_096, 32)`, plus even/odd halves into two more, and runs `CountSketchSpec::new(5, 4_096).assert_contract` (simultaneous L2 band at `delta = 1e-3` with zero violations, marginal `sqrt(3/4096)*\|\|f_-i\|\|_2` rate pinned at `P[Bin(5, 1/3) >= 3] = 51/243`) on the single sketch, on `merge_refs(&[&left, &right])` and on the `from_msgpack` decode; every `topk_heap_items()` entry's `value` must equal `estimate(&item.key)`. Unlike the Count-Min twin, the wire round trip is only re-checked against the band - no per-key equality across the encode/decode is asserted. | | `portable_kll_sketch_satisfies_the_rank_error_characterization_through_merge_and_wire` | The portable KLL facade holds the rank-error characterization single-pass and after a two-shard merge, and its wire round trip is bit-for-bit. | Over `12` compaction seeds `0x5EED_0400 + t * 0x9E37_79B9_7F4A_7C15`, builds `KllSketch::with_seed(200, seed)` plus even/odd shards at `seed ^ 0xAAAA` and `seed ^ 0x5555`, feeds all of them `40_000` uniform values over `1_000_000` (seed `0xC090_5101`), and merges the shards with `merge`. Each seed records two trials - the single-pass sketch and the merged pair - of the worst normalized rank error over the grid `[0.1, 0.25, 0.5, 0.75, 0.9]` against `KllRankSpec::datasketches(200)`'s `eps = 2.446/200^0.9433 = 0.0165`, and the `24` outcomes are accepted by `assert_independent_binomial` at `p = 0.01`. Per seed it also asserts `to_msgpack`/`from_msgpack` preserves `k() == 200`, `count()`, and every grid quantile exactly, and that merging a `k = 400` sketch into the single-pass one errs. | ### Composition: Documented Input Matrix @@ -2126,6 +2110,7 @@ Test file: [`tests/asapv1_golden.rs`](../tests/asapv1_golden.rs) | `cs_i32_regular_2x4_matches_golden` | The i32 counter type reaches the metadata, and an i32 golden is not an i64 sketch. | Verifies `Count::, RegularPath>::from_storage` over `[[0, 127, 128, 65536], [-1, -33, -32768, -2147483648]]` cast to `i32` serializes to `asapv1_golden/cs_i32_regular_2x4.hex`, that decoding it recovers the eight cells and re-serializes identically, that the i32 and i64 regular fixtures have equal length yet differ, and that the i32 fixture fails to decode as `Count, RegularPath>`. | | `cs_i64_fast_2x4_matches_golden` | The fast-path mode reaches the bytes, so the same matrix encodes differently per path. | Verifies `Count::, FastPath>::from_storage` over `[[0, 127, 128, 65536], [-1, -33, -32768, -2147483648]]` serializes to `asapv1_golden/cs_i64_fast_2x4.hex`, that decoding it recovers the eight cells and re-serializes identically, and that the fixture differs from `cs_i64_regular_2x4.hex` built from the same matrix. | | `cs_i64_regular_2x4_matches_golden` | A known signed Count-Sketch matrix serializes to its checked-in golden bytes. | Verifies `Count::, RegularPath>::from_storage` over `[[0, 127, 128, 65536], [-1, -33, -32768, -2147483648]]` — sweeping the msgpack integer widths in both directions — serializes to `asapv1_golden/cs_i64_regular_2x4.hex`, and that decoding the fixture reports `rows()` `2`, `cols()` `4`, the eight cells, and a byte-identical re-serialization. | +| `csheap_i64_regular_2x4_strkeys_matches_golden` | A known CSHeap matrix and heap serialize to their checked-in golden bytes. | Builds `CSHeap::, RegularPath>::from_storage` over `[[0, 127, 128, 65536], [-1, -33, -32768, -2147483648]]` at `k` `5` and seats the heap directly through `heap_mut().update` with `Str` keys `gamma` `-33`, `delta` `127`, `alpha` `4294967296`, `beta` `127`; verifies the bytes equal `asapv1_golden/csheap_i64_regular_2x4_strkeys.hex`, that decoding recovers the eight cells, `rows()` `2`, `cols()` `4`, capacity `5`, four entries each at its count, and a byte-identical re-serialization, and that the fixture fails to decode as `Count, RegularPath>`. | | `hll_classic_p12_matches_golden` | A known P12 Classic register pattern serializes to its golden bytes on both the portable and native paths. | Builds `HllSketch::from_raw(HllVariant::Regular, 12, regs, 0.0, 0.0, 0.0)` where `regs` is 4,096 zeros with `[0] = 1`, `[1] = 7`, `[100] = 42` and `[4095] = 3`, verifies `to_msgpack()` equals `asapv1_golden/hll_classic_p12.hex`, that `HyperLogLogP12::::deserialize_from_bytes` on the fixture recovers those registers and re-serializes identically, and that `HllSketch::from_msgpack` reports the same registers, `precision` `12` and `HllVariant::Regular`. | | `hll_ertl_mle_p12_matches_golden` | The Datafusion variant tag reaches the bytes for the same P12 register pattern. | Builds `HllSketch::from_raw(HllVariant::Datafusion, 12, regs, 0.0, 0.0, 0.0)` over the shared P12 pattern (`[0] = 1`, `[1] = 7`, `[100] = 42`, `[4095] = 3` in 4,096 registers), verifies `to_msgpack()` equals `asapv1_golden/hll_ertl_mle_p12.hex`, that `HyperLogLogP12::::deserialize_from_bytes` recovers the registers and re-serializes identically, and that `HllSketch::from_msgpack` reports `HllVariant::Datafusion`. | | `hll_hip_p12_matches_golden` | The HIP variant's three running scalars travel on the wire beside its registers. | Builds `HllSketch::from_raw(HllVariant::Hip, 12, regs, 1.5, 2.5, 3.0)` over the shared P12 pattern, verifies `to_msgpack()` equals `asapv1_golden/hll_hip_p12.hex`, that `HyperLogLogHIPP12::deserialize_from_bytes` re-serializes the fixture identically, and that `HllSketch::from_msgpack` reports the registers, `HllVariant::Hip`, `hip_kxq0` `1.5`, `hip_kxq1` `2.5` and `hip_est` `3.0`. | diff --git a/src/lib.rs b/src/lib.rs index b062b8b9..92003b6d 100644 --- a/src/lib.rs +++ b/src/lib.rs @@ -46,7 +46,6 @@ pub use message_pack_format::portable::countminsketch_topk::{CmsHeapItem, CountM pub use message_pack_format::portable::countsketch::{ COUNT_SKETCH_TOPK_CAPACITY, CountSketch, CountSketchDelta, }; -pub use message_pack_format::portable::countsketch_topk::{CountSketchWithHeap, CsHeapItem}; pub use message_pack_format::portable::ddsketch::{DDSKETCH_GROW_CHUNK, DdSketch, DdSketchDelta}; pub use message_pack_format::portable::delta_set_aggregator::DeltaResult; pub use message_pack_format::portable::hll::{HllSketch, HllSketchDelta, HllVariant}; diff --git a/src/message_pack_format/portable/countsketch_topk.rs b/src/message_pack_format/portable/countsketch_topk.rs deleted file mode 100644 index 82ec8958..00000000 --- a/src/message_pack_format/portable/countsketch_topk.rs +++ /dev/null @@ -1,471 +0,0 @@ -//! Wire-format-aligned Count Sketch + top-k heap composite. -//! -//! No `sketches::*` equivalent — this combines `sketches::CSHeap` with -//! the wire-format `CountSketch` shape and exposes it as a single -//! Go-interop type. Mirrors [`crate::message_pack_format::portable::countminsketch_topk::CountMinSketchWithHeap`] -//! structurally, but backed by `CSHeap` (median estimator, signed rows) -//! instead of `CMSHeap` (min-over-rows estimator) — the two are -//! different sketch algorithms that happen to produce a same-shaped -//! `rows x cols` matrix, not interchangeable data. - -use serde::{Deserialize, Serialize}; - -use crate::message_pack_format::{Error as MsgPackError, MessagePackCodec}; -use crate::sketches::countsketch_topk::CSHeap; -use crate::{DataInput, RegularPath, Vector2D}; - -/// Wire-format heap item (key, value) used by the dispatch helpers below. -pub struct WireHeapItem { - pub key: String, - pub value: f64, -} - -/// Concrete Count-Sketch-with-Heap type backing the wire-format -/// `CountSketchWithHeap`. -pub type SketchlibCSHeap = CSHeap, RegularPath>; - -/// Creates a fresh CSHeap with the given dimensions and heap capacity. -pub fn new_sketchlib_cs_heap(row_num: usize, col_num: usize, heap_size: usize) -> SketchlibCSHeap { - CSHeap::new(row_num, col_num, heap_size) -} - -/// Builds a CSHeap from an existing sketch matrix and optional heap -/// items. Used when deserializing wire-format state. -pub fn sketchlib_cs_heap_from_matrix_and_heap( - row_num: usize, - col_num: usize, - heap_size: usize, - sketch: &[Vec], - topk_heap: &[WireHeapItem], -) -> SketchlibCSHeap { - let matrix = Vector2D::from_fn(row_num, col_num, |r, c| { - sketch - .get(r) - .and_then(|row| row.get(c)) - .copied() - .unwrap_or(0.0) - .round() as i64 - }); - let mut cs_heap = CSHeap::from_storage(matrix, heap_size); - - for item in topk_heap { - let count = item.value.round() as i64; - if count > 0 { - let input = DataInput::Str(&item.key); - cs_heap.heap_mut().update(&input, count); - } - } - - cs_heap -} - -/// Converts a CSHeap's storage into a `Vec>` matrix. -pub fn matrix_from_sketchlib_cs_heap(cs_heap: &SketchlibCSHeap) -> Vec> { - let storage = cs_heap.cs().as_storage(); - let rows = storage.rows(); - let cols = storage.cols(); - let mut sketch = vec![vec![0.0; cols]; rows]; - - for (r, row) in sketch.iter_mut().enumerate().take(rows) { - for (c, cell) in row.iter_mut().enumerate().take(cols) { - if let Some(v) = storage.get(r, c) { - *cell = *v as f64; - } - } - } - - sketch -} - -/// Converts sketchlib HHHeap items to wire-format (key, value) pairs. -pub fn heap_to_wire(cs_heap: &SketchlibCSHeap) -> Vec { - cs_heap - .heap() - .heap() - .iter() - .map(|hh_item| { - let key = match &hh_item.key { - crate::HeapItem::String(s) => s.clone(), - other => format!("{other:?}"), - }; - WireHeapItem { - key, - value: hh_item.count as f64, - } - }) - .collect() -} - -/// Updates a CSHeap with a weighted key. Automatically updates the heap. -pub fn sketchlib_cs_heap_update(cs_heap: &mut SketchlibCSHeap, key: &str, value: f64) { - let many = value.round() as i64; - if many <= 0 { - return; - } - cs_heap.insert_many(&DataInput::String(key.to_owned()), many); -} - -/// Queries a CSHeap for a key's frequency estimate (median-of-signed-rows, -/// not CMS's min-over-rows). -pub fn sketchlib_cs_heap_query(cs_heap: &SketchlibCSHeap, key: &str) -> f64 { - cs_heap.estimate(&DataInput::String(key.to_owned())) -} - -/// Item in the top-k heap representing a key-value pair. -#[derive(Debug, Clone, Serialize, Deserialize)] -pub struct CsHeapItem { - pub key: String, - pub value: f64, -} - -/// Count Sketch with Heap for top-k tracking. Combines the balanced/ -/// zero-mean-error frequency estimator with efficient top-k maintenance. -pub struct CountSketchWithHeap { - pub rows: usize, - pub cols: usize, - pub heap_size: usize, - pub(crate) backend: SketchlibCSHeap, -} - -impl std::fmt::Debug for CountSketchWithHeap { - fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { - f.debug_struct("CountSketchWithHeap") - .field("rows", &self.rows) - .field("cols", &self.cols) - .field("heap_size", &self.heap_size) - .finish() - } -} - -impl Clone for CountSketchWithHeap { - fn clone(&self) -> Self { - // Structural clone: `CSHeap` derives `Clone` (its `Count` + - // `HHHeap` fields are `Clone`), so copy the backend directly - // instead of extracting the matrix + heap to wire form and - // rebuilding it. - Self { - rows: self.rows, - cols: self.cols, - heap_size: self.heap_size, - backend: self.backend.clone(), - } - } -} - -impl CountSketchWithHeap { - pub fn new(rows: usize, cols: usize, heap_size: usize) -> Self { - Self { - rows, - cols, - heap_size, - backend: new_sketchlib_cs_heap(rows, cols, heap_size), - } - } - - pub fn rows(&self) -> usize { - self.rows - } - - pub fn cols(&self) -> usize { - self.cols - } - - /// Create from a sketch matrix and heap (e.g. from JSON deserialization). - pub fn from_legacy_matrix( - sketch: Vec>, - topk_heap: Vec, - rows: usize, - cols: usize, - heap_size: usize, - ) -> Self { - let wire_heap: Vec = topk_heap - .into_iter() - .map(|h| WireHeapItem { - key: h.key, - value: h.value, - }) - .collect(); - Self { - rows, - cols, - heap_size, - backend: sketchlib_cs_heap_from_matrix_and_heap( - rows, cols, heap_size, &sketch, &wire_heap, - ), - } - } - - /// Get the top-k heap items. - pub fn topk_heap_items(&self) -> Vec { - heap_to_wire(&self.backend) - .into_iter() - .map(|w| CsHeapItem { - key: w.key, - value: w.value, - }) - .collect() - } - - /// Get the sketch matrix. - pub fn sketch_matrix(&self) -> Vec> { - matrix_from_sketchlib_cs_heap(&self.backend) - } - - pub fn update(&mut self, key: &str, value: f64) { - sketchlib_cs_heap_update(&mut self.backend, key, value); - } - - pub fn estimate(&self, key: &str) -> f64 { - sketchlib_cs_heap_query(&self.backend, key) - } - - /// Merge another CountSketchWithHeap into self in place. Both - /// operands must have identical dimensions; the resulting - /// `heap_size` is the minimum of the two. - pub fn merge( - &mut self, - other: &CountSketchWithHeap, - ) -> Result<(), Box> { - if self.rows != other.rows || self.cols != other.cols { - return Err(format!( - "CountSketchWithHeap dimension mismatch: self={}x{}, other={}x{}", - self.rows, self.cols, other.rows, other.cols - ) - .into()); - } - self.backend.merge(&other.backend); - self.heap_size = self.heap_size.min(other.heap_size); - Ok(()) - } - - pub fn merge_refs( - inputs: &[&CountSketchWithHeap], - ) -> Result> { - let first = inputs - .first() - .ok_or("CountSketchWithHeap::merge_refs called with empty input")?; - let mut merged = (*first).clone(); - for h in inputs.iter().skip(1) { - merged.merge(h)?; - } - Ok(merged) - } - - pub fn aggregate_topk( - rows: usize, - cols: usize, - heap_size: usize, - keys: &[&str], - values: &[f64], - ) -> Option> { - if keys.is_empty() { - return None; - } - let mut sketch = Self::new(rows, cols, heap_size); - for (key, &value) in keys.iter().zip(values.iter()) { - sketch.update(key, value); - } - sketch.to_msgpack().ok() - } -} - -// ----- Wire format ----- - -/// Inner CS payload nested inside [`CountSketchWithHeapWire`]. -#[derive(Debug, Clone, Serialize, Deserialize)] -pub struct CountSketchInnerWire { - pub sketch: Vec>, - #[serde(rename = "row_num")] - pub rows: usize, - #[serde(rename = "col_num")] - pub cols: usize, -} - -#[derive(Debug, Clone, Serialize, Deserialize)] -pub struct CountSketchWithHeapWire { - pub sketch: CountSketchInnerWire, - pub topk_heap: Vec, - pub heap_size: usize, -} - -impl MessagePackCodec for CountSketchWithHeap { - fn to_msgpack(&self) -> Result, MsgPackError> { - let wire = CountSketchWithHeapWire { - sketch: CountSketchInnerWire { - sketch: self.sketch_matrix(), - rows: self.rows, - cols: self.cols, - }, - topk_heap: self.topk_heap_items(), - heap_size: self.heap_size, - }; - Ok(rmp_serde::to_vec(&wire)?) - } - - fn from_msgpack(bytes: &[u8]) -> Result { - let wire: CountSketchWithHeapWire = rmp_serde::from_slice(bytes)?; - - let mut sorted_topk_heap = wire.topk_heap; - sorted_topk_heap.sort_by(|a, b| b.value.partial_cmp(&a.value).unwrap()); - - let wire_heap: Vec = sorted_topk_heap - .iter() - .map(|h| WireHeapItem { - key: h.key.clone(), - value: h.value, - }) - .collect(); - let backend = sketchlib_cs_heap_from_matrix_and_heap( - wire.sketch.rows, - wire.sketch.cols, - wire.heap_size, - &wire.sketch.sketch, - &wire_heap, - ); - - Ok(Self { - rows: wire.sketch.rows, - cols: wire.sketch.cols, - heap_size: wire.heap_size, - backend, - }) - } -} - -#[cfg(test)] -mod tests { - use super::*; - - #[test] - fn test_creation() { - let cs = CountSketchWithHeap::new(4, 1000, 20); - assert_eq!(cs.rows, 4); - assert_eq!(cs.cols, 1000); - assert_eq!(cs.heap_size, 20); - assert_eq!(cs.sketch_matrix().len(), 4); - assert_eq!(cs.sketch_matrix()[0].len(), 1000); - assert_eq!(cs.topk_heap_items().len(), 0); - } - - #[test] - fn test_query_empty() { - let cs = CountSketchWithHeap::new(2, 10, 5); - assert_eq!(cs.estimate("anything"), 0.0); - } - - #[test] - fn test_merge() { - let mut sketch1 = vec![vec![0.0; 10]; 2]; - sketch1[0][0] = 10.0; - sketch1[1][1] = 20.0; - let mut cs1 = CountSketchWithHeap::from_legacy_matrix( - sketch1, - vec![ - CsHeapItem { - key: "key1".to_string(), - value: 100.0, - }, - CsHeapItem { - key: "key2".to_string(), - value: 50.0, - }, - ], - 2, - 10, - 5, - ); - let mut sketch2 = vec![vec![0.0; 10]; 2]; - sketch2[0][0] = 5.0; - sketch2[1][1] = 15.0; - let cs2 = CountSketchWithHeap::from_legacy_matrix( - sketch2, - vec![ - CsHeapItem { - key: "key3".to_string(), - value: 75.0, - }, - CsHeapItem { - key: "key1".to_string(), - value: 80.0, - }, - ], - 2, - 10, - 3, - ); - - cs1.merge(&cs2).unwrap(); - - assert_eq!(cs1.sketch_matrix()[0][0], 15.0); - assert_eq!(cs1.sketch_matrix()[1][1], 35.0); - assert_eq!(cs1.heap_size, 3); - assert!(cs1.topk_heap_items().len() <= 3); - } - - #[test] - fn test_merge_dimension_mismatch() { - let mut cs1 = CountSketchWithHeap::new(2, 10, 5); - let cs2 = CountSketchWithHeap::new(3, 10, 5); - assert!(cs1.merge(&cs2).is_err()); - } - - #[test] - fn test_msgpack_round_trip() { - let mut cs = CountSketchWithHeap::new(4, 128, 3); - cs.update("hot", 100.0); - cs.update("cold", 1.0); - - let bytes = cs.to_msgpack().unwrap(); - let deserialized = CountSketchWithHeap::from_msgpack(&bytes).unwrap(); - - assert_eq!(deserialized.rows, 4); - assert_eq!(deserialized.cols, 128); - assert_eq!(deserialized.heap_size, 3); - let items = deserialized.topk_heap_items(); - assert!(!items.is_empty()); - let hot = items - .iter() - .find(|item| item.key == "hot") - .expect("'hot' should be in the heap"); - assert!(hot.value >= 100.0); - assert!(deserialized.estimate("hot") >= 100.0); - assert!(deserialized.estimate("cold") >= 1.0); - } - - #[test] - fn test_aggregate_topk() { - let keys = ["a", "b", "a", "c"]; - let values = [1.0, 2.0, 3.0, 0.5]; - let bytes = CountSketchWithHeap::aggregate_topk(4, 100, 2, &keys, &values).unwrap(); - let cs = CountSketchWithHeap::from_msgpack(&bytes).unwrap(); - assert_eq!(cs.heap_size, 2); - assert!(cs.topk_heap_items().len() <= 2); - } - - #[test] - fn test_aggregate_topk_empty() { - assert!(CountSketchWithHeap::aggregate_topk(4, 100, 10, &[], &[]).is_none()); - } - - /// The whole point of splitting this from `CountMinSketchWithHeap`: - /// CS's median estimator must behave differently from CMS's - /// min-over-rows estimator on the same kind of skewed insert - /// pattern, not just share a struct shape. - #[test] - fn test_median_estimator_differs_from_cms_style_min() { - let mut cs = CountSketchWithHeap::new(5, 64, 10); - for _ in 0..50 { - cs.update("heavy", 1.0); - } - for i in 0..20 { - cs.update(&format!("light-{i}"), 1.0); - } - let est = cs.estimate("heavy"); - assert!( - (40.0..=60.0).contains(&est), - "median estimate {est} should track the true count (50) reasonably \ - closely, not collide-and-inflate the way a single min-over-rows \ - outlier could" - ); - } -} diff --git a/src/message_pack_format/portable/mod.rs b/src/message_pack_format/portable/mod.rs index 97e0d372..6524c04c 100644 --- a/src/message_pack_format/portable/mod.rs +++ b/src/message_pack_format/portable/mod.rs @@ -10,7 +10,6 @@ pub mod countminsketch; pub mod countminsketch_topk; pub mod countsketch; -pub mod countsketch_topk; pub mod ddsketch; pub mod delta_set_aggregator; pub mod hll; diff --git a/tests/asapv1_golden.rs b/tests/asapv1_golden.rs index 30b41651..29acbf67 100644 --- a/tests/asapv1_golden.rs +++ b/tests/asapv1_golden.rs @@ -9,8 +9,8 @@ //! See `asapv1_golden/README.md`. use asap_sketchlib::{ - Classic, Count, CountMin, ErtlMLE, FastPath, HllSketch, HllVariant, HyperLogLogHIPP12, - HyperLogLogP12, KLL, MessagePackCodec, RegularPath, Vector2D, + CSHeap, Classic, Count, CountMin, DataInput, ErtlMLE, FastPath, HllSketch, HllVariant, + HyperLogLogHIPP12, HyperLogLogP12, KLL, MessagePackCodec, RegularPath, Vector2D, }; fn decode_hex(s: &str) -> Vec { @@ -30,6 +30,7 @@ const GOLDEN_CMS_F64: &str = include_str!("../asapv1_golden/cms_f64_fast_2x3.hex const GOLDEN_CS_REGULAR: &str = include_str!("../asapv1_golden/cs_i64_regular_2x4.hex"); const GOLDEN_CS_FAST: &str = include_str!("../asapv1_golden/cs_i64_fast_2x4.hex"); const GOLDEN_CS_I32: &str = include_str!("../asapv1_golden/cs_i32_regular_2x4.hex"); +const GOLDEN_CS_HEAP: &str = include_str!("../asapv1_golden/csheap_i64_regular_2x4_strkeys.hex"); const GOLDEN_KLL_F64: &str = include_str!("../asapv1_golden/kll_f64_k200.hex"); const GOLDEN_KLL_I64: &str = include_str!("../asapv1_golden/kll_i64_k200.hex"); @@ -242,6 +243,59 @@ fn cs_i32_regular_2x4_matches_golden() { ); } +// --------------------------------------------------------------------------- +// CSHeap: the Count Sketch matrix above plus a heap whose entries are seated +// directly, never by inserting through the sketch. +// --------------------------------------------------------------------------- + +/// Heap entries of the CSHeap fixture, in seating order. The counts span +/// uint64, a positive fixint tie (ordered by key on the wire) and a negative +/// int8; a CSHeap heap count is a signed median and may be negative. +const CS_HEAP_ENTRIES: [(&str, i64); 4] = [ + ("gamma", -33), + ("delta", 127), + ("alpha", 4_294_967_296), + ("beta", 127), +]; +const CS_HEAP_K: usize = 5; + +#[test] +fn csheap_i64_regular_2x4_strkeys_matches_golden() { + let want = decode_hex(GOLDEN_CS_HEAP); + + let mut sketch = CSHeap::, RegularPath>::from_storage( + Vector2D::from_fn(2, 4, |r, c| CS_VALS[r][c]), + CS_HEAP_K, + ); + for (key, count) in CS_HEAP_ENTRIES { + sketch.heap_mut().update(&DataInput::Str(key), count); + } + let got = sketch.serialize_to_bytes().expect("serialize"); + assert_eq!(got, want, "CSHeap i64/regular bytes diverge from golden"); + + let decoded = + CSHeap::, RegularPath>::deserialize_from_bytes(&want).expect("decode"); + let flat: Vec = CS_VALS.iter().flatten().copied().collect(); + assert_eq!(decoded.cs().as_storage().as_slice(), flat.as_slice()); + assert_eq!((decoded.rows(), decoded.cols()), (2, 4)); + assert_eq!(decoded.heap().capacity(), CS_HEAP_K); + assert_eq!(decoded.heap().len(), CS_HEAP_ENTRIES.len()); + for (key, count) in CS_HEAP_ENTRIES { + let seat = decoded + .heap() + .find(&DataInput::Str(key)) + .unwrap_or_else(|| panic!("heap key {key:?} missing after decode")); + assert_eq!(decoded.heap().heap()[seat].count, count, "heap key {key:?}"); + } + assert_eq!(decoded.serialize_to_bytes().expect("re-serialize"), want); + + // A CSHeap golden is not a Count Sketch envelope. + assert!( + Count::, RegularPath>::deserialize_from_bytes(&want).is_err(), + "the CSHeap golden must not decode as a plain Count Sketch" + ); +} + // --------------------------------------------------------------------------- // KLL: build known state (k=200, seed 42, integers 1..=50 — below the level-0 // capacity, so no compaction fires and the retained set is deterministic) -> diff --git a/tests/e2e/composition.rs b/tests/e2e/composition.rs index 23e42c41..3c45a728 100644 --- a/tests/e2e/composition.rs +++ b/tests/e2e/composition.rs @@ -16,9 +16,9 @@ use common::specs::{CardinalityConfidenceSpec, CountMinSpec, CountSketchSpec, Kl use common::{FreqTruth, NumericTruth, uniform_u64, zipf_u64}; use asap_sketchlib::{ - Classic, Count, CountMin, CountMinSketchWithHeap, CountSketchWithHeap, DataInput, - EnsembleSketch, ErtlMLE, FastPath, HashSketchEnsemble, HyperLogLog, HyperLogLogHIP, KllSketch, - MessagePackCodec, UnivMonQ, UnivMonQConfig, Vector2D, + Classic, Count, CountMin, CountMinSketchWithHeap, DataInput, EnsembleSketch, ErtlMLE, FastPath, + HashSketchEnsemble, HyperLogLog, HyperLogLogHIP, KllSketch, MessagePackCodec, UnivMonQ, + UnivMonQConfig, Vector2D, }; const ROWS: usize = 3; @@ -636,67 +636,6 @@ fn portable_count_min_with_heap_satisfies_the_count_min_bound_through_merge_and_ ); } -#[test] -fn portable_count_sketch_with_heap_satisfies_the_l2_bound_through_merge_and_wire() { - const HEAP: usize = 32; - const CS_ROWS: usize = 5; - let stream = zipf_u64(N, DOMAIN, 1.1, STREAM_SEED); - let mut truth = FreqTruth::default(); - let mut single = CountSketchWithHeap::new(CS_ROWS, COLS, HEAP); - let mut left = CountSketchWithHeap::new(CS_ROWS, COLS, HEAP); - let mut right = CountSketchWithHeap::new(CS_ROWS, COLS, HEAP); - for (i, k) in stream.iter().enumerate() { - truth.observe(*k as i64); - let key = format!("k{k}"); - single.update(&key, 1.0); - if i % 2 == 0 { - left.update(&key, 1.0); - } else { - right.update(&key, 1.0); - } - } - - let context = format!( - "rows={CS_ROWS} cols={COLS} heap={HEAP} zipf(1.1) domain={DOMAIN} n={N} seed={STREAM_SEED:#x}" - ); - let spec = CountSketchSpec::new(CS_ROWS, COLS); - spec.assert_contract( - "portable CountSketchWithHeap", - &truth, - |k| single.estimate(&format!("k{k}")), - &context, - ); - - let mut heap_tally = Tally::default(); - for item in single.topk_heap_items() { - let est = single.estimate(&item.key); - heap_tally.record(item.value == est, || { - format!( - "key {}: heap holds {} but the sketch estimates {est}", - item.key, item.value - ) - }); - } - heap_tally.assert_none("portable CountSketchWithHeap heap consistency", &context); - - let merged = CountSketchWithHeap::merge_refs(&[&left, &right]).expect("merge"); - spec.assert_contract( - "portable CountSketchWithHeap after merge", - &truth, - |k| merged.estimate(&format!("k{k}")), - &context, - ); - - let bytes = single.to_msgpack().expect("encode"); - let decoded = CountSketchWithHeap::from_msgpack(&bytes).expect("decode"); - spec.assert_contract( - "portable CountSketchWithHeap after a wire round trip", - &truth, - |k| decoded.estimate(&format!("k{k}")), - &context, - ); -} - /// The portable KLL facade under the DataSketches maximum-rank-error /// characterization, seeded so a failure reproduces. `KllSketch::new` seeds /// from the wall clock; `with_seed` is what an accuracy test must use. From 458d02e020003f68257a4316fa79b448b8399371 Mon Sep 17 00:00:00 2001 From: GordonYuanyc Date: Fri, 2 Oct 2026 00:50:17 -0600 Subject: [PATCH 07/36] refactor(cmsheap)!: drop the portable MessagePack format CMSHeap serializes through ASAPv1 only. Removes message_pack_format::portable::countminsketch_topk (CountMinSketchWithHeap, CmsHeapItem, its wire DTOs and sketchlib_cms_heap_* helpers) and the crate-root re-exports. The e2e merge-and-wire test runs on CMSHeap over ASAPv1, and two CMSHeap golden fixtures are added (submodule bump). Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 17 + asapv1_golden | 2 +- docs/asapv1_wire_format.md | 2 - docs/message_pack_format.md | 6 +- docs/tests.md | 22 +- src/lib.rs | 4 +- .../portable/countminsketch_topk.rs | 447 ------------------ .../portable/countsketch_topk.rs | 7 +- src/message_pack_format/portable/mod.rs | 1 - tests/README.md | 2 +- tests/asapv1_golden.rs | 97 +++- tests/e2e/composition.rs | 85 ++-- tests/msgpack_compat.rs | 41 +- 13 files changed, 174 insertions(+), 559 deletions(-) delete mode 100644 src/message_pack_format/portable/countminsketch_topk.rs diff --git a/CHANGELOG.md b/CHANGELOG.md index 80d85f98..740943b6 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -66,6 +66,9 @@ signals a backwards-compatible change. worker's seed and the aggregator's from one seed. The unseeded constructors draw from the thread generator. The seed is not serialized, and the ASAPv1 payload is unchanged. +- **ASAPv1 golden fixtures for CMSHeap** (`03 00`): + `cmsheap_i64_regular_2x3_strkeys` and `cmsheap_i32_fast_2x3_i64keys`, checked + by `tests/asapv1_golden.rs`. ### Changed @@ -88,6 +91,20 @@ signals a backwards-compatible change. as an empty window like any other, and that window takes a place in `max_windows`. +### Removed + +- **CMSHeap's pre-envelope MessagePack format.** `CMSHeap` serializes through + ASAPv1 only (`serialize_to_bytes` / `deserialize_from_bytes`). Removed: the + module `message_pack_format::portable::countminsketch_topk` with + `CountMinSketchWithHeap` (and its `MessagePackCodec` impl, `merge_refs` and + `aggregate_topk`), `CmsHeapItem`, `WireHeapItem`, `SketchlibCMSHeap`, + `CountMinSketchInnerWire`, `CountMinSketchWithHeapWire`, + `new_sketchlib_cms_heap`, `sketchlib_cms_heap_from_matrix_and_heap`, + `matrix_from_sketchlib_cms_heap`, `heap_to_wire`, `sketchlib_cms_heap_update` + and `sketchlib_cms_heap_query`, plus the crate-root re-exports + `CountMinSketchWithHeap` and `CmsHeapItem`. Bytes in that format no longer + decode. + ### Fixed - `KLLDynamic` panicked on every quantile query once a NaN had entered the diff --git a/asapv1_golden b/asapv1_golden index 8287a4ae..821d0ea7 160000 --- a/asapv1_golden +++ b/asapv1_golden @@ -1 +1 @@ -Subproject commit 8287a4ae04e6cfbfb3ed33dafbf1dac77d8f7fc1 +Subproject commit 821d0ea7c1dbd1f4bc8f0c7b3938bd6635a02037 diff --git a/docs/asapv1_wire_format.md b/docs/asapv1_wire_format.md index fd14e70a..0ec65cb1 100644 --- a/docs/asapv1_wire_format.md +++ b/docs/asapv1_wire_format.md @@ -659,8 +659,6 @@ Wire counter types are the base Count-Min's: **`"i32"`, `"i64"` and `"f64"`**. ` **A producer-side constraint: a `NaN` key.** `HHHeap` finds a resident by digest and then compares with `PartialEq`, and `NaN` never equals itself, so offering the same `NaN` key twice seats **two** entries rather than rescoring one. That state is reachable in memory and has **no encoding**: the encode-side duplicate check reads the two keys' bits, sees one key twice, and fails with the same complaint the decoder would make. Nothing repairs it on either door — a producer that inserts float keys must keep `NaN` out of a heap it intends to serialize. -**Not this: `portable/countminsketch_topk.rs`.** Despite the name, that module emits no ASAPv1 bytes. It is a legacy Go-interop struct with its own msgpack shape: an `f64` matrix rounded into the `i64` one on the way in, and heap keys coerced to `String` — a `HeapItem` that is not already a `String` goes through a Rust debug format, which is lossy and reversible by nothing. `0x03 0x00` shares none of that: its matrix travels at the metadata's `counter_type` and its keys travel as the exact `HeapItem` variant `key_type` names. - ### 3.8: DDSketch payload (`0x05 0x00`) `DDSketch` maps a positive value to the logarithmic bucket index `floor(ln(v) / ln(gamma))` and keeps a dense count per bucket. The relative accuracy `alpha` is its one construction parameter and lives in the metadata, so the payload is the bucket store plus the running scalars the buckets do not determine: diff --git a/docs/message_pack_format.md b/docs/message_pack_format.md index 12a2b91c..308a2e42 100644 --- a/docs/message_pack_format.md +++ b/docs/message_pack_format.md @@ -169,9 +169,9 @@ decode → re-encode them byte-identically. The `kind_id` registry is mirrored verbatim with Go's `wire/asapmsgpack/magic_ids.go`, never independently allocated. -That proof covers the six `kind_id`s a fixture exists for — HLL's three -estimators, Count-Min, Count Sketch and compact KLL. Every other implemented -kind has **no ASAPv1 golden and therefore no ASAPv1 drift guard**; +That proof covers the seven `kind_id`s a fixture exists for — HLL's three +estimators, Count-Min, CMSHeap, Count Sketch and compact KLL. Every other +implemented kind has **no ASAPv1 golden and therefore no ASAPv1 drift guard**; [its README](https://github.com/ProjectASAP/sketchlib-golden-bytes#coverage) lists what is covered. The older `portable` path is guarded separately, by the `sketchlib-go` goldens of its own listed above. diff --git a/docs/tests.md b/docs/tests.md index ccb1571c..38687f8f 100644 --- a/docs/tests.md +++ b/docs/tests.md @@ -1240,20 +1240,6 @@ Test file: [`src/message_pack_format/portable/countminsketch.rs`](../src/message | `test_compute_delta_then_apply_matches_current` | A delta between two non-empty snapshots reconstructs the later one. | For a `2x64` base fed 40 updates of `k{i % 8}` and a current fed 30 more, verifies the delta applied to the base reproduces the current matrix. | | `test_compute_delta_matches_go_golden_bytes` | The packed delta bytes are pinned by a golden. | For a `4x2048` sketch fed 50 updates of `flow-{i % 10}` at weight `1.0`, verifies the delta against an empty sketch at threshold `1.0` equals a 239-byte inline golden. | -### Portable CountMinSketchWithHeap - -Test file: [`src/message_pack_format/portable/countminsketch_topk.rs`](../src/message_pack_format/portable/countminsketch_topk.rs) - -| test_name | test_description | what_is_tested | -| --- | --- | --- | -| `test_creation` | A fresh composite has the requested matrix and an empty heap. | For `CountMinSketchWithHeap::new(4, 1000, 20)`, verifies `rows` `4`, `cols` `1000`, `heap_size` `20`, a `sketch_matrix()` of 4 rows of 1,000, and no heap items. | -| `test_query_empty` | An empty composite answers zero. | Verifies a `2x10` composite with a 5-slot heap reports `estimate("anything")` of `0.0`. | -| `test_merge` | Merge sums the matrices and adopts the peer's heap bound. | Merges a `2x10` composite holding `5.0` at `(0,0)`, `15.0` at `(1,1)`, heap `{key3: 75.0, key1: 80.0}` and `heap_size` `3` into one holding `10.0` at `(0,0)`, `20.0` at `(1,1)`, heap `{key1: 100.0, key2: 50.0}` and `heap_size` `5`, and verifies the merged cells read `15.0` and `35.0`, `heap_size` becomes `3`, and the heap holds at most 3 items. | -| `test_merge_dimension_mismatch` | Mismatched row counts are refused. | Verifies merging a `3x10` composite into a `2x10` one fails. | -| `test_msgpack_round_trip` | The round trip carries the matrix, the heap bound, and the heap itself. | Round-trips a `4x128` composite with a 3-slot heap fed `"hot"` at `100.0` and `"cold"` at `1.0`, and verifies the decode reports `rows` `4`, `cols` `128`, `heap_size` `3`, a non-empty heap holding `"hot"` at or above `100.0`, and estimates at or above `100.0` and `1.0`. | -| `test_aggregate_topk` | `aggregate_topk` builds a decodable composite honouring its heap bound. | For `aggregate_topk(4, 100, 2, ["a", "b", "a", "c"], [1.0, 2.0, 3.0, 0.5])`, verifies the decode reports `heap_size` `2` and at most 2 heap items; the heap contents themselves are not checked. | -| `test_aggregate_topk_empty` | An empty stream produces nothing. | Verifies `aggregate_topk(4, 100, 10, &[], &[])` returns `None`. | - ### Portable CountSketch Test file: [`src/message_pack_format/portable/countsketch.rs`](../src/message_pack_format/portable/countsketch.rs) @@ -1950,13 +1936,13 @@ Test file: [`tests/e2e/composition.rs`](../tests/e2e/composition.rs) | `univmonq_with_window_bound_chooses_a_hierarchy_that_satisfies_its_own_inequality` | `with_window_bound` picks the smallest level count whose deepest stratum still fits the candidate table. | For `max_updates` in `10_000`, `1_000_000` and `100_000_000` at `delta = 1e-3`, calls `UnivMonQConfig::default().with_window_bound` and re-derives the Bernstein upper tail `mean + sqrt(2*mean*ln(1/delta)) + (2/3)*ln(1/delta)` with `mean = max_updates / 2^(levels-1)`, asserting it is strictly below `cfg.candidates`; asserts minimality by requiring the same bound at `levels - 1` to be at or above `candidates` whenever `levels > 2`; asserts `levels` is non-decreasing across the three windows; builds each chosen config and asserts `count() == 1_000` after `1_000` updates; and asserts `with_window_bound(1_000, 1.5)` is an error. | | `univmonq_multi_shard_merge_with_distinct_source_ids_covers_the_union` | Shards with distinct source ids merge into a sketch whose exact aggregates cover the union. | Builds `4` shards with `UnivMonQ::with_hasher_and_source_id(UnivMonQConfig::default(), i + 1)`, deals `40_000` uniform values over `1_000_000` (seed `0xC090_5101`) round robin by `i % 4`, asserts each shard's `source_id()` is `i + 1`, merges the other three into the first with `merge` returning `Ok`, and asserts the merged `count()` is `40_000` and `min()`/`max()` equal the whole stream's exact extremes. Only the distinct-id path is exercised: no assertion covers a merge of shards that share a source id, and no estimator is checked. | -### Composition: Portable Sketch-With-Heap Facades +### Composition: Sketch-With-Heap Types and Portable Facades Test file: [`tests/e2e/composition.rs`](../tests/e2e/composition.rs) | test_name | test_description | what_is_tested | | --- | --- | --- | -| `portable_count_min_with_heap_satisfies_the_count_min_bound_through_merge_and_wire` | The portable Count-Min-plus-heap holds Count-Min's bound and an agreeing heap through a merge and a MessagePack round trip. | Feeds `40_000` Zipf(1.1) keys over a `2_048` domain (seed `0xC090_5101`) as `"k{key}"` strings into `CountMinSketchWithHeap::new(3, 4_096, 32)`, plus even/odd halves into two more. Runs `CountMinSpec::new(3, 4_096).assert_contract` on the single sketch, asserts every `topk_heap_items()` entry's `value` equals `estimate(&item.key)` with zero violations, re-runs the contract on `CountMinSketchWithHeap::merge_refs(&[&left, &right])` against the full truth, and after `to_msgpack`/`from_msgpack` asserts the decoded sketch's `estimate` is exactly equal to the original's for every observed key and still satisfies the contract. | +| `cms_heap_satisfies_the_count_min_bound_through_merge_and_wire` | `CMSHeap` holds Count-Min's bound and an agreeing heap through a merge and an ASAPv1 round trip. | Feeds `40_000` Zipf(1.1) keys over a `2_048` domain (seed `0xC090_5101`) as `DataInput::String("k{key}")` into `CMSHeap::, RegularPath>::new(3, 4_096, 32)`, plus even/odd halves into two more. Runs `CountMinSpec::new(3, 4_096).assert_contract` on the single sketch, asserts every heap entry's `count` equals `estimate` of its string key with zero violations, re-runs the contract on the even half merged with the odd half against the full truth, and after `serialize_to_bytes`/`deserialize_from_bytes` asserts the decoded sketch's `estimate` is exactly equal to the original's for every observed key and still satisfies the contract. | | `portable_count_sketch_with_heap_satisfies_the_l2_bound_through_merge_and_wire` | The portable Count-Sketch-plus-heap holds the L2 bound and an agreeing heap through a merge and a MessagePack round trip. | Feeds the same `40_000`-key Zipf(1.1) stream (domain `2_048`, seed `0xC090_5101`) as `"k{key}"` strings into `CountSketchWithHeap::new(5, 4_096, 32)`, plus even/odd halves into two more, and runs `CountSketchSpec::new(5, 4_096).assert_contract` (simultaneous L2 band at `delta = 1e-3` with zero violations, marginal `sqrt(3/4096)*\|\|f_-i\|\|_2` rate pinned at `P[Bin(5, 1/3) >= 3] = 51/243`) on the single sketch, on `merge_refs(&[&left, &right])` and on the `from_msgpack` decode; every `topk_heap_items()` entry's `value` must equal `estimate(&item.key)`. Unlike the Count-Min twin, the wire round trip is only re-checked against the band - no per-key equality across the encode/decode is asserted. | | `portable_kll_sketch_satisfies_the_rank_error_characterization_through_merge_and_wire` | The portable KLL facade holds the rank-error characterization single-pass and after a two-shard merge, and its wire round trip is bit-for-bit. | Over `12` compaction seeds `0x5EED_0400 + t * 0x9E37_79B9_7F4A_7C15`, builds `KllSketch::with_seed(200, seed)` plus even/odd shards at `seed ^ 0xAAAA` and `seed ^ 0x5555`, feeds all of them `40_000` uniform values over `1_000_000` (seed `0xC090_5101`), and merges the shards with `merge`. Each seed records two trials - the single-pass sketch and the merged pair - of the worst normalized rank error over the grid `[0.1, 0.25, 0.5, 0.75, 0.9]` against `KllRankSpec::datasketches(200)`'s `eps = 2.446/200^0.9433 = 0.0165`, and the `24` outcomes are accepted by `assert_independent_binomial` at `p = 0.01`. Per seed it also asserts `to_msgpack`/`from_msgpack` preserves `k() == 200`, `count()`, and every grid quantile exactly, and that merging a `k = 400` sketch into the single-pass one errs. | @@ -2123,6 +2109,8 @@ Test file: [`tests/asapv1_golden.rs`](../tests/asapv1_golden.rs) | --- | --- | --- | | `cms_f64_fast_2x3_matches_golden` | A known f64 fast-path Count-Min matrix serializes to its checked-in golden bytes. | Verifies `CountMin::, FastPath>::from_storage` over the `2x3` matrix `[[0.0, 1.5, 2.25], [3.75, 4.125, 5.0625]]` serializes to `asapv1_golden/cms_f64_fast_2x3.hex`, and that decoding that fixture recovers the six cells in row-major order and re-serializes to the same bytes. | | `cms_i64_regular_2x3_matches_golden` | A known i64 regular-path Count-Min matrix serializes to its checked-in golden bytes. | Verifies `CountMin::, RegularPath>::from_storage` over the `2x3` matrix `[[0, 1, 127], [128, 300, 65536]]` serializes to `asapv1_golden/cms_i64_regular_2x3.hex`, and that decoding the fixture reports `rows()` `2`, `cols()` `3`, the six cells in row-major order, and a byte-identical re-serialization. | +| `cmsheap_i32_fast_2x3_i64keys_matches_golden` | A known i32 fast-path CMSHeap with `i64` keys serializes to its golden bytes. | Builds `CMSHeap::, FastPath>::from_storage` over `[[0, 1, 127], [128, 300, 65536]]` with `k` `3`, seats `HeapItem::I64` entries `4294967296: 3`, `-1: 7` and `-129: 7` with `update_heap_item`, verifies the bytes equal `asapv1_golden/cmsheap_i32_fast_2x3_i64keys.hex`, and that decoding the fixture recovers the six cells, capacity `3`, exactly those three entries, and a byte-identical re-serialization. | +| `cmsheap_i64_regular_2x3_strkeys_matches_golden` | A known i64 regular-path CMSHeap with string keys serializes to its golden bytes, whatever order its entries were seated in. | Builds `CMSHeap::, RegularPath>::from_storage` over `[[0, 1, 127], [128, 300, 65536]]` with `k` `5`, seats `HeapItem::String` entries `cold: 1`, `warm: 300`, `hot: 65536` and `mild: 300` in that order, verifies the bytes equal `asapv1_golden/cmsheap_i64_regular_2x3_strkeys.hex`, and that decoding the fixture reports `rows()` `2`, `cols()` `3`, the six cells, capacity `5`, exactly those four entries, and a byte-identical re-serialization. | | `cs_i32_regular_2x4_matches_golden` | The i32 counter type reaches the metadata, and an i32 golden is not an i64 sketch. | Verifies `Count::, RegularPath>::from_storage` over `[[0, 127, 128, 65536], [-1, -33, -32768, -2147483648]]` cast to `i32` serializes to `asapv1_golden/cs_i32_regular_2x4.hex`, that decoding it recovers the eight cells and re-serializes identically, that the i32 and i64 regular fixtures have equal length yet differ, and that the i32 fixture fails to decode as `Count, RegularPath>`. | | `cs_i64_fast_2x4_matches_golden` | The fast-path mode reaches the bytes, so the same matrix encodes differently per path. | Verifies `Count::, FastPath>::from_storage` over `[[0, 127, 128, 65536], [-1, -33, -32768, -2147483648]]` serializes to `asapv1_golden/cs_i64_fast_2x4.hex`, that decoding it recovers the eight cells and re-serializes identically, and that the fixture differs from `cs_i64_regular_2x4.hex` built from the same matrix. | | `cs_i64_regular_2x4_matches_golden` | A known signed Count-Sketch matrix serializes to its checked-in golden bytes. | Verifies `Count::, RegularPath>::from_storage` over `[[0, 127, 128, 65536], [-1, -33, -32768, -2147483648]]` — sweeping the msgpack integer widths in both directions — serializes to `asapv1_golden/cs_i64_regular_2x4.hex`, and that decoding the fixture reports `rows()` `2`, `cols()` `4`, the eight cells, and a byte-identical re-serialization. | @@ -2140,9 +2128,7 @@ Test file: [`tests/msgpack_compat.rs`](../tests/msgpack_compat.rs) | --- | --- | --- | | `count_min_decodes_go_bytes` | The test asserts nothing. | The test asserts nothing: the body is empty apart from three commented-out lines sketching an `include_bytes!` of `fixtures/msgpack/count_min.msgpack` and a `CountMinSketch::from_msgpack` check, and it is marked `#[ignore]` with the reason that no cross-language msgpack fixture exists in the repository. | | `count_min_sketch_round_trip` | A portable `CountMinSketch` survives a msgpack round trip with its geometry intact. | Updates `CountMinSketch::new(3, 64)` with `alpha` `+1.0`, `beta` `+2.0` and `alpha` `+4.0`, then verifies `from_msgpack(to_msgpack())` reports `rows` `3` and `cols` `64`; the counter matrix and the per-key estimates are not checked. | -| `count_min_sketch_with_heap_round_trip` | A portable `CountMinSketchWithHeap` survives a msgpack round trip with its geometry and heap size intact. | Updates `CountMinSketchWithHeap::new(3, 64, 8)` with `hot` `+100.0` and `warm` `+10.0`, then verifies the decode reports `rows` `3`, `cols` `64` and `heap_size` `8`; the heap's contents are not checked. | | `count_min_wire_shape` | The `CountMinSketchWire` DTO round-trips through `rmp_serde` with its field shape intact. | Serializes `CountMinSketchWire { sketch: vec![vec![1.0, 2.0]; 3], rows: 3, cols: 2 }` with `rmp_serde::to_vec` and reads it back with `rmp_serde::from_slice`, verifying `rows` `3`, `cols` `2` and a `sketch` of length `3`; the cell values are not checked. | -| `count_min_with_heap_wire_shape` | The nested `CountMinSketchWithHeapWire` DTO round-trips through `rmp_serde` with its nesting intact. | Serializes a wire struct holding a `CountMinSketchInnerWire` of `2x4` zeros, one `CmsHeapItem { key: "hot", value: 42.0 }` and `heap_size: 8`, then verifies the `rmp_serde` decode reports `heap_size` `8`, one heap entry and the key `hot`; the item's value and the inner matrix are not re-checked. | | `count_sketch_round_trip` | A portable `CountSketch` survives a msgpack round trip with its geometry intact. | Updates `CountSketch::new(3, 64)` with `k1` `+1.0` and `k2` `+2.0`, then verifies the decode reports `rows` `3` and `cols` `64`; the signed counters and the estimates are not checked. | | `dd_sketch_decodes_go_bytes` | The test asserts nothing. | The test asserts nothing: the body is empty and the test is marked `#[ignore]` with the reason that no cross-language msgpack fixture exists in the repository. | | `dd_sketch_round_trip` | A portable `DdSketch` survives a msgpack round trip carrying its full count in the bucket array. | Updates `DdSketch::new(0.01)` with `1.0`, `10.0` and `100.0`, then verifies the decode's `total_count()` — recovered by summing the buckets, since the count is not on the wire — is `3`; `alpha`, the bucket offset and the quantiles are not checked. | diff --git a/src/lib.rs b/src/lib.rs index b062b8b9..ebf209c9 100644 --- a/src/lib.rs +++ b/src/lib.rs @@ -11,8 +11,7 @@ //! - [`message_pack_format`]: MessagePack/proto wire format shared with //! `sketchlib-go`. Owns the wire-DTO sketch types consumed by the ASAP //! query engine (`CountMinSketch`, `CountSketch`, `KllSketch`, `HllSketch`, -//! `DdSketch`, `HydraKllSketch`, `CountMinSketchWithHeap`, -//! `SetAggregator`, `DeltaResult`). +//! `DdSketch`, `HydraKllSketch`, `SetAggregator`, `DeltaResult`). //! //! Most users can start with the crate-root re-exports such as [`DataInput`], //! [`CountMin`], [`HyperLogLog`], [`KLL`], and [`DDSketch`]. Reach for the @@ -42,7 +41,6 @@ pub mod __private { pub use common::*; pub use message_pack_format::MessagePackCodec; pub use message_pack_format::portable::countminsketch::{CountMinSketch, CountMinSketchDelta}; -pub use message_pack_format::portable::countminsketch_topk::{CmsHeapItem, CountMinSketchWithHeap}; pub use message_pack_format::portable::countsketch::{ COUNT_SKETCH_TOPK_CAPACITY, CountSketch, CountSketchDelta, }; diff --git a/src/message_pack_format/portable/countminsketch_topk.rs b/src/message_pack_format/portable/countminsketch_topk.rs deleted file mode 100644 index 0227824a..00000000 --- a/src/message_pack_format/portable/countminsketch_topk.rs +++ /dev/null @@ -1,447 +0,0 @@ -//! Wire-format-aligned Count-Min sketch + top-k heap composite. -//! -//! No `sketches::*` equivalent — this combines `sketches::CMSHeap` -//! with the wire-format `CountMinSketch` shape and exposes it as a -//! single Go-interop type. - -use serde::{Deserialize, Serialize}; - -use crate::message_pack_format::{Error as MsgPackError, MessagePackCodec}; -use crate::sketches::countminsketch_topk::CMSHeap; -use crate::{DataInput, RegularPath, Vector2D}; - -/// Wire-format heap item (key, value) used by the dispatch helpers below. -pub struct WireHeapItem { - pub key: String, - pub value: f64, -} - -/// Concrete Count-Min-with-Heap type backing the wire-format -/// `CountMinSketchWithHeap`. -pub type SketchlibCMSHeap = CMSHeap, RegularPath>; - -/// Creates a fresh CMSHeap with the given dimensions and heap capacity. -pub fn new_sketchlib_cms_heap( - row_num: usize, - col_num: usize, - heap_size: usize, -) -> SketchlibCMSHeap { - CMSHeap::new(row_num, col_num, heap_size) -} - -/// Builds a CMSHeap from an existing sketch matrix and optional heap -/// items. Used when deserializing wire-format state. -pub fn sketchlib_cms_heap_from_matrix_and_heap( - row_num: usize, - col_num: usize, - heap_size: usize, - sketch: &[Vec], - topk_heap: &[WireHeapItem], -) -> SketchlibCMSHeap { - let matrix = Vector2D::from_fn(row_num, col_num, |r, c| { - sketch - .get(r) - .and_then(|row| row.get(c)) - .copied() - .unwrap_or(0.0) - .round() as i64 - }); - let mut cms_heap = CMSHeap::from_storage(matrix, heap_size); - - for item in topk_heap { - let count = item.value.round() as i64; - if count > 0 { - let input = DataInput::Str(&item.key); - cms_heap.heap_mut().update(&input, count); - } - } - - cms_heap -} - -/// Converts a CMSHeap's storage into a `Vec>` matrix. -pub fn matrix_from_sketchlib_cms_heap(cms_heap: &SketchlibCMSHeap) -> Vec> { - let storage = cms_heap.cms().as_storage(); - let rows = storage.rows(); - let cols = storage.cols(); - let mut sketch = vec![vec![0.0; cols]; rows]; - - for (r, row) in sketch.iter_mut().enumerate().take(rows) { - for (c, cell) in row.iter_mut().enumerate().take(cols) { - if let Some(v) = storage.get(r, c) { - *cell = *v as f64; - } - } - } - - sketch -} - -/// Converts sketchlib HHHeap items to wire-format (key, value) pairs. -pub fn heap_to_wire(cms_heap: &SketchlibCMSHeap) -> Vec { - cms_heap - .heap() - .heap() - .iter() - .map(|hh_item| { - let key = match &hh_item.key { - crate::HeapItem::String(s) => s.clone(), - other => format!("{other:?}"), - }; - WireHeapItem { - key, - value: hh_item.count as f64, - } - }) - .collect() -} - -/// Updates a CMSHeap with a weighted key. Automatically updates the heap. -pub fn sketchlib_cms_heap_update(cms_heap: &mut SketchlibCMSHeap, key: &str, value: f64) { - let many = value.round() as i64; - if many <= 0 { - return; - } - cms_heap.insert_many(&DataInput::String(key.to_owned()), many); -} - -/// Queries a CMSHeap for a key's frequency estimate. -pub fn sketchlib_cms_heap_query(cms_heap: &SketchlibCMSHeap, key: &str) -> f64 { - cms_heap.estimate(&DataInput::String(key.to_owned())) as f64 -} - -/// Item in the top-k heap representing a key-value pair. -#[derive(Debug, Clone, Serialize, Deserialize)] -pub struct CmsHeapItem { - pub key: String, - pub value: f64, -} - -/// Count-Min Sketch with Heap for top-k tracking. Combines probabilistic -/// frequency counting with efficient top-k maintenance. -pub struct CountMinSketchWithHeap { - pub rows: usize, - pub cols: usize, - pub heap_size: usize, - pub(crate) backend: SketchlibCMSHeap, -} - -impl std::fmt::Debug for CountMinSketchWithHeap { - fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { - f.debug_struct("CountMinSketchWithHeap") - .field("rows", &self.rows) - .field("cols", &self.cols) - .field("heap_size", &self.heap_size) - .finish() - } -} - -impl Clone for CountMinSketchWithHeap { - fn clone(&self) -> Self { - // Structural clone: `CMSHeap` derives `Clone` (its `CountMin` + `HHHeap` - // fields are `Clone`), so copy the backend directly instead of - // extracting the matrix + heap to wire form and rebuilding it. - Self { - rows: self.rows, - cols: self.cols, - heap_size: self.heap_size, - backend: self.backend.clone(), - } - } -} - -impl CountMinSketchWithHeap { - pub fn new(rows: usize, cols: usize, heap_size: usize) -> Self { - Self { - rows, - cols, - heap_size, - backend: new_sketchlib_cms_heap(rows, cols, heap_size), - } - } - - pub fn rows(&self) -> usize { - self.rows - } - - pub fn cols(&self) -> usize { - self.cols - } - - /// Create from a sketch matrix and heap (e.g. from JSON deserialization). - pub fn from_legacy_matrix( - sketch: Vec>, - topk_heap: Vec, - rows: usize, - cols: usize, - heap_size: usize, - ) -> Self { - let wire_heap: Vec = topk_heap - .into_iter() - .map(|h| WireHeapItem { - key: h.key, - value: h.value, - }) - .collect(); - Self { - rows, - cols, - heap_size, - backend: sketchlib_cms_heap_from_matrix_and_heap( - rows, cols, heap_size, &sketch, &wire_heap, - ), - } - } - - /// Get the top-k heap items. - pub fn topk_heap_items(&self) -> Vec { - heap_to_wire(&self.backend) - .into_iter() - .map(|w| CmsHeapItem { - key: w.key, - value: w.value, - }) - .collect() - } - - /// Get the sketch matrix. - pub fn sketch_matrix(&self) -> Vec> { - matrix_from_sketchlib_cms_heap(&self.backend) - } - - pub fn update(&mut self, key: &str, value: f64) { - sketchlib_cms_heap_update(&mut self.backend, key, value); - } - - pub fn estimate(&self, key: &str) -> f64 { - sketchlib_cms_heap_query(&self.backend, key) - } - - /// Merge another CountMinSketchWithHeap into self in place. Both - /// operands must have identical dimensions; the resulting `heap_size` - /// is the minimum of the two. - pub fn merge( - &mut self, - other: &CountMinSketchWithHeap, - ) -> Result<(), Box> { - if self.rows != other.rows || self.cols != other.cols { - return Err(format!( - "CountMinSketchWithHeap dimension mismatch: self={}x{}, other={}x{}", - self.rows, self.cols, other.rows, other.cols - ) - .into()); - } - self.backend.merge(&other.backend); - self.heap_size = self.heap_size.min(other.heap_size); - Ok(()) - } - - pub fn merge_refs( - inputs: &[&CountMinSketchWithHeap], - ) -> Result> { - let first = inputs - .first() - .ok_or("CountMinSketchWithHeap::merge_refs called with empty input")?; - let mut merged = (*first).clone(); - for h in inputs.iter().skip(1) { - merged.merge(h)?; - } - Ok(merged) - } - - pub fn aggregate_topk( - rows: usize, - cols: usize, - heap_size: usize, - keys: &[&str], - values: &[f64], - ) -> Option> { - if keys.is_empty() { - return None; - } - let mut sketch = Self::new(rows, cols, heap_size); - for (key, &value) in keys.iter().zip(values.iter()) { - sketch.update(key, value); - } - sketch.to_msgpack().ok() - } -} - -// ----- Wire format ----- - -/// Inner CMS payload nested inside [`CountMinSketchWithHeapWire`]. -#[derive(Debug, Clone, Serialize, Deserialize)] -pub struct CountMinSketchInnerWire { - pub sketch: Vec>, - #[serde(rename = "row_num")] - pub rows: usize, - #[serde(rename = "col_num")] - pub cols: usize, -} - -#[derive(Debug, Clone, Serialize, Deserialize)] -pub struct CountMinSketchWithHeapWire { - pub sketch: CountMinSketchInnerWire, - pub topk_heap: Vec, - pub heap_size: usize, -} - -impl MessagePackCodec for CountMinSketchWithHeap { - fn to_msgpack(&self) -> Result, MsgPackError> { - let wire = CountMinSketchWithHeapWire { - sketch: CountMinSketchInnerWire { - sketch: self.sketch_matrix(), - rows: self.rows, - cols: self.cols, - }, - topk_heap: self.topk_heap_items(), - heap_size: self.heap_size, - }; - Ok(rmp_serde::to_vec(&wire)?) - } - - fn from_msgpack(bytes: &[u8]) -> Result { - let wire: CountMinSketchWithHeapWire = rmp_serde::from_slice(bytes)?; - - let mut sorted_topk_heap = wire.topk_heap; - sorted_topk_heap.sort_by(|a, b| b.value.partial_cmp(&a.value).unwrap()); - - let wire_heap: Vec = sorted_topk_heap - .iter() - .map(|h| WireHeapItem { - key: h.key.clone(), - value: h.value, - }) - .collect(); - let backend = sketchlib_cms_heap_from_matrix_and_heap( - wire.sketch.rows, - wire.sketch.cols, - wire.heap_size, - &wire.sketch.sketch, - &wire_heap, - ); - - Ok(Self { - rows: wire.sketch.rows, - cols: wire.sketch.cols, - heap_size: wire.heap_size, - backend, - }) - } -} - -#[cfg(test)] -mod tests { - use super::*; - - #[test] - fn test_creation() { - let cms = CountMinSketchWithHeap::new(4, 1000, 20); - assert_eq!(cms.rows, 4); - assert_eq!(cms.cols, 1000); - assert_eq!(cms.heap_size, 20); - assert_eq!(cms.sketch_matrix().len(), 4); - assert_eq!(cms.sketch_matrix()[0].len(), 1000); - assert_eq!(cms.topk_heap_items().len(), 0); - } - - #[test] - fn test_query_empty() { - let cms = CountMinSketchWithHeap::new(2, 10, 5); - assert_eq!(cms.estimate("anything"), 0.0); - } - - #[test] - fn test_merge() { - let mut sketch1 = vec![vec![0.0; 10]; 2]; - sketch1[0][0] = 10.0; - sketch1[1][1] = 20.0; - let mut cms1 = CountMinSketchWithHeap::from_legacy_matrix( - sketch1, - vec![ - CmsHeapItem { - key: "key1".to_string(), - value: 100.0, - }, - CmsHeapItem { - key: "key2".to_string(), - value: 50.0, - }, - ], - 2, - 10, - 5, - ); - let mut sketch2 = vec![vec![0.0; 10]; 2]; - sketch2[0][0] = 5.0; - sketch2[1][1] = 15.0; - let cms2 = CountMinSketchWithHeap::from_legacy_matrix( - sketch2, - vec![ - CmsHeapItem { - key: "key3".to_string(), - value: 75.0, - }, - CmsHeapItem { - key: "key1".to_string(), - value: 80.0, - }, - ], - 2, - 10, - 3, - ); - - cms1.merge(&cms2).unwrap(); - - assert_eq!(cms1.sketch_matrix()[0][0], 15.0); - assert_eq!(cms1.sketch_matrix()[1][1], 35.0); - assert_eq!(cms1.heap_size, 3); - assert!(cms1.topk_heap_items().len() <= 3); - } - - #[test] - fn test_merge_dimension_mismatch() { - let mut cms1 = CountMinSketchWithHeap::new(2, 10, 5); - let cms2 = CountMinSketchWithHeap::new(3, 10, 5); - assert!(cms1.merge(&cms2).is_err()); - } - - #[test] - fn test_msgpack_round_trip() { - let mut cms = CountMinSketchWithHeap::new(4, 128, 3); - cms.update("hot", 100.0); - cms.update("cold", 1.0); - - let bytes = cms.to_msgpack().unwrap(); - let deserialized = CountMinSketchWithHeap::from_msgpack(&bytes).unwrap(); - - assert_eq!(deserialized.rows, 4); - assert_eq!(deserialized.cols, 128); - assert_eq!(deserialized.heap_size, 3); - let items = deserialized.topk_heap_items(); - assert!(!items.is_empty()); - let hot = items - .iter() - .find(|item| item.key == "hot") - .expect("'hot' should be in the heap"); - assert!(hot.value >= 100.0); - assert!(deserialized.estimate("hot") >= 100.0); - assert!(deserialized.estimate("cold") >= 1.0); - } - - #[test] - fn test_aggregate_topk() { - let keys = ["a", "b", "a", "c"]; - let values = [1.0, 2.0, 3.0, 0.5]; - let bytes = CountMinSketchWithHeap::aggregate_topk(4, 100, 2, &keys, &values).unwrap(); - let cms = CountMinSketchWithHeap::from_msgpack(&bytes).unwrap(); - assert_eq!(cms.heap_size, 2); - assert!(cms.topk_heap_items().len() <= 2); - } - - #[test] - fn test_aggregate_topk_empty() { - assert!(CountMinSketchWithHeap::aggregate_topk(4, 100, 10, &[], &[]).is_none()); - } -} diff --git a/src/message_pack_format/portable/countsketch_topk.rs b/src/message_pack_format/portable/countsketch_topk.rs index 82ec8958..6cf252ac 100644 --- a/src/message_pack_format/portable/countsketch_topk.rs +++ b/src/message_pack_format/portable/countsketch_topk.rs @@ -2,11 +2,7 @@ //! //! No `sketches::*` equivalent — this combines `sketches::CSHeap` with //! the wire-format `CountSketch` shape and exposes it as a single -//! Go-interop type. Mirrors [`crate::message_pack_format::portable::countminsketch_topk::CountMinSketchWithHeap`] -//! structurally, but backed by `CSHeap` (median estimator, signed rows) -//! instead of `CMSHeap` (min-over-rows estimator) — the two are -//! different sketch algorithms that happen to produce a same-shaped -//! `rows x cols` matrix, not interchangeable data. +//! Go-interop type, backed by `CSHeap` (median estimator, signed rows). use serde::{Deserialize, Serialize}; @@ -447,7 +443,6 @@ mod tests { assert!(CountSketchWithHeap::aggregate_topk(4, 100, 10, &[], &[]).is_none()); } - /// The whole point of splitting this from `CountMinSketchWithHeap`: /// CS's median estimator must behave differently from CMS's /// min-over-rows estimator on the same kind of skewed insert /// pattern, not just share a struct shape. diff --git a/src/message_pack_format/portable/mod.rs b/src/message_pack_format/portable/mod.rs index 97e0d372..f881932a 100644 --- a/src/message_pack_format/portable/mod.rs +++ b/src/message_pack_format/portable/mod.rs @@ -8,7 +8,6 @@ //! prefer a sketch's own `serialize_to_bytes`; do not add callers here. pub mod countminsketch; -pub mod countminsketch_topk; pub mod countsketch; pub mod countsketch_topk; pub mod ddsketch; diff --git a/tests/README.md b/tests/README.md index f18d2bfb..673ee5e9 100644 --- a/tests/README.md +++ b/tests/README.md @@ -23,7 +23,7 @@ its theorem. | `e2e/matrix_instances.rs` | Every built-in `(storage, hashing path)` instance of CountMin / Count / CMSHeap / CSHeap, plus counter-width edges | | `e2e/numeric_types.rs` | Every `NumericalValue` type through `KLL`, `KLLDynamic` and `DDSketch::add` | | `e2e/windows.rs` | Every `EHSketchList` variant in an `ExponentialHistogram`, every `TumblingWindow` payload, and `MicroCM`'s in-cell sliding window | -| `e2e/composition.rs` | `HashSketchEnsemble`, `UnivMonQ`'s config surface, and the portable sketch+heap facade | +| `e2e/composition.rs` | `HashSketchEnsemble`, `UnivMonQ`'s config surface, `CMSHeap`, and the portable sketch+heap facade | | `e2e/nitro.rs` | Every Nitro ingestion path — row-level `CountMin`/`Count`, `NitroBatch::insert` / `insert_cached_step`, and the bare `Vector2D` target — under one sampling model, plus merge, seeding, saturation and serde/context continuation. **All Nitro E2E behaviour lives here**; the sampler's own structural tests are unit tests in `src/common/structure_utils.rs` and `src/sketch_framework/nitro.rs`, where the cursor and skip counter are visible without a public accessor | | `e2e/frameworks.rs` | Hydra's subpopulation lattice and UnivMon composition | | `e2e/heavy_hitters.rs` | Space-Saving's error sandwich, `min_count` ceiling and Stream-Summary lists under sustained eviction, plus CocoSketch's exact mass partitions and multi-seed unbiasedness, and Elastic through the batteries and the heavy-hitter properties no battery models | diff --git a/tests/asapv1_golden.rs b/tests/asapv1_golden.rs index 30b41651..759affd4 100644 --- a/tests/asapv1_golden.rs +++ b/tests/asapv1_golden.rs @@ -9,8 +9,8 @@ //! See `asapv1_golden/README.md`. use asap_sketchlib::{ - Classic, Count, CountMin, ErtlMLE, FastPath, HllSketch, HllVariant, HyperLogLogHIPP12, - HyperLogLogP12, KLL, MessagePackCodec, RegularPath, Vector2D, + CMSHeap, Classic, Count, CountMin, ErtlMLE, FastPath, HeapItem, HllSketch, HllVariant, + HyperLogLogHIPP12, HyperLogLogP12, KLL, MessagePackCodec, RegularPath, Vector2D, }; fn decode_hex(s: &str) -> Vec { @@ -27,6 +27,9 @@ const GOLDEN_ERTL: &str = include_str!("../asapv1_golden/hll_ertl_mle_p12.hex"); const GOLDEN_HIP: &str = include_str!("../asapv1_golden/hll_hip_p12.hex"); const GOLDEN_CMS_I64: &str = include_str!("../asapv1_golden/cms_i64_regular_2x3.hex"); const GOLDEN_CMS_F64: &str = include_str!("../asapv1_golden/cms_f64_fast_2x3.hex"); +const GOLDEN_CMSHEAP_STR: &str = + include_str!("../asapv1_golden/cmsheap_i64_regular_2x3_strkeys.hex"); +const GOLDEN_CMSHEAP_I64: &str = include_str!("../asapv1_golden/cmsheap_i32_fast_2x3_i64keys.hex"); const GOLDEN_CS_REGULAR: &str = include_str!("../asapv1_golden/cs_i64_regular_2x4.hex"); const GOLDEN_CS_FAST: &str = include_str!("../asapv1_golden/cs_i64_fast_2x4.hex"); const GOLDEN_CS_I32: &str = include_str!("../asapv1_golden/cs_i32_regular_2x4.hex"); @@ -162,6 +165,96 @@ fn cms_f64_fast_2x3_matches_golden() { assert_eq!(decoded.serialize_to_bytes().expect("re-serialize"), want); } +// --------------------------------------------------------------------------- +// CMSHeap: the Count-Min i64 matrix plus heap entries set directly. Each heap +// holds a count tie, so the fixtures pin the emitted entry order. +// --------------------------------------------------------------------------- + +/// `(key, count)` entries of the string-keyed fixture, in emitted order. +const CMSHEAP_STR_ENTRIES: [(&str, i64); 4] = + [("hot", 65536), ("mild", 300), ("warm", 300), ("cold", 1)]; + +/// `(key, count)` entries of the i64-keyed fixture, in emitted order. +const CMSHEAP_I64_ENTRIES: [(i64, i64); 3] = [(-129, 7), (-1, 7), (4_294_967_296, 3)]; + +#[test] +fn cmsheap_i64_regular_2x3_strkeys_matches_golden() { + let want = decode_hex(GOLDEN_CMSHEAP_STR); + + let mut sketch = CMSHeap::, RegularPath>::from_storage( + Vector2D::from_fn(2, 3, |r, c| I64_VALS[r][c]), + 5, + ); + for (key, count) in [("cold", 1), ("warm", 300), ("hot", 65536), ("mild", 300)] { + sketch + .heap_mut() + .update_heap_item(&HeapItem::String(key.to_string()), count); + } + let got = sketch.serialize_to_bytes().expect("serialize"); + assert_eq!( + got, want, + "CMSHeap i64/regular/string bytes diverge from golden" + ); + + let decoded = + CMSHeap::, RegularPath>::deserialize_from_bytes(&want).expect("decode"); + let flat: Vec = I64_VALS.iter().flatten().copied().collect(); + assert_eq!(decoded.cms().as_storage().as_slice(), flat.as_slice()); + assert_eq!((decoded.rows(), decoded.cols()), (2, 3)); + assert_eq!(decoded.heap().capacity(), 5); + let mut held: Vec<(String, i64)> = decoded + .heap() + .heap() + .iter() + .map(|item| match &item.key { + HeapItem::String(k) => (k.clone(), item.count), + other => panic!("key {other:?} is not a string"), + }) + .collect(); + held.sort_by(|a, b| b.1.cmp(&a.1).then_with(|| a.0.cmp(&b.0))); + let expected: Vec<(String, i64)> = CMSHEAP_STR_ENTRIES + .iter() + .map(|&(k, c)| (k.to_string(), c)) + .collect(); + assert_eq!(held, expected); + assert_eq!(decoded.serialize_to_bytes().expect("re-serialize"), want); +} + +#[test] +fn cmsheap_i32_fast_2x3_i64keys_matches_golden() { + let want = decode_hex(GOLDEN_CMSHEAP_I64); + + let mut sketch = CMSHeap::, FastPath>::from_storage( + Vector2D::from_fn(2, 3, |r, c| I64_VALS[r][c] as i32), + 3, + ); + for (key, count) in [(4_294_967_296, 3), (-1, 7), (-129, 7)] { + sketch + .heap_mut() + .update_heap_item(&HeapItem::I64(key), count); + } + let got = sketch.serialize_to_bytes().expect("serialize"); + assert_eq!(got, want, "CMSHeap i32/fast/i64 bytes diverge from golden"); + + let decoded = + CMSHeap::, FastPath>::deserialize_from_bytes(&want).expect("decode"); + let flat: Vec = I64_VALS.iter().flatten().map(|&v| v as i32).collect(); + assert_eq!(decoded.cms().as_storage().as_slice(), flat.as_slice()); + assert_eq!(decoded.heap().capacity(), 3); + let mut held: Vec<(i64, i64)> = decoded + .heap() + .heap() + .iter() + .map(|item| match item.key { + HeapItem::I64(k) => (k, item.count), + ref other => panic!("key {other:?} is not an i64"), + }) + .collect(); + held.sort_by(|a, b| b.1.cmp(&a.1).then_with(|| a.0.cmp(&b.0))); + assert_eq!(held, CMSHEAP_I64_ENTRIES); + assert_eq!(decoded.serialize_to_bytes().expect("re-serialize"), want); +} + // --------------------------------------------------------------------------- // Count Sketch: build known matrix state -> serialize == golden, round-trips. // All three fixtures hold the SAME matrix: the i64 pair differs only by `mode` diff --git a/tests/e2e/composition.rs b/tests/e2e/composition.rs index 23e42c41..cd174878 100644 --- a/tests/e2e/composition.rs +++ b/tests/e2e/composition.rs @@ -1,5 +1,5 @@ //! Composition layers: `HashSketchEnsemble`, `UnivMonQ`'s configuration -//! surface, and the portable facade types that pair a sketch with a heap. +//! surface, and the types that pair a sketch with a heap. //! //! Nitro is a composition layer too, but all of its behaviour lives in //! `tests/e2e/nitro.rs`. @@ -16,9 +16,9 @@ use common::specs::{CardinalityConfidenceSpec, CountMinSpec, CountSketchSpec, Kl use common::{FreqTruth, NumericTruth, uniform_u64, zipf_u64}; use asap_sketchlib::{ - Classic, Count, CountMin, CountMinSketchWithHeap, CountSketchWithHeap, DataInput, - EnsembleSketch, ErtlMLE, FastPath, HashSketchEnsemble, HyperLogLog, HyperLogLogHIP, KllSketch, - MessagePackCodec, UnivMonQ, UnivMonQConfig, Vector2D, + CMSHeap, Classic, Count, CountMin, CountSketchWithHeap, DataInput, EnsembleSketch, ErtlMLE, + FastPath, HashSketchEnsemble, HeapItem, HyperLogLog, HyperLogLogHIP, KllSketch, + MessagePackCodec, RegularPath, UnivMonQ, UnivMonQConfig, Vector2D, }; const ROWS: usize = 3; @@ -559,27 +559,29 @@ fn univmonq_multi_shard_merge_with_distinct_source_ids_covers_the_union() { assert_eq!(merged.max(), Some(truth.max()), "merged max must be exact"); } -// -------------------------------------------------------- Portable facade +// -------------------------------------------------------- Sketch plus heap -/// The portable sketch-plus-heap types, on a real stream against exact truth: -/// the point estimate under the right family's bound, the heap consistent with -/// it, and both surviving a MessagePack round trip and a merge. +/// `CMSHeap` on a real stream against exact truth: the point estimate under +/// Count-Min's bound, the heap consistent with it, and both surviving a merge +/// and an ASAPv1 round trip. #[test] -fn portable_count_min_with_heap_satisfies_the_count_min_bound_through_merge_and_wire() { +fn cms_heap_satisfies_the_count_min_bound_through_merge_and_wire() { const HEAP: usize = 32; + type Sketch = CMSHeap, RegularPath>; + let key_of = |k: i64| DataInput::String(format!("k{k}")); let stream = zipf_u64(N, DOMAIN, 1.1, STREAM_SEED); let mut truth = FreqTruth::default(); - let mut single = CountMinSketchWithHeap::new(ROWS, COLS, HEAP); - let mut left = CountMinSketchWithHeap::new(ROWS, COLS, HEAP); - let mut right = CountMinSketchWithHeap::new(ROWS, COLS, HEAP); + let mut single = Sketch::new(ROWS, COLS, HEAP); + let mut left = Sketch::new(ROWS, COLS, HEAP); + let mut right = Sketch::new(ROWS, COLS, HEAP); for (i, k) in stream.iter().enumerate() { truth.observe(*k as i64); - let key = format!("k{k}"); - single.update(&key, 1.0); + let key = key_of(*k as i64); + single.insert(&key); if i % 2 == 0 { - left.update(&key, 1.0); + left.insert(&key); } else { - right.update(&key, 1.0); + right.insert(&key); } } @@ -588,54 +590,65 @@ fn portable_count_min_with_heap_satisfies_the_count_min_bound_through_merge_and_ ); let spec = CountMinSpec::new(ROWS, COLS); spec.assert_contract( - "portable CountMinSketchWithHeap", + "CMSHeap", &truth, - |k| single.estimate(&format!("k{k}")), + |k| single.estimate(&key_of(k)) as f64, &context, ); // The heap must agree with the sketch it sits beside. let mut heap_tally = Tally::default(); - for item in single.topk_heap_items() { - let est = single.estimate(&item.key); - heap_tally.record(item.value == est, || { + for item in single.heap().heap() { + let HeapItem::String(key) = &item.key else { + panic!( + "heap key {:?} is not the string it was inserted as", + item.key + ); + }; + let est = single.estimate(&DataInput::Str(key)); + heap_tally.record(item.count == est, || { format!( - "key {}: heap holds {} but the sketch estimates {est}", - item.key, item.value + "key {key}: heap holds {} but the sketch estimates {est}", + item.count ) }); } - heap_tally.assert_none("portable CountMinSketchWithHeap heap consistency", &context); + heap_tally.assert_none("CMSHeap heap consistency", &context); // Merge, then the same contract. - let merged = CountMinSketchWithHeap::merge_refs(&[&left, &right]).expect("merge"); + let mut merged = left.clone(); + merged.merge(&right); spec.assert_contract( - "portable CountMinSketchWithHeap after merge", + "CMSHeap after merge", &truth, - |k| merged.estimate(&format!("k{k}")), + |k| merged.estimate(&key_of(k)) as f64, &context, ); // Wire round trip, then the same contract again: serialization must not // change a single answer. - let bytes = single.to_msgpack().expect("encode"); - let decoded = CountMinSketchWithHeap::from_msgpack(&bytes).expect("decode"); + let bytes = single.serialize_to_bytes().expect("encode"); + let decoded = Sketch::deserialize_from_bytes(&bytes).expect("decode"); let mut wire_tally = Tally::default(); for (k, _) in truth.pairs() { - let key = format!("k{k}"); - let a = single.estimate(&key); - let b = decoded.estimate(&key); - wire_tally.record(a == b, || format!("key {key}: before {a} after {b}")); + let a = single.estimate(&key_of(k)); + let b = decoded.estimate(&key_of(k)); + wire_tally.record(a == b, || format!("key k{k}: before {a} after {b}")); } - wire_tally.assert_none("portable CountMinSketchWithHeap wire round trip", &context); + wire_tally.assert_none("CMSHeap wire round trip", &context); spec.assert_contract( - "portable CountMinSketchWithHeap after a wire round trip", + "CMSHeap after a wire round trip", &truth, - |k| decoded.estimate(&format!("k{k}")), + |k| decoded.estimate(&key_of(k)) as f64, &context, ); } +// -------------------------------------------------------- Portable facade + +/// The portable sketch-plus-heap type, on a real stream against exact truth: +/// the point estimate under the L2 bound, the heap consistent with it, and +/// both surviving a MessagePack round trip and a merge. #[test] fn portable_count_sketch_with_heap_satisfies_the_l2_bound_through_merge_and_wire() { const HEAP: usize = 32; diff --git a/tests/msgpack_compat.rs b/tests/msgpack_compat.rs index 267a51c6..d51a39db 100644 --- a/tests/msgpack_compat.rs +++ b/tests/msgpack_compat.rs @@ -21,18 +21,14 @@ use std::collections::HashSet; -use asap_sketchlib::CmsHeapItem; use asap_sketchlib::message_pack_format::MessagePackCodec; use asap_sketchlib::message_pack_format::portable::countminsketch::CountMinSketchWire; -use asap_sketchlib::message_pack_format::portable::countminsketch_topk::{ - CountMinSketchInnerWire, CountMinSketchWithHeapWire, -}; use asap_sketchlib::message_pack_format::portable::delta_set_aggregator::DeltaResult; use asap_sketchlib::message_pack_format::portable::hydra_kll::HydraKllSketchWire; use asap_sketchlib::message_pack_format::portable::kll::KllSketchData; use asap_sketchlib::{ - CountMinSketch, CountMinSketchWithHeap, CountSketch, DdSketch, HllSketch, HllVariant, - HydraKllSketch, KllSketch, SetAggregator, + CountMinSketch, CountSketch, DdSketch, HllSketch, HllVariant, HydraKllSketch, KllSketch, + SetAggregator, }; // ===== round-trip: every wire-format-aligned type ===== @@ -49,18 +45,6 @@ fn count_min_sketch_round_trip() { assert_eq!(restored.cols, 64); } -#[test] -fn count_min_sketch_with_heap_round_trip() { - let mut s = CountMinSketchWithHeap::new(3, 64, 8); - s.update("hot", 100.0); - s.update("warm", 10.0); - let bytes = s.to_msgpack().expect("encode"); - let restored = CountMinSketchWithHeap::from_msgpack(&bytes).expect("decode"); - assert_eq!(restored.rows, 3); - assert_eq!(restored.cols, 64); - assert_eq!(restored.heap_size, 8); -} - #[test] fn count_sketch_round_trip() { let mut s = CountSketch::new(3, 64); @@ -163,27 +147,6 @@ fn count_min_wire_shape() { assert_eq!(restored.sketch.len(), 3); } -#[test] -fn count_min_with_heap_wire_shape() { - let wire = CountMinSketchWithHeapWire { - sketch: CountMinSketchInnerWire { - sketch: vec![vec![0.0; 4]; 2], - rows: 2, - cols: 4, - }, - topk_heap: vec![CmsHeapItem { - key: "hot".to_string(), - value: 42.0, - }], - heap_size: 8, - }; - let bytes = rmp_serde::to_vec(&wire).unwrap(); - let restored: CountMinSketchWithHeapWire = rmp_serde::from_slice(&bytes).unwrap(); - assert_eq!(restored.heap_size, 8); - assert_eq!(restored.topk_heap.len(), 1); - assert_eq!(restored.topk_heap[0].key, "hot"); -} - #[test] fn hydra_kll_wire_shape() { let wire = HydraKllSketchWire { From f8a4676ba1cd79bb4ed1fe348ef883e7a35438fe Mon Sep 17 00:00:00 2001 From: GordonYuanyc Date: Fri, 2 Oct 2026 00:50:49 -0600 Subject: [PATCH 08/36] refactor(ddsketch)!: drop the proto and portable formats DDSketch serializes through ASAPv1 only. Removes proto/ddsketch (DDSketchState, DDSketchDelta, DDSketchBucketDelta) and reserves SketchEnvelope field 14; removes message_pack_format::portable::ddsketch (DdSketch, DdSketchDelta, DDSKETCH_GROW_CHUNK, MAX_APPLY_DELTA_SPAN_BUCKETS) with its crate-root re-exports, and the native MessagePackCodec shim. Tests that exercised only the removed formats are deleted; the rest run against DDSketch and ASAPv1. Adds ASAPv1 golden fixtures ddsketch_positive_a001 (metadata version 1) and ddsketch_signed_a001 (metadata version 2) via the asapv1_golden submodule bump. Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 15 + asapv1_golden | 2 +- docs/api/api_ddsketch.md | 10 - docs/asapv1_wire_format.md | 4 +- docs/ddsketch_signed_values.md | 8 +- docs/e2e_testing_harness.md | 9 +- docs/features.md | 2 +- docs/message_pack_format.md | 4 +- docs/tests.md | 47 +- examples/accuracy_probe.rs | 23 +- proto/ddsketch/ddsketch.proto | 75 - proto/sketchlib.proto | 5 +- src/lib.rs | 5 +- src/message_pack_format/native/ddsketch.rs | 17 - src/message_pack_format/native/mod.rs | 1 - src/message_pack_format/portable/ddsketch.rs | 1364 ------------------ src/message_pack_format/portable/mod.rs | 7 +- src/proto/generated/sketchlib.v1.rs | 63 +- src/sketches/ddsketch.rs | 6 +- tests/PBT_COVERAGE.md | 17 +- tests/asapv1_golden.rs | 134 +- tests/bug_verification.rs | 46 +- tests/common/conformance.rs | 6 +- tests/common/specs.rs | 24 +- tests/conformance_kit.rs | 32 - tests/ddsketch_signed.rs | 94 +- tests/e2e/quantiles.rs | 355 +---- tests/msgpack_compat.rs | 22 +- tests/pbt/ddsketch.rs | 126 +- tests/sketches_go_parity_probe.rs | 57 +- tests/xtest_consumer.rs | 131 -- tools/gen-proto/src/main.rs | 1 - 32 files changed, 275 insertions(+), 2437 deletions(-) delete mode 100644 proto/ddsketch/ddsketch.proto delete mode 100644 src/message_pack_format/native/ddsketch.rs delete mode 100644 src/message_pack_format/portable/ddsketch.rs diff --git a/CHANGELOG.md b/CHANGELOG.md index 80d85f98..180401bd 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -66,6 +66,9 @@ signals a backwards-compatible change. worker's seed and the aggregator's from one seed. The unseeded constructors draw from the thread generator. The seed is not serialized, and the ASAPv1 payload is unchanged. +- **DDSketch ASAPv1 golden fixtures**: `ddsketch_positive_a001` (metadata + version 1) and `ddsketch_signed_a001` (metadata version 2, negative store and + zero count), checked by `tests/asapv1_golden.rs`. ### Changed @@ -88,6 +91,18 @@ signals a backwards-compatible change. as an empty window like any other, and that window takes a place in `max_windows`. +### Removed + +- **DDSketch's protobuf and portable formats.** DDSketch serializes through + ASAPv1 only (`DDSketch::serialize_to_bytes` / `deserialize_from_bytes`). + Removed: `proto/ddsketch/ddsketch.proto` and the generated + `proto::sketchlib::{DdSketchState, DdSketchDelta, DdSketchBucketDelta}`; + `SketchEnvelope`'s `ddsketch` field (14), now reserved; + `message_pack_format::portable::ddsketch` (`DdSketch`, `DdSketchDelta`, + `DDSKETCH_GROW_CHUNK`, `MAX_APPLY_DELTA_SPAN_BUCKETS`) and the crate-root + re-exports of the first three; the `MessagePackCodec` impl for `DDSketch`. + There is no ASAPv1 DDSketch delta. + ### Fixed - `KLLDynamic` panicked on every quantile query once a NaN had entered the diff --git a/asapv1_golden b/asapv1_golden index 8287a4ae..1d2d0ffd 160000 --- a/asapv1_golden +++ b/asapv1_golden @@ -1 +1 @@ -Subproject commit 8287a4ae04e6cfbfb3ed33dafbf1dac77d8f7fc1 +Subproject commit 1d2d0ffd3b0f3deb0efa506ad65f6c83e418e38e diff --git a/docs/api/api_ddsketch.md b/docs/api/api_ddsketch.md index d4724210..42baf9db 100644 --- a/docs/api/api_ddsketch.md +++ b/docs/api/api_ddsketch.md @@ -95,16 +95,6 @@ assert!(p50 >= 1.0); retained minimum and maximum rather than to a bucket representative, so the endpoints carry zero error. -This differs from the portable wire twin -`message_pack_format::portable::ddsketch::DdSketch::quantile`, which uses the -lower-quantile convention `floor(q * (n - 1))` of the DDSketch paper and of the -Go reference implementation, and which retains no min/max scalars — its -endpoints are ordinary bucket representatives, accurate to `alpha` like any -other rank. The two agree at most `q` and diverge at small `n` or ragged `q` -(at `n = 3, q = 0.4` this type answers `sorted[1]` and the portable one -`sorted[0]`). Both conventions are deliberate and are pinned by -`ddsketch_core_and_portable_answer_different_order_statistics`. - ## Caveats - Inputs must be positive values. diff --git a/docs/asapv1_wire_format.md b/docs/asapv1_wire_format.md index fd14e70a..97bdd7ae 100644 --- a/docs/asapv1_wire_format.md +++ b/docs/asapv1_wire_format.md @@ -1341,14 +1341,14 @@ Good direction (more compact, higher fidelity, less Rust-internal duplication), 2. **Golden byte-vector fixtures** in one shared repo, [`sketchlib-golden-bytes`](https://github.com/ProjectASAP/sketchlib-golden-bytes), which each implementation mounts at `asapv1_golden/`; both languages decode and re-encode them byte-identically. These replace the `portable`-as-oracle round-trip test. 3. **This registry**, mirrored, never independently allocated. -Fixtures exist for six `kind_id`s — HLL's three estimators, Count-Min, Count Sketch and compact KLL — and `sketchlib-go` mirrors those. Every other kind here has a payload and **no fixture**, so this document is its only contract; `asapv1_golden/README.md` lists the gap. `portable` carries its own Go goldens. +Fixtures exist for seven `kind_id`s — HLL's three estimators, Count-Min, Count Sketch, DDSketch and compact KLL — and `sketchlib-go` mirrors all but DDSketch's. Every other kind here has a payload and **no fixture**, so this document is its only contract; `asapv1_golden/README.md` lists the gap. `portable` carries its own Go goldens. **Hash profile on the Go side.** Rust derives the hash spec from a generic `HashProfile` bound on the hasher type; Go has no generic hasher type, so there is nothing to derive from. On the Go side the profile is simply **written into** the metadata on encode and **read from** it on decode. Go MUST validate the profile it reads (same fail-closed intent as Rust): a sketch is only mergeable/queryable if its `hash_profile_id` + seeds match the profile Go is prepared to reproduce. -Sequencing: (2) covers six `kind_id`s, and the one `native bytes == portable bytes` test is HLL-only, since no other `portable` type emits ASAPv1 bytes. +Sequencing: (2) covers seven `kind_id`s, and the one `native bytes == portable bytes` test is HLL-only, since no other `portable` type emits ASAPv1 bytes. Retire `portable` once the fixtures cover the rest. --- diff --git a/docs/ddsketch_signed_values.md b/docs/ddsketch_signed_values.md index 923eddaa..603629fa 100644 --- a/docs/ddsketch_signed_values.md +++ b/docs/ddsketch_signed_values.md @@ -1,18 +1,16 @@ # Signed DDSketch -Both `DDSketch` (core) and `DdSketch` (portable) retain positive samples, negative samples, and zeros. The design follows [DataDog DDSketch](https://github.com/DataDog/sketches-go/blob/master/ddsketch/ddsketch.go): separate positive and negative magnitude stores, plus a zero count. Bucket indices can be negative in either store; an index's sign does not encode an observation's sign. +`DDSketch` retains positive samples, negative samples, and zeros. The design follows [DataDog DDSketch](https://github.com/DataDog/sketches-go/blob/master/ddsketch/ddsketch.go): separate positive and negative magnitude stores, plus a zero count. Bucket indices can be negative in either store; an index's sign does not encode an observation's sign. -`try_add` / `try_update` return errors for NaN, infinities, or magnitudes above the mapping's maximum. Existing `add` / `update` wrappers keep their return types and ignore those errors. Magnitudes below the mapping's minimum are counted as zero, including subnormal inputs. Such nonzero tiny observations are outside the relative-error guarantee. Original sample values are still compressed into buckets, not retained individually. +`try_add` returns an error for NaN, infinities, or magnitudes above the mapping's maximum. `add` ignores those errors. Magnitudes below the mapping's minimum are counted as zero, including subnormal inputs. Such nonzero tiny observations are outside the relative-error guarantee. Original sample values are still compressed into buckets, not retained individually. -Core preserves its existing ceil(q*n) order-statistic convention and exact ingested extrema. Portable preserves floor(q*(n-1)); `quantile_interpolated` preserves the PromQL interpolation convention. For indexable nonzero order statistics the estimate has absolute error at most `alpha * abs(value)`, for either sign. +Quantiles use the ceil(q*n) order-statistic convention and exact ingested extrema. For indexable nonzero order statistics the estimate has absolute error at most `alpha * abs(value)`, for either sign. If interpolation combines endpoints x and y with weight t, the absolute error is bounded by `alpha * ((1-t)*abs(x) + t*abs(y))`, excluding zero-mapped tiny values. Opposite signs can cancel, so the interpolated result need not have relative error alpha. Dividing quantiles additionally requires a finite nonzero true denominator and valid relative-error bounds for both operands. This change does not establish those planner preconditions. ## Wire and downstream upgrade - Native ASAPv1: positive-only states keep metadata version 1 and their existing bytes. Signed/zero states use metadata version 2 and append negative counts, negative offset, and zero count. Old native readers reject version 2. -- Portable MessagePack: positive-only snapshots keep their three fields; signed snapshots append the same three fields. Positive-only deltas keep their two arrays; signed deltas append negative indices, negative counts, and zero-count increment. New readers accept both layouts. -- Protobuf: `DdSketchState` adds fields 16–18; `DdSketchDelta` adds fields 16–17. `DdSketch::to_proto` / `from_proto` preserve all stores. Old protobuf readers silently ignore unknown fields: upgrade Collector, backend, and Go producer schema/mapping together before emitting signed protobuf. This is not DataDog's binary wire format. - Octo: `DdDelta` now requires a `store: DdStore` discriminator. Existing positive delta constructors should use `DdStore::Positive`. Worker residual keys are `(DdStore, i32)`. Both promotion and flush preserve all three categories; partial counts still need flushing before complete queries. Go implementations of the ASAP wire formats and downstream dependency pins are not changed by this Rust library patch. diff --git a/docs/e2e_testing_harness.md b/docs/e2e_testing_harness.md index 373f6040..35cd7438 100644 --- a/docs/e2e_testing_harness.md +++ b/docs/e2e_testing_harness.md @@ -101,7 +101,7 @@ One spec per *metric*, not per sketch, because the metric is what differs: | `CountSketchSpec` | two-sided L2, rank-independent | marginal `sqrt(3/w)·‖f₋ᵢ‖₂` at `P[Bin(d, 1/3) ≥ ⌈d/2⌉]`; simultaneous at the smallest κ with `P[Bin(d,1/κ) ≥ ⌈d/2⌉] ≤ δ/D` | | `SecondMomentSpec` | F2 from a Count Sketch matrix | `sqrt(2κ/w)`, same median amplification | | `KllRankSpec` | KLL **maximum** rank error over a q grid | `ε(k) = 2.446 / k^0.9433` — an Apache DataSketches *characterization fit*, not a theorem about this code | -| `RelativeQuantileSpec` + `DdRankConvention` | DDSketch relative value error | `α + ULP slack` vs the exact order statistic **of that implementation's own rank convention** | +| `RelativeQuantileSpec` + `DdRankConvention` | DDSketch relative value error | `α + ULP slack` vs the exact `ceil(q·n)` order statistic | | `CardinalityConfidenceSpec` | HLL / KMV | `z · σ_rel`, σ derived exactly, the tail a normal approximation | | `SamplingConfidenceSpec` | Nitro | `z · sqrt(f(p·r(1−r) + (1−p)/p))`, `r = frac(1/p)` | | `PrioritySampleSpec` | `UniformSampling` | `len = ⌈n·rate⌉` exactly; `Var[mean] = (σ_N²/m)(N−m)/(N−1)` | @@ -141,11 +141,8 @@ Keeping these apart is deliberate. There is no shared `QuantileSpec { rank_tol }` that KLL and DDSketch both use, because they do not promise the same thing: a correct KLL can return a value 100× off on a heavy-tailed stream and still be within its rank guarantee, and a correct -DDSketch has no rank guarantee at all. For the same reason the two DDSketch -implementations do not share a truth helper: `DDSketch::get_value_at_quantile` -answers `sorted[ceil(q·n) − 1]` while the portable `DdSketch::quantile` answers -`sorted[floor(q·(n−1))]`, so `DdRankConvention` carries the choice and each is -scored on the question it actually answers. +DDSketch has no rank guarantee at all. `DDSketch::get_value_at_quantile` +answers `sorted[ceil(q·n) − 1]`, and `DdRankConvention` names that truth. Two distinctions are easy to lose: diff --git a/docs/features.md b/docs/features.md index 64f296a3..ee5c93b1 100644 --- a/docs/features.md +++ b/docs/features.md @@ -112,7 +112,7 @@ MessagePack (`rmp-serde`) support. **serde support** means the type derives `Ser | EHSketchList | Yes | Yes | | ExponentialHistogram | No | Yes | -Protobuf (prost): `.proto` definitions exist for CountMin, Count, HLL, DDSketch, KLL, Elastic, Coco, Hydra, and UnivMon. Rust conversion code is in progress. +Protobuf (prost): `.proto` definitions exist for CountMin, Count, HLL, KLL, Elastic, Coco, Hydra, and UnivMon. Rust conversion code is in progress. ### API Stability diff --git a/docs/message_pack_format.md b/docs/message_pack_format.md index 12a2b91c..708a79e8 100644 --- a/docs/message_pack_format.md +++ b/docs/message_pack_format.md @@ -169,8 +169,8 @@ decode → re-encode them byte-identically. The `kind_id` registry is mirrored verbatim with Go's `wire/asapmsgpack/magic_ids.go`, never independently allocated. -That proof covers the six `kind_id`s a fixture exists for — HLL's three -estimators, Count-Min, Count Sketch and compact KLL. Every other implemented +That proof covers the seven `kind_id`s a fixture exists for — HLL's three +estimators, Count-Min, Count Sketch, DDSketch and compact KLL. Every other implemented kind has **no ASAPv1 golden and therefore no ASAPv1 drift guard**; [its README](https://github.com/ProjectASAP/sketchlib-golden-bytes#coverage) lists what is covered. The older `portable` path is guarded separately, by the `sketchlib-go` goldens diff --git a/docs/tests.md b/docs/tests.md index ccb1571c..8256c369 100644 --- a/docs/tests.md +++ b/docs/tests.md @@ -1294,30 +1294,6 @@ Test file: [`src/message_pack_format/portable/countsketch_topk.rs`](../src/messa | `test_aggregate_topk_empty` | An empty stream produces nothing. | Verifies `aggregate_topk(4, 100, 10, &[], &[])` returns `None`. | | `test_median_estimator_differs_from_cms_style_min` | The median-over-rows estimator tracks the true count rather than inflating. | For a `5x64` composite with a 10-slot heap fed `"heavy"` 50 times at weight `1.0` and 20 keys `light-0`..`light-19` once each, verifies `estimate("heavy")` falls in `40.0..=60.0`. | -### Portable DdSketch - -Test file: [`src/message_pack_format/portable/ddsketch.rs`](../src/message_pack_format/portable/ddsketch.rs) - -| test_name | test_description | what_is_tested | -| --- | --- | --- | -| `msgpack_delta_against_empty_round_trips` | A msgpack delta against an empty snapshot carries the bucket store. | For an `alpha` `0.01` sketch fed `1.0..=200.0`, verifies the delta against an empty sketch at threshold `1` applied to a fresh sketch gives the same `total_count` and quantiles at `0.5`, `0.9`, and `0.99` within 1%. | -| `msgpack_delta_wire_layout_is_parallel_arrays` | The delta is a two-element array of parallel index and count arrays. | For an `alpha` `0.01` sketch fed `1.0`, `1.0`, and `1000.0`, verifies the delta bytes equal `rmp_serde::to_vec(&(idx: Vec, d_count: Vec))` recomputed from the non-zero store cells at absolute index, and that decoding reports two buckets. | -| `test_new_empty` | A fresh sketch holds nothing. | For `DdSketch::new(0.01)`, verifies `total_count` is `0` and `store_counts` is empty. | -| `test_merge_aligned_same_offset` | Aligned stores add cell by cell. | Merges `[10, 20, 30]` at offset `-1` into `[1, 2, 3]` at offset `-1` and verifies the store becomes `[11, 22, 33]`, the offset stays `-1`, and `total_count` is `66`. | -| `test_merge_overlapping_offsets` | Merge widens the window across overlapping stores. | Merges `[10, 10, 10]` at offset `0` into `[1, 1, 1]` at offset `-1` and verifies the store becomes `[1, 11, 11, 10]` at offset `-1` with `total_count` `33`. | -| `test_merge_disjoint_offsets` | Merge zero-fills the gap between disjoint stores. | Merges `[3, 4]` at offset `5` into `[1, 2]` at offset `0` and verifies the store becomes `[1, 2, 0, 0, 0, 3, 4]` at offset `0`. | -| `test_apply_delta_additive_inside_store` | Delta buckets inside the store add to their cells. | For a base `[1, 2, 3]` at offset `-1`, applies a delta with buckets `(-1,4)`, `(0,8)`, `(1,12)`, `d_count` `24`, `d_sum` `120.0`, and `new_max` `9.0`, and verifies the store becomes `[5, 10, 15]` with `total_count` `30`. | -| `test_apply_delta_expands_store_on_new_bucket` | A delta bucket past the store grows it. | For a base `[1, 2]` at offset `0`, applies a delta with the single bucket `(4, 7)`, `d_count` `7`, and `d_sum` `35.0`, and verifies the store becomes `[1, 2, 0, 0, 7]` at offset `0` with `total_count` `10`. | -| `test_apply_delta_matches_full_merge` | Applying a delta lands where merging the same addition lands. | For a base `[1, 2, 3]` at offset `0`, verifies merging `[10, 0, 20]` at offset `0` and applying the delta buckets `(0,10)` and `(2,20)` give the same store and `total_count`. | -| `test_merge_alpha_mismatch` | Sketches of different relative accuracy are not mergeable. | Verifies merging an `alpha` `0.02` sketch into an `alpha` `0.01` one fails. | -| `test_msgpack_round_trip` | The msgpack round trip preserves alpha, store, and offset. | Round-trips `from_raw(0.01, [1, 2, 3], -2)` and verifies `alpha`, `store_counts`, `store_offset`, and `total_count` are unchanged. | -| `test_msgpack_is_three_element_array` | The wire form is a three-element array, not a map. | For `from_raw(0.01, [1, 2, 3], -2)`, verifies the first byte of `to_msgpack` is `0x93`, the msgpack fixarray marker for length `3`. | -| `test_insert_and_quantile_lognormal` | Quantiles track a known log-normal distribution. | Feeds 100,000 xorshift64\*/Box-Muller log-normal samples with `mu` `3` and `sigma` `0.7` into an `alpha` `0.01` sketch and verifies `\|ln(P50 / 20.09)\|` and `\|ln(P99 / 102.4)\|` are each below `0.05`; P90 is not asserted. | -| `test_delta_chain_preserves_quantile_accuracy` | A chain of applied deltas stays inside the alpha bound of the direct path. | Over five batches of 10,000 log-normal samples at `alpha` `0.01`, seeds one sketch from the first batch and applies four deltas built by the test's own `compute_dd_delta` helper, then verifies its `total_count` equals the directly fed sketch's exactly and that quantiles at `0.5`, `0.9`, and `0.99` agree within `alpha`; the shipped `compute_delta` is not the path exercised here. | -| `test_compute_delta_against_empty_round_trips` | A delta against an empty snapshot carries the whole store. | For an `alpha` `0.01` sketch fed `1.0..=200.0`, verifies the delta against empty at threshold `1`, applied to a fresh sketch, gives the exact `total_count`, the same count for every non-zero bucket compared at absolute index, and quantiles at `0.5`, `0.9`, and `0.99` within 1%. | -| `test_consecutive_windows_emit_own_state` | Each window emits its own state rather than a cross-window difference. | Builds `win1` from `1.0..=50.0` inserted twice (count `2` per bucket) and `win2` from `1.0..=50.0` once, and verifies `win2`'s delta against empty reconstructs `win2`'s `total_count` and that those bytes differ from `win2`'s delta against `win1`. | -| `test_update_then_envelope_matches_go_golden_bytes` | The proto envelope for a populated store is byte-identical to the golden. | For an `alpha` `0.01` sketch fed `1.0..=50.0`, builds a `DdSketchState` from `wire_alpha()`, `store_counts`, and `store_offset`, wraps it in a `SketchEnvelope` with `format_version` `1`, and verifies the prost encoding equals a 403-byte inline golden carrying a 384-entry store and a `store_offset` of `-64` (sint32 zigzag `0x7f`); the golden constant records that it is the Rust-produced value and still to be reconciled against the Go reference. | - ### Portable DeltaResult Test file: [`src/message_pack_format/portable/delta_set_aggregator.rs`](../src/message_pack_format/portable/delta_set_aggregator.rs) @@ -1698,12 +1674,11 @@ Test file: [`tests/e2e/quantiles.rs`](../tests/e2e/quantiles.rs) | test_name | test_description | what_is_tested | | --- | --- | --- | -| `ddsketch_core_and_portable_satisfy_the_relative_value_error_contract` | Both DDSketch implementations meet the relative-value-error guarantee against the order statistic of their own rank convention. | Crosses `alpha in {0.001, 0.01, 0.05, 0.1}` x `n in {1_000, 20_000, 100_000}` x the six `dds_streams` shapes (`adversarial-bucket-edges` log-uniform on `gamma` edges, positive-filtered `normal(1000, 250)`, `exponential(1e-3)`, `uniform` over `[1e6, 1e7)`, `zipf(1.1)` over 8192 keys in `[1e6, 1e7]`, and `wide-dynamic-range` spanning nine decades). Asserts `DDSketch::get_count()` and `PortableDds::total_count()` both equal `n` (no sample dropped), then per alpha tallies `RelativeQuantileSpec::core(alpha)` on `get_value_at_quantile` and `RelativeQuantileSpec::portable(alpha)` on `quantile` over the q grid `[0.0, 0.01, 0.1, 0.5, 0.9, 0.99, 1.0]` and closes both with `assert_none`: every estimate must be within `alpha + 8*f64::EPSILON*(1 + \|ln v\|)` of `sorted[ceil(q*n) - 1]` for the core and `sorted[floor(q*(n-1))]` for the portable twin. Zero violations tolerated, since the guarantee is deterministic. | -| `ddsketch_core_and_portable_answer_different_order_statistics` | The core and portable quantile conventions differ, and the divergence is pinned rather than absorbed into `alpha`. | Runs 8 `(n, q, core_idx, port_idx)` probes — `(3, 0.4, 1, 0)`, `(4, 0.34, 1, 1)`, `(4, 0.3, 1, 0)`, `(7, 0.2, 1, 1)`, `(7, 0.6, 4, 3)`, `(5, 0.5, 2, 2)`, `(10, 0.25, 2, 2)`, `(10, 0.15, 1, 1)` — on values `100 * gamma^(3i)` at `ALPHA = 0.01`, so one rank of movement is a whole bucket. Asserts `DdRankConvention::CeilNearestRank.index(n, q)` equals `core_idx` and `DdRankConvention::LowerFloor.index(n, q)` equals `port_idx`, that each implementation satisfies its own `RelativeQuantileSpec::check` against `sorted[its own index]`, that the two answers are unequal on every probe whose indices disagree, and that at least `3` probes disagree. | -| `ddsketch_core_endpoints_are_exact_and_portable_endpoints_are_alpha_relative` | The core sketch's endpoints are exact while the portable twin's are only `alpha`-relative. | For each of `alpha in {0.001, 0.01, 0.05, 0.1}` over 5_000 `log_uniform_f64` values (seed `4_242`, exponent range `3..30`), asserts `core.get_value_at_quantile(0.0) == Some(truth.min())`, `(1.0) == Some(truth.max())`, and `core.min()`/`core.max()` equal the exact extremes; the portable `quantile(0.0)` and `quantile(1.0)` are only checked through `RelativeQuantileSpec::portable(alpha).check` against the true min/max, i.e. within `alpha + a few ULP`, with no exactness claimed. | -| `ddsketch_satisfies_the_relative_error_contract_at_bucket_boundaries` | Values on, just below and just above a bucket edge all stay inside `alpha` on both sides. | For each of the four alphas and `k in {-40, -7, 0, 1, 13, 60, 200}` computes `edge = gamma^k` and probes `edge`, `edge*(1 - f64::EPSILON)`, `edge*(1 + f64::EPSILON)`, `edge*sqrt(gamma)` and `edge*gamma*(1 - 1e-12)`. Each probe is the single sample of a fresh core and portable sketch, so both rank conventions land on index 0 and the mapping is isolated; `quantile(0.5)` must satisfy each spec's `alpha + numerical_slack` tolerance against the probe itself. Probes the mapping refuses (`get_count() == 0`) are skipped rather than asserted on. | -| `ddsketch_merge_and_delta_replay_preserve_the_relative_error_contract` | Shard merge and bucket-delta replay both leave the relative-error contract intact on both implementations. | Per `alpha in {0.001, 0.01, 0.05, 0.1}` over `zipf_f64(40_000, 8_192, 1.1, 1e3, 1e7)`: a 4-way round-robin core shard merge must report `get_count() == 40_000` exactly and satisfy the contract over the full q grid; a core sketch rebuilt by replaying every non-zero `store_counts()` entry as an `octo_delta::DdDelta` at `store_offset() + i` must reproduce `get_count()` and satisfy the contract over the interior grid `DDS_QS[1..6] = [0.01, 0.1, 0.5, 0.9, 0.99]` (its endpoints are bucket representatives, so `q=0`/`q=1` are excluded). On the portable side, an even/odd split merged with `merge` is scored over the full grid, and a `DdSketchDelta` carrying the merged sketch's buckets plus `d_count` replayed into an empty sketch is scored over the interior grid. All four tallies use `assert_none`. | -| `ddsketch_satisfies_the_relative_value_error_contract_at_extreme_accuracy_parameters` | The relative-value contract still holds at accuracy parameters three orders of magnitude either side of the usual range. | For `alpha in {1e-5, 1e-4, 0.3, 0.5, 0.9}` (seed `0x0DDA_0000 + i*7919`), takes the six `dds_streams(alpha, 20_000, seed)` shapes filtered to `[min, max]` from `ddsketch::ddsketch_indexable_bounds(alpha)`, skipping any shape left with fewer than 100 values. Asserts core `get_count()` and portable `total_count()` both equal the surviving count, then `assert_none` on `RelativeQuantileSpec::core(alpha)` and `::portable(alpha)` over the 7-point `DDS_QS` grid. | +| `ddsketch_satisfies_the_relative_value_error_contract` | DDSketch meets the relative-value-error guarantee against the `ceil(q*n)` order statistic. | Crosses `alpha in {0.001, 0.01, 0.05, 0.1}` x `n in {1_000, 20_000, 100_000}` x the six `dds_streams` shapes (`adversarial-bucket-edges` log-uniform on `gamma` edges, positive-filtered `normal(1000, 250)`, `exponential(1e-3)`, `uniform` over `[1e6, 1e7)`, `zipf(1.1)` over 8192 keys in `[1e6, 1e7]`, and `wide-dynamic-range` spanning nine decades). Asserts `DDSketch::get_count()` equals `n` (no sample dropped), then per alpha tallies `RelativeQuantileSpec::core(alpha)` on `get_value_at_quantile` over the q grid `[0.0, 0.01, 0.1, 0.5, 0.9, 0.99, 1.0]` and closes with `assert_none`: every estimate must be within `alpha + 8*f64::EPSILON*(1 + \|ln v\|)` of `sorted[ceil(q*n) - 1]`. Zero violations tolerated, since the guarantee is deterministic. | +| `ddsketch_endpoints_are_exact` | The endpoints are the exact extremes. | For each of `alpha in {0.001, 0.01, 0.05, 0.1}` over 5_000 `log_uniform_f64` values (seed `4_242`, exponent range `3..30`), asserts `get_value_at_quantile(0.0) == Some(truth.min())`, `(1.0) == Some(truth.max())`, and `min()`/`max()` equal the exact extremes. | +| `ddsketch_satisfies_the_relative_error_contract_at_bucket_boundaries` | Values on, just below and just above a bucket edge all stay inside `alpha` on both sides. | For each of the four alphas and `k in {-40, -7, 0, 1, 13, 60, 200}` computes `edge = gamma^k` and probes `edge`, `edge*(1 - f64::EPSILON)`, `edge*(1 + f64::EPSILON)`, `edge*sqrt(gamma)` and `edge*gamma*(1 - 1e-12)`. Each probe is the single sample of a fresh sketch, so the mapping is isolated from any rank effect; `get_value_at_quantile(0.5)` must satisfy `alpha + numerical_slack` against the probe itself. Probes the mapping refuses (`get_count() == 0`) are skipped rather than asserted on. | +| `ddsketch_merge_and_delta_replay_preserve_the_relative_error_contract` | Shard merge and bucket-delta replay both leave the relative-error contract intact. | Per `alpha in {0.001, 0.01, 0.05, 0.1}` over `zipf_f64(40_000, 8_192, 1.1, 1e3, 1e7)`: a 4-way round-robin shard merge must report `get_count() == 40_000` exactly and satisfy the contract over the full q grid; a sketch rebuilt by replaying every non-zero `store_counts()` entry as an `octo_delta::DdDelta` at `store_offset() + i` must reproduce `get_count()` and satisfy the contract over the interior grid `DDS_QS[1..6] = [0.01, 0.1, 0.5, 0.9, 0.99]` (its endpoints are bucket representatives, so `q=0`/`q=1` are excluded). Both tallies use `assert_none`. | +| `ddsketch_satisfies_the_relative_value_error_contract_at_extreme_accuracy_parameters` | The relative-value contract still holds at accuracy parameters three orders of magnitude either side of the usual range. | For `alpha in {1e-5, 1e-4, 0.3, 0.5, 0.9}` (seed `0x0DDA_0000 + i*7919`), takes the six `dds_streams(alpha, 20_000, seed)` shapes filtered to `[min, max]` from `ddsketch::ddsketch_indexable_bounds(alpha)`, skipping any shape left with fewer than 100 values. Asserts `get_count()` equals the surviving count, then `assert_none` on `RelativeQuantileSpec::core(alpha)` over the 7-point `DDS_QS` grid. | | `ddsketch_bucket_width_widens_monotonically_with_the_accuracy_parameter` | A looser accuracy parameter never needs more buckets. | Feeds 20_000 values `1.0 + uniform_u64(.., 1_000_000, 0x0DDA_9001)` into a `DDSketch` at each `alpha in {1e-5, 1e-4, 0.3, 0.5, 0.9}` in that order, asserts nothing was dropped (`get_count() == 20_000`), and asserts the occupied-bucket count (`store_counts()` entries above zero) is no larger than the previous alpha's. The check is non-strict, so equal bucket counts pass and no bucket width is measured directly. | ### Quantiles: DDSketch Input Rejection @@ -1712,8 +1687,7 @@ Test file: [`tests/e2e/quantiles.rs`](../tests/e2e/quantiles.rs) | test_name | test_description | what_is_tested | | --- | --- | --- | -| `ddsketch_rejects_untrackable_values_and_mapping_mismatches` | Untrackable values are dropped without allocating or corrupting state, and a mismatched mapping is a runtime error on both types. | At `alpha = 0.01` feeds `NAN`, `-INFINITY`, `INFINITY`, `-5.0`, `0.0`, `f64::MIN_POSITIVE`, `5e-324`, `f64::MAX` and `1e308` into both implementations and asserts `get_count() == 0`, `total_count() == 0`, `port.store_counts.len() < 10_000`, and `port.quantile(0.5) == None`. Then `[1.0, 2.0, 4.0, 8.0, 16.0]` must give `get_count() == 5` / `total_count() == 5` with the store between `1` and `512` buckets. Using the shared `ddsketch_indexable_bounds(0.01)`, `min*(1 + 1e-9)` and `max*(1 - 1e-9)` are both accepted (count `2` in both, and the two counts equal), while `min*0.5` and `max*(1 + 1e-6)` leave the counts at `2`. `DDSketch::merge` and `PortableDds::merge` across `alpha = 0.01` vs `0.05` must both return `Err`. Finally at `alpha = 1e-9`, `1e-300` is dropped by both (`total_count() == 0`, `get_count() == 0`) with `store_counts.len() == 0`, so no allocation occurs for a rejected value. | -| `portable_ddsketch_rejects_hostile_delta_spans` | A delta spanning an implausible bucket range is rejected before any allocation, and a benign one still applies. | Seeds a portable sketch at `alpha = 0.01` with `1.0`, then asserts `apply_delta` on `buckets: [(i32::MAX - 1, 1), (7, 3)]` returns `Err` and leaves both `store_counts.len()` and `store_offset` at their prior values. A benign delta `[(6, 5)]` then applies successfully and `quantile(0.9)` must equal exactly `Some(gamma.powf(6.0) * (1.0 + alpha))`, so the applied count is visible through queries. | +| `ddsketch_rejects_untrackable_values_and_mapping_mismatches` | Untrackable values are dropped without allocating or corrupting state, and a mismatched mapping is a runtime error. | At `alpha = 0.01` feeds `NAN`, `-INFINITY`, `INFINITY`, `f64::MAX` and `1e308` and asserts `get_count() == 0`, an empty store, and `get_value_at_quantile(0.5) == None`. Then `[1.0, 2.0, 4.0, 8.0, 16.0]` must give `get_count() == 5` with the store between `1` and `512` buckets. Using the shared `ddsketch_indexable_bounds(0.01)`, `min*(1 + 1e-9)` and `max*(1 - 1e-9)` are both kept (count `2`); `min*0.5` enters the zero bucket and `max*(1 + 1e-6)` is dropped (count `3`, `zero_count() == 1`). `merge` across `alpha = 0.01` vs `0.05` returns `Err`. At `alpha = 1e-9`, `1e-300` is counted (`get_count() == 1`) with an empty store, so a zero-mapped value allocates nothing. | ### Quantiles: UnivMon-Q Estimators @@ -2126,6 +2100,8 @@ Test file: [`tests/asapv1_golden.rs`](../tests/asapv1_golden.rs) | `cs_i32_regular_2x4_matches_golden` | The i32 counter type reaches the metadata, and an i32 golden is not an i64 sketch. | Verifies `Count::, RegularPath>::from_storage` over `[[0, 127, 128, 65536], [-1, -33, -32768, -2147483648]]` cast to `i32` serializes to `asapv1_golden/cs_i32_regular_2x4.hex`, that decoding it recovers the eight cells and re-serializes identically, that the i32 and i64 regular fixtures have equal length yet differ, and that the i32 fixture fails to decode as `Count, RegularPath>`. | | `cs_i64_fast_2x4_matches_golden` | The fast-path mode reaches the bytes, so the same matrix encodes differently per path. | Verifies `Count::, FastPath>::from_storage` over `[[0, 127, 128, 65536], [-1, -33, -32768, -2147483648]]` serializes to `asapv1_golden/cs_i64_fast_2x4.hex`, that decoding it recovers the eight cells and re-serializes identically, and that the fixture differs from `cs_i64_regular_2x4.hex` built from the same matrix. | | `cs_i64_regular_2x4_matches_golden` | A known signed Count-Sketch matrix serializes to its checked-in golden bytes. | Verifies `Count::, RegularPath>::from_storage` over `[[0, 127, 128, 65536], [-1, -33, -32768, -2147483648]]` — sweeping the msgpack integer widths in both directions — serializes to `asapv1_golden/cs_i64_regular_2x4.hex`, and that decoding the fixture reports `rows()` `2`, `cols()` `4`, the eight cells, and a byte-identical re-serialization. | +| `ddsketch_positive_a001_matches_golden` | A positive-only DDSketch state serializes to its golden bytes as metadata version 1, and the golden decodes back to that state. | Builds `DDSketch` at `alpha = 0.01` through its serde form from the store `[1, 0, 127, 128, 300, 65536, 4294967296]` at offset `-40`, `sum = 2181071000.0`, `min = 0.453125`, `max = 0.5078125`, and verifies `serialize_to_bytes` equals `ddsketch_positive_a001.hex`; the decode reports that `alpha`, store, offset and scalars, an empty negative store, a zero `zero_count`, a `get_count()` equal to the bucket sum, and re-encodes byte-identically. | +| `ddsketch_signed_a001_matches_golden` | A signed DDSketch state serializes to its golden bytes as metadata version 2, and the golden decodes back to that state. | Builds `DDSketch` at `alpha = 0.01` from the positive store `[3, 0, 2]` at offset `310`, the negative store `[5, 1]` at offset `-208`, `zero_count = 7`, `sum = 2523.90625`, `min = -0.016`, `max = 515.0`, and verifies `serialize_to_bytes` equals `ddsketch_signed_a001.hex`; the decode reports both stores, both offsets, the zero count, the scalars and a `get_count()` of `18`, and re-encodes byte-identically. | | `hll_classic_p12_matches_golden` | A known P12 Classic register pattern serializes to its golden bytes on both the portable and native paths. | Builds `HllSketch::from_raw(HllVariant::Regular, 12, regs, 0.0, 0.0, 0.0)` where `regs` is 4,096 zeros with `[0] = 1`, `[1] = 7`, `[100] = 42` and `[4095] = 3`, verifies `to_msgpack()` equals `asapv1_golden/hll_classic_p12.hex`, that `HyperLogLogP12::::deserialize_from_bytes` on the fixture recovers those registers and re-serializes identically, and that `HllSketch::from_msgpack` reports the same registers, `precision` `12` and `HllVariant::Regular`. | | `hll_ertl_mle_p12_matches_golden` | The Datafusion variant tag reaches the bytes for the same P12 register pattern. | Builds `HllSketch::from_raw(HllVariant::Datafusion, 12, regs, 0.0, 0.0, 0.0)` over the shared P12 pattern (`[0] = 1`, `[1] = 7`, `[100] = 42`, `[4095] = 3` in 4,096 registers), verifies `to_msgpack()` equals `asapv1_golden/hll_ertl_mle_p12.hex`, that `HyperLogLogP12::::deserialize_from_bytes` recovers the registers and re-serializes identically, and that `HllSketch::from_msgpack` reports `HllVariant::Datafusion`. | | `hll_hip_p12_matches_golden` | The HIP variant's three running scalars travel on the wire beside its registers. | Builds `HllSketch::from_raw(HllVariant::Hip, 12, regs, 1.5, 2.5, 3.0)` over the shared P12 pattern, verifies `to_msgpack()` equals `asapv1_golden/hll_hip_p12.hex`, that `HyperLogLogHIPP12::deserialize_from_bytes` re-serializes the fixture identically, and that `HllSketch::from_msgpack` reports the registers, `HllVariant::Hip`, `hip_kxq0` `1.5`, `hip_kxq1` `2.5` and `hip_est` `3.0`. | @@ -2144,8 +2120,6 @@ Test file: [`tests/msgpack_compat.rs`](../tests/msgpack_compat.rs) | `count_min_wire_shape` | The `CountMinSketchWire` DTO round-trips through `rmp_serde` with its field shape intact. | Serializes `CountMinSketchWire { sketch: vec![vec![1.0, 2.0]; 3], rows: 3, cols: 2 }` with `rmp_serde::to_vec` and reads it back with `rmp_serde::from_slice`, verifying `rows` `3`, `cols` `2` and a `sketch` of length `3`; the cell values are not checked. | | `count_min_with_heap_wire_shape` | The nested `CountMinSketchWithHeapWire` DTO round-trips through `rmp_serde` with its nesting intact. | Serializes a wire struct holding a `CountMinSketchInnerWire` of `2x4` zeros, one `CmsHeapItem { key: "hot", value: 42.0 }` and `heap_size: 8`, then verifies the `rmp_serde` decode reports `heap_size` `8`, one heap entry and the key `hot`; the item's value and the inner matrix are not re-checked. | | `count_sketch_round_trip` | A portable `CountSketch` survives a msgpack round trip with its geometry intact. | Updates `CountSketch::new(3, 64)` with `k1` `+1.0` and `k2` `+2.0`, then verifies the decode reports `rows` `3` and `cols` `64`; the signed counters and the estimates are not checked. | -| `dd_sketch_decodes_go_bytes` | The test asserts nothing. | The test asserts nothing: the body is empty and the test is marked `#[ignore]` with the reason that no cross-language msgpack fixture exists in the repository. | -| `dd_sketch_round_trip` | A portable `DdSketch` survives a msgpack round trip carrying its full count in the bucket array. | Updates `DdSketch::new(0.01)` with `1.0`, `10.0` and `100.0`, then verifies the decode's `total_count()` — recovered by summing the buckets, since the count is not on the wire — is `3`; `alpha`, the bucket offset and the quantiles are not checked. | | `delta_result_round_trip` | A `DeltaResult` survives a msgpack round trip with both of its sets. | Serializes a `DeltaResult` whose `added` holds `a` and whose `removed` holds `b`, then verifies the decode's `added` contains `a` and its `removed` contains `b`. | | `hll_sketch_decodes_go_bytes` | The test asserts nothing. | The test asserts nothing: the body is empty and the test is marked `#[ignore]` with the reason that no cross-language msgpack fixture exists in the repository. | | `hll_sketch_round_trip` | A portable `HllSketch` survives a msgpack round trip with its register count intact. | Updates `HllSketch::new(HllVariant::Regular, 8)` with `b"a"`, `b"b"` and `b"c"`, then verifies only that the decode's `registers` has the same length as the source's; the register values, the precision and the variant are not checked. | @@ -2166,7 +2140,6 @@ Test file: [`tests/conformance_kit.rs`](../tests/conformance_kit.rs) | `count_l2hh_passes_signed_frequency_and_merge_conformance` | `CountL2HH` passes the two-sided frequency, turnstile and merge-equivalence batteries. | Drives `CountL2HH::with_dimensions(5, 4_096)` over a 60,000-draw Zipf stream (domain 2,048, exponent 1.1, seed `9001`) through `frequency_battery` two-sided at `rel_tol` `0.06` and `abs_tol` `25.0` on every key of true count at least `25`, through `turnstile_battery` on key `42` (`+500` then `-200` must estimate `300` within `1.0`, then `-300` must cancel to within `1e-6`), and through `merge_equivalence_battery`, which shards the stream by parity, merges, and requires agreement with a single-pass sketch within `25.0 + 0.06 * count`. | | `countmin_passes_frequency_and_merge_conformance` | `CountMin` passes the one-sided frequency and merge-equivalence batteries. | Drives `CountMin::, FastPath>::with_dimensions(4, 4096)` over the 60,000-draw Zipf stream (domain 2,048, exponent 1.1, seed `9001`) through `frequency_battery` one-sided at `rel_tol` `0.01` and `abs_tol` `4.0`, which requires the absent sentinel key `i64::MIN` to estimate within `±4.0` and each key of true count at least `25` to land in `[count, count * 1.01 + 4.0]`, then through `merge_equivalence_battery` at the same spec. | | `countsketch_passes_signed_frequency_conformance` | `Count` passes the two-sided frequency, turnstile and merge-equivalence batteries. | Drives `Count::, RegularPath>::with_dimensions(5, 4096)` over the 60,000-draw Zipf stream through `frequency_battery` two-sided at `rel_tol` `0.06` and `abs_tol` `25.0`, `turnstile_battery` on key `42` via `insert_many` (`+500 - 200` must read `300` within `1.0`, then cancel to within `1e-6`), and `merge_equivalence_battery` at the same spec. | -| `ddsketch_passes_relative_quantile_conformance` | The portable `DdSketch` holds its relative-value-error promise under its own rank convention. | Runs `relative_quantile_battery` on `PortableDds::new(0.01)` over the positive samples of `normal_f64(40_000, 500.0, 80.0, 7001)`, requiring at q `0.1`, `0.25`, `0.5`, `0.75` and `0.9` a relative error of at most `0.01` plus a few-ULP numerical slack against the exact `floor(q*(n-1))` order statistic. | | `heap_backed_frequency_sketches_pass_frequency_and_merge_conformance` | Both heap-backed matrix sketches pass the frequency and merge-equivalence batteries. | Over the 60,000-draw Zipf stream (domain 2,048, exponent 1.1, seed `9001`), drives `CMSHeap::, FastPath>::new(4, 4_096, 64)` through `frequency_battery` and `merge_equivalence_battery` one-sided at `rel_tol` `0.01` and `abs_tol` `8.0`, and `CSHeap::, RegularPath>::new(5, 4_096, 64)` through both two-sided at `rel_tol` `0.06` and `abs_tol` `25.0`. | | `hll_variants_pass_cardinality_conformance` | All three native HyperLogLog estimators and the portable one estimate within 3%. | Runs `cardinality_battery` at `rel_tol` `0.03` over 100,000 distinct keys `i.wrapping_mul(0x9E37_79B9_7F4A_7C15)` on `HyperLogLog`, `HyperLogLog`, `HyperLogLogHIP` and the portable `HllSketch::new(HllVariant::Regular, 14)`; each must estimate `100_000` within 3% and stay in band after the whole stream is replayed as duplicates. | | `kll_family_passes_quantile_conformance` | The three KLL adapters hold the shared rank-error band. | Runs `quantile_battery` at the default `rank_tol` of `0.03` over q `0.1`, `0.25`, `0.5`, `0.75` and `0.9` on the positive samples of `normal_f64(40_000, 500.0, 80.0, 7001)` for `KLL::init_kll_with_seed(200, 0x4017_0001)`, `KLLDynamic::::init_kll_with_seed(200, 0x4017_0001)`, and a `KLL` at the same `k` and seed queried through `quantile_cached`. | @@ -2193,7 +2166,6 @@ Test file: [`tests/sketches_go_parity_probe.rs`](../tests/sketches_go_parity_pro | --- | --- | --- | | `sketches_count_fastpath_matches_go_count_sketch_envelope` | A `sketches::Count` fast-path matrix encodes to the checked-in Count-Sketch envelope bytes. | Fills `Count, FastPath, DefaultXxHasher>` at `3x512` with 25 `insert_many` calls over the five keys `k-a`..`k-e`, packs the row-major cells and per-row L2 sums into a `CountSketchState` with `counter_type` `INT64` inside a `SketchEnvelope { format_version: 1 }` with no producer or hash_spec, and verifies the prost encoding equals the 1,577-byte fixture `tests/cs_envelope_golden.hex`. | | `sketches_countmin_fastpath_matches_go_count_min_envelope` | A `sketches::CountMin` fast-path matrix encodes to the checked-in Count-Min envelope bytes. | Fills `CountMin, FastPath, DefaultXxHasher>` at `4x2048` with 50 `insert_many` calls over the ten keys `flow-0`..`flow-9`, packs the cells cast to `i64` plus per-row L1 and L2 sums into a `CountMinState` with `counter_type` `INT64`, and verifies the prost encoding matches the 8,275-byte fixture `src/sketches/testdata/cms_envelope_golden.hex` in length and then byte for byte. | -| `sketches_ddsketch_matches_go_envelope` | A `sketches::DDSketch` encodes to the DDSketch envelope hex constant held in the test file. | Fills `DDSketch::new(0.01)` with `1..=50` as `f64`, encodes a `DdSketchState` carrying `store_counts`, `store_offset` and an alpha round-tripped through gamma (`gamma = (1 + a) / (1 - a)`, `alpha_wire = (gamma - 1) / (gamma + 1)`), and verifies the envelope matches the 403-byte inline hex constant in length and then byte for byte; the comment on that constant records it as this crate's own output rather than a fixture from another implementation, so no cross-language parity is pinned. | | `sketches_hll_classic_matches_go_envelope` | An ErtlMLE P14 register set encodes to the checked-in HyperLogLog envelope bytes. | Feeds `HyperLogLogImpl` the `hash64_seeded(CANONICAL_HASH_SEED, ...)` digests of the IEEE-754 little-endian bytes of `1.0..=50.0` through `insert_with_hash`, encodes a `HyperLogLogState` with `variant` `2`, `precision` `14`, the dense registers, zeroed HIP scalars and no sparse form, and verifies the envelope matches the 16,398-byte fixture `src/sketches/testdata/hll_envelope_golden.hex` in length and then byte for byte; despite the name, the variant under test is `ErtlMLE`, not `Classic`. | ### Cross-Language Proto Consumer @@ -2202,7 +2174,7 @@ Test file: [`tests/xtest_consumer.rs`](../tests/xtest_consumer.rs) | test_name | test_description | what_is_tested | | --- | --- | --- | -| `cross_language_proto` | Protobuf envelopes read from the cross-language fixture directory answer their point queries in band. | Asserts nothing unless fixtures are available: with no `XTEST_DIR` and no discoverable producer directory holding `tests/cross_language/xtest_producer_test.go`, it prints a skip line and returns. With fixtures it decodes nine `SketchEnvelope` files and records a pass or failure for each — `countmin.pb` min-frequency for `item:42` at least `101`, `kll.pb` p50 within 5% of `5000` and p99 within 5% of `9900`, `ddsketch.pb` the same two quantiles within 2%, `hll.pb` an ErtlMLE cardinality inside `[40_000, 65_000]`, `countsketch.pb` `cs:hot` at least `200`, `coco.pb` `coco:hot` at least `500`, `elastic.pb` `elephant` at least `900`, `univmon.pb` a cardinality inside `[1_000, 15_000]`, `hydra.pb` `hydra:42` at least `51` — using the hash seeds `0xcafe3553`, `0x6a09e667` and `0xbb67ae85`, and, when `countmin_sampled.pb` and `hll_sampled.pb` exist, requires `effective_sample_p` of `0.1` and a `rescale_count` by `1/p` within 5% of `100_000` and 6% of `200_000`. A single `assert!` at the end fails if any check failed; a missing or mistyped envelope panics on the spot. | +| `cross_language_proto` | Protobuf envelopes read from the cross-language fixture directory answer their point queries in band. | Asserts nothing unless fixtures are available: with no `XTEST_DIR` and no discoverable producer directory holding `tests/cross_language/xtest_producer_test.go`, it prints a skip line and returns. With fixtures it decodes eight `SketchEnvelope` files and records a pass or failure for each — `countmin.pb` min-frequency for `item:42` at least `101`, `kll.pb` p50 within 5% of `5000` and p99 within 5% of `9900`, `hll.pb` an ErtlMLE cardinality inside `[40_000, 65_000]`, `countsketch.pb` `cs:hot` at least `200`, `coco.pb` `coco:hot` at least `500`, `elastic.pb` `elephant` at least `900`, `univmon.pb` a cardinality inside `[1_000, 15_000]`, `hydra.pb` `hydra:42` at least `51` — using the hash seeds `0xcafe3553`, `0x6a09e667` and `0xbb67ae85`, and, when `countmin_sampled.pb` and `hll_sampled.pb` exist, requires `effective_sample_p` of `0.1` and a `rescale_count` by `1/p` within 5% of `100_000` and 6% of `200_000`. A single `assert!` at the end fails if any check failed; a missing or mistyped envelope panics on the spot. | ### HLL Custom Precision @@ -2223,5 +2195,4 @@ Test file: [`tests/bug_verification.rs`](../tests/bug_verification.rs) | test_name | test_description | what_is_tested | | --- | --- | --- | -| `portable_ddsketch_respects_alpha_at_bucket_edges` | Both DDSketch implementations hold the advertised relative accuracy on a value sitting at a bucket's lower edge. | Sets `alpha = 0.05`, `gamma = (1+alpha)/(1-alpha)`, and feeds the bucket-edge value `gamma^20 * (1 + 1e-6)` 10,000 times into both `message_pack_format::portable::ddsketch::DdSketch::new(0.05)` and `DDSketch::new(0.05)`, then asserts the portable `quantile(0.5)` and the core `get_value_at_quantile(0.5)` are each within `alpha * (1 + 1e-6)` relative error of that value — a representative reported as the bucket log-midpoint `gamma^(k+0.5)` gives about `5.13%` here and fails. Exact agreement between the two is not asserted, because the core sketch clamps representatives to its observed min/max and the portable wire format does not carry them. | | `countl2hh_f2_survives_beyond_i64_max` | `CountL2HH`'s hot-path L2 accumulation stays exact inside `i64` and saturates beyond it. | Builds `CountL2HH::::with_dimensions_and_seed(4, 2048, 7)` and calls `fast_insert_with_count(&DataInput::U32(1), 3_000_000_000)`, asserting `get_l2_sqr()` matches the exact `9e18` to a relative `1e-12`; a second key `U32(2)` at the same count puts the true F2 at `1.8e19`, past `i64::MAX`, and `get_l2_sqr()` must then equal `i64::MAX` (`9.223e18`) to a relative `1e-12` rather than wrapping to `0` or panicking in debug. | diff --git a/examples/accuracy_probe.rs b/examples/accuracy_probe.rs index 643bfc3e..2c2efd94 100644 --- a/examples/accuracy_probe.rs +++ b/examples/accuracy_probe.rs @@ -8,7 +8,6 @@ use asap_sketchlib::common::input::{HydraCounter, HydraQuery}; use asap_sketchlib::message_pack_format::portable::countminsketch::CountMinSketch; -use asap_sketchlib::message_pack_format::portable::ddsketch::DdSketch as PortableDds; use asap_sketchlib::message_pack_format::portable::hll::{HllSketch, HllVariant}; use asap_sketchlib::message_pack_format::portable::hydra_kll::HydraKllSketch; use asap_sketchlib::message_pack_format::portable::kll::KllSketch as PortableKll; @@ -512,66 +511,48 @@ fn quantile_report(qf: impl Fn(f64) -> f64, data: &[f64], qs: &[f64]) -> (f64, S // ---------------------------------------------------------------- DDSketch fn probe_ddsketch() { let mut p = Probe::new(); - println!("--- DDSketch core vs portable (alpha=0.05) ---"); + println!("--- DDSketch (alpha=0.05) ---"); let alpha = 0.05; let gamma = (1.0f64 + alpha) / (1.0 - alpha); // Isolated adversarial probe: one distinct value sitting just above a // bucket's lower edge gamma^k, repeated N times. The only correct answer // for every quantile is v itself. let mut worst_core = 0.0f64; - let mut worst_port = 0.0f64; for k in [10i32, 20, 30] { let v = gamma.powi(k) * (1.0 + 1e-6); // just inside bucket k let n = 10_000u64; let mut core = DDSketch::new(alpha); - let mut port = PortableDds::new(alpha); for _ in 0..n { core.add(&v); - port.update(v); } let rc = rel_err(core.get_value_at_quantile(0.5).unwrap(), v); - let rp = rel_err(port.quantile(0.5).unwrap(), v); worst_core = worst_core.max(rc); - worst_port = worst_port.max(rp); } p.check( "core DDSketch honors alpha=0.05 at bucket edges", format!("max rel err {worst_core:.5}"), worst_core <= alpha * (1.0 + 1e-6), ); - p.check( - "portable DdSketch honors alpha=0.05 at bucket edges", - format!("max rel err {worst_port:.5} (shared gamma^k*(1+alpha) representative)"), - worst_port <= alpha * (1.0 + 1e-6), - ); - // Mixed-stream sanity on both. + // Mixed-stream sanity. let mut rng = StdRng::seed_from_u64(47); let mut core = DDSketch::new(alpha); - let mut port = PortableDds::new(alpha); let mut truth: Vec = Vec::new(); for _ in 0..20_000 { let k = rng.random_range(5..40); let frac = rng.random::(); let v = gamma.powi(k) * (1.0 + frac * (gamma - 1.0)); core.add(&v); - port.update(v); truth.push(v); } truth.sort_by(|a, b| a.partial_cmp(b).unwrap()); let mc = qs_max_rel(&truth, |q| core.get_value_at_quantile(q).unwrap(), &qs_dd()); - let mp = qs_max_rel(&truth, |q| port.quantile(q).unwrap(), &qs_dd()); p.check( "core mixed stream <= alpha", format!("max rel {mc:.5}"), mc <= alpha, ); - p.check( - "portable mixed stream <= alpha", - format!("max rel {mp:.5}"), - mp <= alpha, - ); p.finish("DDSketch"); } diff --git a/proto/ddsketch/ddsketch.proto b/proto/ddsketch/ddsketch.proto deleted file mode 100644 index 02b8beff..00000000 --- a/proto/ddsketch/ddsketch.proto +++ /dev/null @@ -1,75 +0,0 @@ -// ddsketch.proto — Portable wire format for DDSketch quantile sketch state. - -syntax = "proto3"; - -package sketchlib.v1; - -option go_package = "github.com/ProjectASAP/sketchlib-go/proto/ddsketch;ddsketchpb"; - -// ============================================================================ -// DDSketch -// ============================================================================ - -// DDSketchState is the portable state of a DDSketch quantile sketch. -// -// Only alpha is required to reconstruct the index mapping. Consumers must -// recompute all derived mapping values on load: -// gamma = (1.0 + alpha) / (1.0 - alpha) -// log_gamma = ln(gamma) -// inv_log_gamma = 1.0 / log_gamma -// -// Bucket indices: a value v maps to bucket floor(ln(v) * inv_log_gamma). -// Stored bucket absolute_index = array_index + store_offset. -message DDSketchState { - // Relative accuracy guarantee. Required. Must satisfy 0 < alpha < 1. - double alpha = 1; - - // Bucket count array in index order (ascending absolute bucket index). - // Absolute index of store_counts[i] is i + store_offset. - repeated uint64 store_counts = 2 [packed = true]; - - // Absolute bucket index corresponding to store_counts[0]. - // May be negative (values < 1.0 map to negative bucket indices). - sint32 store_offset = 3; - - // Tags 4-7 previously carried the DataPoint-level METRIC scalars - // (count, sum, min, max). They were dropped from the wire format: - // the count is recoverable by summing store_counts and the remaining - // aggregates are carried elsewhere. - reserved 4, 5, 6, 7; - - reserved 8 to 15; - // Signed extension: negative magnitudes use the same mapping as positives. - repeated uint64 negative_store_counts = 16 [packed = true]; - sint32 negative_store_offset = 17; - uint64 zero_count = 18; -} - -// ============================================================================ -// DDSketch Delta -// ============================================================================ - -// DDSketchDelta carries only the buckets that changed above threshold T. -// -// DataPoint-level metric scalars (d_count/d_sum/new_min/new_max/min_changed/ -// max_changed) are intentionally NOT carried on the wire: the count delta is -// recoverable by summing bucket deltas, and min/max/quantiles are derived from -// the bucket distribution within the relative-accuracy guarantee. This message -// is byte-identical to the Go reference implementation's DDSketchDelta so the -// Go and Rust runtimes emit byte-identical delta frames for identical window -// state (cross-language byte parity). -message DDSketchDelta { - repeated DDSketchBucketDelta buckets = 1; - repeated DDSketchBucketDelta negative_buckets = 16; - uint64 zero_count = 17; - - // Fields 2-7 (d_count/d_sum/new_min/new_max/min_changed/max_changed) - // removed: DataPoint-level metric scalars are no longer carried on the wire. - reserved 2, 3, 4, 5, 6, 7; -} - -// DDSketchBucketDelta is the delta for one log-scale bucket. -message DDSketchBucketDelta { - sint32 index = 1; // signed bucket index (negative for values < 1) - uint64 d_count = 2; // Δcount ≥ threshold T -} diff --git a/proto/sketchlib.proto b/proto/sketchlib.proto index 0a6d8193..602da7fc 100644 --- a/proto/sketchlib.proto +++ b/proto/sketchlib.proto @@ -8,7 +8,6 @@ // countsketch/countsketch.proto — CountSketchState, TopKState, HeapEntry // hll/hll.proto — HyperLogLogState, HLLVariant // kll/kll.proto — KLLState, CoinState -// ddsketch/ddsketch.proto — DDSketchState // univmon/univmon.proto — UnivMonState, UnivMonLayer // hydra/hydra.proto — HydraState, HydraCounterType, HydraCell // @@ -33,7 +32,6 @@ import "countminsketch/countminsketch.proto"; import "countsketch/countsketch.proto"; import "hll/hll.proto"; import "kll/kll.proto"; -import "ddsketch/ddsketch.proto"; import "univmon/univmon.proto"; import "hydra/hydra.proto"; import "cocosketch/cocosketch.proto"; @@ -70,13 +68,14 @@ message SketchEnvelope { CountSketchState count_sketch = 11; HyperLogLogState hll = 12; KLLState kll = 13; - DDSketchState ddsketch = 14; UnivMonState univmon = 15; HydraState hydra = 16; CocoSketchState coco = 17; ElasticState elastic = 18; } + reserved 14; + reserved "ddsketch"; reserved 19 to 31; // Reserved for future sketch types. // Geometric / NitroSketch sampling probability for the producing sketch. diff --git a/src/lib.rs b/src/lib.rs index b062b8b9..dd542eb6 100644 --- a/src/lib.rs +++ b/src/lib.rs @@ -11,8 +11,8 @@ //! - [`message_pack_format`]: MessagePack/proto wire format shared with //! `sketchlib-go`. Owns the wire-DTO sketch types consumed by the ASAP //! query engine (`CountMinSketch`, `CountSketch`, `KllSketch`, `HllSketch`, -//! `DdSketch`, `HydraKllSketch`, `CountMinSketchWithHeap`, -//! `SetAggregator`, `DeltaResult`). +//! `HydraKllSketch`, `CountMinSketchWithHeap`, `SetAggregator`, +//! `DeltaResult`). //! //! Most users can start with the crate-root re-exports such as [`DataInput`], //! [`CountMin`], [`HyperLogLog`], [`KLL`], and [`DDSketch`]. Reach for the @@ -47,7 +47,6 @@ pub use message_pack_format::portable::countsketch::{ COUNT_SKETCH_TOPK_CAPACITY, CountSketch, CountSketchDelta, }; pub use message_pack_format::portable::countsketch_topk::{CountSketchWithHeap, CsHeapItem}; -pub use message_pack_format::portable::ddsketch::{DDSKETCH_GROW_CHUNK, DdSketch, DdSketchDelta}; pub use message_pack_format::portable::delta_set_aggregator::DeltaResult; pub use message_pack_format::portable::hll::{HllSketch, HllSketchDelta, HllVariant}; pub use message_pack_format::portable::hydra_kll::HydraKllSketch; diff --git a/src/message_pack_format/native/ddsketch.rs b/src/message_pack_format/native/ddsketch.rs deleted file mode 100644 index 9a88bd74..00000000 --- a/src/message_pack_format/native/ddsketch.rs +++ /dev/null @@ -1,17 +0,0 @@ -//! Native MessagePack codec impl for [`crate::sketches::ddsketch::DDSketch`]. -//! -//! Produces and consumes the ASAPv1 envelope (kind_id `0x05 0x00`): `alpha` in -//! the metadata, `[counts, offset, sum, min, max]` in the payload. - -use crate::message_pack_format::{Error, MessagePackCodec}; -use crate::sketches::ddsketch::DDSketch; - -impl MessagePackCodec for DDSketch { - fn to_msgpack(&self) -> Result, Error> { - Ok(self.serialize_to_bytes()?) - } - - fn from_msgpack(bytes: &[u8]) -> Result { - Ok(Self::deserialize_from_bytes(bytes)?) - } -} diff --git a/src/message_pack_format/native/mod.rs b/src/message_pack_format/native/mod.rs index 0a83ef96..f612a145 100644 --- a/src/message_pack_format/native/mod.rs +++ b/src/message_pack_format/native/mod.rs @@ -10,7 +10,6 @@ pub mod countminsketch; pub mod countsketch; pub mod countsketch_topk; -pub mod ddsketch; pub mod hll; pub mod kll; pub mod kll_dynamic; diff --git a/src/message_pack_format/portable/ddsketch.rs b/src/message_pack_format/portable/ddsketch.rs deleted file mode 100644 index e8961d6f..00000000 --- a/src/message_pack_format/portable/ddsketch.rs +++ /dev/null @@ -1,1364 +0,0 @@ -//! Wire-format-aligned DDSketch types. The wire DTO + runtime ops -//! live together here. - -use serde::{Deserialize, Serialize}; - -use crate::message_pack_format::{Error as MsgPackError, MessagePackCodec}; - -/// Bucket-store growth chunk for the wire-format-aligned [`DdSketch`] -/// variant. Matches the Go reference implementation's `DDSketch.GrowChunk` -/// so the `store_counts` / `store_offset` layout written by -/// [`DdSketch::update`] is byte-identical to the Go `SerializePortable` -/// output for the same input stream. -pub const DDSKETCH_GROW_CHUNK: usize = 128; - -/// Safety limit for a single [`DdSketchDelta`] application: deltas arrive -/// over the wire untrusted, so a span beyond this many buckets is rejected -/// with an error instead of padding the dense store toward a multi-gigabyte -/// allocation. Generous relative to any legitimate producer: at α = 0.01 the -/// entire indexable range spans ~35k buckets. -pub const MAX_APPLY_DELTA_SPAN_BUCKETS: i64 = 1 << 22; - -// ===================================================================== -// Wire-format-aligned variant. -// -// `DdSketch` and `DdSketchDelta` below are the public-field, -// proto-decode-friendly types consumed by the query-engine -// accumulators. The high-throughput in-process variant is -// `crate::sketches::ddsketch::DDSketch`, which keeps its own design. -// ===================================================================== - -/// Sparse delta between two consecutive DDSketch snapshots — the -/// input shape for [`DdSketch::apply_delta`]. Mirrors the -/// `DDSketchDelta` proto (and its Rust bindings). Kept as a plain -/// struct so this crate doesn't need a tonic/prost dependency; proto -/// decode lives in the accumulator. -#[derive(Debug, Clone, Default)] -pub struct DdSketchDelta { - /// `(absolute_bucket_index, Δcount)` pairs, additive. - pub buckets: Vec<(i32, u64)>, - pub negative_buckets: Vec<(i32, u64)>, - pub zero_count: u64, - /// Δ total count. May be negative (signed on the wire). - pub d_count: i64, - /// Δ sum. - pub d_sum: f64, - /// Whether `new_min` carries a meaningful value. Min can only - /// decrease; a delta that didn't lower min sends `false`. - pub min_changed: bool, - pub new_min: f64, - /// Whether `new_max` carries a meaningful value. Max can only - /// increase. - pub max_changed: bool, - pub new_max: f64, -} - -/// Signed DDSketch state: separate magnitude stores and a zero count. -/// Positive-only MessagePack retains `[alpha, counts, offset]`; signed state -/// appends `[negative_counts, negative_offset, zero_count]` to that array. -#[derive(Debug, Clone, Deserialize)] -pub struct DdSketch { - /// Relative accuracy parameter; must satisfy `0 < alpha < 1`. - pub alpha: f64, - /// Bucket counts in absolute-index order. The absolute index of - /// `store_counts[i]` is `i + store_offset`. - pub store_counts: Vec, - /// Absolute bucket index corresponding to `store_counts[0]`. May - /// be negative. - pub store_offset: i32, - /// Negative-value magnitudes, indexed with the same logarithmic mapping. - #[serde(default)] - pub negative_store_counts: Vec, - #[serde(default)] - pub negative_store_offset: i32, - #[serde(default)] - pub zero_count: u64, -} - -impl Serialize for DdSketch { - fn serialize(&self, serializer: S) -> Result { - use serde::ser::SerializeStruct; - let signed = !self.negative_store_counts.is_empty() || self.zero_count != 0; - let mut state = serializer.serialize_struct("DdSketch", if signed { 6 } else { 3 })?; - state.serialize_field("alpha", &self.alpha)?; - state.serialize_field("store_counts", &self.store_counts)?; - state.serialize_field("store_offset", &self.store_offset)?; - if signed { - state.serialize_field("negative_store_counts", &self.negative_store_counts)?; - state.serialize_field("negative_store_offset", &self.negative_store_offset)?; - state.serialize_field("zero_count", &self.zero_count)?; - } - state.end() - } -} - -impl DdSketch { - /// Construct an empty sketch. - pub fn new(alpha: f64) -> Self { - assert!( - alpha > 0.0 && alpha < 1.0, - "alpha must be in (0,1); alpha=0 makes ln(gamma)=0 and every guard degenerate" - ); - Self { - alpha, - store_counts: Vec::new(), - store_offset: 0, - negative_store_counts: Vec::new(), - negative_store_offset: 0, - zero_count: 0, - } - } - - /// Construct from the decoded wire fields. - pub fn from_raw(alpha: f64, store_counts: Vec, store_offset: i32) -> Self { - Self { - alpha, - store_counts, - store_offset, - negative_store_counts: Vec::new(), - negative_store_offset: 0, - zero_count: 0, - } - } - - /// Convert every store to protobuf, including negative and zero observations. - pub fn to_proto(&self) -> crate::proto::sketchlib::DdSketchState { - crate::proto::sketchlib::DdSketchState { - alpha: self.wire_alpha(), - store_counts: self.store_counts.clone(), - store_offset: self.store_offset, - negative_store_counts: self.negative_store_counts.clone(), - negative_store_offset: self.negative_store_offset, - zero_count: self.zero_count, - } - } - - /// Restore all stores from a decoded protobuf state. - pub fn from_proto(state: crate::proto::sketchlib::DdSketchState) -> Self { - Self { - alpha: state.alpha, - store_counts: state.store_counts, - store_offset: state.store_offset, - negative_store_counts: state.negative_store_counts, - negative_store_offset: state.negative_store_offset, - zero_count: state.zero_count, - } - } - - /// Total number of values added, recovered by summing the bucket - /// counts. The wire format carries no `count` scalar, so this is the - /// authoritative count for quantile-rank computation. - pub fn total_count(&self) -> u64 { - self.store_counts - .iter() - .chain(&self.negative_store_counts) - .copied() - .fold(self.zero_count, u64::saturating_add) - } - - /// Merge one other sketch into self by aligning bucket arrays on - /// absolute indices. Both operands must share the same `alpha`. - /// - /// Like [`Self::apply_delta`], the operand is treated as untrusted wire - /// state: if the union span would exceed - /// [`MAX_APPLY_DELTA_SPAN_BUCKETS`] *and* grow beyond both operands' - /// existing stores, `Err` is returned before any allocation. Stores that - /// already legitimately span more than the cap (exotic small-α - /// configurations) merge normally as long as the union stays within - /// what either side already holds. - pub fn merge( - &mut self, - other: &DdSketch, - ) -> Result<(), Box> { - // Validate both store unions before publishing any of the merged state. - let mut merged = self.clone(); - merged.merge_positive(other)?; - let mut negative = Self::from_raw( - self.alpha, - self.negative_store_counts.clone(), - self.negative_store_offset, - ); - negative.merge_positive(&Self::from_raw( - other.alpha, - other.negative_store_counts.clone(), - other.negative_store_offset, - ))?; - merged.negative_store_counts = negative.store_counts; - merged.negative_store_offset = negative.store_offset; - merged.zero_count = self - .zero_count - .checked_add(other.zero_count) - .ok_or("DDSketch zero count overflow")?; - *self = merged; - Ok(()) - } - - fn merge_positive( - &mut self, - other: &DdSketch, - ) -> Result<(), Box> { - if (self.alpha - other.alpha).abs() > f64::EPSILON { - return Err(format!( - "DdSketch alpha mismatch: self={}, other={}", - self.alpha, other.alpha - ) - .into()); - } - - if other.store_counts.is_empty() { - return Ok(()); - } - if self.store_counts.is_empty() { - self.store_counts = other.store_counts.clone(); - self.store_offset = other.store_offset; - } else { - let self_start = self.store_offset as i64; - let self_end = self_start + self.store_counts.len() as i64; - let other_start = other.store_offset as i64; - let other_end = other_start + other.store_counts.len() as i64; - let new_start = self_start.min(other_start); - let new_end = self_end.max(other_end); - let new_len = new_end - new_start; - // Hostile-span guard: a decoded snapshot with store_offset near - // i32::MIN and a short count vector must not allocate a union - // spanning ~2^31 buckets (~17 GiB) in one call. Growth beyond - // both operands' existing spans is capped; stores that already - // legitimately exceed the cap pass through untouched. - let max_existing = - (self.store_counts.len() as i64).max(other.store_counts.len() as i64); - if new_len > MAX_APPLY_DELTA_SPAN_BUCKETS && new_len > max_existing { - return Err(format!( - "DdSketch merge spans {} buckets, exceeding the {}-bucket safety limit", - new_len, MAX_APPLY_DELTA_SPAN_BUCKETS - ) - .into()); - } - let new_len = new_len as usize; - let mut merged = vec![0u64; new_len]; - for (start, counts) in [ - (self_start, &self.store_counts), - (other_start, &other.store_counts), - ] { - for (i, &c) in counts.iter().enumerate() { - let idx = (start + i as i64 - new_start) as usize; - merged[idx] = merged[idx].saturating_add(c); - } - } - self.store_counts = merged; - self.store_offset = new_start as i32; - } - Ok(()) - } - - /// Apply a sparse delta to this sketch in place. Matches the - /// `ApplyDelta` logic in the Go reference implementation: - /// bucket counts add, total count + sum add, min can only decrease - /// and max can only increase. Used by the backend ingest path to - /// reconstitute a full sketch from a base snapshot + subsequent - /// delta-transmission frames. - /// - /// Deltas arrive over the wire and are treated as untrusted: before - /// mutating anything, the union span the delta would require is - /// checked against [`MAX_APPLY_DELTA_SPAN_BUCKETS`]. A hostile or - /// corrupt delta carrying an index near `i32::MAX` would otherwise - /// pad the dense store by ~2·10⁹ buckets (~17 GiB) in one call — - /// independent of α, unlike the `update()` path whose worst case is - /// bounded by the indexable-range guard. - pub fn apply_delta( - &mut self, - delta: &DdSketchDelta, - ) -> Result<(), Box> { - let mut merged = self.clone(); - merged.apply_positive_delta(delta)?; - let mut negative = Self::from_raw( - self.alpha, - self.negative_store_counts.clone(), - self.negative_store_offset, - ); - negative.apply_positive_delta(&DdSketchDelta { - buckets: delta.negative_buckets.clone(), - ..Default::default() - })?; - merged.negative_store_counts = negative.store_counts; - merged.negative_store_offset = negative.store_offset; - merged.zero_count = self - .zero_count - .checked_add(delta.zero_count) - .ok_or("DDSketch zero count overflow")?; - *self = merged; - Ok(()) - } - - fn apply_positive_delta( - &mut self, - delta: &DdSketchDelta, - ) -> Result<(), Box> { - // Pre-validate span before any mutation so a rejected delta - // leaves state untouched. - if !delta.buckets.is_empty() { - let mut min_d = i64::MAX; - let mut max_d = i64::MIN; - for (abs_idx, _) in &delta.buckets { - min_d = min_d.min(*abs_idx as i64); - max_d = max_d.max(*abs_idx as i64); - } - let cur_start = if self.store_counts.is_empty() { - min_d - } else { - self.store_offset as i64 - }; - let cur_end = cur_start + self.store_counts.len() as i64; - let new_start = cur_start.min(min_d); - let new_end = cur_end.max(max_d + 1); - if new_end - new_start > MAX_APPLY_DELTA_SPAN_BUCKETS { - return Err(format!( - "DdSketch delta spans {} buckets, exceeding the {}-bucket safety limit", - new_end - new_start, - MAX_APPLY_DELTA_SPAN_BUCKETS - ) - .into()); - } - } - - for (abs_idx, d_count) in &delta.buckets { - if self.store_counts.is_empty() { - self.store_counts = vec![0u64; 1]; - self.store_offset = *abs_idx; - } - let cur_start = self.store_offset as i64; - let cur_end = cur_start + self.store_counts.len() as i64; - let k = *abs_idx as i64; - if k < cur_start { - // Prepend zeros. - let pad = (cur_start - k) as usize; - let mut buf = vec![0u64; pad]; - buf.append(&mut self.store_counts); - self.store_counts = buf; - self.store_offset = *abs_idx; - } else if k >= cur_end { - let pad = (k - cur_end + 1) as usize; - self.store_counts.extend(std::iter::repeat_n(0u64, pad)); - } - let arr_idx = (k - self.store_offset as i64) as usize; - self.store_counts[arr_idx] = self.store_counts[arr_idx].saturating_add(*d_count); - } - Ok(()) - } - - /// Compute a sparse, proto-marshalled `DDSketchDelta` of `self` - /// against a `snapshot`. A bucket is included when its `Δcount` - /// (self − snapshot, clamped at 0) is `>= threshold`. - /// - /// This is the Rust twin of the Go reference implementation's - /// `ComputeDelta`: it iterates every non-empty bucket in `self`, - /// subtracts the snapshot's count for the same absolute index, and - /// emits the surviving bucket deltas. The returned bytes are a - /// `prost`-encoded [`crate::proto::sketchlib::DdSketchDelta`], - /// byte-identical to the Go `proto.Marshal(DDSketchDelta)` output - /// for the same inputs (cross-language byte parity). - /// - /// Delta-against-empty: when `snapshot` is the empty sketch, every - /// surviving bucket delta equals the window's own bucket count, so - /// the result is this window's full state encoded as a delta (no - /// cross-window subtraction). The DataPoint-level metric scalars are - /// not carried — the count delta is recoverable by summing bucket - /// deltas. - pub fn compute_delta(&self, snapshot: &DdSketch, threshold: u64) -> Vec { - use crate::proto::sketchlib::{DdSketchBucketDelta, DdSketchDelta as ProtoDelta}; - use prost::Message; - - let mut delta = ProtoDelta::default(); - if !self.store_counts.is_empty() { - for (i, &c) in self.store_counts.iter().enumerate() { - if c == 0 { - continue; - } - let k = self.store_offset + i as i32; - let snap_count: u64 = if !snapshot.store_counts.is_empty() { - let idx = k as i64 - snapshot.store_offset as i64; - if idx >= 0 && (idx as usize) < snapshot.store_counts.len() { - snapshot.store_counts[idx as usize] - } else { - 0 - } - } else { - 0 - }; - let dc = c.saturating_sub(snap_count); - if dc >= threshold { - delta.buckets.push(DdSketchBucketDelta { - index: k, - d_count: dc, - }); - } - } - } - let negative = Self::from_raw( - self.alpha, - self.negative_store_counts.clone(), - self.negative_store_offset, - ); - let previous = Self::from_raw( - snapshot.alpha, - snapshot.negative_store_counts.clone(), - snapshot.negative_store_offset, - ); - delta.negative_buckets = sparse_difference(&negative, &previous, threshold) - .into_iter() - .map(|(index, d_count)| DdSketchBucketDelta { index, d_count }) - .collect(); - delta.zero_count = self.zero_count.saturating_sub(snapshot.zero_count); - if delta.zero_count < threshold { - delta.zero_count = 0; - } - delta.encode_to_vec() - } - - /// Apply a `prost`-encoded [`crate::proto::sketchlib::DdSketchDelta`] - /// to this sketch in place (additive bucket merge). The Rust twin of - /// the Go reference implementation's `ApplyDelta`. - /// - /// Returns `Err` if `bytes` is not a valid `DDSketchDelta` proto. - pub fn apply_delta_bytes( - &mut self, - bytes: &[u8], - ) -> Result<(), Box> { - use crate::proto::sketchlib::DdSketchDelta as ProtoDelta; - use prost::Message; - - let proto = ProtoDelta::decode(bytes)?; - let delta = DdSketchDelta { - negative_buckets: proto - .negative_buckets - .into_iter() - .map(|b| (b.index, b.d_count)) - .collect(), - zero_count: proto.zero_count, - buckets: proto - .buckets - .into_iter() - .map(|b| (b.index, b.d_count)) - .collect(), - ..DdSketchDelta::default() - }; - self.apply_delta(&delta) - } - - /// MessagePack twin of [`Self::compute_delta`]. Computes the same - /// sparse bucket delta of `self` against `snapshot` (a bucket is - /// carried when its `Δcount = self − snapshot` clamped at 0 is - /// `>= threshold`), but serializes it as the parallel-array - /// MessagePack layout via [`DdSketchDelta`]'s [`MessagePackCodec`] - /// instead of proto. Same delta-against-empty semantics: against the - /// empty sketch every surviving delta equals this window's own bucket - /// count. - pub fn compute_delta_msgpack(&self, snapshot: &DdSketch, threshold: u64) -> Vec { - let mut delta = DdSketchDelta::default(); - if !self.store_counts.is_empty() { - for (i, &c) in self.store_counts.iter().enumerate() { - if c == 0 { - continue; - } - let k = self.store_offset + i as i32; - let snap_count: u64 = if !snapshot.store_counts.is_empty() { - let idx = k as i64 - snapshot.store_offset as i64; - if idx >= 0 && (idx as usize) < snapshot.store_counts.len() { - snapshot.store_counts[idx as usize] - } else { - 0 - } - } else { - 0 - }; - let dc = c.saturating_sub(snap_count); - if dc >= threshold { - delta.buckets.push((k, dc)); - } - } - } - delta.negative_buckets = sparse_difference( - &Self::from_raw( - self.alpha, - self.negative_store_counts.clone(), - self.negative_store_offset, - ), - &Self::from_raw( - snapshot.alpha, - snapshot.negative_store_counts.clone(), - snapshot.negative_store_offset, - ), - threshold, - ); - delta.zero_count = self.zero_count.saturating_sub(snapshot.zero_count); - if delta.zero_count < threshold { - delta.zero_count = 0; - } - delta - .to_msgpack() - .expect("DdSketchDelta msgpack encode is infallible for owned bucket arrays") - } - - /// MessagePack twin of [`Self::apply_delta_bytes`]. Decodes the - /// parallel-array MessagePack [`DdSketchDelta`] and applies it in - /// place (additive bucket merge). - /// - /// Returns `Err` if `bytes` is not a valid MessagePack `DdSketchDelta`. - pub fn apply_delta_msgpack_bytes( - &mut self, - bytes: &[u8], - ) -> Result<(), Box> { - let delta = DdSketchDelta::from_msgpack(bytes)?; - self.apply_delta(&delta) - } - - /// Merge a slice of references into a single new sketch. Returns - /// `Err` on alpha mismatch or an empty input. - pub fn merge_refs( - inputs: &[&DdSketch], - ) -> Result> { - let first = inputs - .first() - .ok_or("DdSketch::merge_refs called with empty input")?; - let mut merged = DdSketch::new(first.alpha); - for d in inputs { - merged.merge(d)?; - } - Ok(merged) - } - - /// Insert a single positive value. Updates count, sum, min/max - /// and increments the bucket for `floor(ln(v) / ln(gamma))` - /// where `gamma = (1+α)/(1-α)`. Provided primarily so tests can - /// build a ground-truth sketch to compare delta-apply output - /// against. - /// - /// Bucket-store growth mirrors the Go reference implementation's - /// `Buckets.ensure`: the first allocation is a half-chunk-centered - /// `GROW_CHUNK` of zeros, and subsequent expansions extend by - /// `max(needed, GROW_CHUNK)` so the on-the-wire `store_counts` / - /// `store_offset` layout is byte-identical to the Go - /// `SerializePortable` output. Without this chunked layout the - /// `DDSketchState` proto bytes would diverge from the Go producer's - /// payload (cross-language byte parity). - pub fn update(&mut self, value: f64) { - let _ = self.try_update(value); - } - - /// Reject NaN/infinity and values above the mapping range explicitly. - /// Like DataDog, magnitudes below the mapping range enter the zero bucket; - /// the relative-error guarantee does not cover those nonzero tiny values. - pub fn try_update(&mut self, value: f64) -> Result<(), &'static str> { - if !value.is_finite() { - return Err("DDSketch requires finite values"); - } - let (min_v, max_v) = crate::sketches::ddsketch::ddsketch_indexable_bounds(self.alpha); - if value.abs() > max_v { - return Err("DDSketch value exceeds indexable range"); - } - if value.abs() < min_v { - self.zero_count = self - .zero_count - .checked_add(1) - .ok_or("DDSketch zero count overflow")?; - return Ok(()); - } - let negative = value < 0.0; - let value = value.abs(); - let gamma = (1.0 + self.alpha) / (1.0 - self.alpha); - let ln_gamma = gamma.ln(); - // Reuse the positive-store growth routine for the magnitude store. - if negative { - std::mem::swap(&mut self.store_counts, &mut self.negative_store_counts); - std::mem::swap(&mut self.store_offset, &mut self.negative_store_offset); - } - let idx = (value.ln() / ln_gamma).floor() as i32; - self.ensure_bucket(idx); - let arr_idx = (idx as i64 - self.store_offset as i64) as usize; - self.store_counts[arr_idx] = self.store_counts[arr_idx].saturating_add(1); - if negative { - std::mem::swap(&mut self.store_counts, &mut self.negative_store_counts); - std::mem::swap(&mut self.store_offset, &mut self.negative_store_offset); - } - Ok(()) - } - - /// Ensure bucket `k` is addressable in `store_counts`, growing in - /// chunks of [`DDSKETCH_GROW_CHUNK`] to match the Go reference - /// implementation's `Buckets.ensure`. Empty stores are seeded with - /// a half-chunk of zeros centered on `k` (`store_offset = k - - /// GROW_CHUNK/2`); out-of-range expansions extend by `max(needed, - /// GROW_CHUNK)` on either side. The on-the-wire byte layout is - /// therefore identical between the Go and Rust producers fed the - /// same input stream — required for cross-language byte parity. - fn ensure_bucket(&mut self, k: i32) { - if self.store_counts.is_empty() { - self.store_counts = vec![0u64; DDSKETCH_GROW_CHUNK]; - self.store_offset = k - (DDSKETCH_GROW_CHUNK as i32 / 2); - return; - } - let cur_start = self.store_offset as i64; - let cur_end = cur_start + self.store_counts.len() as i64; - let kk = k as i64; - if kk < cur_start { - let needed = (cur_start - kk) as usize; - let grow = needed.max(DDSKETCH_GROW_CHUNK); - let mut buf = vec![0u64; grow]; - buf.extend_from_slice(&self.store_counts); - self.store_counts = buf; - self.store_offset -= grow as i32; - } else if kk >= cur_end { - let needed = (kk - cur_end + 1) as usize; - let grow = needed.max(DDSKETCH_GROW_CHUNK); - self.store_counts.extend(std::iter::repeat_n(0u64, grow)); - } - } - - /// Estimate the quantile at rank `q` ∈ [0, 1]. Walks the bucket - /// array in ascending absolute-index order, accumulating counts - /// until the target rank; returns the bucket's representative - /// value `gamma^k * (1 + alpha)` where `k` is the bucket's absolute - /// index. Returns `None` if the sketch is empty. - /// - /// For indexable nonzero order statistics, absolute error is at most - /// `alpha * abs(true_value)`. Tiny values mapped to zero are excluded. - pub fn quantile(&self, q: f64) -> Option { - let count = self.total_count(); - if count == 0 { - return None; - } - let target = (q * (count.saturating_sub(1)) as f64).floor() as u64; - self.value_at_rank(target) - } - - /// Estimate the linearly interpolated quantile at `q * (n - 1)`. - /// - /// Interpolation across opposite signs can cancel: its absolute error is - /// bounded by alpha times the weighted absolute endpoints, but its relative - /// error need not be bounded by alpha. Zero-mapped tiny values are excluded. - /// This interpolates the two adjacent order-statistic bucket estimates, - /// as required by PromQL quantile and continuous percentiles. The existing - /// `quantile` method retains its lower-order-statistic convention. - /// Returns `None` for invalid q/alpha, empty or overflowing counts, counts - /// above exact Float64 integer precision, or nonfinite bucket estimates. - pub fn quantile_interpolated(&self, q: f64) -> Option { - if !q.is_finite() - || !(0.0..=1.0).contains(&q) - || !self.alpha.is_finite() - || !(0.0..1.0).contains(&self.alpha) - || self.alpha == 0.0 - { - return None; - } - let count = self - .store_counts - .iter() - .chain(&self.negative_store_counts) - .try_fold(self.zero_count, |sum, count| sum.checked_add(*count))?; - if count == 0 || count > (1u64 << 53) { - return None; - } - let rank = q * (count - 1) as f64; - let lower_rank = rank.floor() as u64; - let upper_rank = rank.ceil() as u64; - let lower = self.value_at_rank(lower_rank)?; - let upper = self.value_at_rank(upper_rank)?; - if !lower.is_finite() || !upper.is_finite() { - return None; - } - let fraction = rank - lower_rank as f64; - let result = lower * (1.0 - fraction) + upper * fraction; - result.is_finite().then_some(result) - } - - fn value_at_rank(&self, target: u64) -> Option { - let mut target = target; - let gamma = (1.0 + self.alpha) / (1.0 - self.alpha); - for (i, &count) in self.negative_store_counts.iter().enumerate().rev() { - if target < count { - return Some( - -gamma.powf((self.negative_store_offset as i64 + i as i64) as f64) - * (1.0 + self.alpha), - ); - } - target = target.checked_sub(count)?; - } - if target < self.zero_count { - return Some(0.0); - } - target = target.checked_sub(self.zero_count)?; - let mut cumulative: u64 = 0; - let gamma = (1.0 + self.alpha) / (1.0 - self.alpha); - let mut last_nonempty: Option = None; - for (i, &c) in self.store_counts.iter().enumerate() { - if c > 0 { - last_nonempty = Some(i); - } - cumulative = cumulative.saturating_add(c); - if cumulative > target { - let k = self.store_offset as i64 + i as i64; - return Some(gamma.powf(k as f64) * (1.0 + self.alpha)); - } - } - // Numerical edge case: if we fall off the end (e.g. q == 1.0 and - // rounding lands past the final increment), estimate from the - // highest non-empty bucket. The wire format carries no `max` - // scalar; the representative is within DDSketch's α - // relative-accuracy bound of the true max. - last_nonempty.map(|i| { - let k = (self.store_offset as i64 + i as i64) as f64; - gamma.powf(k) * (1.0 + self.alpha) - }) - } - - /// Return the alpha value as it appears on the wire — round-tripped - /// through gamma exactly the way the Go reference implementation's - /// `DDSketch.SerializePortable` does it (`alpha = (gamma - 1) / - /// (gamma + 1)`, where `gamma = (1+α)/(1-α)`). The roundtrip - /// introduces a small floating-point drift; without applying it the - /// Rust producer's `DDSketchState.alpha` field bytes would diverge - /// from the Go producer's, and cross-language byte parity would fail - /// on the very first proto field. - #[inline] - pub fn wire_alpha(&self) -> f64 { - let gamma = (1.0 + self.alpha) / (1.0 - self.alpha); - (gamma - 1.0) / (gamma + 1.0) - } -} - -fn sparse_difference(current: &DdSketch, previous: &DdSketch, threshold: u64) -> Vec<(i32, u64)> { - current - .store_counts - .iter() - .enumerate() - .filter_map(|(i, count)| { - let key = i64::from(current.store_offset) + i as i64; - let old = key - i64::from(previous.store_offset); - let old_count = usize::try_from(old) - .ok() - .and_then(|i| previous.store_counts.get(i)) - .copied() - .unwrap_or(0); - let delta = count.saturating_sub(old_count); - (delta > 0 && delta >= threshold).then_some((key as i32, delta)) - }) - .collect() -} - -#[derive(Deserialize)] -struct DeltaWire( - Vec, - Vec, - #[serde(default)] Vec, - #[serde(default)] Vec, - #[serde(default)] u64, -); - -impl MessagePackCodec for DdSketch { - fn to_msgpack(&self) -> Result, MsgPackError> { - Ok(rmp_serde::to_vec(self)?) - } - - fn from_msgpack(bytes: &[u8]) -> Result { - Ok(rmp_serde::from_slice(bytes)?) - } -} - -impl MessagePackCodec for DdSketchDelta { - /// Encodes the sparse bucket delta as the 2-element MessagePack array - /// `[ idx:[]i32, d_count:[]u64 ]` — parallel arrays of absolute bucket - /// index → Δcount — byte-identical to Rust - /// `rmp_serde::to_vec(&(Vec, Vec))` (compact mode), the layout - /// the `sketchlib-go` `asapmsgpack` DDSketch delta encoder mirrors. Only - /// the bucket cells cross the wire; the DataPoint-level metric scalars - /// (`d_count`/`d_sum`/`new_min`/`new_max`) are reconstructable and are - /// not carried, matching the proto delta. - fn to_msgpack(&self) -> Result, MsgPackError> { - let idx: Vec = self.buckets.iter().map(|(i, _)| *i).collect(); - let d_count: Vec = self.buckets.iter().map(|(_, c)| *c).collect(); - if self.negative_buckets.is_empty() && self.zero_count == 0 { - Ok(rmp_serde::to_vec(&(idx, d_count))?) - } else { - let ni: Vec<_> = self.negative_buckets.iter().map(|b| b.0).collect(); - let nc: Vec<_> = self.negative_buckets.iter().map(|b| b.1).collect(); - Ok(rmp_serde::to_vec(&(idx, d_count, ni, nc, self.zero_count))?) - } - } - - fn from_msgpack(bytes: &[u8]) -> Result { - let DeltaWire(idx, d_count, ni, nc, zero_count) = rmp_serde::from_slice(bytes)?; - if idx.len() != d_count.len() || ni.len() != nc.len() { - return Err(rmp_serde::decode::Error::Syntax( - "DDSketch delta array lengths differ".into(), - ) - .into()); - } - Ok(DdSketchDelta { - buckets: idx.into_iter().zip(d_count).collect(), - negative_buckets: ni.into_iter().zip(nc).collect(), - zero_count, - ..DdSketchDelta::default() - }) - } -} - -#[cfg(test)] -mod tests { - use super::*; - use crate::message_pack_format::MessagePackCodec; - - #[test] - fn msgpack_delta_against_empty_round_trips() { - let mut w = DdSketch::new(0.01); - for i in 1..=200 { - w.update(i as f64); - } - let empty = DdSketch::new(0.01); - let bytes = w.compute_delta_msgpack(&empty, 1); - let mut recon = DdSketch::new(0.01); - recon.apply_delta_msgpack_bytes(&bytes).unwrap(); - assert_eq!(recon.total_count(), w.total_count()); - for q in [0.5, 0.9, 0.99] { - let got = recon.quantile(q).unwrap(); - let want = w.quantile(q).unwrap(); - assert!( - (got / want - 1.0).abs() <= 0.01, - "q={q}: got={got} want={want}" - ); - } - } - - #[test] - fn msgpack_delta_wire_layout_is_parallel_arrays() { - // Lock the cross-language wire shape: a 2-element array of parallel - // (idx, d_count) arrays, byte-identical to the tuple encoding the Go - // asapmsgpack DDSketch delta mirrors. - let mut w = DdSketch::new(0.01); - w.update(1.0); - w.update(1.0); - w.update(1000.0); - let empty = DdSketch::new(0.01); - let bytes = w.compute_delta_msgpack(&empty, 1); - let mut idx: Vec = Vec::new(); - let mut d_count: Vec = Vec::new(); - for (i, &c) in w.store_counts.iter().enumerate() { - if c != 0 { - idx.push(w.store_offset + i as i32); - d_count.push(c); - } - } - let expected = rmp_serde::to_vec(&(idx, d_count)).unwrap(); - assert_eq!(bytes, expected); - let d = DdSketchDelta::from_msgpack(&bytes).unwrap(); - assert_eq!(d.buckets.len(), 2, "two distinct buckets expected"); - } - - #[test] - fn test_new_empty() { - let d = DdSketch::new(0.01); - assert_eq!(d.total_count(), 0); - assert!(d.store_counts.is_empty()); - } - - #[test] - fn test_merge_aligned_same_offset() { - let mut a = DdSketch::from_raw(0.01, vec![1, 2, 3], -1); - let b = DdSketch::from_raw(0.01, vec![10, 20, 30], -1); - a.merge(&b).unwrap(); - assert_eq!(a.store_counts, vec![11, 22, 33]); - assert_eq!(a.store_offset, -1); - assert_eq!(a.total_count(), 66); - } - - #[test] - fn test_merge_overlapping_offsets() { - // a covers indices [-1, 0, 1]; b covers indices [0, 1, 2] - let mut a = DdSketch::from_raw(0.01, vec![1, 1, 1], -1); - let b = DdSketch::from_raw(0.01, vec![10, 10, 10], 0); - a.merge(&b).unwrap(); - // Merged window is [-1, 0, 1, 2] → [1, 11, 11, 10] - assert_eq!(a.store_counts, vec![1, 11, 11, 10]); - assert_eq!(a.store_offset, -1); - assert_eq!(a.total_count(), 33); - } - - #[test] - fn test_merge_disjoint_offsets() { - let mut a = DdSketch::from_raw(0.01, vec![1, 2], 0); - let b = DdSketch::from_raw(0.01, vec![3, 4], 5); - a.merge(&b).unwrap(); - // Window [0..7) → [1,2,0,0,0,3,4] - assert_eq!(a.store_counts, vec![1, 2, 0, 0, 0, 3, 4]); - assert_eq!(a.store_offset, 0); - } - - #[test] - fn test_apply_delta_additive_inside_store() { - let mut base = DdSketch::from_raw(0.01, vec![1, 2, 3], -1); - let delta = DdSketchDelta { - buckets: vec![(-1, 4), (0, 8), (1, 12)], - d_count: 24, - d_sum: 120.0, - min_changed: false, - new_min: 0.0, - max_changed: true, - new_max: 9.0, - ..Default::default() - }; - base.apply_delta(&delta).unwrap(); - assert_eq!(base.store_counts, vec![5, 10, 15]); - assert_eq!(base.total_count(), 30); - } - - #[test] - fn test_apply_delta_expands_store_on_new_bucket() { - // Base covers [0..2]; delta adds a bucket at absolute index 4. - let mut base = DdSketch::from_raw(0.01, vec![1, 2], 0); - let delta = DdSketchDelta { - buckets: vec![(4, 7)], - d_count: 7, - d_sum: 35.0, - min_changed: false, - new_min: 0.0, - max_changed: true, - new_max: 6.0, - ..Default::default() - }; - base.apply_delta(&delta).unwrap(); - assert_eq!(base.store_counts, vec![1, 2, 0, 0, 7]); - assert_eq!(base.store_offset, 0); - assert_eq!(base.total_count(), 10); - } - - #[test] - fn test_apply_delta_matches_full_merge() { - // Snapshot the sketch, add more samples via a merge, and confirm - // the delta+apply path lands at the same state. - let base = DdSketch::from_raw(0.01, vec![1, 2, 3], 0); - let addition = DdSketch::from_raw(0.01, vec![10, 0, 20], 0); - let mut via_merge = base.clone(); - via_merge.merge(&addition).unwrap(); - - let delta = DdSketchDelta { - buckets: vec![(0, 10), (2, 20)], - d_count: 30, - d_sum: 70.0, - min_changed: true, - new_min: 0.5, - max_changed: true, - new_max: 5.0, - ..Default::default() - }; - let mut via_delta = base; - via_delta.apply_delta(&delta).unwrap(); - - assert_eq!(via_delta.store_counts, via_merge.store_counts); - assert_eq!(via_delta.total_count(), via_merge.total_count()); - } - - #[test] - fn test_merge_alpha_mismatch() { - let mut a = DdSketch::new(0.01); - let b = DdSketch::new(0.02); - assert!(a.merge(&b).is_err()); - } - - #[test] - fn test_msgpack_round_trip() { - let original = DdSketch::from_raw(0.01, vec![1, 2, 3], -2); - let bytes = original.to_msgpack().unwrap(); - let decoded = DdSketch::from_msgpack(&bytes).unwrap(); - assert_eq!(decoded.alpha, original.alpha); - assert_eq!(decoded.store_counts, original.store_counts); - assert_eq!(decoded.store_offset, original.store_offset); - assert_eq!(decoded.total_count(), original.total_count()); - } - - /// The msgpack wire layout MUST be a 3-element array - /// `[alpha, store_counts, store_offset]` after dropping the - /// DataPoint-level METRIC scalars (count/sum/min/max). This pins the - /// element count so the bytes stay parity-aligned with the Go - /// reference implementation. - #[test] - fn test_msgpack_is_three_element_array() { - let sk = DdSketch::from_raw(0.01, vec![1, 2, 3], -2); - let bytes = sk.to_msgpack().unwrap(); - // rmp compact encoding leads with an array marker. A fixarray of - // length 3 is the single byte 0x93 (0b1001_0011). - assert_eq!( - bytes[0], 0x93, - "expected a 3-element msgpack fixarray (0x93), got {:#04x}", - bytes[0] - ); - } - - #[test] - fn test_insert_and_quantile_lognormal() { - // Ground-truth: insert a large i.i.d. log-normal sample into - // a full sketch and sanity-check P50 / P90 / P99. - let mut gt = DdSketch::new(0.01); - let mut rng = 0xdead_beefu64; - let mut next = || { - // xorshift64* - rng ^= rng << 13; - rng ^= rng >> 7; - rng ^= rng << 17; - rng - }; - // Box-Muller normal → log-normal(mu=3, sigma=0.7). - let lognormal = |u: u64, v: u64| -> f64 { - let r1 = (u as f64) / (u64::MAX as f64).max(1.0); - let r2 = (v as f64) / (u64::MAX as f64).max(1.0); - let z = (-2.0 * r1.max(1e-12).ln()).sqrt() * (2.0 * std::f64::consts::PI * r2).cos(); - (3.0 + 0.7 * z).exp() - }; - for _ in 0..100_000 { - gt.update(lognormal(next(), next())); - } - let p50 = gt.quantile(0.5).unwrap(); - let p99 = gt.quantile(0.99).unwrap(); - // Analytical P50 = exp(mu) = e^3 ≈ 20.09; - // P99 ≈ exp(mu + sigma * Φ⁻¹(0.99)) = e^(3 + 0.7×2.326) ≈ 102.4. - assert!( - (p50 / 20.09).ln().abs() < 0.05, - "P50 {p50} not close to 20.09" - ); - assert!( - (p99 / 102.4).ln().abs() < 0.05, - "P99 {p99} not close to 102.4" - ); - } - - /// Building a sketch via `base + apply_delta()` produces quantile - /// estimates within DDSketch's α bound of the ground-truth - /// full-sketch path. - #[test] - fn test_delta_chain_preserves_quantile_accuracy() { - let alpha = 0.01; - let mut rng = 0xcafe_babeu64; - let mut next = || { - rng ^= rng << 13; - rng ^= rng >> 7; - rng ^= rng << 17; - rng - }; - let lognormal = |u: u64, v: u64| -> f64 { - let r1 = (u as f64) / (u64::MAX as f64).max(1.0); - let r2 = (v as f64) / (u64::MAX as f64).max(1.0); - let z = (-2.0 * r1.max(1e-12).ln()).sqrt() * (2.0 * std::f64::consts::PI * r2).cos(); - (3.0 + 0.7 * z).exp() - }; - - // Path A (ground truth): one sketch, 50k samples inserted - // directly. - let mut full = DdSketch::new(alpha); - // Path B (delta chain): a base sketch from the first 10k - // samples, then 4 incremental "flushes" of 10k samples each, - // each transmitted as a delta computed against the previous - // snapshot. - let mut reconstituted = DdSketch::new(alpha); - let mut prev_snapshot = DdSketch::new(alpha); // what the "receiver" has cached - - let batch = 10_000; - let batches = 5; - for b in 0..batches { - let mut this_batch = prev_snapshot.clone(); - for _ in 0..batch { - let v = lognormal(next(), next()); - full.update(v); - this_batch.update(v); - } - // Compute a "delta" = diff of this_batch vs prev_snapshot - // in our in-memory struct shape. Matches what the Go - // reference implementation's `ComputeDelta` would put on - // the wire. - let delta = compute_dd_delta(&prev_snapshot, &this_batch); - if b == 0 { - // First batch seeds the reconstituted sketch. - reconstituted = this_batch.clone(); - } else { - reconstituted.apply_delta(&delta).unwrap(); - } - prev_snapshot = this_batch; - } - - // Both paths should see the same total count (exact), recovered - // by summing bucket counts now that the `count` scalar is gone. - assert_eq!( - reconstituted.total_count(), - full.total_count(), - "count diverged" - ); - - // And P50 / P90 / P99 should agree within α bound. - for q in [0.5, 0.9, 0.99] { - let got = reconstituted.quantile(q).unwrap(); - let want = full.quantile(q).unwrap(); - let rel_err = (got / want - 1.0).abs(); - assert!( - rel_err <= alpha, - "q={q} rel_err={rel_err:.4} exceeds α={alpha}: reconstituted={got}, full={want}", - ); - } - } - - /// Helper used only in the delta-chain test: computes a - /// `DdSketchDelta` from two snapshots. Mirrors the Go reference - /// implementation's `ComputeDelta` logic so the test exercises the - /// wire-format path end-to-end. - fn compute_dd_delta(snapshot: &DdSketch, current: &DdSketch) -> DdSketchDelta { - let mut cells = Vec::new(); - if !current.store_counts.is_empty() { - for (i, &c) in current.store_counts.iter().enumerate() { - if c == 0 { - continue; - } - let k = current.store_offset + i as i32; - let snap_count: u64 = if !snapshot.store_counts.is_empty() { - let idx = k as i64 - snapshot.store_offset as i64; - if idx >= 0 && (idx as usize) < snapshot.store_counts.len() { - snapshot.store_counts[idx as usize] - } else { - 0 - } - } else { - 0 - }; - let dc = c.saturating_sub(snap_count); - if dc > 0 { - cells.push((k, dc)); - } - } - } - let d_count = current.total_count() as i64 - snapshot.total_count() as i64; - DdSketchDelta { - buckets: cells, - d_count, - d_sum: 0.0, - min_changed: false, - new_min: 0.0, - max_changed: false, - new_max: 0.0, - ..Default::default() - } - } - - /// Delta-against-empty: a `compute_delta` against an EMPTY base - /// reconstructs the window's full state when applied (round-trip). - /// With `threshold = 1` every non-empty bucket survives, so applying - /// the delta to a fresh empty sketch yields the same bucket store as - /// the original window. - #[test] - fn test_compute_delta_against_empty_round_trips() { - let mut window = DdSketch::new(0.01); - for i in 1..=200 { - window.update(i as f64); - } - let empty = DdSketch::new(0.01); - - // Delta against empty = the window's full state in delta form. - let delta_bytes = window.compute_delta(&empty, 1); - - // Apply to a fresh empty base. - let mut reconstructed = DdSketch::new(0.01); - reconstructed.apply_delta_bytes(&delta_bytes).unwrap(); - - // Total count recovered exactly. - assert_eq!(reconstructed.total_count(), window.total_count()); - // Every non-empty bucket count matches (compare on absolute index). - for (i, &c) in window.store_counts.iter().enumerate() { - if c == 0 { - continue; - } - let k = window.store_offset + i as i32; - let idx = (k - reconstructed.store_offset) as usize; - assert_eq!(reconstructed.store_counts[idx], c, "bucket k={k}"); - } - // Quantiles agree within the α relative-accuracy bound. - for q in [0.5, 0.9, 0.99] { - let got = reconstructed.quantile(q).unwrap(); - let want = window.quantile(q).unwrap(); - assert!( - (got / want - 1.0).abs() <= 0.01, - "q={q}: reconstructed={got} window={want}" - ); - } - } - - /// Two consecutive windows each emit their OWN state — the second - /// window's delta-against-empty is NOT diffed against the first - /// window (no cross-window subtraction). - #[test] - fn test_consecutive_windows_emit_own_state() { - let empty = DdSketch::new(0.01); - - // Overlapping value ranges so a cross-window subtraction (win2 − - // win1) would genuinely differ from win2's own state — making the - // "no subtraction" assertion meaningful. - let mut win1 = DdSketch::new(0.01); - for i in 1..=50 { - win1.update(i as f64); - win1.update(i as f64); // win1 buckets carry count 2 - } - let mut win2 = DdSketch::new(0.01); - for i in 1..=50 { - win2.update(i as f64); // same buckets, count 1 - } - - // Each window diffs against EMPTY (delta-against-empty), not against - // the prior window. - let d2_against_empty = win2.compute_delta(&empty, 1); - let d2_against_win1 = win2.compute_delta(&win1, 1); - - // The delta-against-empty reconstructs win2 exactly. - let mut recon = DdSketch::new(0.01); - recon.apply_delta_bytes(&d2_against_empty).unwrap(); - assert_eq!(recon.total_count(), win2.total_count()); - - // Sanity: diffing against win1 would have produced a DIFFERENT - // (cross-window-subtracted) frame, proving the two are not the same - // and that diffing against empty avoids the cross-window subtraction. - assert_ne!( - d2_against_empty, d2_against_win1, - "delta-against-empty must differ from cross-window delta" - ); - } - - /// Cross-language byte-parity guard against the Go reference - /// implementation's `DDSketch.SerializePortable` output for the - /// deterministic input `(1..=50)` with `α = 0.01`. The hex blob - /// below was captured from a `proto.Marshal` of the Go envelope - /// (with `Producer` and `HashSpec` cleared). Any change to - /// [`DdSketch::update`]'s bucket-store growth that breaks parity - /// will surface here. - #[test] - fn test_update_then_envelope_matches_go_golden_bytes() { - use crate::proto::sketchlib::{ - DdSketchState, SketchEnvelope, sketch_envelope::SketchState, - }; - use prost::Message; - - let mut sk = DdSketch::new(0.01); - for i in 1..=50 { - sk.update(i as f64); - } - - let state = DdSketchState { - alpha: sk.wire_alpha(), - store_counts: sk.store_counts.clone(), - store_offset: sk.store_offset, - ..Default::default() - }; - let envelope = SketchEnvelope { - format_version: 1, - producer: None, - hash_spec: None, - sample_p: 0.0, - sketch_state: Some(SketchState::Ddsketch(state)), - }; - let mut got = Vec::with_capacity(envelope.encoded_len()); - envelope.encode(&mut got).expect("prost encode"); - - // Byte string for the same `(1..=50)`, α=0.01 input AFTER dropping - // the DataPoint-level METRIC scalars (count/sum/min/max → proto - // tags 4-7 reserved). 403 bytes total: a `SketchEnvelope` proto - // wrapping a `DDSketchState` carrying only `alpha`, `store_counts` - // (Go chunk-128 padded layout, offset = -64, len = 384) and - // `store_offset` (sint32 zigzag 0x7f = -64). - // - // NOTE: this golden is currently the RUST-produced value. It MUST be - // reconciled against the Go reference implementation's regenerated - // golden before declaring cross-language byte parity. - const GOLDEN_HEX: &str = "0801728e03096214ae47e17a843f128003000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000000000001000000000000000000000000000000000000000100000000000000000000000000000100000000000000000000010000000000000000010000000000000001000000000001000000000001000000000001000000010000000001000000010000010000000100000100000100000100000100010000010001000100010001000100010001000100010100010100010100010101000101010100010101010101010100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000187f"; - let want = decode_hex(GOLDEN_HEX); - assert_eq!( - got, - want, - "DDSketch envelope bytes diverge from golden \ - ({} bytes got vs {} bytes want)", - got.len(), - want.len(), - ); - } - - fn decode_hex(s: &str) -> Vec { - let bytes: Vec = s - .as_bytes() - .chunks(2) - .map(|pair| { - let high = hex_nibble(pair[0]); - let low = hex_nibble(pair[1]); - (high << 4) | low - }) - .collect(); - bytes - } - - #[test] - fn interpolated_quantile_preserves_adjacent_ranks_and_legacy_convention() { - let mut sketch = DdSketch::new(0.01); - sketch.update(20.0); - sketch.update(40.0); - let lower = sketch.quantile(0.0).unwrap(); - let upper = sketch.quantile(1.0).unwrap(); - assert_eq!(sketch.quantile(0.9), Some(lower)); - for (q, expected) in [(0.0, 20.0), (0.5, 30.0), (0.9, 38.0), (1.0, 40.0)] { - let actual = sketch.quantile_interpolated(q).unwrap(); - assert!((actual - expected).abs() <= expected * 0.01); - assert_eq!(actual, lower * (1.0 - q) + upper * q); - } - for q in [-0.1, 1.1, f64::NAN, f64::INFINITY] { - assert!(sketch.quantile_interpolated(q).is_none()); - } - assert!(DdSketch::new(0.01).quantile_interpolated(0.5).is_none()); - let mut singleton = DdSketch::new(0.01); - singleton.update(20.0); - assert_eq!( - singleton.quantile_interpolated(0.9), - singleton.quantile(0.0) - ); - for values in [vec![20.0, 20.0], vec![10.0, 10.0, 20.0, 40.0]] { - let mut repeated = DdSketch::new(0.01); - for &value in &values { - repeated.update(value); - } - for q in [0.0, 0.5, 0.9, 1.0] { - let rank = q * (values.len() - 1) as f64; - let fraction = rank - rank.floor(); - let expected = values[rank.floor() as usize] * (1.0 - fraction) - + values[rank.ceil() as usize] * fraction; - assert!( - (repeated.quantile_interpolated(q).unwrap() - expected).abs() - <= 0.01 * expected - ); - } - } - for alpha in [0.0, -0.1, 1.0, f64::NAN] { - assert!( - DdSketch::from_raw(alpha, vec![1], 0) - .quantile_interpolated(0.5) - .is_none() - ); - } - assert!( - DdSketch::from_raw(0.01, vec![(1u64 << 53) + 1], 0) - .quantile_interpolated(0.5) - .is_none() - ); - // Negative and zero inputs retain their ranks during interpolation. - let mut unsupported = DdSketch::new(0.01); - unsupported.update(-20.0); - unsupported.update(0.0); - assert!((unsupported.quantile_interpolated(0.9).unwrap() + 2.0).abs() <= 0.02); - let overflow = DdSketch::from_raw(0.01, vec![u64::MAX, 1], 0); - assert!(overflow.quantile_interpolated(0.5).is_none()); - } - - fn hex_nibble(c: u8) -> u8 { - match c { - b'0'..=b'9' => c - b'0', - b'a'..=b'f' => c - b'a' + 10, - b'A'..=b'F' => c - b'A' + 10, - _ => panic!("non-hex byte {}", c as char), - } - } -} diff --git a/src/message_pack_format/portable/mod.rs b/src/message_pack_format/portable/mod.rs index 97e0d372..9f79df41 100644 --- a/src/message_pack_format/portable/mod.rs +++ b/src/message_pack_format/portable/mod.rs @@ -3,15 +3,14 @@ //! Each per-algorithm submodule holds one sketch's pre-envelope wire type. //! These carry a cross-language contract of their own, on goldens apart //! from ASAPv1's `asapv1_golden/`: HLL, Count-Min, Count Sketch and KLL pin -//! byte parity against Go's serializers, KLL in both directions; DDSketch's -//! golden is a Rust self-pin, not yet reconciled. New code uses ASAPv1: -//! prefer a sketch's own `serialize_to_bytes`; do not add callers here. +//! byte parity against Go's serializers, KLL in both directions. New code +//! uses ASAPv1: prefer a sketch's own `serialize_to_bytes`; do not add +//! callers here. pub mod countminsketch; pub mod countminsketch_topk; pub mod countsketch; pub mod countsketch_topk; -pub mod ddsketch; pub mod delta_set_aggregator; pub mod hll; pub mod hydra_kll; diff --git a/src/proto/generated/sketchlib.v1.rs b/src/proto/generated/sketchlib.v1.rs index 2c098d3f..f4ccaf4b 100644 --- a/src/proto/generated/sketchlib.v1.rs +++ b/src/proto/generated/sketchlib.v1.rs @@ -601,65 +601,6 @@ pub struct CoinState { #[prost(uint32, tag = "3")] pub remaining_bits: u32, } -/// DDSketchState is the portable state of a DDSketch quantile sketch. -/// -/// Only alpha is required to reconstruct the index mapping. Consumers must -/// recompute all derived mapping values on load: -/// gamma = (1.0 + alpha) / (1.0 - alpha) -/// log_gamma = ln(gamma) -/// inv_log_gamma = 1.0 / log_gamma -/// -/// Bucket indices: a value v maps to bucket floor(ln(v) * inv_log_gamma). -/// Stored bucket absolute_index = array_index + store_offset. -#[derive(Clone, PartialEq, ::prost::Message)] -pub struct DdSketchState { - /// Relative accuracy guarantee. Required. Must satisfy 0 < alpha < 1. - #[prost(double, tag = "1")] - pub alpha: f64, - /// Bucket count array in index order (ascending absolute bucket index). - /// Absolute index of store_counts\[i\] is i + store_offset. - #[prost(uint64, repeated, tag = "2")] - pub store_counts: ::prost::alloc::vec::Vec, - /// Absolute bucket index corresponding to store_counts\[0\]. - /// May be negative (values < 1.0 map to negative bucket indices). - #[prost(sint32, tag = "3")] - pub store_offset: i32, - /// Signed extension: negative magnitudes use the same mapping as positives. - #[prost(uint64, repeated, tag = "16")] - pub negative_store_counts: ::prost::alloc::vec::Vec, - #[prost(sint32, tag = "17")] - pub negative_store_offset: i32, - #[prost(uint64, tag = "18")] - pub zero_count: u64, -} -/// DDSketchDelta carries only the buckets that changed above threshold T. -/// -/// DataPoint-level metric scalars (d_count/d_sum/new_min/new_max/min_changed/ -/// max_changed) are intentionally NOT carried on the wire: the count delta is -/// recoverable by summing bucket deltas, and min/max/quantiles are derived from -/// the bucket distribution within the relative-accuracy guarantee. This message -/// is byte-identical to the Go reference implementation's DDSketchDelta so the -/// Go and Rust runtimes emit byte-identical delta frames for identical window -/// state (cross-language byte parity). -#[derive(Clone, PartialEq, ::prost::Message)] -pub struct DdSketchDelta { - #[prost(message, repeated, tag = "1")] - pub buckets: ::prost::alloc::vec::Vec, - #[prost(message, repeated, tag = "16")] - pub negative_buckets: ::prost::alloc::vec::Vec, - #[prost(uint64, tag = "17")] - pub zero_count: u64, -} -/// DDSketchBucketDelta is the delta for one log-scale bucket. -#[derive(Clone, Copy, PartialEq, ::prost::Message)] -pub struct DdSketchBucketDelta { - /// signed bucket index (negative for values < 1) - #[prost(sint32, tag = "1")] - pub index: i32, - /// Δcount ≥ threshold T - #[prost(uint64, tag = "2")] - pub d_count: u64, -} /// UnivMonState is the portable state of a Universal Monitoring (UnivMon) sketch. /// /// UnivMon is a hierarchy of L Count Sketches paired with L heavy-hitter heaps. @@ -892,7 +833,7 @@ pub struct SketchEnvelope { /// The sketch payload. Exactly one field must be set. #[prost( oneof = "sketch_envelope::SketchState", - tags = "10, 11, 12, 13, 14, 15, 16, 17, 18" + tags = "10, 11, 12, 13, 15, 16, 17, 18" )] pub sketch_state: ::core::option::Option, } @@ -909,8 +850,6 @@ pub mod sketch_envelope { Hll(super::HyperLogLogState), #[prost(message, tag = "13")] Kll(super::KllState), - #[prost(message, tag = "14")] - Ddsketch(super::DdSketchState), #[prost(message, tag = "15")] Univmon(super::UnivMonState), #[prost(message, tag = "16")] diff --git a/src/sketches/ddsketch.rs b/src/sketches/ddsketch.rs index bc6dc56c..993c0959 100644 --- a/src/sketches/ddsketch.rs +++ b/src/sketches/ddsketch.rs @@ -212,9 +212,6 @@ impl TryFrom for DDSketch { /// `minIndexableValue`/`maxIndexableValue`. Values outside this range are /// handled without growing the dense store: smaller magnitudes enter the /// zero bucket, while larger magnitudes are rejected. -/// -/// Single source of truth shared by core `DDSketch`, the portable wire twin, -/// and tests, so the two implementations cannot drift algebraically again. pub fn ddsketch_indexable_bounds(alpha: f64) -> (f64, f64) { let gamma = (1.0 + alpha) / (1.0 - alpha); let inv_log_gamma = 1.0 / gamma.ln(); @@ -520,8 +517,7 @@ impl DDSketch { /// This is a REAL runtime check, not a `debug_assert!`: a `debug_assert!` /// compiles out in release builds, leaving a mismatched merge to corrupt /// results with no signal at all. DataDog's `MergeWith` and sketchlib-go's - /// Go `Merge` both return an error here; the portable `DdSketch::merge` in - /// this same crate does too. + /// Go `Merge` both return an error here. pub fn merge(&mut self, other: &DDSketch) -> Result<(), String> { if (self.alpha - other.alpha).abs() >= 1e-12 || (self.gamma - other.gamma).abs() >= 1e-12 { return Err(format!( diff --git a/tests/PBT_COVERAGE.md b/tests/PBT_COVERAGE.md index cddee44f..2758a99c 100644 --- a/tests/PBT_COVERAGE.md +++ b/tests/PBT_COVERAGE.md @@ -2,8 +2,8 @@ One `proptest` law per line. Each compares the sketch against an answer computed without it; what that answer is comes first in every section. One -file per sketch under `tests/pbt/`. `cargo test --test pbt` runs 367 laws; -with `--all-features`, 435, adding the four **experimental** modules. Every +file per sketch under `tests/pbt/`. `cargo test --test pbt` runs 365 laws; +with `--all-features`, 433, adding the four **experimental** modules. Every law was mutation-checked: break the implementation, the law goes red. Shared generators, `grid` and `round_trip!` live in `support.rs`. @@ -105,25 +105,24 @@ Compared against: a key's `(col, sign)` per row, read off a one-key sketch; the - msgpack keeps cells and estimates - ASAPv1 keeps 64 estimates; `edge_rows` x `edge_dimension` -## ddsketch.rs — 14 +## ddsketch.rs — 12 -Compared against: the sorted stream. `DDSketch::get_value_at_quantile` targets the `ceil(q*n)`-th value, portable `DdSketch::quantile` the `floor(q*(n-1))`-th. Streams carry negatives, zeros and positives inside the indexable band. +Compared against: the sorted stream. `DDSketch::get_value_at_quantile` targets the `ceil(q*n)`-th value. Streams carry negatives, zeros and positives inside the indexable band. - placement - negative store, zero counter, positive store hold exactly the three class counts; `get_count() == arrivals` - - portable `DdSketch`: the same three counts - `sum() == stream sum`, bit for bit - - first dense allocation: one 128-bucket chunk, seed value at its center; both impls, both signed stores + - first dense allocation: one 128-bucket chunk, seed value at its center; both signed stores - relative error `<= alpha` - - `|quantile(q) - exact| <= alpha * |exact|`, across the sign boundary, both impls + - `|quantile(q) - exact| <= alpha * |exact|`, across the sign boundary - `quantile(q)` non-decreasing in `q`, across the sign boundary - single value `v`: `quantile(q) == v` for every `q`; `min`, `max`, `q=0`, `q=1` equal the tracked extremes - merge - counts add; both stores, both offsets, zero counter identical either way round - merged `quantile(q)` within `alpha` of the combined stream's exact value - - different `alpha`: refused, receiver count unchanged, both impls; same `alpha`: merges + - different `alpha`: refused, receiver count unchanged; same `alpha`: merges - wire - - msgpack keeps stores, offsets, zero count, quantiles + - ASAPv1 keeps stores, offsets, zero count, `sum`, quantiles - ASAPv1 keeps count, `alpha`, `sum`, quantiles; stream of NaN, ±inf, ±0, f64 extremes ## elastic.rs — 12 diff --git a/tests/asapv1_golden.rs b/tests/asapv1_golden.rs index 30b41651..f57bf89f 100644 --- a/tests/asapv1_golden.rs +++ b/tests/asapv1_golden.rs @@ -9,9 +9,10 @@ //! See `asapv1_golden/README.md`. use asap_sketchlib::{ - Classic, Count, CountMin, ErtlMLE, FastPath, HllSketch, HllVariant, HyperLogLogHIPP12, - HyperLogLogP12, KLL, MessagePackCodec, RegularPath, Vector2D, + Classic, Count, CountMin, DDSketch, ErtlMLE, FastPath, HllSketch, HllVariant, + HyperLogLogHIPP12, HyperLogLogP12, KLL, MessagePackCodec, RegularPath, Vector1D, Vector2D, }; +use serde::Serialize; fn decode_hex(s: &str) -> Vec { let s = s.trim(); @@ -32,6 +33,8 @@ const GOLDEN_CS_FAST: &str = include_str!("../asapv1_golden/cs_i64_fast_2x4.hex" const GOLDEN_CS_I32: &str = include_str!("../asapv1_golden/cs_i32_regular_2x4.hex"); const GOLDEN_KLL_F64: &str = include_str!("../asapv1_golden/kll_f64_k200.hex"); const GOLDEN_KLL_I64: &str = include_str!("../asapv1_golden/kll_i64_k200.hex"); +const GOLDEN_DD_POSITIVE: &str = include_str!("../asapv1_golden/ddsketch_positive_a001.hex"); +const GOLDEN_DD_SIGNED: &str = include_str!("../asapv1_golden/ddsketch_signed_a001.hex"); /// The known P12 register pattern shared by all three HLL fixtures. fn p12_registers() -> Vec { @@ -287,3 +290,130 @@ fn kll_i64_k200_matches_golden() { assert_eq!(decoded.quantile(0.0), 1.0); assert_eq!(decoded.quantile(1.0), 50.0); } + +// --------------------------------------------------------------------------- +// DDSketch: bucket stores, offsets, zero count and scalars set directly +// (DDSketch never hashes). The positive-only state is metadata version 1; the +// signed state is version 2 and adds the negative store and zero count. +// --------------------------------------------------------------------------- + +#[derive(Serialize)] +struct DdStore { + counts: Vector1D, + offset: i32, +} + +/// The field order of `DDSketch`'s serde form. +#[derive(Serialize)] +struct DdState { + alpha: f64, + store: DdStore, + sum: f64, + min: f64, + max: f64, + negative_store: DdStore, + zero_count: u64, +} + +/// Positive-only: `uint` widths fixint / uint8 / uint16 / uint32 / uint64 in +/// the counts and an int8 offset. +const DD_POSITIVE_COUNTS: [u64; 7] = [1, 0, 127, 128, 300, 65536, 4294967296]; +const DD_POSITIVE_OFFSET: i32 = -40; +const DD_POSITIVE_SCALARS: (f64, f64, f64) = (2181071000.0, 0.453125, 0.5078125); + +/// Signed: uint16 positive offset, int16 negative offset, nonzero zero count. +const DD_SIGNED_COUNTS: [u64; 3] = [3, 0, 2]; +const DD_SIGNED_OFFSET: i32 = 310; +const DD_SIGNED_NEGATIVE_COUNTS: [u64; 2] = [5, 1]; +const DD_SIGNED_NEGATIVE_OFFSET: i32 = -208; +const DD_SIGNED_ZERO_COUNT: u64 = 7; +const DD_SIGNED_SCALARS: (f64, f64, f64) = (2523.90625, -0.016, 515.0); + +fn dd_sketch( + (counts, offset): (&[u64], i32), + (negative_counts, negative_offset): (&[u64], i32), + zero_count: u64, + (sum, min, max): (f64, f64, f64), +) -> DDSketch { + let state = DdState { + alpha: 0.01, + store: DdStore { + counts: Vector1D::from_vec(counts.to_vec()), + offset, + }, + sum, + min, + max, + negative_store: DdStore { + counts: Vector1D::from_vec(negative_counts.to_vec()), + offset: negative_offset, + }, + zero_count, + }; + rmp_serde::from_slice(&rmp_serde::to_vec(&state).expect("encode state")) + .expect("a valid DDSketch state") +} + +fn dd_positive() -> DDSketch { + dd_sketch( + (&DD_POSITIVE_COUNTS, DD_POSITIVE_OFFSET), + (&[], 0), + 0, + DD_POSITIVE_SCALARS, + ) +} + +fn dd_signed() -> DDSketch { + dd_sketch( + (&DD_SIGNED_COUNTS, DD_SIGNED_OFFSET), + (&DD_SIGNED_NEGATIVE_COUNTS, DD_SIGNED_NEGATIVE_OFFSET), + DD_SIGNED_ZERO_COUNT, + DD_SIGNED_SCALARS, + ) +} + +#[test] +fn ddsketch_positive_a001_matches_golden() { + let want = decode_hex(GOLDEN_DD_POSITIVE); + + let got = dd_positive().serialize_to_bytes().expect("serialize"); + assert_eq!(got, want, "DDSketch positive bytes diverge from golden"); + + let decoded = DDSketch::deserialize_from_bytes(&want).expect("decode"); + assert_eq!(decoded.alpha(), 0.01); + assert_eq!(decoded.store_counts(), DD_POSITIVE_COUNTS.as_slice()); + assert_eq!(decoded.store_offset(), DD_POSITIVE_OFFSET); + assert!(decoded.negative_store_counts().is_empty()); + assert_eq!(decoded.zero_count(), 0); + let (sum, min, max) = DD_POSITIVE_SCALARS; + assert_eq!(decoded.sum(), sum); + assert_eq!(decoded.min(), Some(min)); + assert_eq!(decoded.max(), Some(max)); + assert_eq!(decoded.get_count(), DD_POSITIVE_COUNTS.iter().sum::()); + assert_eq!(decoded.serialize_to_bytes().expect("re-serialize"), want); +} + +#[test] +fn ddsketch_signed_a001_matches_golden() { + let want = decode_hex(GOLDEN_DD_SIGNED); + + let got = dd_signed().serialize_to_bytes().expect("serialize"); + assert_eq!(got, want, "DDSketch signed bytes diverge from golden"); + + let decoded = DDSketch::deserialize_from_bytes(&want).expect("decode"); + assert_eq!(decoded.alpha(), 0.01); + assert_eq!(decoded.store_counts(), DD_SIGNED_COUNTS.as_slice()); + assert_eq!(decoded.store_offset(), DD_SIGNED_OFFSET); + assert_eq!( + decoded.negative_store_counts(), + DD_SIGNED_NEGATIVE_COUNTS.as_slice() + ); + assert_eq!(decoded.negative_store_offset(), DD_SIGNED_NEGATIVE_OFFSET); + assert_eq!(decoded.zero_count(), DD_SIGNED_ZERO_COUNT); + let (sum, min, max) = DD_SIGNED_SCALARS; + assert_eq!(decoded.sum(), sum); + assert_eq!(decoded.min(), Some(min)); + assert_eq!(decoded.max(), Some(max)); + assert_eq!(decoded.get_count(), 18); + assert_eq!(decoded.serialize_to_bytes().expect("re-serialize"), want); +} diff --git a/tests/bug_verification.rs b/tests/bug_verification.rs index 1de70dd6..d5b69ca7 100644 --- a/tests/bug_verification.rs +++ b/tests/bug_verification.rs @@ -10,51 +10,7 @@ //! in `tests/e2e/nitro.rs`, which names both failure modes on the same run //! that already exercises every ingestion path. -use asap_sketchlib::message_pack_format::portable::ddsketch::DdSketch as PortableDds; -use asap_sketchlib::{CountL2HH, DDSketch, DataInput, DefaultXxHasher}; - -// --------------------------------------------------------------------------- -// Bug 3: portable DdSketch reports gamma^(k+0.5) (bucket log-midpoint) as the -// representative (portable/ddsketch.rs:426). For values sitting at the lower -// edge of bucket k the relative error is sqrt(gamma)-1 ~= alpha + alpha^2/2, -// violating the advertised relative-accuracy guarantee. Core DDSketch already -// uses gamma^k * (1 + alpha) (ddsketch.rs:383-392), so the two disagree on -// identical data. -// -// Synthetic data: value gamma^k * (1 + 1e-6) repeated 10_000x, alpha = 0.05. -// Truth for every quantile: the value itself. Current behavior: 5.13% error. -// --------------------------------------------------------------------------- -#[test] -fn portable_ddsketch_respects_alpha_at_bucket_edges() { - let alpha = 0.05; - let gamma = (1.0f64 + alpha) / (1.0 - alpha); - let v = gamma.powi(20) * (1.0 + 1e-6); - - let mut port = PortableDds::new(alpha); - let mut core = DDSketch::new(alpha); - for _ in 0..10_000 { - port.update(v); - core.add(&v); - } - - let q_port = port.quantile(0.5).unwrap(); - let rel_port = ((q_port - v) / v).abs(); - assert!( - rel_port <= alpha * (1.0 + 1e-6), - "portable DdSketch median off by {rel_port:.5} (> alpha={alpha}) at a \ - bucket-edge value" - ); - - // Both implementations must sit within alpha of the truth on identical - // data. (Exact equality is impossible: core clamps representatives to its - // observed min/max, which the portable wire format does not carry.) - let q_core = core.get_value_at_quantile(0.5).unwrap(); - let rel_core = ((q_core - v) / v).abs(); - assert!( - rel_core <= alpha * (1.0 + 1e-6), - "core DDSketch median off by {rel_core:.5} (> alpha={alpha})" - ); -} +use asap_sketchlib::{CountL2HH, DataInput, DefaultXxHasher}; // --------------------------------------------------------------------------- // Bug 4: CountL2HH's hot-path L2 accumulation ran in plain i64 diff --git a/tests/common/conformance.rs b/tests/common/conformance.rs index e52705b4..5810a644 100644 --- a/tests/common/conformance.rs +++ b/tests/common/conformance.rs @@ -429,10 +429,8 @@ where /// interchangeable: a rank band says nothing about DDSketch's promise, and a /// relative-value band says nothing about KLL's. /// -/// The order statistic compared against comes from the spec's own -/// `DdRankConvention`, so a core `DDSketch` (`ceil(q*n)`) and the portable -/// `DdSketch` (`floor(q*(n-1))`) are each scored on the question they actually -/// answer. +/// The order statistic compared against comes from the spec's +/// `DdRankConvention`. pub fn relative_quantile_battery( sketch: &str, new_sketch: F, diff --git a/tests/common/specs.rs b/tests/common/specs.rs index 54f4aebf..54670d60 100644 --- a/tests/common/specs.rs +++ b/tests/common/specs.rs @@ -996,21 +996,14 @@ pub fn breakpoint_rank_interval_distance(estimated: &[(f64, f64)], sorted_truth: // DDSketch: relative value error // --------------------------------------------------------------------------- -/// Which order statistic a DDSketch implementation answers a quantile query -/// with. The two shipped implementations do not agree, and the truth a test -/// compares against has to follow the implementation it is testing. +/// Which order statistic a DDSketch answers a quantile query with; the truth +/// a test compares against follows it. #[derive(Clone, Copy, Debug, PartialEq, Eq)] pub enum DdRankConvention { /// `DDSketch::get_value_at_quantile`: `rank = ceil(q * n)`, 1-based, so the /// answer is `sorted[ceil(q*n) - 1]`. `q = 0` and `q = 1` short-circuit to /// the exact stored minimum and maximum. CeilNearestRank, - /// Portable `DdSketch::quantile`: `target = floor(q * (n - 1))`, 0-based, - /// so the answer is `sorted[floor(q*(n-1))]` — the lower-quantile - /// convention of the DDSketch paper and of DataDog's reference - /// implementation. No exact minimum or maximum is retained, so the - /// endpoints are bucket representatives like any other rank. - LowerFloor, } impl DdRankConvention { @@ -1024,7 +1017,6 @@ impl DdRankConvention { let q = q.clamp(0.0, 1.0); match self { DdRankConvention::CeilNearestRank => (((q * n as f64).ceil() as usize).clamp(1, n)) - 1, - DdRankConvention::LowerFloor => ((q * (n - 1) as f64).floor() as usize).min(n - 1), } } @@ -1036,7 +1028,6 @@ impl DdRankConvention { pub fn name(self) -> &'static str { match self { DdRankConvention::CeilNearestRank => "ceil(q*n) nearest-rank", - DdRankConvention::LowerFloor => "floor(q*(n-1)) lower-quantile", } } } @@ -1045,8 +1036,7 @@ impl DdRankConvention { /// /// DDSketch's guarantee is on the *value*, not the rank: the returned estimate /// for the `q`-quantile must be within `alpha` relative error of the exact -/// order statistic **at the same `q` under that implementation's own rank -/// convention**: +/// order statistic **at the same `q` under the sketch's rank convention**: /// /// ```text /// |est - true| / |true| <= alpha + numerical_slack(true) @@ -1081,14 +1071,6 @@ impl RelativeQuantileSpec { } } - /// A spec for the portable `DdSketch`. - pub fn portable(alpha: f64) -> Self { - Self { - alpha, - convention: DdRankConvention::LowerFloor, - } - } - /// A few ULP of headroom, scaled by the magnitude of the logarithm because /// that is what `powf`/`ln` round-off actually tracks. /// diff --git a/tests/conformance_kit.rs b/tests/conformance_kit.rs index dd57a219..f82ff7d3 100644 --- a/tests/conformance_kit.rs +++ b/tests/conformance_kit.rs @@ -10,7 +10,6 @@ use common::conformance::{ }; use common::{FreqTruth, zipf_u64}; -use asap_sketchlib::message_pack_format::portable::ddsketch::DdSketch as PortableDds; use asap_sketchlib::message_pack_format::portable::hll::{HllSketch, HllVariant}; use asap_sketchlib::{ Bloom, CMSHeap, CSHeap, CountL2HH, CountMin, DDSketch, DataInput, FastPath, FoldCMS, FoldCS, @@ -175,17 +174,6 @@ impl QuantileOps for KllDynamicAdapter { } } -struct PortableDdsAdapter(PortableDds); - -impl QuantileOps for PortableDdsAdapter { - fn update(&mut self, value: f64) { - self.0.update(value); - } - fn quantile(&self, q: f64) -> f64 { - self.0.quantile(q).expect("non-empty sketch") - } -} - struct UnivMonQAdapter(UnivMonQ); impl QuantileOps for UnivMonQAdapter { @@ -364,26 +352,6 @@ fn kll_family_passes_quantile_conformance() { .assert_ok(); } -/// DDSketch through the **relative-value-error** battery, against exact -/// nearest-rank order statistics — the guarantee it actually makes. -#[test] -fn ddsketch_passes_relative_quantile_conformance() { - const ALPHA: f64 = 0.01; - let values: Vec = common::normal_f64(40_000, 500.0, 80.0, 7001) - .into_iter() - .filter(|v| *v > 0.0) - .collect(); - - conformance::relative_quantile_battery( - "PortableDds", - || PortableDdsAdapter(PortableDds::new(ALPHA)), - &values, - common::specs::RelativeQuantileSpec::portable(ALPHA), - &conformance::DEFAULT_QUANTILE_QS, - ) - .assert_ok(); -} - #[test] fn univmonq_passes_quantile_conformance() { let values: Vec = common::uniform_u64(30_000, 50_000, 7002) diff --git a/tests/ddsketch_signed.rs b/tests/ddsketch_signed.rs index 3cb591c6..0bfa1e2e 100644 --- a/tests/ddsketch_signed.rs +++ b/tests/ddsketch_signed.rs @@ -1,73 +1,31 @@ -use asap_sketchlib::message_pack_format::MessagePackCodec; -use asap_sketchlib::{DDSketch, DdSketch}; +use asap_sketchlib::DDSketch; -// Both public implementations must count negative and zero observations. +// Negative and zero observations are counted. #[test] fn signed_samples_and_zero_are_retained() { - let mut portable = DdSketch::new(0.01); let mut core = DDSketch::new(0.01); for v in [-20.0, -2.0, 0.0, 0.0, 2.0, 20.0] { - portable.update(v); core.add(&v); } - assert_eq!(portable.total_count(), 6); assert_eq!(core.get_count(), 6); - assert_eq!(portable.quantile(0.5), Some(0.0)); assert_eq!(core.get_value_at_quantile(0.5), Some(0.0)); - assert!(portable.quantile(0.0).unwrap() < -19.0); - assert!(portable.quantile(1.0).unwrap() > 19.0); + assert!(core.get_value_at_quantile(0.0).unwrap() < -19.0); + assert!(core.get_value_at_quantile(1.0).unwrap() > 19.0); } -// Signed snapshots must survive both codecs and merge with positive-only state. +// Signed snapshots survive ASAPv1 and merge. #[test] fn signed_merge_and_serialization() { - let mut p = DdSketch::new(0.01); let mut c = DDSketch::new(0.01); for v in [-4.0, 0.0, 4.0] { - p.update(v); c.add(&v); } - let mut p2 = DdSketch::from_msgpack(&p.to_msgpack().unwrap()).unwrap(); let mut c2 = DDSketch::deserialize_from_bytes(&c.serialize_to_bytes().unwrap()).unwrap(); - p2.merge(&p).unwrap(); c2.merge(&c).unwrap(); - assert_eq!(p2.total_count(), 6); assert_eq!(c2.get_count(), 6); - assert_eq!(p2.quantile_interpolated(0.5), Some(0.0)); assert_eq!(c2.get_value_at_quantile(0.5), Some(0.0)); } -#[test] -fn signed_delta_and_proto_round_trips() { - use prost::Message; - let mut base = DdSketch::new(0.01); - for v in [-100., -1., 0., 1., 100.] { - base.update(v); - } - let mut current = base.clone(); - for v in [-1000., -0.1, 0., 0., 0.1, 1000.] { - current.update(v); - } - let mut protobuf = base.clone(); - protobuf - .apply_delta_bytes(¤t.compute_delta(&base, 1)) - .unwrap(); - let mut msgpack = base; - msgpack - .apply_delta_msgpack_bytes(¤t.compute_delta_msgpack(&msgpack, 1)) - .unwrap(); - let bytes = current.to_proto().encode_to_vec(); - let snapshot = DdSketch::from_proto( - asap_sketchlib::proto::sketchlib::DdSketchState::decode(bytes.as_slice()).unwrap(), - ); - for restored in [protobuf, msgpack, snapshot] { - assert_eq!(restored.total_count(), 11); - for q in [0., 0.1, 0.5, 0.9, 1.] { - assert!((restored.quantile(q).unwrap() - current.quantile(q).unwrap()).abs() < 1e-9); - } - } -} - #[test] fn signed_order_statistics_obey_relative_bound() { let mut values: Vec = (-300i32..=300) @@ -80,44 +38,32 @@ fn signed_order_statistics_obey_relative_bound() { }) .collect(); values.sort_by(f64::total_cmp); - let mut portable = DdSketch::new(0.01); let mut core = DDSketch::new(0.01); for v in &values { - portable.try_update(*v).unwrap(); core.try_add(v).unwrap(); } for i in 0..=1000 { let q = i as f64 / 1000.; - let pi = (q * (values.len() - 1) as f64).floor() as usize; let ci = ((q * values.len() as f64).ceil() as usize).max(1) - 1; - for (got, want) in [ - (portable.quantile(q).unwrap(), values[pi]), - (core.get_value_at_quantile(q).unwrap(), values[ci]), - ] { - assert!( - (got - want).abs() <= 0.010000001 * want.abs(), - "q={q}, got={got}, want={want}" - ); - } + let (got, want) = (core.get_value_at_quantile(q).unwrap(), values[ci]); + assert!( + (got - want).abs() <= 0.010000001 * want.abs(), + "q={q}, got={got}, want={want}" + ); } } #[test] fn zero_only_special_values_and_octo_flush() { use asap_sketchlib::sketch_framework::octo::DdWorkerSketch; - let mut portable = DdSketch::new(0.01); let mut core = DDSketch::new(0.01); for v in [0., -0., 0.] { - portable.try_update(v).unwrap(); core.try_add(&v).unwrap(); } for v in [f64::NAN, f64::INFINITY, f64::NEG_INFINITY] { - assert!(portable.try_update(v).is_err()); assert!(core.try_add(&v).is_err()); } - assert_eq!(portable.total_count(), 3); assert_eq!(core.get_count(), 3); - assert_eq!(portable.quantile_interpolated(0.9), Some(0.)); let mut worker = DdWorkerSketch::new(0.01); let mut replay = DDSketch::new(0.01); for v in [-10., -10., -10., -0.1, 0., 0., 0.1, 10.] { @@ -134,20 +80,12 @@ fn zero_only_special_values_and_octo_flush() { #[test] fn tiny_values_are_counted_in_zero_bucket_and_signed_serde_survives() { let mut core = DDSketch::new(0.01); - let mut portable = DdSketch::new(0.01); let tiny = f64::MIN_POSITIVE / 2.; for v in [-tiny, tiny, 0.] { core.try_add(&v).unwrap(); - portable.try_update(v).unwrap(); } assert_eq!(core.get_count(), 3); assert_eq!(core.zero_count(), 3); - assert_eq!(portable.total_count(), 3); - assert_eq!(portable.quantile_interpolated(0.5), Some(0.)); - let named: DdSketch = - rmp_serde::from_slice(&rmp_serde::to_vec_named(&portable).unwrap()).unwrap(); - assert_eq!(named.total_count(), 3); - assert_eq!(named.quantile(0.5), Some(0.)); core.try_add(&-3.).unwrap(); let decoded: DDSketch = rmp_serde::from_slice(&rmp_serde::to_vec(&core).unwrap()).unwrap(); assert_eq!(decoded.get_count(), 4); @@ -155,15 +93,3 @@ fn tiny_values_are_counted_in_zero_bucket_and_signed_serde_survives() { assert_eq!(decoded.min(), core.min()); assert_eq!(decoded.max(), core.max()); } - -#[test] -fn cross_zero_interpolation_is_not_a_relative_error_certificate() { - let mut sketch = DdSketch::new(0.01); - sketch.update(-1.); - sketch.update(1.001); - let exact = (-1. + 1.001) / 2.; - let estimated = sketch.quantile_interpolated(0.5).unwrap(); - // Both magnitudes land in the same bucket, so their representatives cancel. - assert_eq!(estimated, 0.); - assert!((estimated - exact).abs() > 0.01 * exact.abs()); -} diff --git a/tests/e2e/quantiles.rs b/tests/e2e/quantiles.rs index 637a27c4..197c266b 100644 --- a/tests/e2e/quantiles.rs +++ b/tests/e2e/quantiles.rs @@ -31,19 +31,17 @@ //! //! DDSketch's guarantee is deterministic — bucket width alone, no hash and no //! sampling — so its batteries tolerate zero violations and no statistical -//! model applies. The two shipped implementations answer a quantile query with -//! **different order statistics**, so each is compared against the truth for -//! its own convention; see `DdRankConvention`. +//! model applies. Its truth is the `ceil(q*n)` order statistic; see +//! `DdRankConvention`. use crate::common; -use common::specs::{DdRankConvention, KllRankSpec, RelativeQuantileSpec, Tally, rank_error}; +use common::specs::{KllRankSpec, RelativeQuantileSpec, Tally, rank_error}; use common::{ NumericTruth, assert_between, duplicate_heavy_f64, exponential_f64, log_uniform_f64, monotonic_f64, normal_f64, outside_in_ordering, uniform_u64, zipf_f64, }; -use asap_sketchlib::message_pack_format::portable::ddsketch::DdSketch as PortableDds; use asap_sketchlib::message_pack_format::portable::hydra_kll::HydraKllSketch; use asap_sketchlib::{ DDSketch, DataInput, KLL, KLLConfig, KLLDynamic, TumblingWindow, UnivMonQ, UnivMonQConfig, @@ -587,37 +585,26 @@ fn dds_streams(alpha: f64, n: usize, seed: u64) -> Vec<(&'static str, Vec)> ] } -/// DDSketch's relative-value-error guarantee, for the core sketch and the -/// portable wire twin, at every supported alpha. -/// -/// Each implementation is compared against the exact order statistic **of its -/// own rank convention**: `DDSketch::get_value_at_quantile` answers -/// `sorted[ceil(q*n) - 1]`, while the portable `DdSketch::quantile` answers -/// `sorted[floor(q*(n-1))]`. They are different questions (see -/// `ddsketch_core_and_portable_answer_different_order_statistics`), so a single -/// truth helper would score one of them against the other one's question and -/// quietly absorb the difference into `alpha`. +/// DDSketch's relative-value-error guarantee at every supported alpha, +/// against the exact order statistic `DDSketch::get_value_at_quantile` +/// answers: `sorted[ceil(q*n) - 1]`. /// /// The tolerance is `alpha + numerical_slack`, where the slack is a few ULP of /// the logarithmic mapping — never a percentage of alpha, which would license /// breaking the advertised guarantee by that percentage. #[test] -fn ddsketch_core_and_portable_satisfy_the_relative_value_error_contract() { +fn ddsketch_satisfies_the_relative_value_error_contract() { const SAMPLE_SIZES: [usize; 3] = [1_000, 20_000, 100_000]; for &alpha in &DDS_ALPHAS { let core_spec = RelativeQuantileSpec::core(alpha); - let port_spec = RelativeQuantileSpec::portable(alpha); let mut core_tally = Tally::default(); - let mut port_tally = Tally::default(); for (i, &n) in SAMPLE_SIZES.iter().enumerate() { let seed = 3_005_000u64 + i as u64 * 101 + (alpha * 1e6) as u64; for (label, values) in dds_streams(alpha, n, seed) { let mut core = DDSketch::new(alpha); - let mut port = PortableDds::new(alpha); for v in &values { core.add(v); - port.update(*v); } let truth = NumericTruth::new(values.clone()); assert_eq!( @@ -625,17 +612,9 @@ fn ddsketch_core_and_portable_satisfy_the_relative_value_error_contract() { truth.len(), "{label} alpha={alpha} n={n} seed={seed}: core dropped samples" ); - assert_eq!( - port.total_count() as usize, - truth.len(), - "{label} alpha={alpha} n={n} seed={seed}: portable dropped samples" - ); core_spec.tally_into(&mut core_tally, truth.sorted(), &DDS_QS, |q| { core.get_value_at_quantile(q) }); - port_spec.tally_into(&mut port_tally, truth.sorted(), &DDS_QS, |q| { - port.quantile(q) - }); } } // The guarantee is deterministic, not probabilistic: DDSketch's error @@ -649,126 +628,20 @@ fn ddsketch_core_and_portable_satisfy_the_relative_value_error_contract() { .collect::>() ); core_tally.assert_none(&format!("core DDSketch alpha={alpha}"), &context); - port_tally.assert_none(&format!("portable DdSketch alpha={alpha}"), &context); - } -} - -/// The two implementations answer a quantile query with **different order -/// statistics**, and this pins the divergence rather than letting `alpha` -/// absorb it. -/// -/// - `DDSketch::get_value_at_quantile` uses `rank = ceil(q * n)`, 1-based. -/// - Portable `DdSketch::quantile` uses `target = floor(q * (n - 1))`, 0-based -/// — the lower-quantile convention of the DDSketch paper and of DataDog's -/// reference implementation, which is also what the wire format's Go twin -/// answers. -/// -/// **The decision taken here is to keep both.** The portable type exists to be -/// byte- and answer-compatible with `sketchlib-go`, so its convention is fixed -/// by an external contract; the core type's `ceil` convention is what its own -/// callers have been reading for the life of the API, and it is what lets `q=0` -/// and `q=1` return the exactly retained minimum and maximum. Changing either -/// silently moves numbers under existing callers, and the divergence is only -/// observable at small `n` or ragged `q` — precisely the cases pinned below. -/// What is *not* acceptable is leaving it undocumented, or scoring both against -/// one truth helper, which is what the previous revision did. -/// -/// The probes are chosen so the two formulas disagree: at `n = 3, q = 0.4` the -/// core answers `sorted[1]` and the portable `sorted[0]`. -#[test] -fn ddsketch_core_and_portable_answer_different_order_statistics() { - // (n, q, expected core 0-based index, expected portable 0-based index) - const PROBES: [(usize, f64, usize, usize); 8] = [ - (3, 0.4, 1, 0), - (4, 0.34, 1, 1), - (4, 0.3, 1, 0), - (7, 0.2, 1, 1), - (7, 0.6, 4, 3), - (5, 0.5, 2, 2), - (10, 0.25, 2, 2), - (10, 0.15, 1, 1), - ]; - - // Values one bucket apart at the coarsest alpha, so a one-rank difference - // is a different bucket and therefore a different answer — not two ranks - // that happen to share a representative. - const ALPHA: f64 = 0.01; - let gamma = (1.0 + ALPHA) / (1.0 - ALPHA); - - let mut disagreements = 0usize; - for &(n, q, core_idx, port_idx) in &PROBES { - let values: Vec = (0..n).map(|i| 100.0 * gamma.powi(3 * i as i32)).collect(); - let sorted = values.clone(); - - assert_eq!( - DdRankConvention::CeilNearestRank.index(n, q), - core_idx, - "core convention ceil(q*n)-1 at n={n} q={q}" - ); - assert_eq!( - DdRankConvention::LowerFloor.index(n, q), - port_idx, - "portable convention floor(q*(n-1)) at n={n} q={q}" - ); - - let mut core = DDSketch::new(ALPHA); - let mut port = PortableDds::new(ALPHA); - for v in &values { - core.add(v); - port.update(*v); - } - - let core_spec = RelativeQuantileSpec::core(ALPHA); - let port_spec = RelativeQuantileSpec::portable(ALPHA); - let core_est = core.get_value_at_quantile(q).expect("non-empty"); - let port_est = port.quantile(q).expect("non-empty"); - - if let Err(detail) = core_spec.check(q, core_est, sorted[core_idx]) { - panic!("core DDSketch n={n} q={q}: {detail}"); - } - if let Err(detail) = port_spec.check(q, port_est, sorted[port_idx]) { - panic!("portable DdSketch n={n} q={q}: {detail}"); - } - - if core_idx != port_idx { - disagreements += 1; - // Each implementation must be answering *its own* order statistic, - // so the two answers must differ here: if they agreed, one of them - // would have silently changed convention. - assert!( - core_est != port_est, - "n={n} q={q}: the conventions pick different order statistics \ - ({core_idx} vs {port_idx}) but both returned {core_est}" - ); - } } - assert!( - disagreements >= 3, - "the probe set must contain cases where the two conventions genuinely \ - disagree; only {disagreements} did" - ); } -/// Endpoint behaviour, which is **not** the same on the two implementations. -/// -/// - The core sketch tracks the exact minimum and maximum beside the bucket -/// store, and `get_value_at_quantile` short-circuits `q <= 0` and `q >= 1` to -/// them. Its endpoints are therefore *exact*, with zero error. -/// - The portable sketch carries no min/max scalars at all — they were removed -/// from the wire — so its endpoints are ordinary bucket representatives and -/// are only guaranteed within `alpha`. The previous revision of this test -/// claimed it "clamps its bucket representative into [min, max], so its -/// endpoints are exact too", which is not what the code does. +/// The sketch tracks the exact minimum and maximum beside the bucket store, and +/// `get_value_at_quantile` short-circuits `q <= 0` and `q >= 1` to them, so its +/// endpoints are exact. #[test] -fn ddsketch_core_endpoints_are_exact_and_portable_endpoints_are_alpha_relative() { +fn ddsketch_endpoints_are_exact() { for &alpha in &DDS_ALPHAS { let values = log_uniform_f64(5_000, (1.0 + alpha) / (1.0 - alpha), 3..30, 4_242); let truth = NumericTruth::new(values.clone()); let mut core = DDSketch::new(alpha); - let mut port = PortableDds::new(alpha); for v in &values { core.add(v); - port.update(*v); } assert_eq!( core.get_value_at_quantile(0.0), @@ -790,17 +663,6 @@ fn ddsketch_core_endpoints_are_exact_and_portable_endpoints_are_alpha_relative() Some(truth.max()), "core alpha={alpha}: max() must be exact" ); - - // The portable twin holds only buckets, so its endpoints get the same - // relative-value guarantee as any other quantile and nothing more. - let spec = RelativeQuantileSpec::portable(alpha); - let (p0, p1) = (port.quantile(0.0).unwrap(), port.quantile(1.0).unwrap()); - if let Err(detail) = spec.check(0.0, p0, truth.min()) { - panic!("portable alpha={alpha} at q=0: {detail}"); - } - if let Err(detail) = spec.check(1.0, p1, truth.max()) { - panic!("portable alpha={alpha} at q=1: {detail}"); - } } } @@ -813,11 +675,7 @@ fn ddsketch_core_endpoints_are_exact_and_portable_endpoints_are_alpha_relative() fn ddsketch_satisfies_the_relative_error_contract_at_bucket_boundaries() { for &alpha in &DDS_ALPHAS { let gamma = (1.0 + alpha) / (1.0 - alpha); - // A one-sample sketch answers every q with that one sample, so the two - // rank conventions coincide here by construction (both index 0) and the - // probe isolates the mapping from any rank effect. let core_spec = RelativeQuantileSpec::core(alpha); - let port_spec = RelativeQuantileSpec::portable(alpha); for k in [-40i32, -7, 0, 1, 13, 60, 200] { let edge = gamma.powi(k); if edge <= 0.0 || !edge.is_finite() { @@ -832,9 +690,7 @@ fn ddsketch_satisfies_the_relative_error_contract_at_bucket_boundaries() { ]; for probe in probes { let mut core = DDSketch::new(alpha); - let mut port = PortableDds::new(alpha); core.add(&probe); - port.update(probe); if core.get_count() == 0 { continue; // outside the indexable range at this alpha } @@ -844,10 +700,6 @@ fn ddsketch_satisfies_the_relative_error_contract_at_bucket_boundaries() { if let Err(detail) = core_spec.check(0.5, est, probe) { panic!("core DDSketch alpha={alpha} gamma^{k} boundary probe: {detail}"); } - let pest = port.quantile(0.5).unwrap(); - if let Err(detail) = port_spec.check(0.5, pest, probe) { - panic!("portable DdSketch alpha={alpha} gamma^{k} boundary probe: {detail}"); - } } } } @@ -858,13 +710,11 @@ fn ddsketch_satisfies_the_relative_error_contract_at_bucket_boundaries() { /// guarantee is stated over. #[test] fn ddsketch_merge_and_delta_replay_preserve_the_relative_error_contract() { - use asap_sketchlib::message_pack_format::portable::ddsketch::DdSketchDelta; use asap_sketchlib::octo_delta::DdDelta; const N: usize = 40_000; for &alpha in &DDS_ALPHAS { let spec = RelativeQuantileSpec::core(alpha); - let port_spec = RelativeQuantileSpec::portable(alpha); let values = zipf_f64(N, 8_192, 1.1, 1e3, 1e7, 7_654_321 + (alpha * 1e5) as u64); let truth = NumericTruth::new(values.clone()); @@ -929,49 +779,6 @@ fn ddsketch_merge_and_delta_replay_preserve_the_relative_error_contract() { &DDS_QS[1..6] ), ); - - // Portable side: merge, then a delta carrying the same buckets. - let mut pa = PortableDds::new(alpha); - let mut pb = PortableDds::new(alpha); - for (i, v) in values.iter().enumerate() { - if i % 2 == 0 { - pa.update(*v); - } else { - pb.update(*v); - } - } - pa.merge(&pb).expect("same-alpha portable merge"); - let mut port_tally = Tally::default(); - port_spec.tally_into(&mut port_tally, truth.sorted(), &DDS_QS, |q| pa.quantile(q)); - port_tally.assert_none( - &format!("portable DdSketch alpha={alpha} after merge"), - &format!("zipf n={N} over [1e3, 1e7]"), - ); - - let mut pr = PortableDds::new(alpha); - let delta = DdSketchDelta { - buckets: pa - .store_counts - .iter() - .enumerate() - .filter(|(_, c)| **c > 0) - .map(|(i, c)| (pa.store_offset + i as i32, *c)) - .collect(), - d_count: pa.total_count() as i64, - ..DdSketchDelta::default() - }; - pr.apply_delta(&delta).expect("benign delta"); - let mut pr_tally = Tally::default(); - port_spec.tally_into(&mut pr_tally, truth.sorted(), &DDS_QS[1..6], |q| { - pr.quantile(q) - }); - pr_tally.assert_none( - &format!("portable DdSketch alpha={alpha} rebuilt from a delta"), - &format!( - "zipf n={N} over [1e3, 1e7], interior q grid {:?}", - &DDS_QS[1..6] - ), - ); } } @@ -1622,19 +1429,17 @@ fn portable_hydra_kll_per_key_medians_satisfy_the_rank_characterization() { // ------------------------------------------- DDSketch input rejection /// Structural guards, not accuracy: values the mapping cannot index and -/// deltas spanning an implausible bucket range must be rejected without -/// corrupting state. No error bound applies to any of it. +/// merges across mappings must be rejected without corrupting state. No error +/// bound applies to any of it. #[test] fn ddsketch_rejects_untrackable_values_and_mapping_mismatches() { let alpha = 0.01; let mut core = DDSketch::new(alpha); - let mut port = PortableDds::new(alpha); - // Non-finite / above-maximum-magnitude values must be - // dropped by BOTH implementations: silently, without corrupting bucket 0 - // (NaN floor-casts to 0) and without letting one sample force a distant- - // bucket allocation (unguarded, f64::MAX maps ~35k buckets away at - // alpha=0.01 — ~277 KiB of amplification per sample, scaling with 1/lnγ). + // Non-finite / above-maximum-magnitude values are dropped silently, + // without corrupting bucket 0 (NaN floor-casts to 0) and without letting + // one sample force a distant-bucket allocation (unguarded, f64::MAX maps + // ~35k buckets away at alpha=0.01). for v in [ f64::NAN, f64::NEG_INFINITY, @@ -1643,31 +1448,27 @@ fn ddsketch_rejects_untrackable_values_and_mapping_mismatches() { 1e308, ] { core.add(&v); - port.update(v); } assert_eq!(core.get_count(), 0, "core must drop untrackable extremes"); - assert_eq!( - port.total_count(), - 0, - "portable must drop untrackable extremes" - ); assert!( - port.store_counts.len() < 10_000, - "portable store grew to {} buckets from untrackable input", - port.store_counts.len() + core.store_counts().is_empty(), + "store grew to {} buckets from untrackable input", + core.store_counts().len() + ); + assert_eq!( + core.get_value_at_quantile(0.5), + None, + "nothing trackable was added" ); - assert_eq!(port.quantile(0.5), None, "nothing trackable was added"); // Normal samples still work after the rejected inputs. let good = [1.0f64, 2.0, 4.0, 8.0, 16.0]; for v in good { core.add(&v); - port.update(v); } assert_eq!(core.get_count(), 5); - assert_eq!(port.total_count(), 5); assert_between( - port.store_counts.len() as f64, + core.store_counts().len() as f64, 1.0, 2.0 * 256.0, // initial GROW_CHUNK seed + at most one more "store stays compact", @@ -1675,46 +1476,30 @@ fn ddsketch_rejects_untrackable_values_and_mapping_mismatches() { // Boundary acceptance: values just INSIDE the indexable range must still // be tracked, so the guards reject only genuinely unmappable extremes. - // Bounds come from the shared production helper — core and portable MUST - // agree because they compute from the same function. let (min_idx, max_idx) = asap_sketchlib::sketches::ddsketch::ddsketch_indexable_bounds(alpha); - let mut port_boundary = PortableDds::new(alpha); let mut core_boundary = DDSketch::new(alpha); let just_inside_min = min_idx * (1.0 + 1e-9); // a hair above the floor let just_inside_max = max_idx * (1.0 - 1e-9); // a hair below the ceiling - port_boundary.update(just_inside_min); - port_boundary.update(just_inside_max); core_boundary.add(&just_inside_min); core_boundary.add(&just_inside_max); assert_eq!( - port_boundary.total_count(), + core_boundary.get_count(), 2, "in-range extremes near boundaries must be kept" ); - assert_eq!(core_boundary.get_count(), 2, "core boundary agreement"); - assert_eq!(port_boundary.total_count(), core_boundary.get_count()); // Below the minimum counts as zero; above the maximum is rejected. - port_boundary.update(min_idx * 0.5); - port_boundary.update(max_idx * (1.0 + 1e-6)); core_boundary.add(&(min_idx * 0.5)); - assert_eq!( - port_boundary.total_count(), - 3, - "sub-indexable neighbor enters zero bucket" - ); + core_boundary.add(&(max_idx * (1.0 + 1e-6))); assert_eq!( core_boundary.get_count(), 3, "core retains sub-indexable neighbor as zero" ); - - assert_eq!(port_boundary.zero_count, 1); assert_eq!(core_boundary.zero_count(), 1); - // Item 2 contract: mismatched mappings are a runtime error in BOTH types, - // not a debug-only assertion. + // Mismatched mappings are a runtime error, not a debug-only assertion. let other_alpha = 0.05; let mut core_other = DDSketch::new(other_alpha); core_other.add(&1.0); @@ -1722,79 +1507,20 @@ fn ddsketch_rejects_untrackable_values_and_mapping_mismatches() { core.merge(&core_other).is_err(), "core merge must reject alpha mismatch" ); - let mut port_other = PortableDds::new(other_alpha); - port_other.update(1.0); - assert!( - port.merge(&port_other).is_err(), - "portable merge must reject alpha mismatch" - ); - // Tiny-alpha regression: at alpha=1e-9 ln(gamma) ~ 2e-9, and naive - // reciprocal-multiplied guard formulas diverge from core's — admitting - // v=1e-300 whose bucket index saturates i32 and overflows ensure_bucket. - // Both implementations count it as zero without allocating a bucket. - let mut tiny = PortableDds::new(1e-9); + // Tiny-alpha regression: at alpha=1e-9 ln(gamma) ~ 2e-9, and v=1e-300 + // would saturate an i32 bucket index. It is counted as zero without + // allocating a bucket. let mut tiny_core = DDSketch::new(1e-9); - tiny.update(1e-300); tiny_core.add(&1e-300); - assert_eq!( - tiny.total_count(), - 1, - "portable counts sub-indexable value at tiny alpha" - ); assert_eq!( tiny_core.get_count(), 1, "core counts sub-indexable value at tiny alpha" ); - assert_eq!( - tiny.store_counts.len(), - 0, - "no allocation may occur for zero-mapped values" - ); -} - -#[test] -fn portable_ddsketch_rejects_hostile_delta_spans() { - use asap_sketchlib::message_pack_format::portable::ddsketch::DdSketchDelta; - - let alpha = 0.01; - let mut base = PortableDds::new(alpha); - base.update(1.0); - let (len_before, offset_before) = (base.store_counts.len(), base.store_offset); - - // A corrupt/hostile delta pointing near i32::MAX must be rejected with an - // error BEFORE any allocation: the naive pad would be ~2e9 buckets. - let hostile = DdSketchDelta { - buckets: vec![(i32::MAX - 1, 1), (7, 3)], - ..DdSketchDelta::default() - }; assert!( - base.apply_delta(&hostile).is_err(), - "hostile far-span delta must be rejected" - ); - assert_eq!( - base.store_counts.len(), - len_before, - "state untouched on rejection" - ); - assert_eq!( - base.store_offset, offset_before, - "offset untouched on rejection" - ); - - // Benign deltas still apply: bucket index 6 carries count 5 afterward, - // and its representative gamma^6*(1+alpha) becomes visible in queries. - let benign = DdSketchDelta { - buckets: vec![(6, 5)], - ..DdSketchDelta::default() - }; - base.apply_delta(&benign).expect("benign delta"); - let gamma = (1.0 + alpha) / (1.0 - alpha); - assert_eq!( - base.quantile(0.9), - Some(gamma.powf(6.0) * (1.0 + alpha)), - "applied delta count visible through queries" + tiny_core.store_counts().is_empty(), + "no allocation may occur for zero-mapped values" ); } @@ -2020,9 +1746,7 @@ fn ddsketch_satisfies_the_relative_value_error_contract_at_extreme_accuracy_para for (i, &alpha) in DDS_EXTREME_ALPHAS.iter().enumerate() { let core_spec = RelativeQuantileSpec::core(alpha); - let port_spec = RelativeQuantileSpec::portable(alpha); let mut core_tally = Tally::default(); - let mut port_tally = Tally::default(); let seed = 0x0DDA_0000u64 + i as u64 * 7919; let (min_indexable, max_indexable) = asap_sketchlib::sketches::ddsketch::ddsketch_indexable_bounds(alpha); @@ -2035,10 +1759,8 @@ fn ddsketch_satisfies_the_relative_value_error_contract_at_extreme_accuracy_para continue; } let mut core = DDSketch::new(alpha); - let mut port = PortableDds::new(alpha); for v in &values { core.add(v); - port.update(*v); } let truth = NumericTruth::new(values.clone()); assert_eq!( @@ -2046,24 +1768,15 @@ fn ddsketch_satisfies_the_relative_value_error_contract_at_extreme_accuracy_para truth.len(), "{label} alpha={alpha}: core dropped an indexable sample" ); - assert_eq!( - port.total_count() as usize, - truth.len(), - "{label} alpha={alpha}: portable dropped an indexable sample" - ); core_spec.tally_into(&mut core_tally, truth.sorted(), &DDS_QS, |q| { core.get_value_at_quantile(q) }); - port_spec.tally_into(&mut port_tally, truth.sorted(), &DDS_QS, |q| { - port.quantile(q) - }); } let context = format!( "alpha={alpha} n={N} indexable=[{min_indexable:.3e}, {max_indexable:.3e}] \ q grid {DDS_QS:?}" ); core_tally.assert_none(&format!("core DDSketch alpha={alpha}"), &context); - port_tally.assert_none(&format!("portable DdSketch alpha={alpha}"), &context); } } diff --git a/tests/msgpack_compat.rs b/tests/msgpack_compat.rs index 267a51c6..bd110fb3 100644 --- a/tests/msgpack_compat.rs +++ b/tests/msgpack_compat.rs @@ -31,8 +31,8 @@ use asap_sketchlib::message_pack_format::portable::delta_set_aggregator::DeltaRe use asap_sketchlib::message_pack_format::portable::hydra_kll::HydraKllSketchWire; use asap_sketchlib::message_pack_format::portable::kll::KllSketchData; use asap_sketchlib::{ - CountMinSketch, CountMinSketchWithHeap, CountSketch, DdSketch, HllSketch, HllVariant, - HydraKllSketch, KllSketch, SetAggregator, + CountMinSketch, CountMinSketchWithHeap, CountSketch, HllSketch, HllVariant, HydraKllSketch, + KllSketch, SetAggregator, }; // ===== round-trip: every wire-format-aligned type ===== @@ -72,18 +72,6 @@ fn count_sketch_round_trip() { assert_eq!(restored.cols, 64); } -#[test] -fn dd_sketch_round_trip() { - let mut s = DdSketch::new(0.01); - s.update(1.0); - s.update(10.0); - s.update(100.0); - let bytes = s.to_msgpack().expect("encode"); - let restored = DdSketch::from_msgpack(&bytes).expect("decode"); - // `count` is not on the wire; recover it by summing the bucket array. - assert_eq!(restored.total_count(), 3); -} - #[test] fn hll_sketch_round_trip() { let mut s = HllSketch::new(HllVariant::Regular, 8); @@ -210,7 +198,7 @@ fn hydra_kll_wire_shape() { // ===== golden-bytes placeholders: an ignored, uncovered gap ===== // -// The five tests below are **empty and ignored**. They verify nothing today. +// The four tests below are **empty and ignored**. They verify nothing today. // // What they need is a msgpack payload produced by `sketchlib-go` and checked in // under `tests/fixtures/msgpack/`. That fixture cannot be generated here: this @@ -237,10 +225,6 @@ fn count_min_decodes_go_bytes() { // assert_eq!(s.rows, EXPECTED_ROWS); } -#[ignore = "gap: needs a sketchlib-go-produced msgpack fixture; none can be generated in this repo"] -#[test] -fn dd_sketch_decodes_go_bytes() {} - #[ignore = "gap: needs a sketchlib-go-produced msgpack fixture; none can be generated in this repo"] #[test] fn hll_sketch_decodes_go_bytes() {} diff --git a/tests/pbt/ddsketch.rs b/tests/pbt/ddsketch.rs index b389ef9d..d52ba205 100644 --- a/tests/pbt/ddsketch.rs +++ b/tests/pbt/ddsketch.rs @@ -1,13 +1,11 @@ -//! Property tests for DDSketch: the native sketch and the portable `DdSketch` -//! that carries it over the wire. +//! Property tests for DDSketch. //! //! Masson, Rim, Lee, VLDB '19. The model is the paper's relative-error //! guarantee, not a transcription of the bucket arithmetic: an answer sits //! within a relative alpha of the exact order statistic of the stream. use crate::support::{QUANTILES, extreme_values}; -use asap_sketchlib::message_pack_format::MessagePackCodec; -use asap_sketchlib::{DDSketch, DdSketch}; +use asap_sketchlib::DDSketch; use proptest::prelude::*; const QUANTILE_PROBES: [f64; 7] = [0.0, 0.01, 0.25, 0.5, 0.75, 0.99, 1.0]; @@ -74,14 +72,6 @@ fn dd_of(alpha: f64, values: &[f64]) -> DDSketch { s } -fn portable_of(alpha: f64, values: &[f64]) -> DdSketch { - let mut s = DdSketch::new(alpha); - for v in values { - s.update(*v); - } - s -} - fn sorted(values: &[f64]) -> Vec { let mut v = values.to_vec(); v.sort_by(|a, b| a.partial_cmp(b).expect("the generator excludes NaN")); @@ -123,24 +113,6 @@ proptest! { prop_assert_eq!(sketch.get_count(), values.len() as u64, "total count"); } - #[test] - fn ddsketch_portable_keeps_negative_and_zero_observations( - alpha in alpha(), - values in signed_values(120), - ) { - let sketch = portable_of(alpha, &values); - let negatives = values.iter().filter(|v| **v < 0.0).count() as u64; - let zeros = values.iter().filter(|v| **v == 0.0).count() as u64; - let positives = values.iter().filter(|v| **v > 0.0).count() as u64; - - prop_assert_eq!( - sketch.negative_store_counts.iter().sum::(), negatives, "negative store" - ); - prop_assert_eq!(sketch.zero_count, zeros, "zero bucket"); - prop_assert_eq!(sketch.store_counts.iter().sum::(), positives, "positive store"); - prop_assert_eq!(sketch.total_count(), values.len() as u64, "total count"); - } - /// `sum` is carried on the wire and read back through `sum()`, so it is /// the running total of the stream, not an estimate off the buckets. #[test] @@ -159,7 +131,7 @@ proptest! { } /// The first dense allocation is one `GROW_CHUNK` centred on the value's - /// bucket, in both implementations and in both signed stores. The layout + /// bucket, in both signed stores. The layout /// reaches the wire as `store_counts` / `store_offset`, where the Go /// producer's bytes have to match it for the same input. #[test] @@ -168,13 +140,10 @@ proptest! { value in 1e-6f64..1e9, ) { let native = dd_of(alpha, &[value, -value]); - let portable = portable_of(alpha, &[value, -value]); for (label, counts) in [ - ("native positive", native.store_counts()), - ("native negative", native.negative_store_counts()), - ("portable positive", portable.store_counts.as_slice()), - ("portable negative", portable.negative_store_counts.as_slice()), + ("positive", native.store_counts()), + ("negative", native.negative_store_counts()), ] { prop_assert_eq!(counts.len(), GROW_CHUNK, "{} store width", label); prop_assert_eq!(counts[GROW_CHUNK_CENTRE], 1, "{} store centre", label); @@ -259,27 +228,6 @@ proptest! { } } - /// The portable twin answers the floor(q*(n-1)) order statistic, and it - /// carries no min/max, so the bound applies at every q including 0 and 1. - #[test] - fn ddsketch_portable_signed_answers_stay_within_alpha_of_the_exact_order_statistic( - alpha in alpha(), - values in signed_values(50), - ) { - let sketch = portable_of(alpha, &values); - let ordered = sorted(&values); - - for q in probe_quantiles(40) { - let want = ordered[(q * (ordered.len() - 1) as f64).floor() as usize]; - let got = sketch.quantile(q).expect("a non-empty sketch answers"); - prop_assert!( - (got - want).abs() <= want.abs() * (alpha + 1e-12), - "alpha={} q={} n={}: got {}, exact order statistic {}", - alpha, q, ordered.len(), got, want - ); - } - } - // ===== Merge ===== #[test] @@ -288,25 +236,25 @@ proptest! { a in maybe_signed_values(40), b in maybe_signed_values(40), ) { - let mut ab = portable_of(alpha, &a); - ab.merge(&portable_of(alpha, &b)).expect("merge"); - let mut ba = portable_of(alpha, &b); - ba.merge(&portable_of(alpha, &a)).expect("merge"); - - prop_assert_eq!(ab.total_count(), (a.len() + b.len()) as u64); - prop_assert_eq!(ab.total_count(), ba.total_count()); - prop_assert_eq!(&ab.store_counts, &ba.store_counts); - prop_assert_eq!(ab.store_offset, ba.store_offset); - prop_assert_eq!(&ab.negative_store_counts, &ba.negative_store_counts); - prop_assert_eq!(ab.negative_store_offset, ba.negative_store_offset); - prop_assert_eq!(ab.zero_count, ba.zero_count); + let mut ab = dd_of(alpha, &a); + ab.merge(&dd_of(alpha, &b)).expect("merge"); + let mut ba = dd_of(alpha, &b); + ba.merge(&dd_of(alpha, &a)).expect("merge"); + + prop_assert_eq!(ab.get_count(), (a.len() + b.len()) as u64); + prop_assert_eq!(ab.get_count(), ba.get_count()); + prop_assert_eq!(ab.store_counts(), ba.store_counts()); + prop_assert_eq!(ab.store_offset(), ba.store_offset()); + prop_assert_eq!(ab.negative_store_counts(), ba.negative_store_counts()); + prop_assert_eq!(ab.negative_store_offset(), ba.negative_store_offset()); + prop_assert_eq!(ab.zero_count(), ba.zero_count()); prop_assert_eq!( - ab.zero_count, + ab.zero_count(), a.iter().chain(&b).filter(|v| **v == 0.0).count() as u64, "the merged zero bucket" ); prop_assert_eq!( - ab.negative_store_counts.iter().sum::(), + ab.negative_store_counts().iter().sum::(), a.iter().chain(&b).filter(|v| **v < 0.0).count() as u64, "the merged negative store" ); @@ -340,8 +288,8 @@ proptest! { } /// Two alphas are two index mappings, so the bucket indices of one sketch - /// mean nothing under the other's gamma. Both implementations refuse the - /// merge and leave the receiver as it was. + /// mean nothing under the other's gamma. The merge is refused and leaves + /// the receiver as it was. #[test] fn ddsketch_merge_refuses_a_different_index_mapping( alpha in alpha(), @@ -357,13 +305,6 @@ proptest! { ); prop_assert_eq!(native.get_count(), values.len() as u64, "refused merge changed the count"); - let mut portable = portable_of(alpha, &values); - prop_assert!( - portable.merge(&portable_of(other_alpha, &values)).is_err(), - "portable alpha {} merged a sketch built with alpha {}", alpha, other_alpha - ); - prop_assert_eq!(portable.total_count(), values.len() as u64, "refused merge changed the count"); - prop_assert!( dd_of(alpha, &values).merge(&dd_of(alpha, &values)).is_ok(), "a shared mapping must still merge" @@ -377,19 +318,20 @@ proptest! { alpha in alpha(), values in maybe_signed_values(120), ) { - let s = portable_of(alpha, &values); - - let bytes = s.to_msgpack().expect("encode"); - let restored = DdSketch::from_msgpack(&bytes).expect("decode"); - - prop_assert_eq!(restored.total_count(), s.total_count()); - prop_assert_eq!(&restored.store_counts, &s.store_counts); - prop_assert_eq!(restored.store_offset, s.store_offset); - prop_assert_eq!(&restored.negative_store_counts, &s.negative_store_counts); - prop_assert_eq!(restored.negative_store_offset, s.negative_store_offset); - prop_assert_eq!(restored.zero_count, s.zero_count); + let s = dd_of(alpha, &values); + + let bytes = s.serialize_to_bytes().expect("encode"); + let restored = DDSketch::deserialize_from_bytes(&bytes).expect("decode"); + + prop_assert_eq!(restored.get_count(), s.get_count()); + prop_assert_eq!(restored.store_counts(), s.store_counts()); + prop_assert_eq!(restored.store_offset(), s.store_offset()); + prop_assert_eq!(restored.negative_store_counts(), s.negative_store_counts()); + prop_assert_eq!(restored.negative_store_offset(), s.negative_store_offset()); + prop_assert_eq!(restored.zero_count(), s.zero_count()); + prop_assert_eq!(restored.sum().to_bits(), s.sum().to_bits()); for q in QUANTILE_PROBES { - prop_assert_eq!(restored.quantile(q), s.quantile(q), "q={}", q); + prop_assert_eq!(restored.get_value_at_quantile(q), s.get_value_at_quantile(q), "q={}", q); } } diff --git a/tests/sketches_go_parity_probe.rs b/tests/sketches_go_parity_probe.rs index 2997e486..3ad693a8 100644 --- a/tests/sketches_go_parity_probe.rs +++ b/tests/sketches_go_parity_probe.rs @@ -12,12 +12,11 @@ use asap_sketchlib::common::DataInput; use asap_sketchlib::common::hash::CANONICAL_HASH_SEED; use asap_sketchlib::common::structures::HllBucketListP14; use asap_sketchlib::proto::sketchlib::{ - CountMinState, CountSketchState, CounterType, DdSketchState, HyperLogLogState, SketchEnvelope, + CountMinState, CountSketchState, CounterType, HyperLogLogState, SketchEnvelope, sketch_envelope::SketchState, }; use asap_sketchlib::sketches::countminsketch::CountMin; use asap_sketchlib::sketches::countsketch::Count; -use asap_sketchlib::sketches::ddsketch::DDSketch; use asap_sketchlib::sketches::hll::{ErtlMLE, HyperLogLogImpl}; use asap_sketchlib::{DefaultXxHasher, FastPath, Vector2D}; use prost::Message; @@ -39,15 +38,6 @@ const COUNTMIN_GOLDEN_HEX: &str = include_str!("../src/sketches/testdata/cms_env /// the wrapper test already uses. const HLL_GOLDEN_HEX: &str = include_str!("../src/sketches/testdata/hll_envelope_golden.hex"); -/// 403-byte DDSketch envelope for `alpha=0.01`, `(1..=50)` integer-as-f64 -/// input, without the DataPoint-level METRIC scalars (count/sum/min/max → -/// proto tags 4-7 reserved). -/// -/// NOTE: this is currently the RUST-produced value; it MUST be reconciled -/// against the parallel `sketchlib-go` golden regeneration before declaring -/// cross-language byte parity. -const DDSKETCH_GOLDEN_HEX: &str = "0801728e03096214ae47e17a843f128003000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000000000001000000000000000000000000000000000000000100000000000000000000000000000100000000000000000000010000000000000000010000000000000001000000000001000000000001000000000001000000010000000001000000010000010000000100000100000100000100000100010000010001000100010001000100010001000100010100010100010100010101000101010100010101010101010100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000187f"; - #[test] fn sketches_count_fastpath_matches_go_count_sketch_envelope() { let rows = 3usize; @@ -218,51 +208,6 @@ fn sketches_hll_classic_matches_go_envelope() { ); } -#[test] -fn sketches_ddsketch_matches_go_envelope() { - let mut sk = DDSketch::new(0.01); - for i in 1..=50i32 { - sk.add(&(i as f64)); - } - - // Go serializes `alpha` round-tripped through gamma to mirror its - // internal storage: `alpha_wire = (γ-1)/(γ+1)`, `γ = (1+α)/(1-α)`. - // This is a 25-ULP shift from the user-supplied α=0.01. - let alpha = sk.alpha(); - let gamma = (1.0 + alpha) / (1.0 - alpha); - let alpha_wire = (gamma - 1.0) / (gamma + 1.0); - // The DataPoint-level METRIC scalars (count/sum/min/max) are not on the - // wire; only alpha + the bucket array. - let state = DdSketchState { - alpha: alpha_wire, - store_counts: sk.store_counts().to_vec(), - store_offset: sk.store_offset(), - ..Default::default() - }; - let envelope = SketchEnvelope { - format_version: 1, - producer: None, - hash_spec: None, - sample_p: 0.0, - sketch_state: Some(SketchState::Ddsketch(state)), - }; - let mut got = Vec::with_capacity(envelope.encoded_len()); - envelope.encode(&mut got).expect("prost encode"); - - let want = decode_hex(DDSKETCH_GOLDEN_HEX); - assert_eq!( - got.len(), - want.len(), - "DDSketch envelope length: got {} want {}", - got.len(), - want.len(), - ); - assert_eq!( - got, want, - "sketches::DDSketch envelope diverges from Go golden", - ); -} - fn decode_hex(s: &str) -> Vec { s.trim() .as_bytes() diff --git a/tests/xtest_consumer.rs b/tests/xtest_consumer.rs index 0113ff45..7d0069f3 100644 --- a/tests/xtest_consumer.rs +++ b/tests/xtest_consumer.rs @@ -7,7 +7,6 @@ //! Files consumed (from $XTEST_DIR/): //! countmin.pb — CountMinState with float64 counters //! kll.pb — KLLState with quantile items and coin RNG state -//! ddsketch.pb — DDSketchState with alpha + bucket array //! hll.pb — HyperLogLogState (ErtlMLE variant) //! countsketch.pb — CountSketchState with float64 signed counters //! coco.pb — CocoSketchState (hash+val+hasKey buckets) @@ -159,57 +158,6 @@ fn cross_language_proto() { all_ok = false; } - // ----------------------------------------------------------------------- - // DDSketch - // ----------------------------------------------------------------------- - println!(); - println!("[DDSketch] Step 1/3 — Read ddsketch.pb"); - let bytes = read_file(in_dir.join("ddsketch.pb")); - let env = SketchEnvelope::decode(bytes.as_slice()).expect("decode ddsketch envelope"); - - println!( - "[DDSketch] Step 2/3 — Validate envelope (format_version={}, producer={})", - env.format_version, - env.producer.as_ref().map_or("?", |p| &p.library) - ); - - let dd_state = match env.sketch_state { - Some(sketch_envelope::SketchState::Ddsketch(ref s)) => s.clone(), - other => panic!("expected DDSketch sketch_state, got {other:?}"), - }; - - // `count` is not on the wire; recover it by summing the bucket counts. - let dd_total_count: u64 = dd_state.store_counts.iter().copied().sum(); - println!( - "[DDSketch] alpha={:.4} count={} buckets={} offset={}", - dd_state.alpha, - dd_total_count, - dd_state.store_counts.len(), - dd_state.store_offset - ); - - let dd = DdFromProto::from_state(&dd_state); - let p50_dd = dd.quantile(0.50); - let p99_dd = dd.quantile(0.99); - println!( - "[DDSketch] Step 3/3 — p50 ≈ {:.2} (expect ~5000) p99 ≈ {:.2} (expect ~9900)", - p50_dd.unwrap_or(f64::NAN), - p99_dd.unwrap_or(f64::NAN) - ); - - let p50_ok = p50_dd - .map(|v| (v - 5000.0).abs() / 5000.0 < 0.02) - .unwrap_or(false); - let p99_ok = p99_dd - .map(|v| (v - 9900.0).abs() / 9900.0 < 0.02) - .unwrap_or(false); - if p50_ok && p99_ok { - println!("[DDSketch] PASS"); - } else { - eprintln!("[DDSketch] FAIL: p50={p50_dd:?} p99={p99_dd:?}"); - all_ok = false; - } - // ----------------------------------------------------------------------- // HLL (ErtlMLE estimator) // ----------------------------------------------------------------------- @@ -1164,82 +1112,3 @@ impl KllFromProto { samples.last().unwrap().0 } } - -// --------------------------------------------------------------------------- -// Minimal DDSketch deserialization + quantile query -// --------------------------------------------------------------------------- - -struct DdFromProto { - gamma: f64, - #[allow(dead_code)] - inv_log_gamma: f64, - store_counts: Vec, - store_offset: i32, - // Total count is recovered by summing the bucket array; the - // DataPoint-level count/min/max scalars are not on the wire. - count: u64, -} - -impl DdFromProto { - fn from_state(s: &DdSketchState) -> Self { - let alpha = s.alpha; - let gamma = (1.0 + alpha) / (1.0 - alpha); - let log_gamma = gamma.ln(); - let inv_log_gamma = 1.0 / log_gamma; - Self { - gamma, - inv_log_gamma, - store_counts: s.store_counts.clone(), - store_offset: s.store_offset, - count: s.store_counts.iter().copied().sum(), - } - } - - fn bin_representative(&self, k: i32) -> f64 { - self.gamma.powf(k as f64 + 0.5) - } - - /// Representative value of the lowest non-empty bucket — the - /// α-bounded estimate of the minimum now that the `min` scalar is - /// gone from the wire. - fn min_estimate(&self) -> Option { - self.store_counts - .iter() - .position(|&c| c > 0) - .map(|i| self.bin_representative(self.store_offset + i as i32)) - } - - /// Representative value of the highest non-empty bucket — the - /// α-bounded estimate of the maximum. - fn max_estimate(&self) -> Option { - self.store_counts - .iter() - .rposition(|&c| c > 0) - .map(|i| self.bin_representative(self.store_offset + i as i32)) - } - - fn quantile(&self, q: f64) -> Option { - if self.count == 0 { - return None; - } - if q <= 0.0 { - return self.min_estimate(); - } - if q >= 1.0 { - return self.max_estimate(); - } - let rank = (q * self.count as f64).ceil() as u64; - let mut seen = 0u64; - for (i, &c) in self.store_counts.iter().enumerate() { - if c == 0 { - continue; - } - seen += c; - if seen >= rank { - let bin = self.store_offset + i as i32; - return Some(self.bin_representative(bin)); - } - } - self.max_estimate() - } -} diff --git a/tools/gen-proto/src/main.rs b/tools/gen-proto/src/main.rs index 1de6ad15..9c293417 100644 --- a/tools/gen-proto/src/main.rs +++ b/tools/gen-proto/src/main.rs @@ -32,7 +32,6 @@ const PROTO_FILES: &[&str] = &[ "countsketch/countsketch.proto", "hll/hll.proto", "kll/kll.proto", - "ddsketch/ddsketch.proto", "univmon/univmon.proto", "hydra/hydra.proto", "cocosketch/cocosketch.proto", From e0705a4a05eea726b66c6ba5228d606dbc9a7e46 Mon Sep 17 00:00:00 2001 From: GordonYuanyc Date: Fri, 2 Oct 2026 00:51:21 -0600 Subject: [PATCH 09/36] refactor(hll): tighten wire visibility and proto comments Make the HLL wire DTOs, kind_id constants and metadata builder private to wire.rs, state only what the remaining HLL proto messages are, shorten the HIP registers_of doc, and drop a stray proptest regression file. Co-Authored-By: Claude Opus 5.5 --- proto/hll/hll.proto | 38 +++++---------------------- src/proto/generated/sketchlib.v1.rs | 38 +++++---------------------- src/sketches/hll/wire.rs | 40 ++++++++++++++--------------- tests/pbt/hll.rs | 8 +++--- tests/proptest-regressions/hll.txt | 7 ----- 5 files changed, 37 insertions(+), 94 deletions(-) delete mode 100644 tests/proptest-regressions/hll.txt diff --git a/proto/hll/hll.proto b/proto/hll/hll.proto index eba61943..6a8f0793 100644 --- a/proto/hll/hll.proto +++ b/proto/hll/hll.proto @@ -10,14 +10,8 @@ option go_package = "github.com/ProjectASAP/sketchlib-go/proto/hll;hllpb"; // HyperLogLog // ============================================================================ -// HyperLogLogState is the portable state of a HyperLogLog cardinality sketch. -// -// Precision p determines the register count: num_registers = 2^precision. -// Both libraries default to precision = 14 (16 384 registers). -// -// The variant field is required because the three estimator algorithms are -// not interchangeable: loading an ErtlMLE state into a Regular estimator -// (or vice versa) produces incorrect cardinality estimates. +// HyperLogLogState is the state of one HyperLogLog cell. +// num_registers = 2^precision; variant selects the estimator. message HyperLogLogState { // HLL estimator variant. Required. HLLVariant variant = 1; @@ -26,14 +20,8 @@ message HyperLogLogState { // Default = 14 → 16 384 registers. uint32 precision = 2; - // DENSE register encoding: raw register values, length = 2^precision. - // Each byte stores the maximum (leading-zeros + 1) seen for that bucket. - // Stored as a raw byte string for compact encoding (1 byte per register). - // - // Exactly one of `registers` (dense) or `registers_sparse` (sparse) is - // populated by a producer. A decoder MUST read whichever is present; if - // both are empty the sketch is all-zero. This field stays at tag 3 so old - // dense-only readers keep working unchanged. + // Dense registers, one byte each, length 2^precision. Exactly one of + // `registers` and `registers_sparse` is set; both empty means all zero. bytes registers = 3; // ---- HIP-variant extra fields (populated only when variant = HIP) -------- @@ -47,13 +35,7 @@ message HyperLogLogState { // HIP running cardinality estimate. double hip_est = 6; - // SPARSE register encoding (additive, tag 7). Populated instead of - // `registers` (tag 3) when the number of non-zero registers is below the - // producer's dense/sparse crossover. Old readers that predate this field - // ignore tag 7 (proto skips unknown fields), so adding it is wire-backward- - // compatible. New readers decode BOTH representations — see - // HLLSparseRegisters. Reconstruction yields the identical 2^precision-byte - // dense register array, so cardinality estimation is unaffected. + // Sparse registers, set instead of `registers`; see HLLSparseRegisters. HLLSparseRegisters registers_sparse = 7; reserved 8 to 15; @@ -67,13 +49,9 @@ message HyperLogLogState { // uvarint(index - prev_index) // prev_index starts at 0; deltas are >= 0 // uvarint(value) // 1..=Q+1, always 1 byte in practice // -// For low/medium cardinality (most registers zero) this is far smaller than -// the dense 2^precision-byte array. Decoding allocates a zero-filled dense -// array of length `num_registers` and writes each decoded (index, value). +// Registers absent from `packed` are zero. message HLLSparseRegisters { - // Total number of registers in the reconstructed dense array (= 2^precision). - // Carried explicitly so the decoder can size the dense buffer without - // re-deriving it from precision. + // Length of the dense register array (= 2^precision). uint32 num_registers = 1; // Varint-packed (index_delta, value) pairs for non-zero registers, in @@ -89,8 +67,6 @@ enum HLLVariant { HLL_VARIANT_REGULAR = 1; // Otmar Ertl's MLE estimator (arXiv:1702.01284). - // Originally derived from Apache DataFusion's HLL implementation. - // Not compatible with REGULAR registers. HLL_VARIANT_ERTL_MLE = 2; // Kevin Lang's HyperLogLog++ Improved Proposal (HIP). diff --git a/src/proto/generated/sketchlib.v1.rs b/src/proto/generated/sketchlib.v1.rs index 8b1303a4..0383e092 100644 --- a/src/proto/generated/sketchlib.v1.rs +++ b/src/proto/generated/sketchlib.v1.rs @@ -392,14 +392,8 @@ pub struct CountSketchCell { #[prost(double, tag = "3")] pub d_count: f64, } -/// HyperLogLogState is the portable state of a HyperLogLog cardinality sketch. -/// -/// Precision p determines the register count: num_registers = 2^precision. -/// Both libraries default to precision = 14 (16 384 registers). -/// -/// The variant field is required because the three estimator algorithms are -/// not interchangeable: loading an ErtlMLE state into a Regular estimator -/// (or vice versa) produces incorrect cardinality estimates. +/// HyperLogLogState is the state of one HyperLogLog cell. +/// num_registers = 2^precision; variant selects the estimator. #[derive(Clone, PartialEq, ::prost::Message)] pub struct HyperLogLogState { /// HLL estimator variant. Required. @@ -409,14 +403,8 @@ pub struct HyperLogLogState { /// Default = 14 → 16 384 registers. #[prost(uint32, tag = "2")] pub precision: u32, - /// DENSE register encoding: raw register values, length = 2^precision. - /// Each byte stores the maximum (leading-zeros + 1) seen for that bucket. - /// Stored as a raw byte string for compact encoding (1 byte per register). - /// - /// Exactly one of `registers` (dense) or `registers_sparse` (sparse) is - /// populated by a producer. A decoder MUST read whichever is present; if - /// both are empty the sketch is all-zero. This field stays at tag 3 so old - /// dense-only readers keep working unchanged. + /// Dense registers, one byte each, length 2^precision. Exactly one of + /// `registers` and `registers_sparse` is set; both empty means all zero. #[prost(bytes = "vec", tag = "3")] pub registers: ::prost::alloc::vec::Vec, /// HIP accumulator component kxq0. @@ -428,13 +416,7 @@ pub struct HyperLogLogState { /// HIP running cardinality estimate. #[prost(double, tag = "6")] pub hip_est: f64, - /// SPARSE register encoding (additive, tag 7). Populated instead of - /// `registers` (tag 3) when the number of non-zero registers is below the - /// producer's dense/sparse crossover. Old readers that predate this field - /// ignore tag 7 (proto skips unknown fields), so adding it is wire-backward- - /// compatible. New readers decode BOTH representations — see - /// HLLSparseRegisters. Reconstruction yields the identical 2^precision-byte - /// dense register array, so cardinality estimation is unaffected. + /// Sparse registers, set instead of `registers`; see HLLSparseRegisters. #[prost(message, optional, tag = "7")] pub registers_sparse: ::core::option::Option, } @@ -446,14 +428,10 @@ pub struct HyperLogLogState { /// uvarint(index - prev_index) // prev_index starts at 0; deltas are >= 0 /// uvarint(value) // 1..=Q+1, always 1 byte in practice /// -/// For low/medium cardinality (most registers zero) this is far smaller than -/// the dense 2^precision-byte array. Decoding allocates a zero-filled dense -/// array of length `num_registers` and writes each decoded (index, value). +/// Registers absent from `packed` are zero. #[derive(Clone, PartialEq, ::prost::Message)] pub struct HllSparseRegisters { - /// Total number of registers in the reconstructed dense array (= 2^precision). - /// Carried explicitly so the decoder can size the dense buffer without - /// re-deriving it from precision. + /// Length of the dense register array (= 2^precision). #[prost(uint32, tag = "1")] pub num_registers: u32, /// Varint-packed (index_delta, value) pairs for non-zero registers, in @@ -469,8 +447,6 @@ pub enum HllVariant { /// Classic HLL with bias-correction tables (Flajolet et al. 2007). Regular = 1, /// Otmar Ertl's MLE estimator (arXiv:1702.01284). - /// Originally derived from Apache DataFusion's HLL implementation. - /// Not compatible with REGULAR registers. ErtlMle = 2, /// Kevin Lang's HyperLogLog++ Improved Proposal (HIP). /// Requires hip_kxq0, hip_kxq1, hip_est to be populated. diff --git a/src/sketches/hll/wire.rs b/src/sketches/hll/wire.rs index bb4e0f5e..9a5f93f2 100644 --- a/src/sketches/hll/wire.rs +++ b/src/sketches/hll/wire.rs @@ -36,25 +36,25 @@ impl HllWireVariant for ErtlMLE { /// positional). `registers` is a msgpack `bin` (one byte per register, matching /// Go's `[]byte`) via `serde_bytes` rather than serde's default `array`. #[derive(Debug, Serialize, Deserialize)] -pub(crate) struct HllPayloadPlain { +struct HllPayloadPlain { #[serde(with = "serde_bytes")] - pub(crate) registers: Vec, + registers: Vec, } /// HIP payload: the register bin plus the three HIP running scalars. #[derive(Debug, Serialize, Deserialize)] -pub(crate) struct HllPayloadHip { +struct HllPayloadHip { #[serde(with = "serde_bytes")] - pub(crate) registers: Vec, - pub(crate) hip_kxq0: f64, - pub(crate) hip_kxq1: f64, - pub(crate) hip_est: f64, + registers: Vec, + hip_kxq0: f64, + hip_kxq1: f64, + hip_est: f64, } const HLL_KIND_FAMILY: u8 = 0x01; -pub(crate) const HLL_KIND_CLASSIC: &[u8] = &[HLL_KIND_FAMILY, 0x01]; -pub(crate) const HLL_KIND_ERTL_MLE: &[u8] = &[HLL_KIND_FAMILY, 0x02]; -pub(crate) const HLL_KIND_HIP: &[u8] = &[HLL_KIND_FAMILY, 0x03]; +const HLL_KIND_CLASSIC: &[u8] = &[HLL_KIND_FAMILY, 0x01]; +const HLL_KIND_ERTL_MLE: &[u8] = &[HLL_KIND_FAMILY, 0x02]; +const HLL_KIND_HIP: &[u8] = &[HLL_KIND_FAMILY, 0x03]; /// Descriptor metadata for an HLL sketch (ASAPv1 §2), serialized as a msgpack /// **map** (`to_vec_named`) with keys in this declaration order — the canonical @@ -64,21 +64,21 @@ pub(crate) const HLL_KIND_HIP: &[u8] = &[HLL_KIND_FAMILY, 0x03]; /// decode fail closed on any unexpected key rather than silently dropping it. #[derive(Debug, PartialEq, Serialize, Deserialize)] #[serde(deny_unknown_fields)] -pub(crate) struct HllMetadata { - pub(crate) metadata_version: u8, - pub(crate) hash_profile_id: String, - pub(crate) hash_algorithm: String, - pub(crate) seed_derivation: String, - pub(crate) input_encoding: String, - pub(crate) seed_list: Vec, - pub(crate) canonical_seed_index: u32, - pub(crate) precision: u32, +struct HllMetadata { + metadata_version: u8, + hash_profile_id: String, + hash_algorithm: String, + seed_derivation: String, + input_encoding: String, + seed_list: Vec, + canonical_seed_index: u32, + precision: u32, } /// Builds the HLL descriptor metadata from the hasher's [`HashProfile`], so the /// wire bytes truthfully describe how the sketch was hashed (rather than /// hardcoding the standard profile). -pub(crate) fn hll_metadata(precision: u32) -> HllMetadata { +fn hll_metadata(precision: u32) -> HllMetadata { HllMetadata { metadata_version: 1, hash_profile_id: H::PROFILE_ID.to_string(), diff --git a/tests/pbt/hll.rs b/tests/pbt/hll.rs index 7159350a..373bf3c3 100644 --- a/tests/pbt/hll.rs +++ b/tests/pbt/hll.rs @@ -442,11 +442,9 @@ macro_rules! hip_laws { const M: usize = <$buckets>::NUM_REGISTERS; const P: u32 = <$buckets>::PRECISION as u32; - /// The registers, read back out of the ASAPv1 payload. The struct - /// holds them privately behind the `kxq0`/`kxq1` accumulators, and - /// those accumulators are what the increment law checks, so the - /// envelope is the one view of the registers that stays independent - /// of it. + /// The registers, read out of the ASAPv1 payload: the struct keeps + /// them private, and this view is independent of the `kxq0`/`kxq1` + /// accumulators the increment law checks. fn registers_of(sketch: &S) -> Vec { let bytes = sketch.serialize_to_bytes().expect("encode"); let lengths_at = 8 + bytes[7] as usize; diff --git a/tests/proptest-regressions/hll.txt b/tests/proptest-regressions/hll.txt deleted file mode 100644 index cb8fb6e2..00000000 --- a/tests/proptest-regressions/hll.txt +++ /dev/null @@ -1,7 +0,0 @@ -# Seeds for failure cases proptest has generated in the past. It is -# automatically read and these particular cases re-run before any -# novel cases are generated. -# -# It is recommended to check this file in to source control so that -# everyone who runs the test benefits from these saved cases. -cc 115f567853acb20cbde319427f1d20dbe1769471bdccb8f153dab74ff382dc7a # shrinks to stream = [] From 0153ee4dc3b25fd62e9bf7210b0db7d136a6480d Mon Sep 17 00:00:00 2001 From: GordonYuanyc Date: Fri, 2 Oct 2026 00:52:07 -0600 Subject: [PATCH 10/36] test(kll): ASAPv1 goldens for KLLDynamic; check decoded KLL state - asapv1_golden -> 13797db: kll_dynamic_f64_k200 and kll_dynamic_i64_k200 (kind_id 0x06 0x01, seed 42, no compaction; i64 items cross every msgpack integer width in both signs). - tests/asapv1_golden.rs: all four KLL cases read the golden payload by hand and compare levels, items and coin to the known state; compact decodes compare wire_levels/wire_items/wire_coin to the source; each KLL kind_id is rejected by the other variant's decoder. - KLLDynamic has seeded constructors: correct the wire.rs comment and the Q-KLL-SEED wording; fixture counts in the docs say seven kind_ids. - CHANGELOG: HydraKllSketch's cell type is reachable but unnameable; Added entry for the fixtures. - Re-wrap kll/wire.rs and e2e_testing_harness.md; drop kll.proto from the sketchlib.proto header list. Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 7 +- asapv1_golden | 2 +- docs/asapv1_wire_format.md | 6 +- docs/e2e_testing_harness.md | 5 +- docs/message_pack_format.md | 6 +- docs/tests.md | 6 +- proto/sketchlib.proto | 1 - src/sketches/kll/wire.rs | 3 +- src/sketches/kll_dynamic/wire.rs | 4 +- tests/asapv1_golden.rs | 157 +++++++++++++++++++++++++++++-- 10 files changed, 174 insertions(+), 23 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index bc0a2911..0f3d0237 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -66,6 +66,9 @@ signals a backwards-compatible change. worker's seed and the aggregator's from one seed. The unseeded constructors draw from the thread generator. The seed is not serialized, and the ASAPv1 payload is unchanged. +- **ASAPv1 golden fixtures for `KLLDynamic`** (kind_id `0x06 0x01`): + `kll_dynamic_f64_k200` and `kll_dynamic_i64_k200`, the i64 items crossing + every msgpack integer width in both signs, checked in `tests/asapv1_golden.rs`. ### Changed @@ -105,7 +108,9 @@ signals a backwards-compatible change. `MessagePackCodec` impls for `KllSketch` and `KllSketchData`, the value-offset codec (`KLL_SCALE_SWEEP`, `encode_value_offset`, `decode_value_offset`, `KllProtoItems`), and the `unsafe impl Send/Sync - for KllSketch`. `HydraKllSketch` keeps its per-cell type crate-internally; + for KllSketch`. `HydraKllSketch`'s per-cell type stays reachable through + the public fields `HydraKllSketch.sketch` and + `HydraKllSketchWire.sketches`, but can no longer be named; - `message_pack_format::native::{kll, kll_dynamic}`, the `MessagePackCodec` impls for `KLL` and `KLLDynamic`; - `KLL::from_portable_state`. diff --git a/asapv1_golden b/asapv1_golden index 8287a4ae..13797db1 160000 --- a/asapv1_golden +++ b/asapv1_golden @@ -1 +1 @@ -Subproject commit 8287a4ae04e6cfbfb3ed33dafbf1dac77d8f7fc1 +Subproject commit 13797db10c2782946ccc17ad9b1233c636faa1d9 diff --git a/docs/asapv1_wire_format.md b/docs/asapv1_wire_format.md index c4586bb9..7046da2d 100644 --- a/docs/asapv1_wire_format.md +++ b/docs/asapv1_wire_format.md @@ -1341,14 +1341,14 @@ Good direction (more compact, higher fidelity, less Rust-internal duplication), 2. **Golden byte-vector fixtures** in one shared repo, [`sketchlib-golden-bytes`](https://github.com/ProjectASAP/sketchlib-golden-bytes), which each implementation mounts at `asapv1_golden/`; both languages decode and re-encode them byte-identically. These replace the `portable`-as-oracle round-trip test. 3. **This registry**, mirrored, never independently allocated. -Fixtures exist for six `kind_id`s — HLL's three estimators, Count-Min, Count Sketch and compact KLL — and `sketchlib-go` mirrors those. Every other kind here has a payload and **no fixture**, so this document is its only contract; `asapv1_golden/README.md` lists the gap. `portable` carries its own Go goldens. +Fixtures exist for seven `kind_id`s — HLL's three estimators, Count-Min, Count Sketch and both KLL variants — and `sketchlib-go` mirrors those. Every other kind here has a payload and **no fixture**, so this document is its only contract; `asapv1_golden/README.md` lists the gap. `portable` carries its own Go goldens. **Hash profile on the Go side.** Rust derives the hash spec from a generic `HashProfile` bound on the hasher type; Go has no generic hasher type, so there is nothing to derive from. On the Go side the profile is simply **written into** the metadata on encode and **read from** it on decode. Go MUST validate the profile it reads (same fail-closed intent as Rust): a sketch is only mergeable/queryable if its `hash_profile_id` + seeds match the profile Go is prepared to reproduce. -Sequencing: (2) covers six `kind_id`s, and the one `native bytes == portable bytes` test is HLL-only, since no other `portable` type emits ASAPv1 bytes. +Sequencing: (2) covers seven `kind_id`s, and the one `native bytes == portable bytes` test is HLL-only, since no other `portable` type emits ASAPv1 bytes. Retire `portable` once the fixtures cover the rest. --- @@ -1361,7 +1361,7 @@ Retire `portable` once the fixtures cover the rest. - **Q-PROFILE**: the hash-spec metadata is **derived from the hasher's `HashProfile`** (`hll_metadata::` / `cms_metadata::`) and never hardcoded, so it is always truthful to the hasher. Custom hash profiles are **supported and self-describing**. Merge compatibility is hash-spec equality, so a custom-profile sketch is not mergeable with a standard one. - **Q-CMS**: Count-Min is one `kind_id` (`0x02 0x00`); counter type and mode live in the metadata, so the id stays single. The counter-type domain is `"i32"`, `"i64"` and `"f64"`, with no `i128` (no msgpack integer form); `i32` is recorded at its own width and pinned on decode. - **Q-KLL**: KLL metadata carries **no hash-spec group** — KLL is comparison-based and never hashes, so those fields have no truthful value. Its metadata is structural-only (`metadata_version`, `k`, `m`, `item_type`, optional `seed`) and is *not* `HashProfile`-derived. The two KLL variants (compact `0x06 0x00`, dynamic `0x06 0x01`) share one payload `[levels, items, coin]` and differ only by `kind_id`. `item_type` (`"f64"`/`"i64"`) is a metadata param, not a separate `kind_id` (mirrors Q-CMS's `counter_type`). Retained samples use the top-most-level-first layout that matches `sketchlib-go`. -- **Q-KLL-SEED**: KLL records its reproducible compaction `seed` as an **optional** metadata key. It is construction config (so metadata, not payload, per the config→metadata rule), and it is the first optional key in v1: present only when the sketch carries a seed, omitted otherwise. Rationale: the payload's `coin` already carries the RNG's *current* position (enough to resume compaction), but `seed` is what a later `clear()` re-seeds from — so serializing it lets a decoded sketch keep `clear()`-reproducibility instead of falling back to wall-clock. Cost of omitting it is bounded (only a decoded-then-`clear()`ed sketch loses cross-run byte reproducibility — never correctness), but it is cheap to carry and future-proofs the checkpoint/restore path. `KLLDynamic` has no seed concept and never emits the key; the two variants are deliberately **not** forced to be symmetric here. Go carries and preserves the key without interpreting it. +- **Q-KLL-SEED**: KLL records its reproducible compaction `seed` as an **optional** metadata key. It is construction config (so metadata, not payload, per the config→metadata rule), and it is the first optional key in v1: present only when the sketch carries a seed, omitted otherwise. Rationale: the payload's `coin` already carries the RNG's *current* position (enough to resume compaction), but `seed` is what a later `clear()` re-seeds from — so serializing it lets a decoded sketch keep `clear()`-reproducibility instead of falling back to wall-clock. Cost of omitting it is bounded (only a decoded-then-`clear()`ed sketch loses cross-run byte reproducibility — never correctness), but it is cheap to carry and future-proofs the checkpoint/restore path. `KLLDynamic` never emits the key, even when built with a seed; the two variants are deliberately **not** forced to be symmetric here. Go carries and preserves the key without interpreting it. - **Q-CS**: Count Sketch is one `kind_id` (`0x04 0x00`) and mirrors Count-Min's metadata and `[counts]` payload, including `counter_type`. Its type domain differs: `"i32"` and `"i64"`, with no `"f64"` — not because a float fails a bound, but because **no `CountSketchCounter` impl exists for it**: the trait covers `i32`, `i64` and `i128` and nothing else, so Count-Min's `f64` has no Count Sketch counterpart to serialize — and no `i128` (no msgpack integer form), which leaves the two `CsWireCounter` covers. Structural-param order matches Q-CMS: `... matrix_seed_index, rows, cols, counter_type, mode`. **`i32` is not widened to `i64`** — unlike Count-Min, whose counters are plain numbers, a Count Sketch appears *nested* inside `HydraCounter` and `EHSketchList` as `Vector2D` and must decode back into that variant, so the width is identity and is recorded exactly. Cells are signed, so a decoder must not assume monotonicity. - **Q-CMS-DIMS**: Count-Min `rows`/`cols` are **metadata** and the payload omits them. They are configuration that shapes the payload (like HLL's `precision`), so per the config-to-metadata rule they belong in the descriptor. The payload is then just `[counts]`. Canonical structural-param order: `... matrix_seed_index, rows, cols, counter_type, mode`. - **Q-VER**: no payload version field. A new incompatible encoding gets a **new `kind_id`**; retired ids are reserved forever and never recycled. diff --git a/docs/e2e_testing_harness.md b/docs/e2e_testing_harness.md index e7848ca8..db1f2caf 100644 --- a/docs/e2e_testing_harness.md +++ b/docs/e2e_testing_harness.md @@ -325,8 +325,9 @@ error against its bound. - No assertion depends on unseeded randomness — no `rand::rng()`, no wall-clock seeding, no implicit RNG inside a constructor. An unseeded - sketch appears only where the assertion holds for every draw. `KLL::init_kll_with_seed`, - `KLLDynamic::init_kll_with_seed`, `NitroBatch::with_target_and_seed`, `UniformSampling::with_seed`, + sketch appears only where the assertion holds for every draw. + `KLL::init_kll_with_seed`, `KLLDynamic::init_kll_with_seed`, + `NitroBatch::with_target_and_seed`, `UniformSampling::with_seed`, `Coco::init_with_size_and_seed` and `CocoOctoPlan::with_seed` exist for this. - An estimator whose guarantee is unbiasedness gets no per-key band: its accuracy is the mean over independent seeds, within standard errors of the diff --git a/docs/message_pack_format.md b/docs/message_pack_format.md index 56e2da00..f1847daf 100644 --- a/docs/message_pack_format.md +++ b/docs/message_pack_format.md @@ -169,9 +169,9 @@ decode → re-encode them byte-identically. The `kind_id` registry is mirrored verbatim with Go's `wire/asapmsgpack/magic_ids.go`, never independently allocated. -That proof covers the six `kind_id`s a fixture exists for — HLL's three -estimators, Count-Min, Count Sketch and compact KLL. Every other implemented -kind has **no ASAPv1 golden and therefore no ASAPv1 drift guard**; +That proof covers the seven `kind_id`s a fixture exists for — HLL's three +estimators, Count-Min, Count Sketch and both KLL variants. Every other +implemented kind has **no ASAPv1 golden and therefore no ASAPv1 drift guard**; [its README](https://github.com/ProjectASAP/sketchlib-golden-bytes#coverage) lists what is covered. The older `portable` path is guarded separately, by the `sketchlib-go` goldens of its own listed above. diff --git a/docs/tests.md b/docs/tests.md index ed4984b7..42396907 100644 --- a/docs/tests.md +++ b/docs/tests.md @@ -2106,8 +2106,10 @@ Test file: [`tests/asapv1_golden.rs`](../tests/asapv1_golden.rs) | `hll_classic_p12_matches_golden` | A known P12 Classic register pattern serializes to its golden bytes on both the portable and native paths. | Builds `HllSketch::from_raw(HllVariant::Regular, 12, regs, 0.0, 0.0, 0.0)` where `regs` is 4,096 zeros with `[0] = 1`, `[1] = 7`, `[100] = 42` and `[4095] = 3`, verifies `to_msgpack()` equals `asapv1_golden/hll_classic_p12.hex`, that `HyperLogLogP12::::deserialize_from_bytes` on the fixture recovers those registers and re-serializes identically, and that `HllSketch::from_msgpack` reports the same registers, `precision` `12` and `HllVariant::Regular`. | | `hll_ertl_mle_p12_matches_golden` | The Datafusion variant tag reaches the bytes for the same P12 register pattern. | Builds `HllSketch::from_raw(HllVariant::Datafusion, 12, regs, 0.0, 0.0, 0.0)` over the shared P12 pattern (`[0] = 1`, `[1] = 7`, `[100] = 42`, `[4095] = 3` in 4,096 registers), verifies `to_msgpack()` equals `asapv1_golden/hll_ertl_mle_p12.hex`, that `HyperLogLogP12::::deserialize_from_bytes` recovers the registers and re-serializes identically, and that `HllSketch::from_msgpack` reports `HllVariant::Datafusion`. | | `hll_hip_p12_matches_golden` | The HIP variant's three running scalars travel on the wire beside its registers. | Builds `HllSketch::from_raw(HllVariant::Hip, 12, regs, 1.5, 2.5, 3.0)` over the shared P12 pattern, verifies `to_msgpack()` equals `asapv1_golden/hll_hip_p12.hex`, that `HyperLogLogHIPP12::deserialize_from_bytes` re-serializes the fixture identically, and that `HllSketch::from_msgpack` reports the registers, `HllVariant::Hip`, `hip_kxq0` `1.5`, `hip_kxq1` `2.5` and `hip_est` `3.0`. | -| `kll_f64_k200_matches_golden` | A deterministic f64 KLL serializes to its golden bytes, which round-trip unchanged. | Verifies `KLL::::init_kll_with_seed(200, 42)` fed `1..=50` (below the level-0 capacity, so no compaction fires) serializes to `asapv1_golden/kll_f64_k200.hex`, and that decoding the fixture re-serializes to the same bytes and reports `quantile(0.0)` `1.0` and `quantile(1.0)` `50.0`. | -| `kll_i64_k200_matches_golden` | A deterministic i64 KLL serializes to its golden bytes, which round-trip unchanged. | Verifies `KLL::::init_kll_with_seed(200, 42)` fed `1..=50` serializes to `asapv1_golden/kll_i64_k200.hex`, and that decoding the fixture re-serializes to the same bytes and reports `quantile(0.0)` `1.0` and `quantile(1.0)` `50.0`. | +| `kll_dynamic_f64_k200_matches_golden` | A deterministic f64 `KLLDynamic` serializes to its golden bytes under kind_id `0x06 0x01`, and the golden decodes to that state. | Verifies `KLLDynamic::::init_kll_with_seed(200, 42)` fed `[2.5, -1.0, 0.0, 1e300, -0.125, 42.0, 3.0e-5]` in order (no compaction) serializes to `asapv1_golden/kll_dynamic_f64_k200.hex`; that the fixture's own payload, read from the envelope by hand, is kind_id `0x06 0x01`, `levels` `[0, 7]`, those items bit for bit in input order, and coin `(42, 0, 0)`; that `deserialize_from_bytes` reports `count()` `7`, `quantile(0.0)` `-1.0`, `quantile(1.0)` `1e300` and the source's `rank` at every input, and re-serializes identically; and that the compact `KLL` decoder rejects the fixture while the `KLLDynamic` decoder rejects `kll_f64_k200.hex`. | +| `kll_dynamic_i64_k200_matches_golden` | A deterministic i64 `KLLDynamic` whose items cross every msgpack integer width serializes to its golden bytes, and the golden decodes to that state. | Verifies `KLLDynamic::::init_kll_with_seed(200, 42)` fed 21 values from `0` through `i64::MAX` and `i64::MIN` — positive fixint / uint8 / uint16 / uint32 / uint64 and negative fixint / int8 / int16 / int32 / int64 — serializes to `asapv1_golden/kll_dynamic_i64_k200.hex`; that the hand-read payload is kind_id `0x06 0x01`, `levels` `[0, 21]`, those items in input order, and coin `(42, 0, 0)`; that the decode reports `count()` `21`, `quantile(0.0)` `i64::MIN`, `quantile(1.0)` `i64::MAX` and the source's `rank` at every input, and re-serializes identically; and that each KLL kind_id is rejected by the other variant's decoder. | +| `kll_f64_k200_matches_golden` | A deterministic f64 KLL serializes to its golden bytes, and the golden decodes to that state. | Verifies `KLL::::init_kll_with_seed(200, 42)` fed `1..=50` (below the level-0 capacity, so no compaction fires) serializes to `asapv1_golden/kll_f64_k200.hex`; that the hand-read payload is kind_id `0x06 0x00`, `levels` `[0, 50]`, items `1.0..=50.0` and coin `(42, 0, 0)`; and that decoding the fixture gives the source's `wire_levels()`, `wire_items()` and `wire_coin()`, `count()` `50`, `quantile(0.0)` `1.0` and `quantile(1.0)` `50.0`, and re-serializes to the same bytes. | +| `kll_i64_k200_matches_golden` | A deterministic i64 KLL serializes to its golden bytes, and the golden decodes to that state. | Verifies `KLL::::init_kll_with_seed(200, 42)` fed `1..=50` serializes to `asapv1_golden/kll_i64_k200.hex`; that the hand-read payload is kind_id `0x06 0x00`, `levels` `[0, 50]`, items `1..=50` and coin `(42, 0, 0)`; and that decoding the fixture gives the source's `wire_levels()`, `wire_items()` and `wire_coin()`, `count()` `50`, `quantile(0.0)` `1.0` and `quantile(1.0)` `50.0`, and re-serializes to the same bytes. | ### MessagePack Envelope Compatibility diff --git a/proto/sketchlib.proto b/proto/sketchlib.proto index 6c618327..702541b9 100644 --- a/proto/sketchlib.proto +++ b/proto/sketchlib.proto @@ -7,7 +7,6 @@ // countminsketch/countminsketch.proto — CountMinState // countsketch/countsketch.proto — CountSketchState, TopKState, HeapEntry // hll/hll.proto — HyperLogLogState, HLLVariant -// kll/kll.proto — KLLState, CoinState // ddsketch/ddsketch.proto — DDSketchState // univmon/univmon.proto — UnivMonState, UnivMonLayer // hydra/hydra.proto — HydraState, HydraCounterType, HydraCell diff --git a/src/sketches/kll/wire.rs b/src/sketches/kll/wire.rs index 30deba8a..af6c8b78 100644 --- a/src/sketches/kll/wire.rs +++ b/src/sketches/kll/wire.rs @@ -21,7 +21,8 @@ //! //! The payload's `levels` / `items` use the **top-most-level-first** layout, //! byte-for-byte matching `sketchlib-go` (index `i` in `levels` maps to -//! compactor level `num_levels - 1 - i`; level 0's run is in input order). For the compact KLL this is exactly what [`KLL::wire_levels`] / +//! compactor level `num_levels - 1 - i`; level 0's run is in input order). For +//! the compact KLL this is exactly what [`KLL::wire_levels`] / //! [`KLL::wire_items`] emit (they reverse the leftward-grown L0 buffer back to //! input order); decode inverts that mapping. diff --git a/src/sketches/kll_dynamic/wire.rs b/src/sketches/kll_dynamic/wire.rs index 207dd6a2..b5b56b4f 100644 --- a/src/sketches/kll_dynamic/wire.rs +++ b/src/sketches/kll_dynamic/wire.rs @@ -37,8 +37,8 @@ where /// top-most-level-first / input-order-L0 layout, so `levels` and `items` are /// copied straight out. pub fn serialize_to_bytes(&self) -> Result, RmpEncodeError> { - // `KLLDynamic` has no reproducible-seed concept, so `seed` is always - // `None` and the metadata key is omitted. + // `KLLDynamic` never emits the `seed` metadata key, even when built + // with one. let metadata = rmp_serde::to_vec_named(&kll_metadata::(self.k as u32, self.m as u32, None))?; let (state, bit_cache, remaining_bits) = self.co.to_wire(); diff --git a/tests/asapv1_golden.rs b/tests/asapv1_golden.rs index 186d0993..fdc4bd1a 100644 --- a/tests/asapv1_golden.rs +++ b/tests/asapv1_golden.rs @@ -10,8 +10,9 @@ use asap_sketchlib::{ Classic, Count, CountMin, ErtlMLE, FastPath, HllSketch, HllVariant, HyperLogLogHIPP12, - HyperLogLogP12, KLL, MessagePackCodec, RegularPath, Vector2D, + HyperLogLogP12, KLL, KLLDynamic, MessagePackCodec, RegularPath, Vector2D, }; +use serde::de::DeserializeOwned; fn decode_hex(s: &str) -> Vec { let s = s.trim(); @@ -32,6 +33,8 @@ const GOLDEN_CS_FAST: &str = include_str!("../asapv1_golden/cs_i64_fast_2x4.hex" const GOLDEN_CS_I32: &str = include_str!("../asapv1_golden/cs_i32_regular_2x4.hex"); const GOLDEN_KLL_F64: &str = include_str!("../asapv1_golden/kll_f64_k200.hex"); const GOLDEN_KLL_I64: &str = include_str!("../asapv1_golden/kll_i64_k200.hex"); +const GOLDEN_KLL_DYN_F64: &str = include_str!("../asapv1_golden/kll_dynamic_f64_k200.hex"); +const GOLDEN_KLL_DYN_I64: &str = include_str!("../asapv1_golden/kll_dynamic_i64_k200.hex"); /// The known P12 register pattern shared by all three HLL fixtures. fn p12_registers() -> Vec { @@ -243,11 +246,33 @@ fn cs_i32_regular_2x4_matches_golden() { } // --------------------------------------------------------------------------- -// KLL: build known state (k=200, seed 42, integers 1..=50 — below the level-0 -// capacity, so no compaction fires and the retained set is deterministic) -> -// serialize == golden, and golden round-trips. +// KLL: build known state (k=200, seed 42, inputs below the level-0 capacity, so +// no compaction fires and the retained set is deterministic) -> serialize == +// golden; golden decodes to that state and re-encodes byte-identically. // --------------------------------------------------------------------------- +/// The coin of a sketch seeded with 42 that has not compacted. +const KLL_COIN_SEED_42: (u64, u64, u32) = (42, 0, 0); + +/// A KLL payload `[levels, items, coin]`, read straight from the envelope. +type KllPayloadView = (Vec, Vec, (u64, u64, u32)); + +/// Splits an ASAPv1 envelope by hand and decodes its KLL payload. +fn kll_envelope(bytes: &[u8]) -> (Vec, KllPayloadView) { + assert_eq!(&bytes[..7], b"ASAPv1\x01", "magic and version"); + let kind_len = bytes[7] as usize; + let kind_id = bytes[8..8 + kind_len].to_vec(); + let at = 8 + kind_len; + let meta_len = u32::from_be_bytes(bytes[at..at + 4].try_into().unwrap()) as usize; + let payload_len = u32::from_be_bytes(bytes[at + 4..at + 8].try_into().unwrap()) as usize; + let payload = &bytes[at + 8 + meta_len..]; + assert_eq!(payload.len(), payload_len, "payload length prefix"); + ( + kind_id, + rmp_serde::from_slice(payload).expect("KLL payload"), + ) +} + /// A k=200 KLL over `1..=50` with a fixed compaction seed: fully deterministic. fn kll_1to50 + asap_sketchlib::NumericalValue>(seed: u64) -> KLL { let mut sketch = KLL::::init_kll_with_seed(200, seed); @@ -265,11 +290,20 @@ fn kll_f64_k200_matches_golden() { let got = sketch.serialize_to_bytes().expect("serialize"); assert_eq!(got, want, "KLL f64 bytes diverge from golden"); - // Golden round-trips: decode, and re-encode is byte-identical. + let (kind_id, (levels, items, coin)) = kll_envelope::(&want); + assert_eq!(kind_id, [0x06, 0x00]); + assert_eq!(levels, [0, 50]); + assert_eq!(items, (1..=50).map(f64::from).collect::>()); + assert_eq!(coin, KLL_COIN_SEED_42); + let decoded = KLL::::deserialize_from_bytes(&want).expect("decode"); - assert_eq!(decoded.serialize_to_bytes().expect("re-serialize"), want); + assert_eq!(decoded.wire_levels(), sketch.wire_levels()); + assert_eq!(decoded.wire_items(), sketch.wire_items()); + assert_eq!(decoded.wire_coin(), sketch.wire_coin()); + assert_eq!(decoded.count(), 50); assert_eq!(decoded.quantile(0.0), 1.0); assert_eq!(decoded.quantile(1.0), 50.0); + assert_eq!(decoded.serialize_to_bytes().expect("re-serialize"), want); } #[test] @@ -280,8 +314,117 @@ fn kll_i64_k200_matches_golden() { let got = sketch.serialize_to_bytes().expect("serialize"); assert_eq!(got, want, "KLL i64 bytes diverge from golden"); + let (kind_id, (levels, items, coin)) = kll_envelope::(&want); + assert_eq!(kind_id, [0x06, 0x00]); + assert_eq!(levels, [0, 50]); + assert_eq!(items, (1..=50).collect::>()); + assert_eq!(coin, KLL_COIN_SEED_42); + let decoded = KLL::::deserialize_from_bytes(&want).expect("decode"); - assert_eq!(decoded.serialize_to_bytes().expect("re-serialize"), want); + assert_eq!(decoded.wire_levels(), sketch.wire_levels()); + assert_eq!(decoded.wire_items(), sketch.wire_items()); + assert_eq!(decoded.wire_coin(), sketch.wire_coin()); + assert_eq!(decoded.count(), 50); assert_eq!(decoded.quantile(0.0), 1.0); assert_eq!(decoded.quantile(1.0), 50.0); + assert_eq!(decoded.serialize_to_bytes().expect("re-serialize"), want); +} + +/// Negative, zero, fractional and extreme f64 samples, in insertion order. +const KLL_DYN_F64_VALS: [f64; 7] = [2.5, -1.0, 0.0, 1e300, -0.125, 42.0, 3.0e-5]; + +/// i64 samples crossing every msgpack integer width: positive fixint / uint8 / +/// uint16 / uint32 / uint64 and negative fixint / int8 / int16 / int32 / int64. +const KLL_DYN_I64_VALS: [i64; 21] = [ + 0, + 1, + -1, + 127, + -32, + 128, + -33, + 255, + -128, + 256, + -129, + 65535, + -32768, + 65536, + -32769, + 4294967295, + -2147483648, + 4294967296, + -2147483649, + i64::MAX, + i64::MIN, +]; + +fn kll_dynamic_of(values: &[T]) -> KLLDynamic { + let mut sketch = KLLDynamic::::init_kll_with_seed(200, 42); + for v in values { + sketch.update(v); + } + sketch +} + +#[test] +fn kll_dynamic_f64_k200_matches_golden() { + let want = decode_hex(GOLDEN_KLL_DYN_F64); + + let sketch = kll_dynamic_of(&KLL_DYN_F64_VALS); + let got = sketch.serialize_to_bytes().expect("serialize"); + assert_eq!(got, want, "KLLDynamic f64 bytes diverge from golden"); + + let (kind_id, (levels, items, coin)) = kll_envelope::(&want); + assert_eq!(kind_id, [0x06, 0x01]); + assert_eq!(levels, [0, KLL_DYN_F64_VALS.len() as u32]); + assert_eq!( + items.iter().map(|v| v.to_bits()).collect::>(), + KLL_DYN_F64_VALS + .iter() + .map(|v| v.to_bits()) + .collect::>() + ); + assert_eq!(coin, KLL_COIN_SEED_42); + + // Encode copies the buffer, levels and coin verbatim, so equal bytes mean + // equal state. + let decoded = KLLDynamic::::deserialize_from_bytes(&want).expect("decode"); + assert_eq!(decoded.count(), KLL_DYN_F64_VALS.len()); + assert_eq!(decoded.quantile(0.0), -1.0); + assert_eq!(decoded.quantile(1.0), 1e300); + for v in KLL_DYN_F64_VALS { + assert_eq!(decoded.rank(v), sketch.rank(v), "rank({v})"); + } + assert_eq!(decoded.serialize_to_bytes().expect("re-serialize"), want); + + assert!(KLL::::deserialize_from_bytes(&want).is_err()); + assert!(KLLDynamic::::deserialize_from_bytes(&decode_hex(GOLDEN_KLL_F64)).is_err()); +} + +#[test] +fn kll_dynamic_i64_k200_matches_golden() { + let want = decode_hex(GOLDEN_KLL_DYN_I64); + + let sketch = kll_dynamic_of(&KLL_DYN_I64_VALS); + let got = sketch.serialize_to_bytes().expect("serialize"); + assert_eq!(got, want, "KLLDynamic i64 bytes diverge from golden"); + + let (kind_id, (levels, items, coin)) = kll_envelope::(&want); + assert_eq!(kind_id, [0x06, 0x01]); + assert_eq!(levels, [0, KLL_DYN_I64_VALS.len() as u32]); + assert_eq!(items, KLL_DYN_I64_VALS); + assert_eq!(coin, KLL_COIN_SEED_42); + + let decoded = KLLDynamic::::deserialize_from_bytes(&want).expect("decode"); + assert_eq!(decoded.count(), KLL_DYN_I64_VALS.len()); + assert_eq!(decoded.quantile(0.0), i64::MIN as f64); + assert_eq!(decoded.quantile(1.0), i64::MAX as f64); + for v in KLL_DYN_I64_VALS { + assert_eq!(decoded.rank(v as f64), sketch.rank(v as f64), "rank({v})"); + } + assert_eq!(decoded.serialize_to_bytes().expect("re-serialize"), want); + + assert!(KLL::::deserialize_from_bytes(&want).is_err()); + assert!(KLLDynamic::::deserialize_from_bytes(&decode_hex(GOLDEN_KLL_I64)).is_err()); } From 9c84ed535a7d4b764863b9ecc5799a262fe23ee1 Mon Sep 17 00:00:00 2001 From: GordonYuanyc Date: Fri, 2 Oct 2026 00:56:28 -0600 Subject: [PATCH 11/36] test(cms): pin the f64 fast-path cells against sketchlib-go's layout Adds cm_f64_fast_path_places_keys_in_go_cells: the 40 non-zero cells of sketchlib-go's 4x2048 Count-Min after 50 inserts of flow-{i % 10}, listed inline, reproduced by CountMin, FastPath, DefaultXxHasher>. Records in the CHANGELOG that Count-Min deltas, hh_keys and the envelope sample_p have no ASAPv1 carrier, and drops stale CountMin mentions from the sample_p proto doc and xtest_consumer. Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 3 +- docs/tests.md | 1 + proto/sketchlib.proto | 2 +- src/proto/generated/sketchlib.v1.rs | 2 +- src/sketches/countminsketch.rs | 61 +++++++++++++++++++++++++++++ tests/sketches_go_parity_probe.rs | 5 ++- tests/xtest_consumer.rs | 2 +- 7 files changed, 70 insertions(+), 6 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 8934690b..06be37b1 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -103,7 +103,8 @@ signals a backwards-compatible change. `message_pack_format::native::countminsketch`; and the `src/sketches/testdata/cms_envelope_golden.hex` proto golden. The `CountMinState` proto message stays as the nested Count-Min state of - `HydraCell` and `ElasticState`. + `HydraCell` and `ElasticState`. Count-Min deltas (including `hh_keys`) and + the envelope `sample_p` for Count-Min have no ASAPv1 carrier. ### Fixed diff --git a/docs/tests.md b/docs/tests.md index da587df6..6d9b57f9 100644 --- a/docs/tests.md +++ b/docs/tests.md @@ -40,6 +40,7 @@ Wire tests: [`src/sketches/countminsketch/wire.rs`](../src/sketches/countminsket | `countmin_apply_delta_increments_parent_counter` | Apply delta increments parent counter. | Constructs a `CmDelta{row=1, col=5, value=CM_PROMASK}`, applies it to a `3x64` parent CMS, and verifies the target counter at `(1,5)` equals `CM_PROMASK`. | | `cm_regular_path_correctness` | Regular-path hashing, counters, and estimates are exact on a deterministic stream. | Recomputes expected counter indices for `I32(0..9)` using per-row hashing, asserts exact full-matrix equality after one pass, doubled counters after second pass, and estimate `== 2` for each inserted key. | | `cm_fast_path_correctness` | Fast-path counter placement matches bit-sliced hash mapping. | Recomputes expected fast-path indices for `I32(0..9)` from one hash plus row bit-slices/mask bits and asserts exact full-matrix equality. | +| `cm_f64_fast_path_places_keys_in_go_cells` | An `f64` fast-path sketch puts string keys in the cells sketchlib-go's Count-Min does. | Fills `CountMin, FastPath, DefaultXxHasher>` at `4x2048` with 50 `insert_many(String("flow-{i % 10}"), 1.0)` calls and verifies the matrix holds `5.0` in exactly the 40 `(row, col)` cells sketchlib-go's `CountMinSketch` fills for the same inserts, ten per row, and `0.0` everywhere else. | | `count_min_round_trip_serialization` | Serialization round trip preserves full sketch state. | Serializes/deserializes a populated `3x8` regular-path sketch and verifies dimensions plus the full counter array are unchanged. | | `count_min_custom_hasher_profile_round_trips_and_is_self_describing` | The metadata describes the hasher that built the sketch rather than a hardcoded profile. | For a `3x8` sketch over a hasher declaring its own `HashProfile`, verifies the counter array round-trips, the bytes differ from the standard-profile sketch's over the same inserts, and a standard-profile decode rejects them. | | `count_min_f64_and_mode_in_metadata_round_trip` | The `f64` counter type and the `mode` both travel in the metadata. | Round-trips a `4x16` fast-path `Vector2D` sketch fed fractional weights, verifies the counter array is preserved, then that the same bytes fail to decode as an `i64` regular-path sketch. | diff --git a/proto/sketchlib.proto b/proto/sketchlib.proto index 2b5ad278..6389d542 100644 --- a/proto/sketchlib.proto +++ b/proto/sketchlib.proto @@ -85,7 +85,7 @@ message SketchEnvelope { // geometric skip-sampling, statistically identical to per-update Bernoulli(p)) // and stores the RAW SAMPLED sketch state — never the rescaled state. The // consumer applies the `× 1/sample_p` rescale at QUERY time on the count-like - // estimators (HLL cardinality, CountMin / CountSketch frequency, SUM / COUNT); + // estimators (HLL cardinality, CountSketch frequency, SUM / COUNT); // quantile estimators (KLL, DDSketch) are scale-invariant under uniform // sampling so they carry `sample_p` but need no rescale. // diff --git a/src/proto/generated/sketchlib.v1.rs b/src/proto/generated/sketchlib.v1.rs index c10445dc..84c81225 100644 --- a/src/proto/generated/sketchlib.v1.rs +++ b/src/proto/generated/sketchlib.v1.rs @@ -802,7 +802,7 @@ pub struct SketchEnvelope { /// geometric skip-sampling, statistically identical to per-update Bernoulli(p)) /// and stores the RAW SAMPLED sketch state — never the rescaled state. The /// consumer applies the `× 1/sample_p` rescale at QUERY time on the count-like - /// estimators (HLL cardinality, CountMin / CountSketch frequency, SUM / COUNT); + /// estimators (HLL cardinality, CountSketch frequency, SUM / COUNT); /// quantile estimators (KLL, DDSketch) are scale-invariant under uniform /// sampling so they carry `sample_p` but need no rescale. /// diff --git a/src/sketches/countminsketch.rs b/src/sketches/countminsketch.rs index eae02952..cc2bc2a2 100644 --- a/src/sketches/countminsketch.rs +++ b/src/sketches/countminsketch.rs @@ -880,4 +880,65 @@ mod tests { assert_eq!(storage.as_slice(), expected_once.as_slice()); } + + /// The non-zero cells of sketchlib-go's `CountMinSketch` at `4x2048` after + /// 50 inserts of `flow-{i % 10}`: every listed cell holds `5`. + const GO_FLOW_CELLS: [(usize, usize); 40] = [ + (0, 193), + (0, 306), + (0, 1078), + (0, 1079), + (0, 1160), + (0, 1212), + (0, 1730), + (0, 1734), + (0, 1937), + (0, 1979), + (1, 159), + (1, 525), + (1, 550), + (1, 576), + (1, 726), + (1, 901), + (1, 1193), + (1, 1426), + (1, 1808), + (1, 1863), + (2, 429), + (2, 515), + (2, 700), + (2, 896), + (2, 962), + (2, 1356), + (2, 1410), + (2, 1621), + (2, 1982), + (2, 1997), + (3, 226), + (3, 286), + (3, 333), + (3, 359), + (3, 935), + (3, 1208), + (3, 1312), + (3, 1602), + (3, 1774), + (3, 2046), + ]; + + #[test] + fn cm_f64_fast_path_places_keys_in_go_cells() { + let (rows, cols) = (4usize, 2048usize); + let mut sk: CountMin, FastPath, DefaultXxHasher> = + CountMin::with_dimensions(rows, cols); + for i in 0..50u64 { + sk.insert_many(&DataInput::String(format!("flow-{}", i % 10)), 1.0); + } + + let mut want = vec![0.0f64; rows * cols]; + for (r, c) in GO_FLOW_CELLS { + want[r * cols + c] = 5.0; + } + assert_eq!(sk.as_storage().as_slice(), want.as_slice()); + } } diff --git a/tests/sketches_go_parity_probe.rs b/tests/sketches_go_parity_probe.rs index 98f78683..76fdf1b0 100644 --- a/tests/sketches_go_parity_probe.rs +++ b/tests/sketches_go_parity_probe.rs @@ -4,8 +4,9 @@ //! //! Approach: reuse the exact same Go-golden envelopes that the //! `message_pack_format::portable` parity tests use, but build the -//! matrix via `sketches::Count` with `FastPath` and `DefaultXxHasher`. Identical bytes confirm the shared FastPath -//! math is sufficient — no hashspec bypass needed. +//! matrix via `sketches::Count` with `FastPath` and `DefaultXxHasher`. +//! Identical bytes confirm the shared FastPath math is sufficient — no +//! hashspec bypass needed. use asap_sketchlib::common::DataInput; use asap_sketchlib::common::hash::CANONICAL_HASH_SEED; diff --git a/tests/xtest_consumer.rs b/tests/xtest_consumer.rs index d7909ea5..c0721a79 100644 --- a/tests/xtest_consumer.rs +++ b/tests/xtest_consumer.rs @@ -675,7 +675,7 @@ fn count_sketch_query_float(state: &CountSketchState, hash: u64) -> f64 { let bits_per_row = col_bits(cols); let mask = (cols as u64) - 1; // Read whichever counter field the producer populated (int vs float), - // discriminated by counter_type — same as CountMin. + // discriminated by counter_type. let counts: Vec = if !state.counts_float.is_empty() { state.counts_float.clone() } else { From d9744b9344529c5de4298173db4c58a1154afe40 Mon Sep 17 00:00:00 2001 From: GordonYuanyc Date: Fri, 2 Oct 2026 00:57:02 -0600 Subject: [PATCH 12/36] refactor(hydra)!: drop the proto and portable formats Hydra serializes as ASAPv1 only. Remove the portable HydraKllSketch and HydraKllSketchWire, hydra/hydra.proto (HydraState, HydraCell, HydraCounterType) and SketchEnvelope's hydra field (16, now reserved), with the tests, example section and docs that used them. The two portable property laws give way to a KLL-counter ASAPv1 round-trip law on the native Hydra. Add ASAPv1 golden fixtures for all five Hydra counter variants (0x07 0x00-0x07 0x04), bumping asapv1_golden. Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 17 + asapv1_golden | 2 +- docs/asapv1_wire_format.md | 6 +- docs/features.md | 2 +- docs/message_pack_format.md | 5 +- docs/tests.md | 27 +- examples/accuracy_probe.rs | 26 -- proto/hydra/hydra.proto | 80 ----- proto/sketchlib.proto | 5 +- src/lib.rs | 3 +- src/message_pack_format/portable/hydra_kll.rs | 292 ------------------ src/message_pack_format/portable/kll.rs | 7 +- src/message_pack_format/portable/mod.rs | 1 - src/proto/generated/sketchlib.v1.rs | 106 +------ tests/PBT_COVERAGE.md | 10 +- tests/asapv1_golden.rs | 231 +++++++++++++- tests/e2e/quantiles.rs | 61 ---- tests/msgpack_compat.rs | 46 +-- tests/pbt/hydra.rs | 166 ++++------ tests/xtest_consumer.rs | 132 -------- tools/gen-proto/src/main.rs | 1 - 21 files changed, 340 insertions(+), 886 deletions(-) delete mode 100644 proto/hydra/hydra.proto delete mode 100644 src/message_pack_format/portable/hydra_kll.rs diff --git a/CHANGELOG.md b/CHANGELOG.md index 80d85f98..7a5d9c9d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -66,6 +66,10 @@ signals a backwards-compatible change. worker's seed and the aggregator's from one seed. The unseeded constructors draw from the thread generator. The seed is not serialized, and the ASAPv1 payload is unchanged. +- **ASAPv1 golden fixtures for Hydra's five counter variants** (`0x07 0x00` + KLL, `0x07 0x01` Count-Min, `0x07 0x02` Count Sketch, `0x07 0x03` HLL, + `0x07 0x04` UnivMon), each a small grid whose cell states are set directly, + checked by `tests/asapv1_golden.rs`. ### Changed @@ -88,6 +92,19 @@ signals a backwards-compatible change. as an empty window like any other, and that window takes a place in `max_windows`. +### Removed + +- **Hydra's proto and portable formats.** `Hydra` serializes as ASAPv1 only + (`serialize_to_bytes` / `deserialize_from_bytes`, kinds `0x07 0x00`-`0x07 + 0x04`). Removed: the portable `HydraKllSketch` and `HydraKllSketchWire` + (module `message_pack_format::portable::hydra_kll`, and the crate-root + `HydraKllSketch` re-export), with `HydraKllSketch`'s `MessagePackCodec` impl, + `with_seed`, `merge_refs` and `aggregate_hydrakll`; the proto file + `hydra/hydra.proto` with `HydraState`, `HydraCell` and `HydraCounterType` + (and their generated `proto::sketchlib` types); and `SketchEnvelope`'s + `hydra` field (16, now reserved). A matrix of KLLs keyed by string is + `Hydra` with a one-column schema and `HydraCounter::KLL`. + ### Fixed - `KLLDynamic` panicked on every quantile query once a NaN had entered the diff --git a/asapv1_golden b/asapv1_golden index 8287a4ae..35d7647d 160000 --- a/asapv1_golden +++ b/asapv1_golden @@ -1 +1 @@ -Subproject commit 8287a4ae04e6cfbfb3ed33dafbf1dac77d8f7fc1 +Subproject commit 35d7647de530ccca3382a9a89050d267ba288060 diff --git a/docs/asapv1_wire_format.md b/docs/asapv1_wire_format.md index fd14e70a..ab9b7433 100644 --- a/docs/asapv1_wire_format.md +++ b/docs/asapv1_wire_format.md @@ -707,8 +707,6 @@ A Hydra is a `rows x cols` grid of counters over a fixed set of named key column **Counters are inlined, not nested.** A cell's raw state goes straight into Hydra's positional array in the shape that counter's own section fixes; no cell carries an envelope, a magic or a metadata map of its own. -**Not this: `portable/hydra_kll.rs`.** `HydraKllSketch` is a legacy Go-interop type, not `0x07 0x00`, and it is the counter-example this rule exists to rule out: every cell of its matrix carries a **full ASAPv1 KLL envelope** — magic, version, kind_id, both length prefixes and a complete metadata map — plus `k` repeated once per cell beside it, in a msgpack shape of its own with no ASAPv1 envelope around the whole. That is exactly what Q-NEST forbids, and `0x07 0x00` does none of it: one envelope for the grid, `counter_k` / `counter_m` / `counter_item_type` carried once in its metadata, and each cell just the `[levels, items, coin]` array §3.3 fixes. - **`0x07 0x01` Count-Min counter, `0x07 0x02` Count Sketch counter** — the fixed-size matrix counters tile one array: | Pos | Field | Type | Notes | @@ -1341,14 +1339,14 @@ Good direction (more compact, higher fidelity, less Rust-internal duplication), 2. **Golden byte-vector fixtures** in one shared repo, [`sketchlib-golden-bytes`](https://github.com/ProjectASAP/sketchlib-golden-bytes), which each implementation mounts at `asapv1_golden/`; both languages decode and re-encode them byte-identically. These replace the `portable`-as-oracle round-trip test. 3. **This registry**, mirrored, never independently allocated. -Fixtures exist for six `kind_id`s — HLL's three estimators, Count-Min, Count Sketch and compact KLL — and `sketchlib-go` mirrors those. Every other kind here has a payload and **no fixture**, so this document is its only contract; `asapv1_golden/README.md` lists the gap. `portable` carries its own Go goldens. +Fixtures exist for eleven `kind_id`s — HLL's three estimators, Count-Min, Count Sketch, compact KLL and Hydra's five counter variants — and `sketchlib-go` mirrors all but Hydra's. Every other kind here has a payload and **no fixture**, so this document is its only contract; `asapv1_golden/README.md` lists the gap. `portable` carries its own Go goldens. **Hash profile on the Go side.** Rust derives the hash spec from a generic `HashProfile` bound on the hasher type; Go has no generic hasher type, so there is nothing to derive from. On the Go side the profile is simply **written into** the metadata on encode and **read from** it on decode. Go MUST validate the profile it reads (same fail-closed intent as Rust): a sketch is only mergeable/queryable if its `hash_profile_id` + seeds match the profile Go is prepared to reproduce. -Sequencing: (2) covers six `kind_id`s, and the one `native bytes == portable bytes` test is HLL-only, since no other `portable` type emits ASAPv1 bytes. +Sequencing: (2) covers eleven `kind_id`s, and the one `native bytes == portable bytes` test is HLL-only, since no other `portable` type emits ASAPv1 bytes. Retire `portable` once the fixtures cover the rest. --- diff --git a/docs/features.md b/docs/features.md index 64f296a3..e095a1c8 100644 --- a/docs/features.md +++ b/docs/features.md @@ -112,7 +112,7 @@ MessagePack (`rmp-serde`) support. **serde support** means the type derives `Ser | EHSketchList | Yes | Yes | | ExponentialHistogram | No | Yes | -Protobuf (prost): `.proto` definitions exist for CountMin, Count, HLL, DDSketch, KLL, Elastic, Coco, Hydra, and UnivMon. Rust conversion code is in progress. +Protobuf (prost): `.proto` definitions exist for CountMin, Count, HLL, DDSketch, KLL, Elastic, Coco, and UnivMon. Rust conversion code is in progress. ### API Stability diff --git a/docs/message_pack_format.md b/docs/message_pack_format.md index 12a2b91c..8c3a6b05 100644 --- a/docs/message_pack_format.md +++ b/docs/message_pack_format.md @@ -169,8 +169,9 @@ decode → re-encode them byte-identically. The `kind_id` registry is mirrored verbatim with Go's `wire/asapmsgpack/magic_ids.go`, never independently allocated. -That proof covers the six `kind_id`s a fixture exists for — HLL's three -estimators, Count-Min, Count Sketch and compact KLL. Every other implemented +Fixtures exist for eleven `kind_id`s — HLL's three estimators, Count-Min, +Count Sketch, compact KLL and Hydra's five counter variants; `sketchlib-go` +checks all but Hydra's. Every other implemented kind has **no ASAPv1 golden and therefore no ASAPv1 drift guard**; [its README](https://github.com/ProjectASAP/sketchlib-golden-bytes#coverage) lists what is covered. The older `portable` path is guarded separately, by the `sketchlib-go` goldens diff --git a/docs/tests.md b/docs/tests.md index ccb1571c..66077092 100644 --- a/docs/tests.md +++ b/docs/tests.md @@ -1357,20 +1357,6 @@ Test file: [`src/message_pack_format/portable/hll.rs`](../src/message_pack_forma | `test_compute_delta_matches_go_golden_bytes` | The packed register delta bytes are pinned by a golden. | For a `Datafusion` precision-14 sketch fed the little-endian `f64` bytes of `1..=50`, verifies the delta against an empty sketch at threshold `0` equals a 132-byte inline golden in the packed `(index_delta, value)` form. | | `test_compute_subwindow_delta_matches_go_golden_bytes` | A sub-window delta's bytes are pinned by a golden and still reconstruct the window. | For a `Datafusion` precision-14 base fed `0..2000` as `f64` little-endian bytes and a current fed `2000..5000` on top, verifies the delta carries only the registers that grew and equals a 5,097-byte inline golden, then that applying it to the base reproduces the current register array. | -### Portable HydraKllSketch - -Test file: [`src/message_pack_format/portable/hydra_kll.rs`](../src/message_pack_format/portable/hydra_kll.rs) - -| test_name | test_description | what_is_tested | -| --- | --- | --- | -| `test_creation` | A fresh matrix-of-KLL has the requested shape. | For `HydraKllSketch::new(2, 3, 200)`, verifies `rows` is `2`, `cols` is `3`, and the cell matrix is 2 rows of 3. | -| `test_update_and_query` | A per-key quantile query answers after inserts. | For a `2x10` matrix at `k` `200` fed `"key1"` at `5.0` and `10.0`, verifies `quantile("key1", 0.5)` is at or above `0.0`; the returned value itself is not pinned. | -| `test_merge` | Merge keeps the matrix geometry. | Merges a `2x5` matrix at `k` `200` fed `"key1"` with `6.0..=10.0` into one fed `1.0..=5.0` and verifies only that `rows` stays `2` and `cols` stays `5`; no quantile is checked. | -| `test_merge_dimension_mismatch` | Mismatched row counts are refused. | Verifies merging a `3x5` matrix into a `2x5` one fails. | -| `test_msgpack_round_trip` | The round trip preserves the matrix geometry. | Round-trips a `2x3` matrix at `k` `200` fed `"key1"` at `5.0` and `"key2"` at `10.0` and verifies the decode reports `rows` `2` and `cols` `3`; the per-cell KLL state is not checked. | -| `test_aggregate_hydrakll` | `aggregate_hydrakll` builds a decodable matrix from parallel key and value arrays. | For `aggregate_hydrakll(2, 5, 200, ["a", "b", "a"], [1.0, 2.0, 3.0])`, verifies the returned bytes decode and report `rows` `2` and `cols` `5`; no quantile is checked. | -| `test_aggregate_hydrakll_empty` | An empty stream produces nothing. | Verifies `aggregate_hydrakll(2, 5, 200, &[], &[])` returns `None`. | - ### Portable KllSketch Test file: [`src/message_pack_format/portable/kll.rs`](../src/message_pack_format/portable/kll.rs) @@ -1731,14 +1717,13 @@ Test file: [`tests/e2e/quantiles.rs`](../tests/e2e/quantiles.rs) | `univmonq_occurrence_entropy_is_present_only_when_ordered_samples_are_configured` | Occurrence entropy is reported only when an ordered sample is configured. | Feeds the same 120_000-value `zipf(1.2)` stream over 4_096 keys (seed `0x0DDE_9201`) into two sketches, one with `ordered_samples: 1_024` and one with `ordered_samples: 0`, and asserts `estimate_entropy_occurrence()` is `None` for the unsampled one and `Some` for the sampled one, plus that `estimate_entropy()` stays finite. The returned occurrence entropy is only checked to be finite and non-negative; its value is never compared to the exact entropy, so accuracy is not pinned. | | `univmonq_universal_rank_is_exact_outside_the_observed_range_and_banded_inside_it` | `estimate_rank_universal` is exact outside the observed range and inside a wide band within it. | On 120_000 `zipf(1.2)` values over 2_048 keys (seed `0x0DDE_9301`, default config), asserts `estimate_rank_universal(min - 1.0) == Some(0)`, `(max) == Some(120_000)`, `(max + 1.0) == Some(120_000)`, and that an empty sketch returns `None`. Interior probes at the truth quantiles `0.25`, `0.5` and `0.75` are only required to have a normalized rank error of at most `0.25` (a quarter of the whole stream), recorded in a tally closed with `assert_none`. | -### Quantiles: Windowed and Per-Key KLL +### Quantiles: Windowed KLL Test file: [`tests/e2e/quantiles.rs`](../tests/e2e/quantiles.rs) | test_name | test_description | what_is_tested | | --- | --- | --- | | `tumbling_kll_windows_are_exact_and_answers_satisfy_the_rank_contract` | Window membership is exact arithmetic while the answers inside a window carry KLL's rank error. | Drives 4_000 uniform values over `[0, 1M)` (seed `3009`) into a `TumblingWindow` with window `400`, `16` retained, `k=200`, `m=8`, pool size `4`, and asserts `closed_count() == 9` by equality. Then 16 trials, each with its own compaction seed `kll_trial_seed(0x7717_0000 + t)`, reduce the three window views — `query_all` against the whole stream, `query_recent(1)` against the last 800 values, and `active_sketch()` against the last 400 — to one worst rank error over the q grid `[0.1, 0.25, 0.5, 0.75, 0.9]`, which must not exceed `eps(200) ~ 0.0165`; accepted by `assert_independent_binomial` at `p = 0.01`. Rotation is then checked by a band, not an equality: `query_all().count()` must drift from 4_000 by at most `eps(200)`, so a dropped window (12.5% of the stream) would fail but the retained mass is not pinned exactly. | -| `portable_hydra_kll_per_key_medians_satisfy_the_rank_characterization` | Each Hydra KLL cell answers its own key under the same rank characterization as a standalone KLL. | Builds `HydraKllSketch::with_seed(3, 256, 200, seed)` for 12 trials with prototype seeds `kll_trial_seed(0x5EED_0500 + t)`, feeding two keys of 4_000 values each in sorted order: `svc-a` from `\|normal(100, 5)\|` (seed `3010`) and `svc-b` from `\|normal(900, 45)\|` (seed `3011`). Because every cell is cloned from one seeded prototype the two keys are not independent, so each grid is reduced to its worst rank error across both keys over the q grid `[0.1, 0.25, 0.5, 0.75, 0.9]` — wider than the median the name mentions — checked against `eps(200) ~ 0.0165` and accepted by `assert_independent_binomial` at `p = 0.01` over the twelve prototype seeds. | ### Quantiles: Documented Input Matrix @@ -2129,6 +2114,11 @@ Test file: [`tests/asapv1_golden.rs`](../tests/asapv1_golden.rs) | `hll_classic_p12_matches_golden` | A known P12 Classic register pattern serializes to its golden bytes on both the portable and native paths. | Builds `HllSketch::from_raw(HllVariant::Regular, 12, regs, 0.0, 0.0, 0.0)` where `regs` is 4,096 zeros with `[0] = 1`, `[1] = 7`, `[100] = 42` and `[4095] = 3`, verifies `to_msgpack()` equals `asapv1_golden/hll_classic_p12.hex`, that `HyperLogLogP12::::deserialize_from_bytes` on the fixture recovers those registers and re-serializes identically, and that `HllSketch::from_msgpack` reports the same registers, `precision` `12` and `HllVariant::Regular`. | | `hll_ertl_mle_p12_matches_golden` | The Datafusion variant tag reaches the bytes for the same P12 register pattern. | Builds `HllSketch::from_raw(HllVariant::Datafusion, 12, regs, 0.0, 0.0, 0.0)` over the shared P12 pattern (`[0] = 1`, `[1] = 7`, `[100] = 42`, `[4095] = 3` in 4,096 registers), verifies `to_msgpack()` equals `asapv1_golden/hll_ertl_mle_p12.hex`, that `HyperLogLogP12::::deserialize_from_bytes` recovers the registers and re-serializes identically, and that `HllSketch::from_msgpack` reports `HllVariant::Datafusion`. | | `hll_hip_p12_matches_golden` | The HIP variant's three running scalars travel on the wire beside its registers. | Builds `HllSketch::from_raw(HllVariant::Hip, 12, regs, 1.5, 2.5, 3.0)` over the shared P12 pattern, verifies `to_msgpack()` equals `asapv1_golden/hll_hip_p12.hex`, that `HyperLogLogHIPP12::deserialize_from_bytes` re-serializes the fixture identically, and that `HllSketch::from_msgpack` reports the registers, `HllVariant::Hip`, `hip_kxq0` `1.5`, `hip_kxq1` `2.5` and `hip_est` `3.0`. | +| `hydra_cm_2x2_counter_2x2_matches_golden` | A Count-Min-counter Hydra grid serializes to its golden bytes under `0x07 0x01`, and the golden round-trips. | Builds a `2x2` `Hydra` over `["region", "service"]` whose cells are `CountMin::from_storage` over the four `2x2` `i32` matrices `[[0, 1], [127, 128]]`, `[[255, 256], [300, 65535]]`, `[[65536, 1_000_000], [i32::MAX, 0]]` and all zeros, verifies `serialize_to_bytes` equals `asapv1_golden/hydra_cm_2x2_counter_2x2.hex`, that `Hydra::deserialize_from_bytes` restores the grid shape, the schema and every cell's matrix, and that the decode re-serializes identically. | +| `hydra_cs_2x2_counter_2x2_matches_golden` | A Count-Sketch-counter Hydra grid with signed cells serializes to its golden bytes under `0x07 0x02`, and the golden round-trips. | The same check over `Count::from_storage` cells `[[0, -1], [127, -32]]`, `[[-33, 128], [-128, -129]]`, `[[-32768, 65536], [-32769, i32::MAX]]` and `[[i32::MIN, 1], [0, 0]]`, against `asapv1_golden/hydra_cs_2x2_counter_2x2.hex`. | +| `hydra_hll_1x2_p14_matches_golden` | An HLL-counter Hydra grid serializes to its golden bytes under `0x07 0x03`, and the golden round-trips. | Builds a `1x2` grid of P14 `HyperLogLog` cells whose registers are set through `insert_with_hash` with crafted hashes (and asserted): cell 0 `[0] = 1`, `[1] = 7`, `[100] = 42`, `[16383] = 3`; cell 1 `[0] = 2`, `[8192] = 51`. Verifies the bytes equal `asapv1_golden/hydra_hll_1x2_p14.hex`, the decode restores both register arrays, and re-serializes identically. | +| `hydra_kll_2x2_k200_matches_golden` | A KLL-counter Hydra grid serializes to its golden bytes under `0x07 0x00`, and the golden round-trips. | Builds a `2x2` grid of `KLL::init_kll_with_seed(200, seed)` cells with seeds `1..=4` holding `[1.0..=5.0]`, nothing, `[2.5, -1.0, 0.0, 1e300, -0.125]` and `[3.0e-5]`, verifies the bytes equal `asapv1_golden/hydra_kll_2x2_k200.hex`, that each decoded cell has the same `k`, `m`, levels, items and coin, and that the decode re-serializes identically. | +| `hydra_univmon_1x2_matches_golden` | A UnivMon-counter Hydra grid serializes to its golden bytes under `0x07 0x04`, and the golden round-trips. | Builds a `1x2` grid of `UnivMon::init_univmon(2, 1, 2, 2)` cells; cell 0 takes `CountDelta`s `+5` at layer 0 column 0, `-3` at layer 0 column 1 and `+2` at layer 1 column 1, heap entries `{7: 5, 300: 2}` on layer 0 and `{4294967296: 2}` on layer 1, `set_total_weight(7)` and layer 0 marked incomplete; cell 1 stays empty. Verifies the bytes equal `asapv1_golden/hydra_univmon_1x2.hex`, that each decoded cell re-encodes to the same UnivMon bytes, and that the decode re-serializes identically. | | `kll_f64_k200_matches_golden` | A deterministic f64 KLL serializes to its golden bytes, which round-trip unchanged. | Verifies `KLL::::init_kll_with_seed(200, 42)` fed `1..=50` (below the level-0 capacity, so no compaction fires) serializes to `asapv1_golden/kll_f64_k200.hex`, and that decoding the fixture re-serializes to the same bytes and reports `quantile(0.0)` `1.0` and `quantile(1.0)` `50.0`. | | `kll_i64_k200_matches_golden` | A deterministic i64 KLL serializes to its golden bytes, which round-trip unchanged. | Verifies `KLL::::init_kll_with_seed(200, 42)` fed `1..=50` serializes to `asapv1_golden/kll_i64_k200.hex`, and that decoding the fixture re-serializes to the same bytes and reports `quantile(0.0)` `1.0` and `quantile(1.0)` `50.0`. | @@ -2149,9 +2139,6 @@ Test file: [`tests/msgpack_compat.rs`](../tests/msgpack_compat.rs) | `delta_result_round_trip` | A `DeltaResult` survives a msgpack round trip with both of its sets. | Serializes a `DeltaResult` whose `added` holds `a` and whose `removed` holds `b`, then verifies the decode's `added` contains `a` and its `removed` contains `b`. | | `hll_sketch_decodes_go_bytes` | The test asserts nothing. | The test asserts nothing: the body is empty and the test is marked `#[ignore]` with the reason that no cross-language msgpack fixture exists in the repository. | | `hll_sketch_round_trip` | A portable `HllSketch` survives a msgpack round trip with its register count intact. | Updates `HllSketch::new(HllVariant::Regular, 8)` with `b"a"`, `b"b"` and `b"c"`, then verifies only that the decode's `registers` has the same length as the source's; the register values, the precision and the variant are not checked. | -| `hydra_kll_decodes_go_bytes` | The test asserts nothing. | The test asserts nothing: the body is empty and the test is marked `#[ignore]` with the reason that no cross-language msgpack fixture exists in the repository. | -| `hydra_kll_sketch_round_trip` | A portable `HydraKllSketch` survives a msgpack round trip with its grid shape intact. | Updates `HydraKllSketch::with_seed(2, 4, 200, 0x5EED_0900)` with `("a", 1.0)`, `("a", 2.0)` and `("b", 3.0)`, then verifies the decode reports `rows` `2` and `cols` `4`; the nested KLL payloads are not checked. | -| `hydra_kll_wire_shape` | The `HydraKllSketchWire` DTO round-trips through `rmp_serde` with its two levels of nesting intact. | Serializes a wire struct of `rows: 2`, `cols: 3` and a `2x3` grid of `KllSketchData { k: 200, sketch_bytes: vec![] }`, then verifies the `rmp_serde` decode reports `rows` `2`, `cols` `3`, an outer length of `2` and an inner length of `3`. | | `kll_sketch_decodes_go_bytes` | The test asserts nothing. | The test asserts nothing: the body is empty and the test is marked `#[ignore]` with the reason that no cross-language msgpack fixture exists in the repository. | | `kll_sketch_round_trip` | A portable `KllSketch` survives a msgpack round trip with its `k` and retained mass intact. | Feeds `KllSketch::with_seed(200, 0x5EED_0800)` the values `0..100` as `f64`, then verifies the decode reports `k` `200` and `count()` `100`; the retained items and the quantile answers are not checked. | | `set_aggregator_round_trip` | A `SetAggregator` survives a msgpack round trip with its members. | Updates a `SetAggregator::new()` with `web` and `api`, then verifies the decode holds two values, one of which is `web`. | @@ -2202,7 +2189,7 @@ Test file: [`tests/xtest_consumer.rs`](../tests/xtest_consumer.rs) | test_name | test_description | what_is_tested | | --- | --- | --- | -| `cross_language_proto` | Protobuf envelopes read from the cross-language fixture directory answer their point queries in band. | Asserts nothing unless fixtures are available: with no `XTEST_DIR` and no discoverable producer directory holding `tests/cross_language/xtest_producer_test.go`, it prints a skip line and returns. With fixtures it decodes nine `SketchEnvelope` files and records a pass or failure for each — `countmin.pb` min-frequency for `item:42` at least `101`, `kll.pb` p50 within 5% of `5000` and p99 within 5% of `9900`, `ddsketch.pb` the same two quantiles within 2%, `hll.pb` an ErtlMLE cardinality inside `[40_000, 65_000]`, `countsketch.pb` `cs:hot` at least `200`, `coco.pb` `coco:hot` at least `500`, `elastic.pb` `elephant` at least `900`, `univmon.pb` a cardinality inside `[1_000, 15_000]`, `hydra.pb` `hydra:42` at least `51` — using the hash seeds `0xcafe3553`, `0x6a09e667` and `0xbb67ae85`, and, when `countmin_sampled.pb` and `hll_sampled.pb` exist, requires `effective_sample_p` of `0.1` and a `rescale_count` by `1/p` within 5% of `100_000` and 6% of `200_000`. A single `assert!` at the end fails if any check failed; a missing or mistyped envelope panics on the spot. | +| `cross_language_proto` | Protobuf envelopes read from the cross-language fixture directory answer their point queries in band. | Asserts nothing unless fixtures are available: with no `XTEST_DIR` and no discoverable producer directory holding `tests/cross_language/xtest_producer_test.go`, it prints a skip line and returns. With fixtures it decodes nine `SketchEnvelope` files and records a pass or failure for each — `countmin.pb` min-frequency for `item:42` at least `101`, `kll.pb` p50 within 5% of `5000` and p99 within 5% of `9900`, `ddsketch.pb` the same two quantiles within 2%, `hll.pb` an ErtlMLE cardinality inside `[40_000, 65_000]`, `countsketch.pb` `cs:hot` at least `200`, `coco.pb` `coco:hot` at least `500`, `elastic.pb` `elephant` at least `900`, `univmon.pb` a cardinality inside `[1_000, 15_000]` — using the hash seeds `0xcafe3553` and `0x6a09e667`, and, when `countmin_sampled.pb` and `hll_sampled.pb` exist, requires `effective_sample_p` of `0.1` and a `rescale_count` by `1/p` within 5% of `100_000` and 6% of `200_000`. A single `assert!` at the end fails if any check failed; a missing or mistyped envelope panics on the spot. | ### HLL Custom Precision diff --git a/examples/accuracy_probe.rs b/examples/accuracy_probe.rs index 643bfc3e..8aa6cf93 100644 --- a/examples/accuracy_probe.rs +++ b/examples/accuracy_probe.rs @@ -10,7 +10,6 @@ use asap_sketchlib::common::input::{HydraCounter, HydraQuery}; use asap_sketchlib::message_pack_format::portable::countminsketch::CountMinSketch; use asap_sketchlib::message_pack_format::portable::ddsketch::DdSketch as PortableDds; use asap_sketchlib::message_pack_format::portable::hll::{HllSketch, HllVariant}; -use asap_sketchlib::message_pack_format::portable::hydra_kll::HydraKllSketch; use asap_sketchlib::message_pack_format::portable::kll::KllSketch as PortableKll; use asap_sketchlib::{ CMSHeap, CSHeap, Count as CoreCount, CountL2HH, CountMin, DDSketch, DataInput, Hydra, @@ -1074,30 +1073,5 @@ fn probe_portable_wire_types() { (med - t).abs() / t < 0.05, ); - // HydraKllSketch: per-key quantiles, median across rows. - let mut hk = HydraKllSketch::new(3, 256, 200); - let mut per_key: HashMap<&str, Vec> = HashMap::new(); - let mut rng2 = StdRng::seed_from_u64(56); - for key in ["svc-a", "svc-b"] { - let base = if key == "svc-a" { 100.0 } else { 900.0 }; - let vs: Vec = (0..2_000) - .map(|_| base + rng2.random::() * 50.0) - .collect(); - for v in &vs { - hk.update(key, *v); - } - per_key.insert(key, vs); - } - for (key, vs) in per_key { - let mut s = vs.clone(); - s.sort_by(|a, b| a.partial_cmp(b).unwrap()); - let t = s[s.len() / 2]; - let got = hk.quantile(key, 0.5); - p.check( - &format!("HydraKll median({key})"), - format!("expected ~{t:.1}, got {got:.1}"), - (got - t).abs() / t < 0.05, - ); - } p.finish("Portable wire types"); } diff --git a/proto/hydra/hydra.proto b/proto/hydra/hydra.proto deleted file mode 100644 index 116bb439..00000000 --- a/proto/hydra/hydra.proto +++ /dev/null @@ -1,80 +0,0 @@ -// hydra.proto — Portable wire format for Hydra multi-sketch framework state. - -syntax = "proto3"; - -package sketchlib.v1; - -import "countminsketch/countminsketch.proto"; -import "countsketch/countsketch.proto"; -import "hll/hll.proto"; -import "kll/kll.proto"; -import "univmon/univmon.proto"; - -option go_package = "github.com/ProjectASAP/sketchlib-go/proto/hydra;hydrapb"; - -// ============================================================================ -// Hydra Sketch -// ============================================================================ - -// HydraState is the portable state of a Hydra multi-sketch framework. -// -// Hydra maintains a grid of row_num child sketches (cells), each of the -// same type (counter_type). Items are routed to a specific row cell using a -// position hash derived from the item's subkey. An optional global "big -// counter" sketch aggregates the full stream. -// -// Current implementations set col_num = 1 (one sketch per row). The field is -// included for schema completeness and future extensions. -message HydraState { - // Number of sketch rows (D / number of subpopulations). - uint32 row_num = 1; - - // Number of sketch columns per row. Currently always 1. - uint32 col_num = 2; - - // The sketch type used for every cell. - HydraCounterType counter_type = 3; - - // Serialized cells in row-major order, length = row_num * col_num. - // Each cell's oneof field must match counter_type. - repeated HydraCell cells = 4; - - // Optional global sketch covering the entire stream. - // Absent when global counter is disabled (enable_global_counter = false). - HydraCell big_counter = 5; - - // Seed index used for subpopulation routing (Go: seedHydra, Rust: HYDRA_SEED). - // Both libraries default to 6. - uint32 seed_index = 6; - - // True when per-cell TopK tracking is enabled. - bool enable_topk = 7; - - // True when fanout subkey expansion is active. - bool fanout_subkeys = 8; - - reserved 9 to 15; -} - -// HydraCounterType identifies the sketch algorithm used inside Hydra cells. -enum HydraCounterType { - HYDRA_COUNTER_TYPE_UNSPECIFIED = 0; - HYDRA_COUNTER_TYPE_COUNT_MIN = 1; - HYDRA_COUNTER_TYPE_COUNT_SKETCH = 2; - HYDRA_COUNTER_TYPE_HLL = 3; - HYDRA_COUNTER_TYPE_KLL = 4; - HYDRA_COUNTER_TYPE_UNIVMON = 5; -} - -// HydraCell holds the serialized state of one Hydra grid cell. -// Exactly one field in the oneof must be set, and it must match the parent -// HydraState.counter_type. -message HydraCell { - oneof sketch { - CountMinState count_min = 1; - CountSketchState count_sketch = 2; - HyperLogLogState hll = 3; - KLLState kll = 4; - UnivMonState univmon = 5; - } -} diff --git a/proto/sketchlib.proto b/proto/sketchlib.proto index 0a6d8193..953834d1 100644 --- a/proto/sketchlib.proto +++ b/proto/sketchlib.proto @@ -10,7 +10,6 @@ // kll/kll.proto — KLLState, CoinState // ddsketch/ddsketch.proto — DDSketchState // univmon/univmon.proto — UnivMonState, UnivMonLayer -// hydra/hydra.proto — HydraState, HydraCounterType, HydraCell // // Design goals: // - Cross-language: Go (sketchlib-go) ↔ Rust (asap_sketchlib) round-trips. @@ -35,7 +34,6 @@ import "hll/hll.proto"; import "kll/kll.proto"; import "ddsketch/ddsketch.proto"; import "univmon/univmon.proto"; -import "hydra/hydra.proto"; import "cocosketch/cocosketch.proto"; import "elasticsketch/elasticsketch.proto"; @@ -72,11 +70,12 @@ message SketchEnvelope { KLLState kll = 13; DDSketchState ddsketch = 14; UnivMonState univmon = 15; - HydraState hydra = 16; CocoSketchState coco = 17; ElasticState elastic = 18; } + reserved 16; + reserved "hydra"; reserved 19 to 31; // Reserved for future sketch types. // Geometric / NitroSketch sampling probability for the producing sketch. diff --git a/src/lib.rs b/src/lib.rs index b062b8b9..7944644d 100644 --- a/src/lib.rs +++ b/src/lib.rs @@ -11,7 +11,7 @@ //! - [`message_pack_format`]: MessagePack/proto wire format shared with //! `sketchlib-go`. Owns the wire-DTO sketch types consumed by the ASAP //! query engine (`CountMinSketch`, `CountSketch`, `KllSketch`, `HllSketch`, -//! `DdSketch`, `HydraKllSketch`, `CountMinSketchWithHeap`, +//! `DdSketch`, `CountMinSketchWithHeap`, //! `SetAggregator`, `DeltaResult`). //! //! Most users can start with the crate-root re-exports such as [`DataInput`], @@ -50,7 +50,6 @@ pub use message_pack_format::portable::countsketch_topk::{CountSketchWithHeap, C pub use message_pack_format::portable::ddsketch::{DDSKETCH_GROW_CHUNK, DdSketch, DdSketchDelta}; pub use message_pack_format::portable::delta_set_aggregator::DeltaResult; pub use message_pack_format::portable::hll::{HllSketch, HllSketchDelta, HllVariant}; -pub use message_pack_format::portable::hydra_kll::HydraKllSketch; pub use message_pack_format::portable::kll::{KllSketch, KllSketchData}; pub use message_pack_format::portable::sampling::{ effective_sample_p, is_quantile_scale_invariant, rescale_count, rescale_count_with_env, diff --git a/src/message_pack_format/portable/hydra_kll.rs b/src/message_pack_format/portable/hydra_kll.rs deleted file mode 100644 index a7c8233e..00000000 --- a/src/message_pack_format/portable/hydra_kll.rs +++ /dev/null @@ -1,292 +0,0 @@ -//! Wire-format-aligned HydraKLL sketch type. -//! -//! `HydraKllSketch` has no `sketches::*` counterpart — it's a -//! matrix-of-KLL composite owned entirely by this module. - -use std::cmp::Ordering; - -use serde::{Deserialize, Serialize}; -use xxhash_rust::xxh32::xxh32; - -use crate::message_pack_format::portable::kll::{KllSketch, KllSketchData}; -use crate::message_pack_format::{Error as MsgPackError, MessagePackCodec}; - -#[derive(Debug, Clone)] -pub struct HydraKllSketch { - pub sketch: Vec>, - pub rows: usize, - pub cols: usize, -} - -impl HydraKllSketch { - pub fn new(rows: usize, cols: usize, k: u16) -> Self { - let sketch = vec![vec![KllSketch::new(k); cols]; rows]; - Self { sketch, rows, cols } - } - - /// [`HydraKllSketch::new`] with an explicit compaction-coin seed shared by - /// every cell. - /// - /// `new` seeds from the wall clock, so two runs over the same input give - /// different per-key quantiles. Pass a seed when the result has to be - /// reproducible — replaying a stream, comparing two processes, or - /// asserting an accuracy bound in a test. - pub fn with_seed(rows: usize, cols: usize, k: u16, seed: u64) -> Self { - let sketch = vec![vec![KllSketch::with_seed(k, seed); cols]; rows]; - Self { sketch, rows, cols } - } - - /// Number of hash rows in the sketch matrix. - pub fn rows(&self) -> usize { - self.rows - } - - /// Number of columns in the sketch matrix. - pub fn cols(&self) -> usize { - self.cols - } - - pub fn update(&mut self, key: &str, value: f64) { - let key_bytes = key.as_bytes(); - for i in 0..self.rows { - let hash_value = xxh32(key_bytes, i as u32); - let col_index = (hash_value as usize) % self.cols; - self.sketch[i][col_index].update(value); - } - } - - /// Estimate the value at the given quantile `q` for `key` — the - /// median across rows of each row's KLL quantile estimate at the - /// hashed cell. - pub fn quantile(&self, key: &str, q: f64) -> f64 { - let key_bytes = key.as_bytes(); - let mut quantiles = Vec::with_capacity(self.rows); - - for i in 0..self.rows { - let hash_value = xxh32(key_bytes, i as u32); - let col_index = (hash_value as usize) % self.cols; - quantiles.push(self.sketch[i][col_index].quantile(q)); - } - - if quantiles.is_empty() { - return 0.0; - } - - quantiles.sort_by(|a, b| match a.partial_cmp(b) { - Some(ordering) => ordering, - None => Ordering::Equal, - }); - - let mid = quantiles.len() / 2; - if quantiles.len() % 2 == 0 { - (quantiles[mid - 1] + quantiles[mid]) / 2.0 - } else { - quantiles[mid] - } - } - - /// Merge another HydraKllSketch into self in place. Both operands - /// must have identical dimensions. - pub fn merge( - &mut self, - other: &HydraKllSketch, - ) -> Result<(), Box> { - if self.rows != other.rows || self.cols != other.cols { - return Err(format!( - "HydraKllSketch dimension mismatch: self={}x{}, other={}x{}", - self.rows, self.cols, other.rows, other.cols - ) - .into()); - } - for i in 0..self.rows { - for j in 0..self.cols { - self.sketch[i][j].merge(&other.sketch[i][j])?; - } - } - Ok(()) - } - - /// Merge from references, returning a new sketch — convenience for - /// batch reduction at API edges. - pub fn merge_refs( - inputs: &[&HydraKllSketch], - ) -> Result> { - let first = inputs - .first() - .ok_or("HydraKllSketch::merge_refs called with empty input")?; - let mut merged = (*first).clone(); - for h in inputs.iter().skip(1) { - merged.merge(h)?; - } - Ok(merged) - } - - /// One-shot aggregation: build a sketch from parallel keys/values. - pub fn aggregate_hydrakll( - rows: usize, - cols: usize, - k: u16, - keys: &[&str], - values: &[f64], - ) -> Option> { - if keys.is_empty() { - return None; - } - let mut sketch = Self::new(rows, cols, k); - for (key, &value) in keys.iter().zip(values.iter()) { - sketch.update(key, value); - } - sketch.to_msgpack().ok() - } -} - -// ----- Wire format ----- - -#[derive(Debug, Clone, Serialize, Deserialize)] -pub struct HydraKllSketchWire { - #[serde(rename = "row_num")] - pub rows: usize, - #[serde(rename = "col_num")] - pub cols: usize, - pub sketches: Vec>, -} - -impl MessagePackCodec for HydraKllSketch { - fn to_msgpack(&self) -> Result, MsgPackError> { - let mut sketches = Vec::with_capacity(self.rows); - for row in &self.sketch { - let mut row_data = Vec::with_capacity(self.cols); - for cell in row { - row_data.push(KllSketchData { - k: cell.k, - sketch_bytes: cell.sketch_bytes(), - }); - } - sketches.push(row_data); - } - let wire = HydraKllSketchWire { - rows: self.rows, - cols: self.cols, - sketches, - }; - Ok(rmp_serde::to_vec(&wire)?) - } - - fn from_msgpack(bytes: &[u8]) -> Result { - use crate::sketches::kll::KLL; - use rmp_serde::decode::Error as RmpDecodeError; - - let wire: HydraKllSketchWire = rmp_serde::from_slice(bytes)?; - - if wire.sketches.len() != wire.rows { - return Err(MsgPackError::Decode(RmpDecodeError::Uncategorized( - format!( - "HydraKLL row count mismatch: expected {}, got {}", - wire.rows, - wire.sketches.len() - ), - ))); - } - - let mut sketch: Vec> = Vec::with_capacity(wire.rows); - for (row_idx, row) in wire.sketches.into_iter().enumerate() { - if row.len() != wire.cols { - return Err(MsgPackError::Decode(RmpDecodeError::Uncategorized( - format!( - "HydraKLL column count mismatch in row {}: expected {}, got {}", - row_idx, - wire.cols, - row.len() - ), - ))); - } - let mut accum_row: Vec = Vec::with_capacity(wire.cols); - for cell in row { - let backend = KLL::deserialize_from_bytes(&cell.sketch_bytes)?; - accum_row.push(KllSketch { k: cell.k, backend }); - } - sketch.push(accum_row); - } - - Ok(Self { - sketch, - rows: wire.rows, - cols: wire.cols, - }) - } -} - -#[cfg(test)] -mod tests { - use super::*; - - #[test] - fn test_creation() { - let h = HydraKllSketch::new(2, 3, 200); - assert_eq!(h.rows, 2); - assert_eq!(h.cols, 3); - assert_eq!(h.sketch.len(), 2); - assert_eq!(h.sketch[0].len(), 3); - } - - #[test] - fn test_update_and_query() { - let mut h = HydraKllSketch::new(2, 10, 200); - h.update("key1", 5.0); - h.update("key1", 10.0); - let q = h.quantile("key1", 0.5); - assert!(q >= 0.0); - } - - #[test] - fn test_merge() { - let mut h1 = HydraKllSketch::new(2, 5, 200); - let mut h2 = HydraKllSketch::new(2, 5, 200); - - for i in 1..=5 { - h1.update("key1", i as f64); - } - for i in 6..=10 { - h2.update("key1", i as f64); - } - - h1.merge(&h2).unwrap(); - assert_eq!(h1.rows, 2); - assert_eq!(h1.cols, 5); - } - - #[test] - fn test_merge_dimension_mismatch() { - let mut h1 = HydraKllSketch::new(2, 5, 200); - let h2 = HydraKllSketch::new(3, 5, 200); - assert!(h1.merge(&h2).is_err()); - } - - #[test] - fn test_msgpack_round_trip() { - let mut h = HydraKllSketch::new(2, 3, 200); - h.update("key1", 5.0); - h.update("key2", 10.0); - - let bytes = h.to_msgpack().unwrap(); - let deserialized = HydraKllSketch::from_msgpack(&bytes).unwrap(); - - assert_eq!(deserialized.rows, 2); - assert_eq!(deserialized.cols, 3); - } - - #[test] - fn test_aggregate_hydrakll() { - let keys = ["a", "b", "a"]; - let values = [1.0, 2.0, 3.0]; - let bytes = HydraKllSketch::aggregate_hydrakll(2, 5, 200, &keys, &values).unwrap(); - let h = HydraKllSketch::from_msgpack(&bytes).unwrap(); - assert_eq!(h.rows, 2); - assert_eq!(h.cols, 5); - } - - #[test] - fn test_aggregate_hydrakll_empty() { - assert!(HydraKllSketch::aggregate_hydrakll(2, 5, 200, &[], &[]).is_none()); - } -} diff --git a/src/message_pack_format/portable/kll.rs b/src/message_pack_format/portable/kll.rs index 11fa2ea2..ff082343 100644 --- a/src/message_pack_format/portable/kll.rs +++ b/src/message_pack_format/portable/kll.rs @@ -3,9 +3,7 @@ //! `KllSketch` is a thin facade over the pure-Rust [`crate::sketches::kll::KLL`] //! (parity already locked in by the existing //! `test_update_then_envelope_matches_sketchlib_go_bytes` in this file -//! against `sketchlib-go::KLLSketch.SerializePortable`). The wire shape -//! also serves as a nested field for -//! [`crate::message_pack_format::portable::hydra_kll`]. +//! against `sketchlib-go::KLLSketch.SerializePortable`). use serde::{Deserialize, Serialize}; @@ -352,8 +350,7 @@ impl KllProtoItems { // ----- Wire format ----- -/// Wire DTO for [`KllSketch`]. Also referenced as a nested field by -/// [`crate::message_pack_format::portable::hydra_kll::HydraKllSketchWire`]. +/// Wire DTO for [`KllSketch`]. #[derive(Debug, Clone, Serialize, Deserialize)] pub struct KllSketchData { pub k: u16, diff --git a/src/message_pack_format/portable/mod.rs b/src/message_pack_format/portable/mod.rs index 97e0d372..55c7a3b9 100644 --- a/src/message_pack_format/portable/mod.rs +++ b/src/message_pack_format/portable/mod.rs @@ -14,7 +14,6 @@ pub mod countsketch_topk; pub mod ddsketch; pub mod delta_set_aggregator; pub mod hll; -pub mod hydra_kll; pub mod kll; pub mod sampling; pub mod set_aggregator; diff --git a/src/proto/generated/sketchlib.v1.rs b/src/proto/generated/sketchlib.v1.rs index 2c098d3f..94275c29 100644 --- a/src/proto/generated/sketchlib.v1.rs +++ b/src/proto/generated/sketchlib.v1.rs @@ -698,108 +698,6 @@ pub struct UnivMonLayer { #[prost(message, optional, tag = "2")] pub heap: ::core::option::Option, } -/// HydraState is the portable state of a Hydra multi-sketch framework. -/// -/// Hydra maintains a grid of row_num child sketches (cells), each of the -/// same type (counter_type). Items are routed to a specific row cell using a -/// position hash derived from the item's subkey. An optional global "big -/// counter" sketch aggregates the full stream. -/// -/// Current implementations set col_num = 1 (one sketch per row). The field is -/// included for schema completeness and future extensions. -#[derive(Clone, PartialEq, ::prost::Message)] -pub struct HydraState { - /// Number of sketch rows (D / number of subpopulations). - #[prost(uint32, tag = "1")] - pub row_num: u32, - /// Number of sketch columns per row. Currently always 1. - #[prost(uint32, tag = "2")] - pub col_num: u32, - /// The sketch type used for every cell. - #[prost(enumeration = "HydraCounterType", tag = "3")] - pub counter_type: i32, - /// Serialized cells in row-major order, length = row_num * col_num. - /// Each cell's oneof field must match counter_type. - #[prost(message, repeated, tag = "4")] - pub cells: ::prost::alloc::vec::Vec, - /// Optional global sketch covering the entire stream. - /// Absent when global counter is disabled (enable_global_counter = false). - #[prost(message, optional, tag = "5")] - pub big_counter: ::core::option::Option, - /// Seed index used for subpopulation routing (Go: seedHydra, Rust: HYDRA_SEED). - /// Both libraries default to 6. - #[prost(uint32, tag = "6")] - pub seed_index: u32, - /// True when per-cell TopK tracking is enabled. - #[prost(bool, tag = "7")] - pub enable_topk: bool, - /// True when fanout subkey expansion is active. - #[prost(bool, tag = "8")] - pub fanout_subkeys: bool, -} -/// HydraCell holds the serialized state of one Hydra grid cell. -/// Exactly one field in the oneof must be set, and it must match the parent -/// HydraState.counter_type. -#[derive(Clone, PartialEq, ::prost::Message)] -pub struct HydraCell { - #[prost(oneof = "hydra_cell::Sketch", tags = "1, 2, 3, 4, 5")] - pub sketch: ::core::option::Option, -} -/// Nested message and enum types in `HydraCell`. -pub mod hydra_cell { - #[derive(Clone, PartialEq, ::prost::Oneof)] - pub enum Sketch { - #[prost(message, tag = "1")] - CountMin(super::CountMinState), - #[prost(message, tag = "2")] - CountSketch(super::CountSketchState), - #[prost(message, tag = "3")] - Hll(super::HyperLogLogState), - #[prost(message, tag = "4")] - Kll(super::KllState), - #[prost(message, tag = "5")] - Univmon(super::UnivMonState), - } -} -/// HydraCounterType identifies the sketch algorithm used inside Hydra cells. -#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash, PartialOrd, Ord, ::prost::Enumeration)] -#[repr(i32)] -pub enum HydraCounterType { - Unspecified = 0, - CountMin = 1, - CountSketch = 2, - Hll = 3, - Kll = 4, - Univmon = 5, -} -impl HydraCounterType { - /// String value of the enum field names used in the ProtoBuf definition. - /// - /// The values are not transformed in any way and thus are considered stable - /// (if the ProtoBuf definition does not change) and safe for programmatic use. - pub fn as_str_name(&self) -> &'static str { - match self { - Self::Unspecified => "HYDRA_COUNTER_TYPE_UNSPECIFIED", - Self::CountMin => "HYDRA_COUNTER_TYPE_COUNT_MIN", - Self::CountSketch => "HYDRA_COUNTER_TYPE_COUNT_SKETCH", - Self::Hll => "HYDRA_COUNTER_TYPE_HLL", - Self::Kll => "HYDRA_COUNTER_TYPE_KLL", - Self::Univmon => "HYDRA_COUNTER_TYPE_UNIVMON", - } - } - /// Creates an enum from field names used in the ProtoBuf definition. - pub fn from_str_name(value: &str) -> ::core::option::Option { - match value { - "HYDRA_COUNTER_TYPE_UNSPECIFIED" => Some(Self::Unspecified), - "HYDRA_COUNTER_TYPE_COUNT_MIN" => Some(Self::CountMin), - "HYDRA_COUNTER_TYPE_COUNT_SKETCH" => Some(Self::CountSketch), - "HYDRA_COUNTER_TYPE_HLL" => Some(Self::Hll), - "HYDRA_COUNTER_TYPE_KLL" => Some(Self::Kll), - "HYDRA_COUNTER_TYPE_UNIVMON" => Some(Self::Univmon), - _ => None, - } - } -} /// CocoSketchState is the portable state of a CocoSketch flow-size estimator. /// /// The bucket table is stored flat in row-major order (d rows × width cols). @@ -892,7 +790,7 @@ pub struct SketchEnvelope { /// The sketch payload. Exactly one field must be set. #[prost( oneof = "sketch_envelope::SketchState", - tags = "10, 11, 12, 13, 14, 15, 16, 17, 18" + tags = "10, 11, 12, 13, 14, 15, 17, 18" )] pub sketch_state: ::core::option::Option, } @@ -913,8 +811,6 @@ pub mod sketch_envelope { Ddsketch(super::DdSketchState), #[prost(message, tag = "15")] Univmon(super::UnivMonState), - #[prost(message, tag = "16")] - Hydra(super::HydraState), #[prost(message, tag = "17")] Coco(super::CocoSketchState), #[prost(message, tag = "18")] diff --git a/tests/PBT_COVERAGE.md b/tests/PBT_COVERAGE.md index cddee44f..98968446 100644 --- a/tests/PBT_COVERAGE.md +++ b/tests/PBT_COVERAGE.md @@ -2,8 +2,8 @@ One `proptest` law per line. Each compares the sketch against an answer computed without it; what that answer is comes first in every section. One -file per sketch under `tests/pbt/`. `cargo test --test pbt` runs 367 laws; -with `--all-features`, 435, adding the four **experimental** modules. Every +file per sketch under `tests/pbt/`. `cargo test --test pbt` runs 366 laws; +with `--all-features`, 434, adding the four **experimental** modules. Every law was mutation-checked: break the implementation, the law goes red. Shared generators, `grid` and `round_trip!` live in `support.rs`. @@ -257,7 +257,7 @@ Models use only what the paper fixes; register indexing and rank mapping stay fr - msgpack keeps variant, precision, registers, estimate; `Regular`, `Datafusion`, `Hip`; precisions 4-13 - ASAPv1 on `HyperLogLog` keeps registers and estimate; empty included -## hydra.rs — 7 +## hydra.rs — 6 Compared against: the map rebuilt from the subkey encoding (`label ":" value`, joined by `";"`, escaped) and the public matrix hash at `HYDRA_SEED`. All records share one payload value, so each cell's Count-Min is exact. Labels and values include the escaped characters. Cols in {1, 2, 3, 8, 64, 251, 256}. @@ -268,11 +268,9 @@ Compared against: the map rebuilt from the subkey encoding (`label ":" value`, j - an answer is a median of the subkey's cells: at least half the rows `<=` it and at least half `>=` it. Probes: every subset of every record, every single-column equality, one absent value - merge - `merge(S(xs), S(ys)) == S(xs ++ ys)`, cell for cell -- `HydraKllSketch`, no schema, no fan-out - - one key hits one cell per row at `xxh32(key, row) % cols`, read back as retained count - - query == median of its row cells, as an order statistic; an absent key reads 0 - wire - ASAPv1 keeps every subpopulation answer and the schema + - ASAPv1 on a KLL-counter grid (k=8, so busy cells compact) keeps every cell's quantiles, each record's median and the schema ## kll.rs — 14 diff --git a/tests/asapv1_golden.rs b/tests/asapv1_golden.rs index 30b41651..338f93c8 100644 --- a/tests/asapv1_golden.rs +++ b/tests/asapv1_golden.rs @@ -8,9 +8,11 @@ //! //! See `asapv1_golden/README.md`. +use asap_sketchlib::common::input::HydraCounter; use asap_sketchlib::{ - Classic, Count, CountMin, ErtlMLE, FastPath, HllSketch, HllVariant, HyperLogLogHIPP12, - HyperLogLogP12, KLL, MessagePackCodec, RegularPath, Vector2D, + Classic, Count, CountDelta, CountMin, DataInput, ErtlMLE, FastPath, HllSketch, HllVariant, + Hydra, HyperLogLog, HyperLogLogHIPP12, HyperLogLogP12, KLL, MessagePackCodec, RegularPath, + UnivMon, Vector2D, }; fn decode_hex(s: &str) -> Vec { @@ -32,6 +34,11 @@ const GOLDEN_CS_FAST: &str = include_str!("../asapv1_golden/cs_i64_fast_2x4.hex" const GOLDEN_CS_I32: &str = include_str!("../asapv1_golden/cs_i32_regular_2x4.hex"); const GOLDEN_KLL_F64: &str = include_str!("../asapv1_golden/kll_f64_k200.hex"); const GOLDEN_KLL_I64: &str = include_str!("../asapv1_golden/kll_i64_k200.hex"); +const GOLDEN_HYDRA_KLL: &str = include_str!("../asapv1_golden/hydra_kll_2x2_k200.hex"); +const GOLDEN_HYDRA_CM: &str = include_str!("../asapv1_golden/hydra_cm_2x2_counter_2x2.hex"); +const GOLDEN_HYDRA_CS: &str = include_str!("../asapv1_golden/hydra_cs_2x2_counter_2x2.hex"); +const GOLDEN_HYDRA_HLL: &str = include_str!("../asapv1_golden/hydra_hll_1x2_p14.hex"); +const GOLDEN_HYDRA_UNIVMON: &str = include_str!("../asapv1_golden/hydra_univmon_1x2.hex"); /// The known P12 register pattern shared by all three HLL fixtures. fn p12_registers() -> Vec { @@ -287,3 +294,223 @@ fn kll_i64_k200_matches_golden() { assert_eq!(decoded.quantile(0.0), 1.0); assert_eq!(decoded.quantile(1.0), 50.0); } + +// --------------------------------------------------------------------------- +// Hydra: one fixture per counter variant. Each cell's state is set directly +// (matrix storage, crafted register hashes, KLL samples, UnivMon layer deltas +// and heap entries), so no subkey or value is ever hashed. +// --------------------------------------------------------------------------- + +/// The key columns every Hydra fixture declares. +const HYDRA_SCHEMA: [&str; 2] = ["region", "service"]; + +/// A grid over [`HYDRA_SCHEMA`] whose cells are `cells`, row-major. +fn hydra_grid( + rows: usize, + cols: usize, + prototype: HydraCounter, + cells: Vec, +) -> Hydra { + let mut hydra = Hydra::with_schema(rows, cols, HYDRA_SCHEMA, prototype).expect("valid schema"); + let mut cells = cells.into_iter(); + hydra.sketches = Vector2D::from_fn(rows, cols, |_, _| cells.next().expect("one cell each")); + assert!(cells.next().is_none(), "one cell per grid position"); + hydra +} + +/// Asserts the decoded grid's shape and schema, and that each decoded cell +/// holds the same state as `want`'s: KLL's retained state and coin, the +/// matrices, the registers, and UnivMon's own encoding. +fn assert_same_grid(got: &Hydra, want: &Hydra) { + assert_eq!((got.row_num, got.col_num), (want.row_num, want.col_num)); + assert_eq!(got.schema(), want.schema()); + let cells = got.sketches.as_slice().iter().zip(want.sketches.as_slice()); + for (i, (g, w)) in cells.enumerate() { + let same = match (g, w) { + (HydraCounter::KLL(g), HydraCounter::KLL(w)) => { + (g.wire_k(), g.wire_m(), g.wire_levels(), g.wire_coin()) + == (w.wire_k(), w.wire_m(), w.wire_levels(), w.wire_coin()) + && g.wire_items() == w.wire_items() + } + (HydraCounter::CM(g), HydraCounter::CM(w)) => { + g.as_storage().as_slice() == w.as_storage().as_slice() + } + (HydraCounter::CS(g), HydraCounter::CS(w)) => { + g.as_storage().as_slice() == w.as_storage().as_slice() + } + (HydraCounter::HLL(g), HydraCounter::HLL(w)) => { + g.registers_as_slice() == w.registers_as_slice() + } + (HydraCounter::UNIVERSAL(g), HydraCounter::UNIVERSAL(w)) => { + g.serialize_to_bytes().unwrap() == w.serialize_to_bytes().unwrap() + } + _ => false, + }; + assert!(same, "cell {i} differs after decode"); + } +} + +/// KLL cells (k=200, so no compaction fires), each with its own compaction +/// seed: `[1..=5]`, empty, `[2.5, -1.0, 0.0, 1e300, -0.125]`, `[3.0e-5]`. +fn hydra_kll_fixture() -> Hydra { + let cell = |seed: u64, values: &[f64]| { + let mut kll = KLL::init_kll_with_seed(200, seed); + for v in values { + kll.update(v); + } + HydraCounter::KLL(kll) + }; + hydra_grid( + 2, + 2, + HydraCounter::KLL(KLL::init_kll_with_seed(200, 0)), + vec![ + cell(1, &[1.0, 2.0, 3.0, 4.0, 5.0]), + cell(2, &[]), + cell(3, &[2.5, -1.0, 0.0, 1e300, -0.125]), + cell(4, &[3.0e-5]), + ], + ) +} + +/// Count-Min cells, each a 2x2 `i32` matrix, spanning positive fixint / uint8 / +/// uint16 / uint32 up to `i32::MAX`; the last cell is empty. +const HYDRA_CM_CELLS: [[[i32; 2]; 2]; 4] = [ + [[0, 1], [127, 128]], + [[255, 256], [300, 65535]], + [[65536, 1_000_000], [i32::MAX, 0]], + [[0, 0], [0, 0]], +]; + +/// Count Sketch cells, each a 2x2 `i32` matrix, spanning negative fixint / +/// int8 / int16 / int32 down to `i32::MIN` beside positive values. +const HYDRA_CS_CELLS: [[[i32; 2]; 2]; 4] = [ + [[0, -1], [127, -32]], + [[-33, 128], [-128, -129]], + [[-32768, 65536], [-32769, i32::MAX]], + [[i32::MIN, 1], [0, 0]], +]; + +fn hydra_cm_fixture() -> Hydra { + let cell = |m: &[[i32; 2]; 2]| { + HydraCounter::CM(CountMin::from_storage(Vector2D::from_fn(2, 2, |r, c| { + m[r][c] + }))) + }; + hydra_grid( + 2, + 2, + HydraCounter::CM(CountMin::with_dimensions(2, 2)), + HYDRA_CM_CELLS.iter().map(cell).collect(), + ) +} + +fn hydra_cs_fixture() -> Hydra { + let cell = |m: &[[i32; 2]; 2]| { + HydraCounter::CS(Count::from_storage(Vector2D::from_fn(2, 2, |r, c| m[r][c]))) + }; + hydra_grid( + 2, + 2, + HydraCounter::CS(Count::with_dimensions(2, 2)), + HYDRA_CS_CELLS.iter().map(cell).collect(), + ) +} + +/// The P14 register `(index, value)` pairs each HLL cell sets. +const HYDRA_HLL_CELLS: [&[(usize, u8)]; 2] = [ + &[(0, 1), (1, 7), (100, 42), (16383, 3)], + &[(0, 2), (8192, 51)], +]; + +/// A P14 Ertl-MLE HLL whose registers are exactly `set`, zero elsewhere. Each +/// register is written by a pre-hashed value whose top 14 bits name the index +/// and whose leading zeros below them give the value. +fn hll_p14_with(set: &[(usize, u8)]) -> HyperLogLog { + let mut hll = HyperLogLog::::default(); + for &(index, value) in set { + let rank_bit = if value == 51 { 0 } else { 1u64 << (50 - value) }; + hll.insert_with_hash(((index as u64) << 50) | rank_bit); + } + let mut want = vec![0u8; 1 << 14]; + for &(index, value) in set { + want[index] = value; + } + assert_eq!(hll.registers_as_slice(), want.as_slice()); + hll +} + +fn hydra_hll_fixture() -> Hydra { + hydra_grid( + 1, + 2, + HydraCounter::HLL(HyperLogLog::::default()), + HYDRA_HLL_CELLS + .iter() + .map(|set| HydraCounter::HLL(hll_p14_with(set))) + .collect(), + ) +} + +/// UnivMon cells of 2 layers of 1x2 counters, heap capacity 2. Cell 0: layer 0 +/// `[5, -3]`, full heap `{7: 5, 300: 2}`, incomplete; layer 1 `[0, 2]`, heap +/// `{4294967296: 2}`; total weight 7. Cell 1 is empty. +fn hydra_univmon_fixture() -> Hydra { + let fresh = || UnivMon::init_univmon(2, 1, 2, 2); + let mut um = fresh(); + for (layer, col, value) in [(0usize, 0u32, 5i32), (0, 1, -3), (1, 1, 2)] { + um.l2_sketch_layers[layer].apply_delta(CountDelta { row: 0, col, value }); + } + um.hh_layers[0].update(&DataInput::U64(7), 5); + um.hh_layers[0].update(&DataInput::U64(300), 2); + um.hh_layers[1].update(&DataInput::U64(4_294_967_296), 2); + um.set_total_weight(7); + um.mark_layer_candidates_incomplete(0); + hydra_grid( + 1, + 2, + HydraCounter::UNIVERSAL(fresh()), + vec![ + HydraCounter::UNIVERSAL(um), + HydraCounter::UNIVERSAL(fresh()), + ], + ) +} + +/// Serialize the fixture == golden; decode golden -> the fixture's state; +/// re-encode == golden. +fn assert_hydra_golden(golden: &str, fixture: Hydra, kind_id: [u8; 2]) { + let want = decode_hex(golden); + assert_eq!(&want[7..10], &[2, kind_id[0], kind_id[1]], "kind_id"); + let got = fixture.serialize_to_bytes().expect("serialize"); + assert_eq!(got, want, "Hydra {kind_id:02x?} bytes diverge from golden"); + + let decoded = Hydra::deserialize_from_bytes(&want).expect("decode"); + assert_same_grid(&decoded, &fixture); + assert_eq!(decoded.serialize_to_bytes().expect("re-serialize"), want); +} + +#[test] +fn hydra_kll_2x2_k200_matches_golden() { + assert_hydra_golden(GOLDEN_HYDRA_KLL, hydra_kll_fixture(), [0x07, 0x00]); +} + +#[test] +fn hydra_cm_2x2_counter_2x2_matches_golden() { + assert_hydra_golden(GOLDEN_HYDRA_CM, hydra_cm_fixture(), [0x07, 0x01]); +} + +#[test] +fn hydra_cs_2x2_counter_2x2_matches_golden() { + assert_hydra_golden(GOLDEN_HYDRA_CS, hydra_cs_fixture(), [0x07, 0x02]); +} + +#[test] +fn hydra_hll_1x2_p14_matches_golden() { + assert_hydra_golden(GOLDEN_HYDRA_HLL, hydra_hll_fixture(), [0x07, 0x03]); +} + +#[test] +fn hydra_univmon_1x2_matches_golden() { + assert_hydra_golden(GOLDEN_HYDRA_UNIVMON, hydra_univmon_fixture(), [0x07, 0x04]); +} diff --git a/tests/e2e/quantiles.rs b/tests/e2e/quantiles.rs index 637a27c4..dc111b26 100644 --- a/tests/e2e/quantiles.rs +++ b/tests/e2e/quantiles.rs @@ -44,7 +44,6 @@ use common::{ }; use asap_sketchlib::message_pack_format::portable::ddsketch::DdSketch as PortableDds; -use asap_sketchlib::message_pack_format::portable::hydra_kll::HydraKllSketch; use asap_sketchlib::{ DDSketch, DataInput, KLL, KLLConfig, KLLDynamic, TumblingWindow, UnivMonQ, UnivMonQConfig, }; @@ -1559,66 +1558,6 @@ fn tumbling_kll_windows_are_exact_and_answers_satisfy_the_rank_contract() { ); } -// --------------------------------------------- Portable HydraKll per-key - -/// Hydra routes each key to its own KLL cell, so each cell answers under the -/// same rank characterization as a standalone KLL over that key's observations. -/// -/// Every cell of a `HydraKllSketch` is cloned from one seeded prototype, so the -/// two keys of a single grid share a compaction seed and are **not** independent -/// trials. Each grid is therefore reduced to its worst rank error across both -/// keys and the whole q grid, and the binomial runs over twelve independent -/// prototype seeds. -#[test] -fn portable_hydra_kll_per_key_medians_satisfy_the_rank_characterization() { - const K: usize = 200; - const TRIALS: usize = 12; - const QS: [f64; 5] = [0.1, 0.25, 0.5, 0.75, 0.9]; - - let spec = KllRankSpec::datasketches(K); - let mut truths: Vec<(&str, NumericTruth)> = Vec::new(); - for (name, base, seed) in [("svc-a", 100.0f64, 3010u64), ("svc-b", 900.0, 3011)] { - let vals: Vec = normal_f64(4000, base, base * 0.05, seed) - .into_iter() - .map(f64::abs) // HydraKll cells are KLL: positive domain - .collect(); - truths.push((name, NumericTruth::new(vals))); - } - - let mut tally = Tally::default(); - for t in 0..TRIALS { - let seed = kll_trial_seed(0x5EED_0500 + t as u64); - let mut hk = HydraKllSketch::with_seed(3, 256, K as u16, seed); - for (name, truth) in &truths { - for v in truth.sorted() { - hk.update(name, *v); - } - } - let mut worst = 0.0f64; - let mut detail = String::new(); - for (name, truth) in &truths { - let (e, d) = spec.max_rank_error(truth.sorted(), &QS, |q| hk.quantile(name, q)); - if e >= worst { - worst = e; - detail = format!("key {name}: {d}"); - } - } - let eps = spec.epsilon(); - tally.record(worst <= eps, || { - format!("seed={seed:#x}: max rank error {worst:.6} > eps(k={K}) = {eps:.6}; {detail}") - }); - } - tally.assert_independent_binomial( - "portable HydraKll per-key quantiles / maximum rank error per prototype seed", - spec.trial_failure_probability, - &format!( - "k={K}, 3x256 grid, normal(100, 5) and normal(900, 45), stream seeds 3010/3011, \ - {TRIALS} independent prototype seeds from kll_trial_seed(0x5EED_0500..), \ - q grid {QS:?}" - ), - ); -} - // ------------------------------------------- DDSketch input rejection /// Structural guards, not accuracy: values the mapping cannot index and diff --git a/tests/msgpack_compat.rs b/tests/msgpack_compat.rs index 267a51c6..37dda380 100644 --- a/tests/msgpack_compat.rs +++ b/tests/msgpack_compat.rs @@ -28,11 +28,9 @@ use asap_sketchlib::message_pack_format::portable::countminsketch_topk::{ CountMinSketchInnerWire, CountMinSketchWithHeapWire, }; use asap_sketchlib::message_pack_format::portable::delta_set_aggregator::DeltaResult; -use asap_sketchlib::message_pack_format::portable::hydra_kll::HydraKllSketchWire; -use asap_sketchlib::message_pack_format::portable::kll::KllSketchData; use asap_sketchlib::{ CountMinSketch, CountMinSketchWithHeap, CountSketch, DdSketch, HllSketch, HllVariant, - HydraKllSketch, KllSketch, SetAggregator, + KllSketch, SetAggregator, }; // ===== round-trip: every wire-format-aligned type ===== @@ -107,18 +105,6 @@ fn kll_sketch_round_trip() { assert_eq!(restored.count(), 100); } -#[test] -fn hydra_kll_sketch_round_trip() { - let mut s = HydraKllSketch::with_seed(2, 4, 200, 0x5EED_0900); - s.update("a", 1.0); - s.update("a", 2.0); - s.update("b", 3.0); - let bytes = s.to_msgpack().expect("encode"); - let restored = HydraKllSketch::from_msgpack(&bytes).expect("decode"); - assert_eq!(restored.rows, 2); - assert_eq!(restored.cols, 4); -} - #[test] fn set_aggregator_round_trip() { let mut s = SetAggregator::new(); @@ -184,33 +170,9 @@ fn count_min_with_heap_wire_shape() { assert_eq!(restored.topk_heap[0].key, "hot"); } -#[test] -fn hydra_kll_wire_shape() { - let wire = HydraKllSketchWire { - rows: 2, - cols: 3, - sketches: vec![ - vec![ - KllSketchData { - k: 200, - sketch_bytes: vec![], - }; - 3 - ]; - 2 - ], - }; - let bytes = rmp_serde::to_vec(&wire).unwrap(); - let restored: HydraKllSketchWire = rmp_serde::from_slice(&bytes).unwrap(); - assert_eq!(restored.rows, 2); - assert_eq!(restored.cols, 3); - assert_eq!(restored.sketches.len(), 2); - assert_eq!(restored.sketches[0].len(), 3); -} - // ===== golden-bytes placeholders: an ignored, uncovered gap ===== // -// The five tests below are **empty and ignored**. They verify nothing today. +// The four tests below are **empty and ignored**. They verify nothing today. // // What they need is a msgpack payload produced by `sketchlib-go` and checked in // under `tests/fixtures/msgpack/`. That fixture cannot be generated here: this @@ -248,7 +210,3 @@ fn hll_sketch_decodes_go_bytes() {} #[ignore = "gap: needs a sketchlib-go-produced msgpack fixture; none can be generated in this repo"] #[test] fn kll_sketch_decodes_go_bytes() {} - -#[ignore = "gap: needs a sketchlib-go-produced msgpack fixture; none can be generated in this repo"] -#[test] -fn hydra_kll_decodes_go_bytes() {} diff --git a/tests/pbt/hydra.rs b/tests/pbt/hydra.rs index f25ecd0d..33944ec5 100644 --- a/tests/pbt/hydra.rs +++ b/tests/pbt/hydra.rs @@ -20,19 +20,15 @@ //! subkey names. Everything else about a subpopulation answer — that it never //! falls below the records it counts, that a generalization never reads below //! its specialization — follows from those two. -//! -//! `HydraKllSketch` is a second, unrelated grid in the portable module: no -//! schema and no fan-out, one string key routed by `xxh32(key, row)` and a -//! median of the row KLLs. Its accuracy is characterized in the end-to-end -//! quantile battery; the two laws here pin its routing and its reduction. use asap_sketchlib::common::input::{HydraCounter, HydraQuery}; use asap_sketchlib::{ - CountMin, DataInput, FastPath, HYDRA_SEED, Hydra, HydraKllSketch, MatrixFastHash, Vector2D, + CountMin, DataInput, FastPath, HYDRA_SEED, Hydra, KLL, MatrixFastHash, Vector2D, hash_for_matrix_seeded, }; use proptest::prelude::*; -use xxhash_rust::xxh32::xxh32; + +use crate::support::QUANTILES; /// The value every record carries. const PAYLOAD: &str = "pkt"; @@ -260,35 +256,47 @@ fn grid_cols() -> impl Strategy { ] } -/// The compaction-coin seed every portable grid is built with, so a cell's -/// answer is a function of its stream alone. +/// The compaction-coin seed every KLL cell is built with. const KLL_SEED: u64 = 0x5EED_2000; -/// Wide enough that the streams below are retained whole, so no cell compacts. -const KLL_K: u16 = 256; - -/// Keys the portable grid is streamed and probed with. Few and short, so a -/// narrow grid makes them share cells. -const KLL_KEYS: [&str; 5] = ["a", "b", "c", "", "a:b"]; +/// Small enough that a short stream compacts the busier cells. +const KLL_K: i32 = 8; -/// The column a portable key occupies on each row. -fn kll_columns(rows: usize, cols: usize, key: &str) -> Vec { - (0..rows) - .map(|row| xxh32(key.as_bytes(), row as u32) as usize % cols) - .collect() -} - -/// A stream of `(key index, value)` for the portable grid. -fn kll_stream(max: usize) -> impl Strategy> { - prop::collection::vec((0usize..KLL_KEYS.len(), 0.0f64..1000.0), 1..max) +/// Streams the records into a grid of KLL cells, record `i` carrying +/// `values[i % values.len()]`. +fn kll_filled( + rows: usize, + cols: usize, + labels: &[String], + records: &[Vec], + values: &[f64], +) -> Hydra { + let counter = HydraCounter::KLL(KLL::init_kll_with_seed(KLL_K, KLL_SEED)); + let mut hydra = Hydra::with_schema(rows, cols, labels.to_vec(), counter).expect("valid schema"); + for (record, value) in records.iter().zip(values.iter().cycle()) { + hydra + .update(&borrow(record), &DataInput::F64(*value), None) + .expect("schema arity"); + } + hydra } -fn kll_filled(rows: usize, cols: usize, stream: &[(usize, f64)]) -> HydraKllSketch { - let mut grid = HydraKllSketch::with_seed(rows, cols, KLL_K, KLL_SEED); - for (key, value) in stream { - grid.update(KLL_KEYS[*key], *value); - } - grid +/// Every cell's answer at each probed quantile, row-major. +fn cell_quantiles(hydra: &Hydra) -> Vec> { + hydra + .sketches + .as_slice() + .iter() + .map(|cell| { + QUANTILES + .iter() + .map(|q| { + let answer = cell.query(&HydraQuery::Quantile(*q)); + answer.expect("KLL answers a quantile").to_bits() + }) + .collect() + }) + .collect() } proptest! { @@ -415,73 +423,6 @@ proptest! { } } - // ===== The portable grid routes each row by its own seeded hash ========= - // - // `HydraKllSketch` carries no schema and does not fan out: one key reaches - // one cell per row, at `xxh32(key, row) % cols`. The oracle counts the - // stream into that map and reads each cell's retained count back, so a row - // seeded alike as its neighbours — or a routing that ignores the row — is - // an unequal count rather than a merely worse estimate. - - #[test] - fn portable_hydra_kll_routes_each_update_to_the_cell_its_row_hash_names( - rows in 1usize..5, - cols in 1usize..9, - stream in kll_stream(24), - ) { - let grid = kll_filled(rows, cols, &stream); - - let mut oracle = vec![vec![0u64; cols]; rows]; - for (key, _) in &stream { - for (row, col) in kll_columns(rows, cols, KLL_KEYS[*key]).into_iter().enumerate() { - oracle[row][col] += 1; - } - } - - for (r, row) in oracle.iter().enumerate() { - for (c, want) in row.iter().enumerate() { - prop_assert_eq!( - grid.sketch[r][c].count(), *want, - "cell ({}, {}) of a {}x{} portable grid", r, c, rows, cols - ); - } - } - } - - // ===== The portable grid answers a median of its row cells ============== - // - // As above, an order statistic: the tie rule for an even row count is not - // part of the contract. An empty cell answers 0, which is below every value - // the stream carries, so a query for an absent key is a real probe of the - // reduction rather than a vacuous one. - - #[test] - fn portable_hydra_kll_answers_a_median_of_its_row_cells( - rows in 1usize..5, - cols in 1usize..9, - stream in kll_stream(24), - q in 0.0f64..=1.0, - ) { - let grid = kll_filled(rows, cols, &stream); - - for key in KLL_KEYS { - let row_estimates: Vec = kll_columns(rows, cols, key) - .into_iter() - .enumerate() - .map(|(r, c)| grid.sketch[r][c].quantile(q)) - .collect(); - - let seen = grid.quantile(key, q); - let at_or_below = row_estimates.iter().filter(|e| **e <= seen).count(); - let at_or_above = row_estimates.iter().filter(|e| **e >= seen).count(); - prop_assert!( - 2 * at_or_below >= rows && 2 * at_or_above >= rows, - "key {:?} at q {} read {}, no median of its row estimates {:?}", - key, q, seen, row_estimates - ); - } - } - // ===== Wire ===== // // A grid of counters cut to size costs no more to build than any other @@ -521,4 +462,33 @@ proptest! { |h: &Hydra| h.schema().to_vec(), ); } + + // A KLL cell holds a variable-length state, so the cells travel one array + // element each rather than tiled. Every cell must come back with its + // retained samples, its level layout and its compaction coin. + + #[test] + fn hydra_kll_counter_round_trips_through_its_asapv1_envelope( + rows in 1usize..6, + cols in grid_cols(), + (labels, records) in schema_and_records(16), + values in prop::collection::vec(-1.0e6f64..1.0e6, 1..16), + ) { + let hydra = kll_filled(rows, cols, &labels, &records, &values); + + round_trip!( + Hydra, + hydra, + cell_quantiles, + |h: &Hydra| records + .iter() + .map(|record| { + let key: Vec> = record.iter().map(|v| Some(v.as_str())).collect(); + let median = h.query_key(&key, &HydraQuery::Quantile(0.5)); + median.expect("well-formed query").to_bits() + }) + .collect::>(), + |h: &Hydra| h.schema().to_vec(), + ); + } } diff --git a/tests/xtest_consumer.rs b/tests/xtest_consumer.rs index 0113ff45..d0171887 100644 --- a/tests/xtest_consumer.rs +++ b/tests/xtest_consumer.rs @@ -13,7 +13,6 @@ //! coco.pb — CocoSketchState (hash+val+hasKey buckets) //! elastic.pb — ElasticState (heavy buckets + light CountMin) //! univmon.pb — UnivMonState (layered CountSketch + TopK heaps) -//! hydra.pb — HydraState (CM-cell grid) //! //! Usage: //! XTEST_DIR= cargo test --test xtest_consumer -- --nocapture @@ -395,47 +394,6 @@ fn cross_language_proto() { all_ok = false; } - // ----------------------------------------------------------------------- - // HydraSketch - // ----------------------------------------------------------------------- - println!(); - println!("[Hydra] Step 1/3 — Read hydra.pb"); - let bytes = read_file(in_dir.join("hydra.pb")); - let env = SketchEnvelope::decode(bytes.as_slice()).expect("decode hydra envelope"); - - println!( - "[Hydra] Step 2/3 — Validate envelope (format_version={}, producer={})", - env.format_version, - env.producer.as_ref().map_or("?", |p| &p.library) - ); - - let hydra_state = match env.sketch_state { - Some(sketch_envelope::SketchState::Hydra(ref s)) => s.clone(), - other => panic!("expected Hydra sketch_state, got {other:?}"), - }; - - println!( - "[Hydra] row_num={} col_num={} counter_type={} cells={}", - hydra_state.row_num, - hydra_state.col_num, - hydra_state.counter_type, - hydra_state.cells.len() - ); - - // Query "hydra:42" — inserted 51 times (1 base + 50 extra). - // Routing: subkey_hash = xxh3_64_seeded(seedList[6]=0xbb67ae85, b"hydra:42") - // Value hash: value_hash = xxh3_64_seeded(seedList[0]=0xcafe3553, b"hydra:42") - let hydra_subkey_hash = xxh3_64_seeded(SEED_6, b"hydra:42"); - let hydra_value_hash = xxh3_64_seeded(SEED_0, b"hydra:42"); - let hydra_est = hydra_query_cm(&hydra_state, hydra_subkey_hash, hydra_value_hash); - println!("[Hydra] Step 3/3 — 'hydra:42' est = {hydra_est:.0} (expect ≥ 51)"); - if hydra_est >= 51.0 { - println!("[Hydra] PASS"); - } else { - eprintln!("[Hydra] FAIL: estimate {hydra_est:.0} < 51"); - all_ok = false; - } - // ----------------------------------------------------------------------- // Sampled CountMin (geometric skip-sampling, p=0.1) — exercises the // sample_p envelope field + the backend ×1/p rescale at query. @@ -554,7 +512,6 @@ fn cross_language_proto() { const SEED_0: u64 = 0xcafe3553; // seedList[0] — Hash64 / default hash const SEED_5: u64 = 0x6a09e667; // seedList[5] — CanonicalHashSeed -const SEED_6: u64 = 0xbb67ae85; // seedList[6] — defaultHydraSeed // --------------------------------------------------------------------------- // Helpers @@ -1008,95 +965,6 @@ fn median_of_three_f64(a: f64, b: f64, c: f64) -> f64 { } } -// --------------------------------------------------------------------------- -// HydraSketch CountMin frequency query -// --------------------------------------------------------------------------- -// Routing (mirrors Go's fillPositionsFromHash with default seeds): -// seedCM1 = 0x1111111111111111, seedCM2 = 0x2222222222222222 -// x = subkey_hash ^ seedCM1; y = subkey_hash ^ seedCM2 -// for r in 0..D: xorshift both; pos[r] = (x ^ (y<<1)) % W -// For each row: query CM cell at cells[r*W + pos[r]] with value_hash. -// CM query: min across rows of count at col=(value_hash>>(r*bits))&mask. -// Final result: median of per-Hydra-row CM estimates. - -const HYDRA_SEED_CM1: u64 = 0x1111111111111111; -const HYDRA_SEED_CM2: u64 = 0x2222222222222222; - -fn xorshift64(x: &mut u64) { - *x ^= *x << 13; - *x ^= *x >> 7; - *x ^= *x << 17; -} - -fn hydra_fill_positions(subkey_hash: u64, d: usize, w: usize) -> Vec { - let mut x = subkey_hash ^ HYDRA_SEED_CM1; - let mut y = subkey_hash ^ HYDRA_SEED_CM2; - if x == 0 { - x = HYDRA_SEED_CM1; - } - if y == 0 { - y = HYDRA_SEED_CM2 | 1; - } - - let mut pos = Vec::with_capacity(d); - for _ in 0..d { - xorshift64(&mut x); - xorshift64(&mut y); - pos.push(((x ^ (y << 1)) % w as u64) as usize); - } - pos -} - -fn hydra_query_cm(state: &HydraState, subkey_hash: u64, value_hash: u64) -> f64 { - let d = state.row_num as usize; - let w = state.col_num as usize; - let cells = &state.cells; - - let pos = hydra_fill_positions(subkey_hash, d, w); - - let mut estimates = Vec::with_capacity(d); - for (r, &p) in pos.iter().enumerate().take(d) { - let cell_idx = r * w + p; - if cell_idx >= cells.len() { - estimates.push(0.0f64); - continue; - } - let cell = &cells[cell_idx]; - let freq = match &cell.sketch { - Some(hydra_cell::Sketch::CountMin(cm)) => cm_query_min(cm, value_hash), - _ => 0.0, - }; - estimates.push(freq); - } - - median_f64(&mut estimates) -} - -/// CountMin min-frequency query with packed hash. -fn cm_query_min(cm: &CountMinState, hash: u64) -> f64 { - let rows = cm.rows as usize; - let cols = cm.cols as usize; - let bits_per_row = col_bits(cols); - let mask = (cols as u64) - 1; - let counts: Vec = if !cm.counts_float.is_empty() { - cm.counts_float.clone() - } else { - cm.counts_int.iter().map(|&v| v as f64).collect() - }; - let counts = &counts; - - let mut min_val = f64::MAX; - for r in 0..rows { - let shift = (r as u64) * bits_per_row; - let col = ((hash >> shift) & mask) as usize; - let v = counts[r * cols + col]; - if v < min_val { - min_val = v; - } - } - if min_val == f64::MAX { 0.0 } else { min_val } -} - // --------------------------------------------------------------------------- // Minimal KLL deserialization + CDF query // --------------------------------------------------------------------------- diff --git a/tools/gen-proto/src/main.rs b/tools/gen-proto/src/main.rs index 1de6ad15..e999febb 100644 --- a/tools/gen-proto/src/main.rs +++ b/tools/gen-proto/src/main.rs @@ -34,7 +34,6 @@ const PROTO_FILES: &[&str] = &[ "kll/kll.proto", "ddsketch/ddsketch.proto", "univmon/univmon.proto", - "hydra/hydra.proto", "cocosketch/cocosketch.proto", "elasticsketch/elasticsketch.proto", "sketchlib.proto", From 52b9de69a854fde2d5c68d5db7941bbfd62afbb1 Mon Sep 17 00:00:00 2001 From: GordonYuanyc Date: Fri, 2 Oct 2026 00:57:11 -0600 Subject: [PATCH 13/36] docs: defer the ASAPv1 fixture list to the golden README Co-Authored-By: Claude Opus 5.5 --- docs/asapv1_wire_format.md | 4 ++-- docs/message_pack_format.md | 8 ++++---- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/docs/asapv1_wire_format.md b/docs/asapv1_wire_format.md index f58c9221..3e538119 100644 --- a/docs/asapv1_wire_format.md +++ b/docs/asapv1_wire_format.md @@ -1341,14 +1341,14 @@ Good direction (more compact, higher fidelity, less Rust-internal duplication), 2. **Golden byte-vector fixtures** in one shared repo, [`sketchlib-golden-bytes`](https://github.com/ProjectASAP/sketchlib-golden-bytes), which each implementation mounts at `asapv1_golden/`; both languages decode and re-encode them byte-identically. These replace the `portable`-as-oracle round-trip test. 3. **This registry**, mirrored, never independently allocated. -Fixtures exist for seven `kind_id`s — HLL's three estimators, Count-Min, Count Sketch, compact KLL and CSHeap — and `sketchlib-go` mirrors all but CSHeap's. Every other kind here has a payload and **no fixture**, so this document is its only contract; `asapv1_golden/README.md` lists the gap. `portable` carries its own Go goldens. +`asapv1_golden/README.md`'s Coverage section lists which `kind_id`s have fixtures; for every other kind, this document is the only contract. `portable` carries its own Go goldens. **Hash profile on the Go side.** Rust derives the hash spec from a generic `HashProfile` bound on the hasher type; Go has no generic hasher type, so there is nothing to derive from. On the Go side the profile is simply **written into** the metadata on encode and **read from** it on decode. Go MUST validate the profile it reads (same fail-closed intent as Rust): a sketch is only mergeable/queryable if its `hash_profile_id` + seeds match the profile Go is prepared to reproduce. -Sequencing: (2) covers seven `kind_id`s, and the one `native bytes == portable bytes` test is HLL-only, since no other `portable` type emits ASAPv1 bytes. +Sequencing: the one `native bytes == portable bytes` test is HLL-only, since no other `portable` type emits ASAPv1 bytes. Retire `portable` once the fixtures cover the rest. --- diff --git a/docs/message_pack_format.md b/docs/message_pack_format.md index 35c77252..8dae3de7 100644 --- a/docs/message_pack_format.md +++ b/docs/message_pack_format.md @@ -169,10 +169,10 @@ decode → re-encode them byte-identically. The `kind_id` registry is mirrored verbatim with Go's `wire/asapmsgpack/magic_ids.go`, never independently allocated. -That proof covers the seven `kind_id`s a fixture exists for — HLL's three -estimators, Count-Min, Count Sketch, compact KLL and CSHeap. Every other implemented -kind has **no ASAPv1 golden and therefore no ASAPv1 drift guard**; -[its README](https://github.com/ProjectASAP/sketchlib-golden-bytes#coverage) lists what is covered. +That proof covers only the `kind_id`s with a fixture, which +[its README's Coverage section](https://github.com/ProjectASAP/sketchlib-golden-bytes#coverage) +lists. Every other implemented kind has **no ASAPv1 golden and therefore no +ASAPv1 drift guard**. The older `portable` path is guarded separately, by the `sketchlib-go` goldens of its own listed above. From b033ba2b6f6a874d372e442a07464126718df976 Mon Sep 17 00:00:00 2001 From: GordonYuanyc Date: Fri, 2 Oct 2026 00:59:39 -0600 Subject: [PATCH 14/36] refactor(coco)!: drop the proto format Delete proto/cocosketch/cocosketch.proto (CocoSketchState), reserve the SketchEnvelope oneof field 17 ("coco"), regenerate, and remove the Coco section of the cross-language proto consumer. Coco serializes only as ASAPv1. Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 8 +++ docs/features.md | 2 +- docs/tests.md | 2 +- proto/cocosketch/cocosketch.proto | 35 ------------- proto/sketchlib.proto | 5 +- src/proto/generated/sketchlib.v1.rs | 27 +--------- tests/xtest_consumer.rs | 80 ----------------------------- tools/gen-proto/src/main.rs | 1 - 8 files changed, 14 insertions(+), 146 deletions(-) delete mode 100644 proto/cocosketch/cocosketch.proto diff --git a/CHANGELOG.md b/CHANGELOG.md index 80d85f98..f35c5ce5 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -88,6 +88,14 @@ signals a backwards-compatible change. as an empty window like any other, and that window takes a place in `max_windows`. +### Removed + +- **Coco's protobuf format.** `proto/cocosketch/cocosketch.proto` and the + generated `asap_sketchlib::proto::sketchlib::CocoSketchState` are gone, and + the `SketchEnvelope` oneof field `coco` (17) is reserved, so + `sketch_envelope::SketchState::Coco` is gone too. Coco serializes only as + ASAPv1 (`Coco::serialize_to_bytes` / `deserialize_from_bytes`). + ### Fixed - `KLLDynamic` panicked on every quantile query once a NaN had entered the diff --git a/docs/features.md b/docs/features.md index 64f296a3..1d87ce7a 100644 --- a/docs/features.md +++ b/docs/features.md @@ -112,7 +112,7 @@ MessagePack (`rmp-serde`) support. **serde support** means the type derives `Ser | EHSketchList | Yes | Yes | | ExponentialHistogram | No | Yes | -Protobuf (prost): `.proto` definitions exist for CountMin, Count, HLL, DDSketch, KLL, Elastic, Coco, Hydra, and UnivMon. Rust conversion code is in progress. +Protobuf (prost): `.proto` definitions exist for CountMin, Count, HLL, DDSketch, KLL, Elastic, Hydra, and UnivMon. Rust conversion code is in progress. ### API Stability diff --git a/docs/tests.md b/docs/tests.md index ccb1571c..e577a012 100644 --- a/docs/tests.md +++ b/docs/tests.md @@ -2202,7 +2202,7 @@ Test file: [`tests/xtest_consumer.rs`](../tests/xtest_consumer.rs) | test_name | test_description | what_is_tested | | --- | --- | --- | -| `cross_language_proto` | Protobuf envelopes read from the cross-language fixture directory answer their point queries in band. | Asserts nothing unless fixtures are available: with no `XTEST_DIR` and no discoverable producer directory holding `tests/cross_language/xtest_producer_test.go`, it prints a skip line and returns. With fixtures it decodes nine `SketchEnvelope` files and records a pass or failure for each — `countmin.pb` min-frequency for `item:42` at least `101`, `kll.pb` p50 within 5% of `5000` and p99 within 5% of `9900`, `ddsketch.pb` the same two quantiles within 2%, `hll.pb` an ErtlMLE cardinality inside `[40_000, 65_000]`, `countsketch.pb` `cs:hot` at least `200`, `coco.pb` `coco:hot` at least `500`, `elastic.pb` `elephant` at least `900`, `univmon.pb` a cardinality inside `[1_000, 15_000]`, `hydra.pb` `hydra:42` at least `51` — using the hash seeds `0xcafe3553`, `0x6a09e667` and `0xbb67ae85`, and, when `countmin_sampled.pb` and `hll_sampled.pb` exist, requires `effective_sample_p` of `0.1` and a `rescale_count` by `1/p` within 5% of `100_000` and 6% of `200_000`. A single `assert!` at the end fails if any check failed; a missing or mistyped envelope panics on the spot. | +| `cross_language_proto` | Protobuf envelopes read from the cross-language fixture directory answer their point queries in band. | Asserts nothing unless fixtures are available: with no `XTEST_DIR` and no discoverable producer directory holding `tests/cross_language/xtest_producer_test.go`, it prints a skip line and returns. With fixtures it decodes eight `SketchEnvelope` files and records a pass or failure for each — `countmin.pb` min-frequency for `item:42` at least `101`, `kll.pb` p50 within 5% of `5000` and p99 within 5% of `9900`, `ddsketch.pb` the same two quantiles within 2%, `hll.pb` an ErtlMLE cardinality inside `[40_000, 65_000]`, `countsketch.pb` `cs:hot` at least `200`, `elastic.pb` `elephant` at least `900`, `univmon.pb` a cardinality inside `[1_000, 15_000]`, `hydra.pb` `hydra:42` at least `51` — using the hash seeds `0xcafe3553`, `0x6a09e667` and `0xbb67ae85`, and, when `countmin_sampled.pb` and `hll_sampled.pb` exist, requires `effective_sample_p` of `0.1` and a `rescale_count` by `1/p` within 5% of `100_000` and 6% of `200_000`. A single `assert!` at the end fails if any check failed; a missing or mistyped envelope panics on the spot. | ### HLL Custom Precision diff --git a/proto/cocosketch/cocosketch.proto b/proto/cocosketch/cocosketch.proto deleted file mode 100644 index 041b3c04..00000000 --- a/proto/cocosketch/cocosketch.proto +++ /dev/null @@ -1,35 +0,0 @@ -// cocosketch.proto — Portable wire format for CocoSketch flow-size estimator state. - -syntax = "proto3"; - -package sketchlib.v1; - -option go_package = "github.com/ProjectASAP/sketchlib-go/proto/cocosketch;cocosketchpb"; - -// ============================================================================ -// CocoSketch -// ============================================================================ - -// CocoSketchState is the portable state of a CocoSketch flow-size estimator. -// -// The bucket table is stored flat in row-major order (d rows × width cols). -// Element [r][c] is at index r * width + c. -message CocoSketchState { - // Number of hash rows. - uint32 d = 1; - - // Number of buckets per row. - uint32 width = 2; - - // Full hash stored in each bucket, length = d * width. - // Zero means the bucket is empty (has_keys[i] == false). - repeated uint64 hashes = 3 [packed = true]; - - // Cumulative count value per bucket, length = d * width. - repeated uint64 vals = 4 [packed = true]; - - // Whether each bucket holds a valid key entry, length = d * width. - repeated bool has_keys = 5 [packed = true]; - - reserved 6 to 15; -} diff --git a/proto/sketchlib.proto b/proto/sketchlib.proto index 0a6d8193..3bc37e2d 100644 --- a/proto/sketchlib.proto +++ b/proto/sketchlib.proto @@ -36,7 +36,6 @@ import "kll/kll.proto"; import "ddsketch/ddsketch.proto"; import "univmon/univmon.proto"; import "hydra/hydra.proto"; -import "cocosketch/cocosketch.proto"; import "elasticsketch/elasticsketch.proto"; option go_package = "github.com/ProjectASAP/sketchlib-go/proto/sketch_envelope;sketch_envelopepb"; @@ -73,10 +72,12 @@ message SketchEnvelope { DDSketchState ddsketch = 14; UnivMonState univmon = 15; HydraState hydra = 16; - CocoSketchState coco = 17; ElasticState elastic = 18; } + reserved 17; + reserved "coco"; + reserved 19 to 31; // Reserved for future sketch types. // Geometric / NitroSketch sampling probability for the producing sketch. diff --git a/src/proto/generated/sketchlib.v1.rs b/src/proto/generated/sketchlib.v1.rs index 2c098d3f..c1503e2f 100644 --- a/src/proto/generated/sketchlib.v1.rs +++ b/src/proto/generated/sketchlib.v1.rs @@ -800,29 +800,6 @@ impl HydraCounterType { } } } -/// CocoSketchState is the portable state of a CocoSketch flow-size estimator. -/// -/// The bucket table is stored flat in row-major order (d rows × width cols). -/// Element [r][c] is at index r * width + c. -#[derive(Clone, PartialEq, ::prost::Message)] -pub struct CocoSketchState { - /// Number of hash rows. - #[prost(uint32, tag = "1")] - pub d: u32, - /// Number of buckets per row. - #[prost(uint32, tag = "2")] - pub width: u32, - /// Full hash stored in each bucket, length = d * width. - /// Zero means the bucket is empty (has_keys\[i\] == false). - #[prost(uint64, repeated, tag = "3")] - pub hashes: ::prost::alloc::vec::Vec, - /// Cumulative count value per bucket, length = d * width. - #[prost(uint64, repeated, tag = "4")] - pub vals: ::prost::alloc::vec::Vec, - /// Whether each bucket holds a valid key entry, length = d * width. - #[prost(bool, repeated, tag = "5")] - pub has_keys: ::prost::alloc::vec::Vec, -} /// ElasticState is the portable state of an Elastic Sketch. /// /// The Elastic Sketch has a heavy part (one bucket per index) and a light @@ -892,7 +869,7 @@ pub struct SketchEnvelope { /// The sketch payload. Exactly one field must be set. #[prost( oneof = "sketch_envelope::SketchState", - tags = "10, 11, 12, 13, 14, 15, 16, 17, 18" + tags = "10, 11, 12, 13, 14, 15, 16, 18" )] pub sketch_state: ::core::option::Option, } @@ -915,8 +892,6 @@ pub mod sketch_envelope { Univmon(super::UnivMonState), #[prost(message, tag = "16")] Hydra(super::HydraState), - #[prost(message, tag = "17")] - Coco(super::CocoSketchState), #[prost(message, tag = "18")] Elastic(super::ElasticState), } diff --git a/tests/xtest_consumer.rs b/tests/xtest_consumer.rs index 0113ff45..c702eb3f 100644 --- a/tests/xtest_consumer.rs +++ b/tests/xtest_consumer.rs @@ -10,7 +10,6 @@ //! ddsketch.pb — DDSketchState with alpha + bucket array //! hll.pb — HyperLogLogState (ErtlMLE variant) //! countsketch.pb — CountSketchState with float64 signed counters -//! coco.pb — CocoSketchState (hash+val+hasKey buckets) //! elastic.pb — ElasticState (heavy buckets + light CountMin) //! univmon.pb — UnivMonState (layered CountSketch + TopK heaps) //! hydra.pb — HydraState (CM-cell grid) @@ -283,45 +282,6 @@ fn cross_language_proto() { all_ok = false; } - // ----------------------------------------------------------------------- - // CocoSketch - // ----------------------------------------------------------------------- - println!(); - println!("[CocoSketch] Step 1/3 — Read coco.pb"); - let bytes = read_file(in_dir.join("coco.pb")); - let env = SketchEnvelope::decode(bytes.as_slice()).expect("decode coco envelope"); - - println!( - "[CocoSketch] Step 2/3 — Validate envelope (format_version={}, producer={})", - env.format_version, - env.producer.as_ref().map_or("?", |p| &p.library) - ); - - let coco_state = match env.sketch_state { - Some(sketch_envelope::SketchState::Coco(ref s)) => s.clone(), - other => panic!("expected CocoSketch sketch_state, got {other:?}"), - }; - - println!( - "[CocoSketch] d={} width={} buckets={}", - coco_state.d, - coco_state.width, - coco_state.hashes.len() - ); - - // Query "coco:hot" — inserted with val=500. - // Go uses: hash = common.Hash64([]byte("coco:hot")) = xxh3_64_seeded(seedList[0], ...) - // DeriveIndex(hash, row, width): col = (hash >> (row * maskBitsForWidth(width))) & (width-1) - let coco_hash = xxh3_64_seeded(SEED_0, b"coco:hot"); - let coco_est = coco_estimate(&coco_state, coco_hash); - println!("[CocoSketch] Step 3/3 — 'coco:hot' est = {coco_est} (expect ≥ 500)"); - if coco_est >= 500 { - println!("[CocoSketch] PASS"); - } else { - eprintln!("[CocoSketch] FAIL: estimate {coco_est} < 500"); - all_ok = false; - } - // ----------------------------------------------------------------------- // ElasticSketch // ----------------------------------------------------------------------- @@ -698,21 +658,6 @@ fn col_bits(cols: usize) -> u64 { width.trailing_zeros() as u64 } -/// maskBitsForWidth — mirrors Go's common.maskBitsForWidth. -/// Returns the number of bits needed to represent (width-1). -fn mask_bits_for_width(width: usize) -> u64 { - if width <= 1 { - return 1; - } - let mut u = width - 1; - let mut bits = 0u64; - while u > 0 { - bits += 1; - u >>= 1; - } - bits -} - /// XXH3-64 with explicit seed, matching Go's `hash64_seeded(seed, key)`. fn xxh3_64_seeded(seed: u64, data: &[u8]) -> u64 { XxHash3_64::oneshot_with_seed(seed, data) @@ -835,31 +780,6 @@ fn median_f64(v: &mut [f64]) -> f64 { } } -// --------------------------------------------------------------------------- -// CocoSketch estimate -// --------------------------------------------------------------------------- -// Mirrors Go's CocoSketch.EstimateHash: -// for row i: col = DeriveIndex(hash, i, width); if b.HasKey && b.Hash==hash: total += b.Val -// DeriveIndex(hash, row, width) = (hash >> (row * maskBitsForWidth(width))) & (width-1) - -fn coco_estimate(state: &CocoSketchState, hash: u64) -> u64 { - let d = state.d as usize; - let width = state.width as usize; - let mbw = mask_bits_for_width(width); - let mask = (width as u64) - 1; - let mut total = 0u64; - - for i in 0..d { - let shift = (i as u64) * mbw; - let col = ((hash >> shift) & mask) as usize; - let idx = i * width + col; - if state.has_keys[idx] && state.hashes[idx] == hash { - total += state.vals[idx]; - } - } - total -} - // --------------------------------------------------------------------------- // ElasticSketch query // --------------------------------------------------------------------------- diff --git a/tools/gen-proto/src/main.rs b/tools/gen-proto/src/main.rs index 1de6ad15..7fb4d9b3 100644 --- a/tools/gen-proto/src/main.rs +++ b/tools/gen-proto/src/main.rs @@ -35,7 +35,6 @@ const PROTO_FILES: &[&str] = &[ "ddsketch/ddsketch.proto", "univmon/univmon.proto", "hydra/hydra.proto", - "cocosketch/cocosketch.proto", "elasticsketch/elasticsketch.proto", "sketchlib.proto", ]; From fde0256836f34ec500cd823c9c61a9a31ff0d61a Mon Sep 17 00:00:00 2001 From: GordonYuanyc Date: Fri, 2 Oct 2026 00:59:39 -0600 Subject: [PATCH 15/36] test(coco): add an ASAPv1 golden fixture Bump asapv1_golden to the commit adding coco_3x8.hex, a 3x8 bucket table set directly: values at both ends of every msgpack uint width, empty, 31-byte, 32-byte and multi-byte UTF-8 keys, and unoccupied buckets. Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 2 ++ asapv1_golden | 2 +- docs/asapv1_wire_format.md | 2 +- docs/message_pack_format.md | 4 +-- docs/tests.md | 1 + tests/asapv1_golden.rs | 64 +++++++++++++++++++++++++++++++++++-- 6 files changed, 69 insertions(+), 6 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index f35c5ce5..898d5544 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -66,6 +66,8 @@ signals a backwards-compatible change. worker's seed and the aggregator's from one seed. The unseeded constructors draw from the thread generator. The seed is not serialized, and the ASAPv1 payload is unchanged. +- **ASAPv1 golden fixture for Coco** (`0c 00`): `asapv1_golden/coco_3x8.hex`, + a 3x8 bucket table set directly, checked by `tests/asapv1_golden.rs`. ### Changed diff --git a/asapv1_golden b/asapv1_golden index 8287a4ae..f9bbc3d7 160000 --- a/asapv1_golden +++ b/asapv1_golden @@ -1 +1 @@ -Subproject commit 8287a4ae04e6cfbfb3ed33dafbf1dac77d8f7fc1 +Subproject commit f9bbc3d7f7a093e046d5389f9d3afc26b8c5285d diff --git a/docs/asapv1_wire_format.md b/docs/asapv1_wire_format.md index fd14e70a..7f1853c7 100644 --- a/docs/asapv1_wire_format.md +++ b/docs/asapv1_wire_format.md @@ -1341,7 +1341,7 @@ Good direction (more compact, higher fidelity, less Rust-internal duplication), 2. **Golden byte-vector fixtures** in one shared repo, [`sketchlib-golden-bytes`](https://github.com/ProjectASAP/sketchlib-golden-bytes), which each implementation mounts at `asapv1_golden/`; both languages decode and re-encode them byte-identically. These replace the `portable`-as-oracle round-trip test. 3. **This registry**, mirrored, never independently allocated. -Fixtures exist for six `kind_id`s — HLL's three estimators, Count-Min, Count Sketch and compact KLL — and `sketchlib-go` mirrors those. Every other kind here has a payload and **no fixture**, so this document is its only contract; `asapv1_golden/README.md` lists the gap. `portable` carries its own Go goldens. +Fixtures exist for seven `kind_id`s — HLL's three estimators, Count-Min, Count Sketch, compact KLL and Coco. Every other kind here has a payload and **no fixture**, so this document is its only contract; `asapv1_golden/README.md` lists the gap. `portable` carries its own Go goldens. **Hash profile on the Go side.** Rust derives the hash spec from a generic `HashProfile` bound on the hasher type; Go has no generic hasher type, so there is nothing to derive from. diff --git a/docs/message_pack_format.md b/docs/message_pack_format.md index 12a2b91c..ae238e7e 100644 --- a/docs/message_pack_format.md +++ b/docs/message_pack_format.md @@ -169,8 +169,8 @@ decode → re-encode them byte-identically. The `kind_id` registry is mirrored verbatim with Go's `wire/asapmsgpack/magic_ids.go`, never independently allocated. -That proof covers the six `kind_id`s a fixture exists for — HLL's three -estimators, Count-Min, Count Sketch and compact KLL. Every other implemented +That proof covers the seven `kind_id`s a fixture exists for — HLL's three +estimators, Count-Min, Count Sketch, compact KLL and Coco. Every other implemented kind has **no ASAPv1 golden and therefore no ASAPv1 drift guard**; [its README](https://github.com/ProjectASAP/sketchlib-golden-bytes#coverage) lists what is covered. The older `portable` path is guarded separately, by the `sketchlib-go` goldens diff --git a/docs/tests.md b/docs/tests.md index e577a012..26dbd0de 100644 --- a/docs/tests.md +++ b/docs/tests.md @@ -2123,6 +2123,7 @@ Test file: [`tests/asapv1_golden.rs`](../tests/asapv1_golden.rs) | --- | --- | --- | | `cms_f64_fast_2x3_matches_golden` | A known f64 fast-path Count-Min matrix serializes to its checked-in golden bytes. | Verifies `CountMin::, FastPath>::from_storage` over the `2x3` matrix `[[0.0, 1.5, 2.25], [3.75, 4.125, 5.0625]]` serializes to `asapv1_golden/cms_f64_fast_2x3.hex`, and that decoding that fixture recovers the six cells in row-major order and re-serializes to the same bytes. | | `cms_i64_regular_2x3_matches_golden` | A known i64 regular-path Count-Min matrix serializes to its checked-in golden bytes. | Verifies `CountMin::, RegularPath>::from_storage` over the `2x3` matrix `[[0, 1, 127], [128, 300, 65536]]` serializes to `asapv1_golden/cms_i64_regular_2x3.hex`, and that decoding the fixture reports `rows()` `2`, `cols()` `3`, the six cells in row-major order, and a byte-identical re-serialization. | +| `coco_3x8_matches_golden` | A known Coco bucket table serializes to its checked-in golden bytes. | Sets 14 of the 24 buckets of `Coco::init_with_size(8, 3)` directly — each key in the column its row hashes it to, values spanning positive fixint through uint64 at both ends of each width, keys including `""`, a 31-byte and a 32-byte key and a multi-byte UTF-8 key, and `"zero"` holding `0` — verifies it serializes to `asapv1_golden/coco_3x8.hex`, and that decoding the fixture reports `d` `3`, `w` `8`, all 24 buckets in row-major order with the other ten unoccupied, and a byte-identical re-serialization. | | `cs_i32_regular_2x4_matches_golden` | The i32 counter type reaches the metadata, and an i32 golden is not an i64 sketch. | Verifies `Count::, RegularPath>::from_storage` over `[[0, 127, 128, 65536], [-1, -33, -32768, -2147483648]]` cast to `i32` serializes to `asapv1_golden/cs_i32_regular_2x4.hex`, that decoding it recovers the eight cells and re-serializes identically, that the i32 and i64 regular fixtures have equal length yet differ, and that the i32 fixture fails to decode as `Count, RegularPath>`. | | `cs_i64_fast_2x4_matches_golden` | The fast-path mode reaches the bytes, so the same matrix encodes differently per path. | Verifies `Count::, FastPath>::from_storage` over `[[0, 127, 128, 65536], [-1, -33, -32768, -2147483648]]` serializes to `asapv1_golden/cs_i64_fast_2x4.hex`, that decoding it recovers the eight cells and re-serializes identically, and that the fixture differs from `cs_i64_regular_2x4.hex` built from the same matrix. | | `cs_i64_regular_2x4_matches_golden` | A known signed Count-Sketch matrix serializes to its checked-in golden bytes. | Verifies `Count::, RegularPath>::from_storage` over `[[0, 127, 128, 65536], [-1, -33, -32768, -2147483648]]` — sweeping the msgpack integer widths in both directions — serializes to `asapv1_golden/cs_i64_regular_2x4.hex`, and that decoding the fixture reports `rows()` `2`, `cols()` `4`, the eight cells, and a byte-identical re-serialization. | diff --git a/tests/asapv1_golden.rs b/tests/asapv1_golden.rs index 30b41651..69f9e307 100644 --- a/tests/asapv1_golden.rs +++ b/tests/asapv1_golden.rs @@ -9,8 +9,8 @@ //! See `asapv1_golden/README.md`. use asap_sketchlib::{ - Classic, Count, CountMin, ErtlMLE, FastPath, HllSketch, HllVariant, HyperLogLogHIPP12, - HyperLogLogP12, KLL, MessagePackCodec, RegularPath, Vector2D, + Classic, Coco, CocoBucket, Count, CountMin, ErtlMLE, FastPath, HllSketch, HllVariant, + HyperLogLogHIPP12, HyperLogLogP12, KLL, MessagePackCodec, RegularPath, Vector2D, }; fn decode_hex(s: &str) -> Vec { @@ -32,6 +32,7 @@ const GOLDEN_CS_FAST: &str = include_str!("../asapv1_golden/cs_i64_fast_2x4.hex" const GOLDEN_CS_I32: &str = include_str!("../asapv1_golden/cs_i32_regular_2x4.hex"); const GOLDEN_KLL_F64: &str = include_str!("../asapv1_golden/kll_f64_k200.hex"); const GOLDEN_KLL_I64: &str = include_str!("../asapv1_golden/kll_i64_k200.hex"); +const GOLDEN_COCO: &str = include_str!("../asapv1_golden/coco_3x8.hex"); /// The known P12 register pattern shared by all three HLL fixtures. fn p12_registers() -> Vec { @@ -287,3 +288,62 @@ fn kll_i64_k200_matches_golden() { assert_eq!(decoded.quantile(0.0), 1.0); assert_eq!(decoded.quantile(1.0), 50.0); } + +// --------------------------------------------------------------------------- +// Coco: build a known bucket table -> serialize == golden, round-trips. +// --------------------------------------------------------------------------- + +/// The occupied buckets of the 3x8 Coco fixture as `(row, col, key, value)`; +/// every other bucket is unoccupied. Each key sits in the column its row hashes +/// it to, which the codec checks on both sides. +const COCO_CELLS: [(usize, usize, &str, u64); 14] = [ + (0, 0, "uint16-max", 65535), + (0, 1, "fixint-max", 127), + (0, 2, "clé-ünïcode-流量", 4), + (0, 4, "fixstr-max-31-bytes-0123456789a", 2), + (0, 5, "uint8-min", 128), + (0, 6, "uint64-max", u64::MAX), + (0, 7, "", 1), + (1, 0, "uint64-min", 4294967296), + (1, 1, "uint32-min", 65536), + (1, 3, "str8-min-32-bytes-0123456789abcd", 3), + (1, 5, "zero", 0), + (1, 6, "uint32-max", 4294967295), + (1, 7, "uint8-max", 255), + (2, 5, "uint16-min", 256), +]; + +/// The fixture's 24 buckets in row-major order. +fn coco_buckets() -> Vec<(Option, u64)> { + let mut buckets = vec![(None, 0); 24]; + for (r, c, key, val) in COCO_CELLS { + buckets[r * 8 + c] = (Some(key.to_string()), val); + } + buckets +} + +#[test] +fn coco_3x8_matches_golden() { + let want = decode_hex(GOLDEN_COCO); + + let mut sketch: Coco = Coco::init_with_size(8, 3); + for (r, c, key, val) in COCO_CELLS { + sketch.table[r][c] = CocoBucket { + full_key: Some(key.to_string()), + val, + }; + } + let got = sketch.serialize_to_bytes().expect("serialize"); + assert_eq!(got, want, "Coco bytes diverge from golden"); + + let decoded: Coco = Coco::deserialize_from_bytes(&want).expect("decode"); + assert_eq!((decoded.d, decoded.w), (3, 8)); + let cells: Vec<(Option, u64)> = decoded + .table + .as_slice() + .iter() + .map(|b| (b.full_key.clone(), b.val)) + .collect(); + assert_eq!(cells, coco_buckets()); + assert_eq!(decoded.serialize_to_bytes().expect("re-serialize"), want); +} From f7b65df50a852deceb70f1c2c935afe636a40290 Mon Sep 17 00:00:00 2001 From: GordonYuanyc Date: Fri, 2 Oct 2026 00:59:53 -0600 Subject: [PATCH 16/36] refactor(elastic)!: drop the proto format and add ASAPv1 goldens Delete proto/elasticsketch/elasticsketch.proto (ElasticState) and reserve SketchEnvelope field 18 ("elastic"); regenerate src/proto/generated. Drop the Elastic check from tests/xtest_consumer.rs. ASAPv1 (0x0b 0x00) is Elastic's only serialization. Add the elastic_4b_2x4 and elastic_4b_2x4_stale golden fixtures (bump asapv1_golden) and their cases in tests/asapv1_golden.rs. Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 11 +++ asapv1_golden | 2 +- docs/asapv1_wire_format.md | 4 +- docs/features.md | 2 +- docs/message_pack_format.md | 4 +- docs/tests.md | 4 +- proto/elasticsketch/elasticsketch.proto | 41 ----------- proto/sketchlib.proto | 5 +- src/proto/generated/sketchlib.v1.rs | 31 +------- tests/asapv1_golden.rs | 91 ++++++++++++++++++++++- tests/xtest_consumer.rs | 96 ------------------------- tools/gen-proto/src/main.rs | 1 - 12 files changed, 113 insertions(+), 179 deletions(-) delete mode 100644 proto/elasticsketch/elasticsketch.proto diff --git a/CHANGELOG.md b/CHANGELOG.md index 80d85f98..b21c27da 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -66,6 +66,8 @@ signals a backwards-compatible change. worker's seed and the aggregator's from one seed. The unseeded constructors draw from the thread generator. The seed is not serialized, and the ASAPv1 payload is unchanged. +- **ASAPv1 golden fixtures for Elastic**: `elastic_4b_2x4` and + `elastic_4b_2x4_stale` in `asapv1_golden/`, checked by `tests/asapv1_golden.rs`. ### Changed @@ -88,6 +90,15 @@ signals a backwards-compatible change. as an empty window like any other, and that window takes a place in `max_windows`. +### Removed + +- **BREAKING: Elastic's protobuf form.** `proto/elasticsketch/elasticsketch.proto`, + its message `ElasticState` (`asap_sketchlib::proto::sketchlib::ElasticState`), + and the `SketchEnvelope` oneof variant `elastic` + (`sketch_envelope::SketchState::Elastic`, field 18, now reserved) are gone. + `Elastic::serialize_to_bytes` / `deserialize_from_bytes` (ASAPv1, `0x0b 0x00`) + is Elastic's only serialization. + ### Fixed - `KLLDynamic` panicked on every quantile query once a NaN had entered the diff --git a/asapv1_golden b/asapv1_golden index 8287a4ae..0f0d62b6 160000 --- a/asapv1_golden +++ b/asapv1_golden @@ -1 +1 @@ -Subproject commit 8287a4ae04e6cfbfb3ed33dafbf1dac77d8f7fc1 +Subproject commit 0f0d62b68e5686a3f03b6390f90f7bd2ea36c927 diff --git a/docs/asapv1_wire_format.md b/docs/asapv1_wire_format.md index fd14e70a..82c05ae4 100644 --- a/docs/asapv1_wire_format.md +++ b/docs/asapv1_wire_format.md @@ -1341,14 +1341,14 @@ Good direction (more compact, higher fidelity, less Rust-internal duplication), 2. **Golden byte-vector fixtures** in one shared repo, [`sketchlib-golden-bytes`](https://github.com/ProjectASAP/sketchlib-golden-bytes), which each implementation mounts at `asapv1_golden/`; both languages decode and re-encode them byte-identically. These replace the `portable`-as-oracle round-trip test. 3. **This registry**, mirrored, never independently allocated. -Fixtures exist for six `kind_id`s — HLL's three estimators, Count-Min, Count Sketch and compact KLL — and `sketchlib-go` mirrors those. Every other kind here has a payload and **no fixture**, so this document is its only contract; `asapv1_golden/README.md` lists the gap. `portable` carries its own Go goldens. +Fixtures exist for seven `kind_id`s — HLL's three estimators, Count-Min, Count Sketch, compact KLL and Elastic — and `sketchlib-go` mirrors all but Elastic. Every other kind here has a payload and **no fixture**, so this document is its only contract; `asapv1_golden/README.md` lists the gap. `portable` carries its own Go goldens. **Hash profile on the Go side.** Rust derives the hash spec from a generic `HashProfile` bound on the hasher type; Go has no generic hasher type, so there is nothing to derive from. On the Go side the profile is simply **written into** the metadata on encode and **read from** it on decode. Go MUST validate the profile it reads (same fail-closed intent as Rust): a sketch is only mergeable/queryable if its `hash_profile_id` + seeds match the profile Go is prepared to reproduce. -Sequencing: (2) covers six `kind_id`s, and the one `native bytes == portable bytes` test is HLL-only, since no other `portable` type emits ASAPv1 bytes. +Sequencing: (2) covers seven `kind_id`s, and the one `native bytes == portable bytes` test is HLL-only, since no other `portable` type emits ASAPv1 bytes. Retire `portable` once the fixtures cover the rest. --- diff --git a/docs/features.md b/docs/features.md index 64f296a3..2049da12 100644 --- a/docs/features.md +++ b/docs/features.md @@ -112,7 +112,7 @@ MessagePack (`rmp-serde`) support. **serde support** means the type derives `Ser | EHSketchList | Yes | Yes | | ExponentialHistogram | No | Yes | -Protobuf (prost): `.proto` definitions exist for CountMin, Count, HLL, DDSketch, KLL, Elastic, Coco, Hydra, and UnivMon. Rust conversion code is in progress. +Protobuf (prost): `.proto` definitions exist for CountMin, Count, HLL, DDSketch, KLL, Coco, Hydra, and UnivMon. Rust conversion code is in progress. ### API Stability diff --git a/docs/message_pack_format.md b/docs/message_pack_format.md index 12a2b91c..7b25c084 100644 --- a/docs/message_pack_format.md +++ b/docs/message_pack_format.md @@ -169,8 +169,8 @@ decode → re-encode them byte-identically. The `kind_id` registry is mirrored verbatim with Go's `wire/asapmsgpack/magic_ids.go`, never independently allocated. -That proof covers the six `kind_id`s a fixture exists for — HLL's three -estimators, Count-Min, Count Sketch and compact KLL. Every other implemented +That proof covers the seven `kind_id`s a fixture exists for — HLL's three +estimators, Count-Min, Count Sketch, compact KLL and Elastic. Every other implemented kind has **no ASAPv1 golden and therefore no ASAPv1 drift guard**; [its README](https://github.com/ProjectASAP/sketchlib-golden-bytes#coverage) lists what is covered. The older `portable` path is guarded separately, by the `sketchlib-go` goldens diff --git a/docs/tests.md b/docs/tests.md index ccb1571c..a5f8355c 100644 --- a/docs/tests.md +++ b/docs/tests.md @@ -2126,6 +2126,8 @@ Test file: [`tests/asapv1_golden.rs`](../tests/asapv1_golden.rs) | `cs_i32_regular_2x4_matches_golden` | The i32 counter type reaches the metadata, and an i32 golden is not an i64 sketch. | Verifies `Count::, RegularPath>::from_storage` over `[[0, 127, 128, 65536], [-1, -33, -32768, -2147483648]]` cast to `i32` serializes to `asapv1_golden/cs_i32_regular_2x4.hex`, that decoding it recovers the eight cells and re-serializes identically, that the i32 and i64 regular fixtures have equal length yet differ, and that the i32 fixture fails to decode as `Count, RegularPath>`. | | `cs_i64_fast_2x4_matches_golden` | The fast-path mode reaches the bytes, so the same matrix encodes differently per path. | Verifies `Count::, FastPath>::from_storage` over `[[0, 127, 128, 65536], [-1, -33, -32768, -2147483648]]` serializes to `asapv1_golden/cs_i64_fast_2x4.hex`, that decoding it recovers the eight cells and re-serializes identically, and that the fixture differs from `cs_i64_regular_2x4.hex` built from the same matrix. | | `cs_i64_regular_2x4_matches_golden` | A known signed Count-Sketch matrix serializes to its checked-in golden bytes. | Verifies `Count::, RegularPath>::from_storage` over `[[0, 127, 128, 65536], [-1, -33, -32768, -2147483648]]` — sweeping the msgpack integer widths in both directions — serializes to `asapv1_golden/cs_i64_regular_2x4.hex`, and that decoding the fixture reports `rows()` `2`, `cols()` `4`, the eight cells, and a byte-identical re-serialization. | +| `elastic_4b_2x4_matches_golden` | A known Elastic heavy table and light layer serialize to their golden bytes. | Sets a 4-bucket `2x4` sketch's buckets to a free one, a 31-byte flow id with votes `127`/`128`, an empty flow id with votes `1`/`65535` and the eviction flag, and a 32-byte flow id with votes `i32::MAX`/`256` and the flag, and its light cells to `[[0, 255, 65536, i32::MAX], [-1, -33, -32768, i32::MIN]]`; verifies the bytes equal `asapv1_golden/elastic_4b_2x4.hex`, and that decoding the fixture recovers every bucket, `bktlen` `4`, the light dimensions and cells, and a byte-identical re-serialization. | +| `elastic_4b_2x4_stale_matches_golden` | `stale_copies` reaches the bytes, and nothing else differs. | Builds the same state on a 2-bucket sketch after `expand_heavy()`, verifies the bytes equal `asapv1_golden/elastic_4b_2x4_stale.hex`, that decoding recovers the same state and re-serializes identically, and that the two fixtures have equal length and differ in exactly one byte, `0xc2` against `0xc3`. | | `hll_classic_p12_matches_golden` | A known P12 Classic register pattern serializes to its golden bytes on both the portable and native paths. | Builds `HllSketch::from_raw(HllVariant::Regular, 12, regs, 0.0, 0.0, 0.0)` where `regs` is 4,096 zeros with `[0] = 1`, `[1] = 7`, `[100] = 42` and `[4095] = 3`, verifies `to_msgpack()` equals `asapv1_golden/hll_classic_p12.hex`, that `HyperLogLogP12::::deserialize_from_bytes` on the fixture recovers those registers and re-serializes identically, and that `HllSketch::from_msgpack` reports the same registers, `precision` `12` and `HllVariant::Regular`. | | `hll_ertl_mle_p12_matches_golden` | The Datafusion variant tag reaches the bytes for the same P12 register pattern. | Builds `HllSketch::from_raw(HllVariant::Datafusion, 12, regs, 0.0, 0.0, 0.0)` over the shared P12 pattern (`[0] = 1`, `[1] = 7`, `[100] = 42`, `[4095] = 3` in 4,096 registers), verifies `to_msgpack()` equals `asapv1_golden/hll_ertl_mle_p12.hex`, that `HyperLogLogP12::::deserialize_from_bytes` recovers the registers and re-serializes identically, and that `HllSketch::from_msgpack` reports `HllVariant::Datafusion`. | | `hll_hip_p12_matches_golden` | The HIP variant's three running scalars travel on the wire beside its registers. | Builds `HllSketch::from_raw(HllVariant::Hip, 12, regs, 1.5, 2.5, 3.0)` over the shared P12 pattern, verifies `to_msgpack()` equals `asapv1_golden/hll_hip_p12.hex`, that `HyperLogLogHIPP12::deserialize_from_bytes` re-serializes the fixture identically, and that `HllSketch::from_msgpack` reports the registers, `HllVariant::Hip`, `hip_kxq0` `1.5`, `hip_kxq1` `2.5` and `hip_est` `3.0`. | @@ -2202,7 +2204,7 @@ Test file: [`tests/xtest_consumer.rs`](../tests/xtest_consumer.rs) | test_name | test_description | what_is_tested | | --- | --- | --- | -| `cross_language_proto` | Protobuf envelopes read from the cross-language fixture directory answer their point queries in band. | Asserts nothing unless fixtures are available: with no `XTEST_DIR` and no discoverable producer directory holding `tests/cross_language/xtest_producer_test.go`, it prints a skip line and returns. With fixtures it decodes nine `SketchEnvelope` files and records a pass or failure for each — `countmin.pb` min-frequency for `item:42` at least `101`, `kll.pb` p50 within 5% of `5000` and p99 within 5% of `9900`, `ddsketch.pb` the same two quantiles within 2%, `hll.pb` an ErtlMLE cardinality inside `[40_000, 65_000]`, `countsketch.pb` `cs:hot` at least `200`, `coco.pb` `coco:hot` at least `500`, `elastic.pb` `elephant` at least `900`, `univmon.pb` a cardinality inside `[1_000, 15_000]`, `hydra.pb` `hydra:42` at least `51` — using the hash seeds `0xcafe3553`, `0x6a09e667` and `0xbb67ae85`, and, when `countmin_sampled.pb` and `hll_sampled.pb` exist, requires `effective_sample_p` of `0.1` and a `rescale_count` by `1/p` within 5% of `100_000` and 6% of `200_000`. A single `assert!` at the end fails if any check failed; a missing or mistyped envelope panics on the spot. | +| `cross_language_proto` | Protobuf envelopes read from the cross-language fixture directory answer their point queries in band. | Asserts nothing unless fixtures are available: with no `XTEST_DIR` and no discoverable producer directory holding `tests/cross_language/xtest_producer_test.go`, it prints a skip line and returns. With fixtures it decodes eight `SketchEnvelope` files and records a pass or failure for each — `countmin.pb` min-frequency for `item:42` at least `101`, `kll.pb` p50 within 5% of `5000` and p99 within 5% of `9900`, `ddsketch.pb` the same two quantiles within 2%, `hll.pb` an ErtlMLE cardinality inside `[40_000, 65_000]`, `countsketch.pb` `cs:hot` at least `200`, `coco.pb` `coco:hot` at least `500`, `univmon.pb` a cardinality inside `[1_000, 15_000]`, `hydra.pb` `hydra:42` at least `51` — using the hash seeds `0xcafe3553` and `0xbb67ae85`, and, when `countmin_sampled.pb` and `hll_sampled.pb` exist, requires `effective_sample_p` of `0.1` and a `rescale_count` by `1/p` within 5% of `100_000` and 6% of `200_000`. A single `assert!` at the end fails if any check failed; a missing or mistyped envelope panics on the spot. | ### HLL Custom Precision diff --git a/proto/elasticsketch/elasticsketch.proto b/proto/elasticsketch/elasticsketch.proto deleted file mode 100644 index 94f1f465..00000000 --- a/proto/elasticsketch/elasticsketch.proto +++ /dev/null @@ -1,41 +0,0 @@ -// elasticsketch.proto — Portable wire format for Elastic Sketch state. - -syntax = "proto3"; - -package sketchlib.v1; - -import "countminsketch/countminsketch.proto"; - -option go_package = "github.com/ProjectASAP/sketchlib-go/proto/elasticsketch;elasticsketchpb"; - -// ============================================================================ -// ElasticSketch -// ============================================================================ - -// ElasticState is the portable state of an Elastic Sketch. -// -// The Elastic Sketch has a heavy part (one bucket per index) and a light -// Count-Min layer. The heavy part is serialized as parallel arrays. -// The light layer is stored as a CountMinState with FLOAT64 counters. -message ElasticState { - // Number of heavy buckets. - uint32 bucket_count = 1; - - // Per-bucket flow identifiers (strings), length = bucket_count. - // Empty string means the bucket is unoccupied. - repeated string flow_ids = 2; - - // Positive vote counts per bucket, length = bucket_count. - repeated sint32 vote_pos = 3 [packed = true]; - - // Negative vote counts per bucket, length = bucket_count. - repeated sint32 vote_neg = 4 [packed = true]; - - // Whether each bucket is in eviction state, length = bucket_count. - repeated bool evictions = 5 [packed = true]; - - // Light Count-Min layer (FLOAT64 counters). - CountMinState light = 6; - - reserved 7 to 15; -} diff --git a/proto/sketchlib.proto b/proto/sketchlib.proto index 0a6d8193..864a15d1 100644 --- a/proto/sketchlib.proto +++ b/proto/sketchlib.proto @@ -37,7 +37,6 @@ import "ddsketch/ddsketch.proto"; import "univmon/univmon.proto"; import "hydra/hydra.proto"; import "cocosketch/cocosketch.proto"; -import "elasticsketch/elasticsketch.proto"; option go_package = "github.com/ProjectASAP/sketchlib-go/proto/sketch_envelope;sketch_envelopepb"; @@ -74,9 +73,11 @@ message SketchEnvelope { UnivMonState univmon = 15; HydraState hydra = 16; CocoSketchState coco = 17; - ElasticState elastic = 18; } + reserved 18; + reserved "elastic"; + reserved 19 to 31; // Reserved for future sketch types. // Geometric / NitroSketch sampling probability for the producing sketch. diff --git a/src/proto/generated/sketchlib.v1.rs b/src/proto/generated/sketchlib.v1.rs index 2c098d3f..0f109fd8 100644 --- a/src/proto/generated/sketchlib.v1.rs +++ b/src/proto/generated/sketchlib.v1.rs @@ -823,33 +823,6 @@ pub struct CocoSketchState { #[prost(bool, repeated, tag = "5")] pub has_keys: ::prost::alloc::vec::Vec, } -/// ElasticState is the portable state of an Elastic Sketch. -/// -/// The Elastic Sketch has a heavy part (one bucket per index) and a light -/// Count-Min layer. The heavy part is serialized as parallel arrays. -/// The light layer is stored as a CountMinState with FLOAT64 counters. -#[derive(Clone, PartialEq, ::prost::Message)] -pub struct ElasticState { - /// Number of heavy buckets. - #[prost(uint32, tag = "1")] - pub bucket_count: u32, - /// Per-bucket flow identifiers (strings), length = bucket_count. - /// Empty string means the bucket is unoccupied. - #[prost(string, repeated, tag = "2")] - pub flow_ids: ::prost::alloc::vec::Vec<::prost::alloc::string::String>, - /// Positive vote counts per bucket, length = bucket_count. - #[prost(sint32, repeated, tag = "3")] - pub vote_pos: ::prost::alloc::vec::Vec, - /// Negative vote counts per bucket, length = bucket_count. - #[prost(sint32, repeated, tag = "4")] - pub vote_neg: ::prost::alloc::vec::Vec, - /// Whether each bucket is in eviction state, length = bucket_count. - #[prost(bool, repeated, tag = "5")] - pub evictions: ::prost::alloc::vec::Vec, - /// Light Count-Min layer (FLOAT64 counters). - #[prost(message, optional, tag = "6")] - pub light: ::core::option::Option, -} /// SketchEnvelope is the portable container for all sketch transfers. /// Producers fill exactly one field inside sketch_state. /// Consumers check format_version first, then dispatch on the oneof. @@ -892,7 +865,7 @@ pub struct SketchEnvelope { /// The sketch payload. Exactly one field must be set. #[prost( oneof = "sketch_envelope::SketchState", - tags = "10, 11, 12, 13, 14, 15, 16, 17, 18" + tags = "10, 11, 12, 13, 14, 15, 16, 17" )] pub sketch_state: ::core::option::Option, } @@ -917,7 +890,5 @@ pub mod sketch_envelope { Hydra(super::HydraState), #[prost(message, tag = "17")] Coco(super::CocoSketchState), - #[prost(message, tag = "18")] - Elastic(super::ElasticState), } } diff --git a/tests/asapv1_golden.rs b/tests/asapv1_golden.rs index 30b41651..59576aba 100644 --- a/tests/asapv1_golden.rs +++ b/tests/asapv1_golden.rs @@ -9,8 +9,8 @@ //! See `asapv1_golden/README.md`. use asap_sketchlib::{ - Classic, Count, CountMin, ErtlMLE, FastPath, HllSketch, HllVariant, HyperLogLogHIPP12, - HyperLogLogP12, KLL, MessagePackCodec, RegularPath, Vector2D, + Classic, Count, CountMin, Elastic, ErtlMLE, FastPath, HeavyBucket, HllSketch, HllVariant, + HyperLogLogHIPP12, HyperLogLogP12, KLL, MessagePackCodec, RegularPath, Vector2D, }; fn decode_hex(s: &str) -> Vec { @@ -32,6 +32,8 @@ const GOLDEN_CS_FAST: &str = include_str!("../asapv1_golden/cs_i64_fast_2x4.hex" const GOLDEN_CS_I32: &str = include_str!("../asapv1_golden/cs_i32_regular_2x4.hex"); const GOLDEN_KLL_F64: &str = include_str!("../asapv1_golden/kll_f64_k200.hex"); const GOLDEN_KLL_I64: &str = include_str!("../asapv1_golden/kll_i64_k200.hex"); +const GOLDEN_ELASTIC: &str = include_str!("../asapv1_golden/elastic_4b_2x4.hex"); +const GOLDEN_ELASTIC_STALE: &str = include_str!("../asapv1_golden/elastic_4b_2x4_stale.hex"); /// The known P12 register pattern shared by all three HLL fixtures. fn p12_registers() -> Vec { @@ -287,3 +289,88 @@ fn kll_i64_k200_matches_golden() { assert_eq!(decoded.quantile(0.0), 1.0); assert_eq!(decoded.quantile(1.0), 50.0); } + +// --------------------------------------------------------------------------- +// Elastic: heavy buckets and light counters set directly, never inserted. +// The two fixtures hold the same state and differ only by `stale_copies`. +// --------------------------------------------------------------------------- + +/// Heavy buckets `(flow_id, vote_pos, vote_neg, eviction)`: a free bucket, a +/// 31-byte fixstr id, an empty id and a 32-byte str8 id; the votes sweep +/// positive fixint / uint8 / uint16 / uint32. +const ELASTIC_HEAVY: [(&str, i32, i32, bool); 4] = [ + ("", 0, 0, false), + ("10.0.0.1:443>192.168.10.20:5123", 127, 128, false), + ("", 1, 65535, true), + ("10.0.0.1:443>192.168.10.20:51234", 2147483647, 256, true), +]; + +/// Light layer: row 0 sweeps the unsigned widths up to `i32::MAX`, row 1 the +/// signed widths down to `i32::MIN`. +const ELASTIC_LIGHT: [[i32; 4]; 2] = [[0, 255, 65536, 2147483647], [-1, -33, -32768, -2147483648]]; + +/// Sets the known heavy and light state on a 4-bucket, `2x4` sketch. +fn elastic_known_state(mut sketch: Elastic) -> Elastic { + assert_eq!(sketch.heavy.len(), ELASTIC_HEAVY.len()); + for (bucket, &(id, pos, neg, eviction)) in sketch.heavy.iter_mut().zip(&ELASTIC_HEAVY) { + *bucket = HeavyBucket { + flow_id: id.to_string(), + vote_pos: pos, + vote_neg: neg, + eviction, + }; + } + sketch.light = CountMin::from_storage(Vector2D::from_fn(2, 4, |r, c| ELASTIC_LIGHT[r][c])); + sketch +} + +fn assert_elastic_known_state(sketch: &Elastic) { + let buckets: Vec<(&str, i32, i32, bool)> = sketch + .heavy + .iter() + .map(|b| (b.flow_id.as_str(), b.vote_pos, b.vote_neg, b.eviction)) + .collect(); + assert_eq!(buckets, ELASTIC_HEAVY); + assert_eq!(sketch.bktlen, 4); + assert_eq!((sketch.light.rows(), sketch.light.cols()), (2, 4)); + let flat: Vec = ELASTIC_LIGHT.iter().flatten().copied().collect(); + assert_eq!(sketch.light.as_storage().as_slice(), flat.as_slice()); +} + +#[test] +fn elastic_4b_2x4_matches_golden() { + let want = decode_hex(GOLDEN_ELASTIC); + + let sketch = elastic_known_state(Elastic::init_with_dimensions(4, 2, 4)); + let got = sketch.serialize_to_bytes().expect("serialize"); + assert_eq!(got, want, "Elastic bytes diverge from golden"); + + let decoded = Elastic::deserialize_from_bytes(&want).expect("decode"); + assert_elastic_known_state(&decoded); + assert_eq!(decoded.serialize_to_bytes().expect("re-serialize"), want); +} + +#[test] +fn elastic_4b_2x4_stale_matches_golden() { + let want = decode_hex(GOLDEN_ELASTIC_STALE); + + let mut expanded: Elastic = Elastic::init_with_dimensions(2, 2, 4); + expanded.expand_heavy(); + let sketch = elastic_known_state(expanded); + let got = sketch.serialize_to_bytes().expect("serialize"); + assert_eq!(got, want, "Elastic stale-copies bytes diverge from golden"); + + let decoded = Elastic::deserialize_from_bytes(&want).expect("decode"); + assert_elastic_known_state(&decoded); + assert_eq!(decoded.serialize_to_bytes().expect("re-serialize"), want); + + let fresh = decode_hex(GOLDEN_ELASTIC); + assert_eq!(want.len(), fresh.len()); + let differing: Vec = (0..want.len()).filter(|&i| want[i] != fresh[i]).collect(); + assert_eq!( + differing.len(), + 1, + "the fixtures differ only by stale_copies" + ); + assert_eq!((fresh[differing[0]], want[differing[0]]), (0xc2, 0xc3)); +} diff --git a/tests/xtest_consumer.rs b/tests/xtest_consumer.rs index 0113ff45..1a07680f 100644 --- a/tests/xtest_consumer.rs +++ b/tests/xtest_consumer.rs @@ -11,7 +11,6 @@ //! hll.pb — HyperLogLogState (ErtlMLE variant) //! countsketch.pb — CountSketchState with float64 signed counters //! coco.pb — CocoSketchState (hash+val+hasKey buckets) -//! elastic.pb — ElasticState (heavy buckets + light CountMin) //! univmon.pb — UnivMonState (layered CountSketch + TopK heaps) //! hydra.pb — HydraState (CM-cell grid) //! @@ -322,44 +321,6 @@ fn cross_language_proto() { all_ok = false; } - // ----------------------------------------------------------------------- - // ElasticSketch - // ----------------------------------------------------------------------- - println!(); - println!("[ElasticSketch] Step 1/3 — Read elastic.pb"); - let bytes = read_file(in_dir.join("elastic.pb")); - let env = SketchEnvelope::decode(bytes.as_slice()).expect("decode elastic envelope"); - - println!( - "[ElasticSketch] Step 2/3 — Validate envelope (format_version={}, producer={})", - env.format_version, - env.producer.as_ref().map_or("?", |p| &p.library) - ); - - let elastic_state = match env.sketch_state { - Some(sketch_envelope::SketchState::Elastic(ref s)) => s.clone(), - other => panic!("expected ElasticSketch sketch_state, got {other:?}"), - }; - - println!( - "[ElasticSketch] bucket_count={} light_rows={} light_cols={}", - elastic_state.bucket_count, - elastic_state.light.as_ref().map_or(0, |l| l.rows), - elastic_state.light.as_ref().map_or(0, |l| l.cols) - ); - - // Query "elephant" — inserted 1000 times. - // Go uses CanonicalHashSeed = seedList[5] = 0x6a09e667 - let elephant_hash = xxh3_64_seeded(SEED_5, b"elephant"); - let elephant_est = elastic_query(&elastic_state, "elephant", elephant_hash); - println!("[ElasticSketch] Step 3/3 — 'elephant' est = {elephant_est} (expect ≥ 900)"); - if elephant_est >= 900 { - println!("[ElasticSketch] PASS"); - } else { - eprintln!("[ElasticSketch] FAIL: estimate {elephant_est} < 900"); - all_ok = false; - } - // ----------------------------------------------------------------------- // UnivMon // ----------------------------------------------------------------------- @@ -553,7 +514,6 @@ fn cross_language_proto() { // --------------------------------------------------------------------------- const SEED_0: u64 = 0xcafe3553; // seedList[0] — Hash64 / default hash -const SEED_5: u64 = 0x6a09e667; // seedList[5] — CanonicalHashSeed const SEED_6: u64 = 0xbb67ae85; // seedList[6] — defaultHydraSeed // --------------------------------------------------------------------------- @@ -860,62 +820,6 @@ fn coco_estimate(state: &CocoSketchState, hash: u64) -> u64 { total } -// --------------------------------------------------------------------------- -// ElasticSketch query -// --------------------------------------------------------------------------- -// Mirrors Go's ElasticSketch.queryLocked: -// hash = HashIt(CanonicalHashSeed, []byte(id)) → SEED_5 -// idx = hash % bucket_count -// if flow_ids[idx] == id: if !eviction: return vote_pos[idx] -// else: return vote_pos[idx] + light_estimate -// else: return light_estimate -// Light layer: rows=5, cols=2048, bits=11, mask=2047 -// col_r = (hash >> (r * 11)) & 2047; min across rows - -fn elastic_query(state: &ElasticState, id: &str, hash: u64) -> i64 { - let n = state.bucket_count as usize; - let idx = (hash % n as u64) as usize; - - let heavy_match = idx < state.flow_ids.len() && state.flow_ids[idx] == id; - - if heavy_match { - let vpos = state.vote_pos.get(idx).copied().unwrap_or(0) as i64; - let evicted = state.evictions.get(idx).copied().unwrap_or(false); - if !evicted { - return vpos; - } - return vpos + elastic_light_min(state, hash); - } - elastic_light_min(state, hash) -} - -fn elastic_light_min(state: &ElasticState, hash: u64) -> i64 { - let light = match &state.light { - Some(l) => l, - None => return 0, - }; - let rows = light.rows as usize; - let cols = light.cols as usize; - let bits = col_bits(cols); // trailing zeros of cols = 11 for 2048 - let mask = (cols as u64) - 1; - let counts = &light.counts_float; - - let mut min_val = f64::MAX; - for r in 0..rows { - let shift = (r as u64) * bits; - let col = ((hash >> shift) & mask) as usize; - let v = counts[r * cols + col]; - if v < min_val { - min_val = v; - } - } - if min_val == f64::MAX { - 0 - } else { - min_val as i64 - } -} - // --------------------------------------------------------------------------- // UnivMon cardinality (g-sum heuristic) // --------------------------------------------------------------------------- diff --git a/tools/gen-proto/src/main.rs b/tools/gen-proto/src/main.rs index 1de6ad15..ab74620f 100644 --- a/tools/gen-proto/src/main.rs +++ b/tools/gen-proto/src/main.rs @@ -36,7 +36,6 @@ const PROTO_FILES: &[&str] = &[ "univmon/univmon.proto", "hydra/hydra.proto", "cocosketch/cocosketch.proto", - "elasticsketch/elasticsketch.proto", "sketchlib.proto", ]; From 25cc77eb0482b4a68379322c124d3cd5a6155ade Mon Sep 17 00:00:00 2001 From: GordonYuanyc Date: Fri, 2 Oct 2026 00:59:57 -0600 Subject: [PATCH 17/36] test(cmsheap): pin the heap key tie order and the empty heap in goldens The spec states the key comparison: signed keys by their two's-complement bit pattern read unsigned, floats by their bits, strings and bytes byte-wise with a proper prefix first. Three new golden fixtures pin i64 ties across zero and i64::MIN, string ties across case, length and prefix, and the empty-heap form (submodule bump). The docs defer the fixture list to the golden README. Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 9 +- asapv1_golden | 2 +- docs/asapv1_wire_format.md | 6 +- docs/message_pack_format.md | 7 +- docs/tests.md | 3 + .../portable/countsketch_topk.rs | 3 +- tests/asapv1_golden.rs | 105 ++++++++++++++++++ 7 files changed, 123 insertions(+), 12 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 740943b6..8026b6f8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -67,8 +67,13 @@ signals a backwards-compatible change. draw from the thread generator. The seed is not serialized, and the ASAPv1 payload is unchanged. - **ASAPv1 golden fixtures for CMSHeap** (`03 00`): - `cmsheap_i64_regular_2x3_strkeys` and `cmsheap_i32_fast_2x3_i64keys`, checked - by `tests/asapv1_golden.rs`. + `cmsheap_i64_regular_2x3_strkeys`, `cmsheap_i32_fast_2x3_i64keys`, the + key-tie fixtures `cmsheap_i64_regular_2x3_i64tie` and + `cmsheap_i64_regular_2x3_strtie`, and the empty heap + `cmsheap_i64_regular_2x3_empty`, checked by `tests/asapv1_golden.rs`. +- The ASAPv1 spec states the heap key tie order: a signed key compares as its + two's-complement bit pattern read unsigned, a float by its bits, a string or + bytes key byte-wise with a proper prefix first. ### Changed diff --git a/asapv1_golden b/asapv1_golden index 821d0ea7..105be8ba 160000 --- a/asapv1_golden +++ b/asapv1_golden @@ -1 +1 @@ -Subproject commit 821d0ea7c1dbd1f4bc8f0c7b3938bd6635a02037 +Subproject commit 105be8baba44227c0cb06d9e7342df6e61bdce66 diff --git a/docs/asapv1_wire_format.md b/docs/asapv1_wire_format.md index 0ec65cb1..3a521763 100644 --- a/docs/asapv1_wire_format.md +++ b/docs/asapv1_wire_format.md @@ -562,7 +562,7 @@ Two summaries have no encoding and **fail to serialize** rather than being coerc **Empty summary.** A summary that monitors nothing has no variant to report. It emits `key_type = "u64"` with three empty arrays, so an empty summary has exactly one encoding rather than one per producer. -**Emitted order (cross-language contract).** The payload is **order-defined**: entries are written in **descending `count`**, ties broken by a **total order over the key** (variant tag first, then the value). This is required, not cosmetic: `entries()` order follows the counter arena and `top_k` order follows the bucket walk, and neither survives a rebuild, so an unordered payload would re-serialize to different bytes than it decoded from. With the order pinned, two summaries holding the same triples emit the same bytes whatever order they were seated in, and re-serializing a decoded summary reproduces its bytes exactly. +**Emitted order (cross-language contract).** The payload is **order-defined**: entries are written in **descending `count`**, ties broken by a **total order over the key** (variant tag first, then the value). The value compares as unsigned bits: a signed key by its two's-complement pattern, so every negative key sorts after every non-negative one; a float by its IEEE-754 bits; a string or bytes key byte-wise, a proper prefix first. This is required, not cosmetic: `entries()` order follows the counter arena and `top_k` order follows the bucket walk, and neither survives a rebuild, so an unordered payload would re-serialize to different bytes than it decoded from. With the order pinned, two summaries holding the same triples emit the same bytes whatever order they were seated in, and re-serializing a decoded summary reproduces its bytes exactly. **Decode rules.** Fail **closed** on each, with an error and never a panic: @@ -642,7 +642,7 @@ Wire counter types are the base Count-Min's: **`"i32"`, `"i64"` and `"f64"`**. ` **`key_type` names the exact key variant, and is never widened.** This is §3.5's rule, and the fourteen names and their `HeapItem` variants are the table there. A heap whose keys are of different variants has no single `key_type` and **fails to serialize** with an error naming the mismatch; `HeapItem::I128` / `U128` have no msgpack integer form and are not wire types. An empty heap has no variant to report: it emits `key_type = "u64"` with two empty arrays, so an empty heap has exactly one encoding rather than one per producer. -**Emitted order (cross-language contract).** `counts` is row-major. Heap entries are written in **descending `heap_counts`**, ties broken by a **total order over the key**: the variant tag first, in `HeapItem`'s declaration order, so every string sorts after every numeric key; then the value. This is required, not cosmetic: the heap's in-memory array order follows the sift path taken while it filled, and that order does not survive a rebuild, so an unordered payload would re-serialize to different bytes than it decoded from. With the order pinned, two heaps holding the same entries emit the same bytes whatever order they were seated in. Go must mirror the same comparator. +**Emitted order (cross-language contract).** `counts` is row-major. Heap entries are written in **descending `heap_counts`**, ties broken by a **total order over the key**: the variant tag first, in `HeapItem`'s declaration order, so every string sorts after every numeric key; then the value, compared as §3.5 defines it. This is required, not cosmetic: the heap's in-memory array order follows the sift path taken while it filled, and that order does not survive a rebuild, so an unordered payload would re-serialize to different bytes than it decoded from. With the order pinned, two heaps holding the same entries emit the same bytes whatever order they were seated in. Go must mirror the same comparator. **Decode rules.** Fail **closed** on each, with an error and never a panic: @@ -1339,7 +1339,7 @@ Good direction (more compact, higher fidelity, less Rust-internal duplication), 2. **Golden byte-vector fixtures** in one shared repo, [`sketchlib-golden-bytes`](https://github.com/ProjectASAP/sketchlib-golden-bytes), which each implementation mounts at `asapv1_golden/`; both languages decode and re-encode them byte-identically. These replace the `portable`-as-oracle round-trip test. 3. **This registry**, mirrored, never independently allocated. -Fixtures exist for six `kind_id`s — HLL's three estimators, Count-Min, Count Sketch and compact KLL — and `sketchlib-go` mirrors those. Every other kind here has a payload and **no fixture**, so this document is its only contract; `asapv1_golden/README.md` lists the gap. `portable` carries its own Go goldens. +`asapv1_golden/README.md` lists the `kind_id`s with fixtures, which `sketchlib-go` mirrors; for every other kind this document is the only contract. `portable` carries its own Go goldens. **Hash profile on the Go side.** Rust derives the hash spec from a generic `HashProfile` bound on the hasher type; Go has no generic hasher type, so there is nothing to derive from. diff --git a/docs/message_pack_format.md b/docs/message_pack_format.md index 308a2e42..87f91eee 100644 --- a/docs/message_pack_format.md +++ b/docs/message_pack_format.md @@ -169,10 +169,9 @@ decode → re-encode them byte-identically. The `kind_id` registry is mirrored verbatim with Go's `wire/asapmsgpack/magic_ids.go`, never independently allocated. -That proof covers the seven `kind_id`s a fixture exists for — HLL's three -estimators, Count-Min, CMSHeap, Count Sketch and compact KLL. Every other -implemented kind has **no ASAPv1 golden and therefore no ASAPv1 drift guard**; -[its README](https://github.com/ProjectASAP/sketchlib-golden-bytes#coverage) lists what is covered. +That proof covers only the `kind_id`s a fixture exists for, listed in +[its README](https://github.com/ProjectASAP/sketchlib-golden-bytes#coverage); +a kind without one has **no ASAPv1 drift guard**. The older `portable` path is guarded separately, by the `sketchlib-go` goldens of its own listed above. diff --git a/docs/tests.md b/docs/tests.md index 38687f8f..91203144 100644 --- a/docs/tests.md +++ b/docs/tests.md @@ -2110,6 +2110,9 @@ Test file: [`tests/asapv1_golden.rs`](../tests/asapv1_golden.rs) | `cms_f64_fast_2x3_matches_golden` | A known f64 fast-path Count-Min matrix serializes to its checked-in golden bytes. | Verifies `CountMin::, FastPath>::from_storage` over the `2x3` matrix `[[0.0, 1.5, 2.25], [3.75, 4.125, 5.0625]]` serializes to `asapv1_golden/cms_f64_fast_2x3.hex`, and that decoding that fixture recovers the six cells in row-major order and re-serializes to the same bytes. | | `cms_i64_regular_2x3_matches_golden` | A known i64 regular-path Count-Min matrix serializes to its checked-in golden bytes. | Verifies `CountMin::, RegularPath>::from_storage` over the `2x3` matrix `[[0, 1, 127], [128, 300, 65536]]` serializes to `asapv1_golden/cms_i64_regular_2x3.hex`, and that decoding the fixture reports `rows()` `2`, `cols()` `3`, the six cells in row-major order, and a byte-identical re-serialization. | | `cmsheap_i32_fast_2x3_i64keys_matches_golden` | A known i32 fast-path CMSHeap with `i64` keys serializes to its golden bytes. | Builds `CMSHeap::, FastPath>::from_storage` over `[[0, 1, 127], [128, 300, 65536]]` with `k` `3`, seats `HeapItem::I64` entries `4294967296: 3`, `-1: 7` and `-129: 7` with `update_heap_item`, verifies the bytes equal `asapv1_golden/cmsheap_i32_fast_2x3_i64keys.hex`, and that decoding the fixture recovers the six cells, capacity `3`, exactly those three entries, and a byte-identical re-serialization. | +| `cmsheap_i64_regular_2x3_empty_matches_golden` | An empty CMSHeap heap has one encoding. | Builds `CMSHeap::, RegularPath>::from_storage` over `[[0, 1, 127], [128, 300, 65536]]` with `k` `4` and no entries, verifies the bytes equal `asapv1_golden/cmsheap_i64_regular_2x3_empty.hex`, that the payload ends with two empty arrays, and that decoding the fixture recovers the six cells, capacity `4`, an empty heap, and a byte-identical re-serialization. | +| `cmsheap_i64_regular_2x3_i64tie_matches_golden` | Signed keys tie by their two's-complement bit pattern read unsigned. | Over the same matrix with `k` `5`, seats `HeapItem::I64` entries `-1: 5`, `1: 5`, `0: 5`, `i64::MIN: 5` and `2: 9`, forward and reversed; verifies both encode to `asapv1_golden/cmsheap_i64_regular_2x3_i64tie.hex`, that its payload ends with keys `[2, 0, 1, i64::MIN, -1]` and counts `[9, 5, 5, 5, 5]`, and that decoding recovers the cells, capacity `5`, exactly those entries, and a byte-identical re-serialization. | +| `cmsheap_i64_regular_2x3_strtie_matches_golden` | String keys tie byte-wise, a proper prefix first. | Over the same matrix with `k` `5`, seats `HeapItem::String` entries `b: 5`, `aa: 5`, `Z: 5`, `a: 5` and `hot: 9`, forward and reversed; verifies both encode to `asapv1_golden/cmsheap_i64_regular_2x3_strtie.hex`, that its payload ends with keys `["hot", "Z", "a", "aa", "b"]` and counts `[9, 5, 5, 5, 5]`, and that decoding recovers the cells, capacity `5`, exactly those entries, and a byte-identical re-serialization. | | `cmsheap_i64_regular_2x3_strkeys_matches_golden` | A known i64 regular-path CMSHeap with string keys serializes to its golden bytes, whatever order its entries were seated in. | Builds `CMSHeap::, RegularPath>::from_storage` over `[[0, 1, 127], [128, 300, 65536]]` with `k` `5`, seats `HeapItem::String` entries `cold: 1`, `warm: 300`, `hot: 65536` and `mild: 300` in that order, verifies the bytes equal `asapv1_golden/cmsheap_i64_regular_2x3_strkeys.hex`, and that decoding the fixture reports `rows()` `2`, `cols()` `3`, the six cells, capacity `5`, exactly those four entries, and a byte-identical re-serialization. | | `cs_i32_regular_2x4_matches_golden` | The i32 counter type reaches the metadata, and an i32 golden is not an i64 sketch. | Verifies `Count::, RegularPath>::from_storage` over `[[0, 127, 128, 65536], [-1, -33, -32768, -2147483648]]` cast to `i32` serializes to `asapv1_golden/cs_i32_regular_2x4.hex`, that decoding it recovers the eight cells and re-serializes identically, that the i32 and i64 regular fixtures have equal length yet differ, and that the i32 fixture fails to decode as `Count, RegularPath>`. | | `cs_i64_fast_2x4_matches_golden` | The fast-path mode reaches the bytes, so the same matrix encodes differently per path. | Verifies `Count::, FastPath>::from_storage` over `[[0, 127, 128, 65536], [-1, -33, -32768, -2147483648]]` serializes to `asapv1_golden/cs_i64_fast_2x4.hex`, that decoding it recovers the eight cells and re-serializes identically, and that the fixture differs from `cs_i64_regular_2x4.hex` built from the same matrix. | diff --git a/src/message_pack_format/portable/countsketch_topk.rs b/src/message_pack_format/portable/countsketch_topk.rs index 6cf252ac..47f65216 100644 --- a/src/message_pack_format/portable/countsketch_topk.rs +++ b/src/message_pack_format/portable/countsketch_topk.rs @@ -444,8 +444,7 @@ mod tests { } /// CS's median estimator must behave differently from CMS's - /// min-over-rows estimator on the same kind of skewed insert - /// pattern, not just share a struct shape. + /// min-over-rows estimator on the same kind of skewed insert pattern. #[test] fn test_median_estimator_differs_from_cms_style_min() { let mut cs = CountSketchWithHeap::new(5, 64, 10); diff --git a/tests/asapv1_golden.rs b/tests/asapv1_golden.rs index 759affd4..c6ae06bd 100644 --- a/tests/asapv1_golden.rs +++ b/tests/asapv1_golden.rs @@ -30,6 +30,12 @@ const GOLDEN_CMS_F64: &str = include_str!("../asapv1_golden/cms_f64_fast_2x3.hex const GOLDEN_CMSHEAP_STR: &str = include_str!("../asapv1_golden/cmsheap_i64_regular_2x3_strkeys.hex"); const GOLDEN_CMSHEAP_I64: &str = include_str!("../asapv1_golden/cmsheap_i32_fast_2x3_i64keys.hex"); +const GOLDEN_CMSHEAP_I64_TIE: &str = + include_str!("../asapv1_golden/cmsheap_i64_regular_2x3_i64tie.hex"); +const GOLDEN_CMSHEAP_STR_TIE: &str = + include_str!("../asapv1_golden/cmsheap_i64_regular_2x3_strtie.hex"); +const GOLDEN_CMSHEAP_EMPTY: &str = + include_str!("../asapv1_golden/cmsheap_i64_regular_2x3_empty.hex"); const GOLDEN_CS_REGULAR: &str = include_str!("../asapv1_golden/cs_i64_regular_2x4.hex"); const GOLDEN_CS_FAST: &str = include_str!("../asapv1_golden/cs_i64_fast_2x4.hex"); const GOLDEN_CS_I32: &str = include_str!("../asapv1_golden/cs_i32_regular_2x4.hex"); @@ -255,6 +261,105 @@ fn cmsheap_i32_fast_2x3_i64keys_matches_golden() { assert_eq!(decoded.serialize_to_bytes().expect("re-serialize"), want); } +/// The i64 regular-path CMSHeap over the Count-Min i64 matrix with these +/// entries seated in the given order. +fn cmsheap_i64_regular( + k: usize, + entries: &[(HeapItem, i64)], +) -> CMSHeap, RegularPath> { + let mut sketch = CMSHeap::, RegularPath>::from_storage( + Vector2D::from_fn(2, 3, |r, c| I64_VALS[r][c]), + k, + ); + for (key, count) in entries { + assert!(sketch.heap_mut().update_heap_item(key, *count)); + } + sketch +} + +/// Asserts the golden's payload ends with exactly these `keys` and +/// `heap_counts` arrays, in this order. +fn assert_heap_tail(golden: &[u8], keys: &[K], counts: &[i64]) { + let mut tail = rmp_serde::to_vec(keys).expect("keys"); + tail.extend(rmp_serde::to_vec(counts).expect("counts")); + assert!( + golden.ends_with(&tail), + "golden heap arrays are not in the pinned order" + ); +} + +/// Encodes the entries seated forward and reversed, asserts both equal the +/// golden, then decodes it and asserts the held entries and a byte-identical +/// re-encode. +fn check_cmsheap_golden(want: &[u8], k: usize, entries: &[(HeapItem, i64)]) { + let reversed: Vec<(HeapItem, i64)> = entries.iter().rev().cloned().collect(); + for seated in [entries, reversed.as_slice()] { + let got = cmsheap_i64_regular(k, seated) + .serialize_to_bytes() + .expect("serialize"); + assert_eq!(got, want, "CMSHeap bytes diverge from golden"); + } + + let decoded = + CMSHeap::, RegularPath>::deserialize_from_bytes(want).expect("decode"); + let flat: Vec = I64_VALS.iter().flatten().copied().collect(); + assert_eq!(decoded.cms().as_storage().as_slice(), flat.as_slice()); + assert_eq!(decoded.heap().capacity(), k); + let mut held: Vec<(HeapItem, i64)> = decoded + .heap() + .heap() + .iter() + .map(|item| (item.key.clone(), item.count)) + .collect(); + let mut expected = entries.to_vec(); + let by_debug = + |a: &(HeapItem, i64), b: &(HeapItem, i64)| format!("{a:?}").cmp(&format!("{b:?}")); + held.sort_by(by_debug); + expected.sort_by(by_debug); + assert_eq!(held, expected); + assert_eq!(decoded.serialize_to_bytes().expect("re-serialize"), want); +} + +/// Signed keys tie by their two's-complement bit pattern read unsigned, so the +/// negatives follow every non-negative key. +#[test] +fn cmsheap_i64_regular_2x3_i64tie_matches_golden() { + let want = decode_hex(GOLDEN_CMSHEAP_I64_TIE); + let entries: Vec<(HeapItem, i64)> = [(-1, 5), (1, 5), (0, 5), (i64::MIN, 5), (2, 9)] + .into_iter() + .map(|(k, c)| (HeapItem::I64(k), c)) + .collect(); + check_cmsheap_golden(&want, 5, &entries); + assert_heap_tail(&want, &[2i64, 0, 1, i64::MIN, -1], &[9, 5, 5, 5, 5]); +} + +/// String keys tie byte-wise, a proper prefix first: case and length do not +/// enter except through the bytes. +#[test] +fn cmsheap_i64_regular_2x3_strtie_matches_golden() { + let want = decode_hex(GOLDEN_CMSHEAP_STR_TIE); + let entries: Vec<(HeapItem, i64)> = [("b", 5), ("aa", 5), ("Z", 5), ("a", 5), ("hot", 9)] + .into_iter() + .map(|(k, c)| (HeapItem::String(k.to_string()), c)) + .collect(); + check_cmsheap_golden(&want, 5, &entries); + assert_heap_tail(&want, &["hot", "Z", "a", "aa", "b"], &[9, 5, 5, 5, 5]); +} + +/// An empty heap has one encoding: `key_type` `"u64"` and two empty arrays. +#[test] +fn cmsheap_i64_regular_2x3_empty_matches_golden() { + let want = decode_hex(GOLDEN_CMSHEAP_EMPTY); + check_cmsheap_golden(&want, 4, &[]); + assert_heap_tail::(&want, &[], &[]); + assert!( + CMSHeap::, RegularPath>::deserialize_from_bytes(&want) + .expect("decode") + .heap() + .is_empty() + ); +} + // --------------------------------------------------------------------------- // Count Sketch: build known matrix state -> serialize == golden, round-trips. // All three fixtures hold the SAME matrix: the i64 pair differs only by `mode` From d6578394b49ca2108983c704184fb1011b218d5a Mon Sep 17 00:00:00 2001 From: GordonYuanyc Date: Fri, 2 Oct 2026 01:00:54 -0600 Subject: [PATCH 18/36] refactor(univmon)!: drop the proto format; add ASAPv1 goldens UnivMon serializes only as ASAPv1 (kind_id 0x10 0x00). - proto: reserve SketchEnvelope field 15 ("univmon"); UnivMonState and UnivMonLayer stay in univmon.proto because hydra.proto's HydraCell still embeds them. Regenerated src/proto/generated. - tests/xtest_consumer.rs: drop the univmon.pb check and its g-sum helpers. - asapv1_golden -> e2086f0: univmon_str_l2_2x4_h2, univmon_i64_l2_2x4_h2 and univmon_empty_l2_2x4_h2, built through L2HH::apply_delta and HHHeap::update (no hashing). tests/asapv1_golden.rs checks serialize == golden, the hand-read metadata and payload, the decoded state and the re-encode. - docs: fixture counts say seven kind_ids; features.md and tests.md follow. Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 12 ++ asapv1_golden | 2 +- docs/asapv1_wire_format.md | 4 +- docs/features.md | 2 +- docs/message_pack_format.md | 6 +- docs/tests.md | 5 +- proto/sketchlib.proto | 5 +- src/proto/generated/sketchlib.v1.rs | 4 +- tests/asapv1_golden.rs | 249 +++++++++++++++++++++++++++- tests/xtest_consumer.rs | 128 -------------- 10 files changed, 273 insertions(+), 144 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 80d85f98..c10a7b16 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -66,6 +66,9 @@ signals a backwards-compatible change. worker's seed and the aggregator's from one seed. The unseeded constructors draw from the thread generator. The seed is not serialized, and the ASAPv1 payload is unchanged. +- **ASAPv1 golden fixtures for `UnivMon`** (kind_id `0x10 0x00`): + `univmon_str_l2_2x4_h2`, `univmon_i64_l2_2x4_h2` and + `univmon_empty_l2_2x4_h2`, checked in `tests/asapv1_golden.rs`. ### Changed @@ -88,6 +91,15 @@ signals a backwards-compatible change. as an empty window like any other, and that window takes a place in `max_windows`. +### Removed + +- **UnivMon's proto format.** `UnivMon` serializes only as ASAPv1 + (`serialize_to_bytes` / `deserialize_from_bytes`, kind_id `0x10 0x00`). + Removed the `univmon` field (15) of `SketchEnvelope` (now reserved) and with + it `sketch_envelope::SketchState::Univmon`; `UnivMonState` and + `UnivMonLayer` stay in `proto/univmon/univmon.proto` only as + `HydraCell.univmon`. + ### Fixed - `KLLDynamic` panicked on every quantile query once a NaN had entered the diff --git a/asapv1_golden b/asapv1_golden index 8287a4ae..e2086f05 160000 --- a/asapv1_golden +++ b/asapv1_golden @@ -1 +1 @@ -Subproject commit 8287a4ae04e6cfbfb3ed33dafbf1dac77d8f7fc1 +Subproject commit e2086f053724cfe894f240f8d9e8fb03a540689e diff --git a/docs/asapv1_wire_format.md b/docs/asapv1_wire_format.md index fd14e70a..695cf324 100644 --- a/docs/asapv1_wire_format.md +++ b/docs/asapv1_wire_format.md @@ -1341,14 +1341,14 @@ Good direction (more compact, higher fidelity, less Rust-internal duplication), 2. **Golden byte-vector fixtures** in one shared repo, [`sketchlib-golden-bytes`](https://github.com/ProjectASAP/sketchlib-golden-bytes), which each implementation mounts at `asapv1_golden/`; both languages decode and re-encode them byte-identically. These replace the `portable`-as-oracle round-trip test. 3. **This registry**, mirrored, never independently allocated. -Fixtures exist for six `kind_id`s — HLL's three estimators, Count-Min, Count Sketch and compact KLL — and `sketchlib-go` mirrors those. Every other kind here has a payload and **no fixture**, so this document is its only contract; `asapv1_golden/README.md` lists the gap. `portable` carries its own Go goldens. +Fixtures exist for seven `kind_id`s — HLL's three estimators, Count-Min, Count Sketch, compact KLL and UnivMon — and `sketchlib-go` mirrors those. Every other kind here has a payload and **no fixture**, so this document is its only contract; `asapv1_golden/README.md` lists the gap. `portable` carries its own Go goldens. **Hash profile on the Go side.** Rust derives the hash spec from a generic `HashProfile` bound on the hasher type; Go has no generic hasher type, so there is nothing to derive from. On the Go side the profile is simply **written into** the metadata on encode and **read from** it on decode. Go MUST validate the profile it reads (same fail-closed intent as Rust): a sketch is only mergeable/queryable if its `hash_profile_id` + seeds match the profile Go is prepared to reproduce. -Sequencing: (2) covers six `kind_id`s, and the one `native bytes == portable bytes` test is HLL-only, since no other `portable` type emits ASAPv1 bytes. +Sequencing: (2) covers seven `kind_id`s, and the one `native bytes == portable bytes` test is HLL-only, since no other `portable` type emits ASAPv1 bytes. Retire `portable` once the fixtures cover the rest. --- diff --git a/docs/features.md b/docs/features.md index 64f296a3..5a58639d 100644 --- a/docs/features.md +++ b/docs/features.md @@ -112,7 +112,7 @@ MessagePack (`rmp-serde`) support. **serde support** means the type derives `Ser | EHSketchList | Yes | Yes | | ExponentialHistogram | No | Yes | -Protobuf (prost): `.proto` definitions exist for CountMin, Count, HLL, DDSketch, KLL, Elastic, Coco, Hydra, and UnivMon. Rust conversion code is in progress. +Protobuf (prost): `.proto` definitions exist for CountMin, Count, HLL, DDSketch, KLL, Elastic, Coco, and Hydra. Rust conversion code is in progress. ### API Stability diff --git a/docs/message_pack_format.md b/docs/message_pack_format.md index 12a2b91c..01556231 100644 --- a/docs/message_pack_format.md +++ b/docs/message_pack_format.md @@ -169,9 +169,9 @@ decode → re-encode them byte-identically. The `kind_id` registry is mirrored verbatim with Go's `wire/asapmsgpack/magic_ids.go`, never independently allocated. -That proof covers the six `kind_id`s a fixture exists for — HLL's three -estimators, Count-Min, Count Sketch and compact KLL. Every other implemented -kind has **no ASAPv1 golden and therefore no ASAPv1 drift guard**; +That proof covers the seven `kind_id`s a fixture exists for — HLL's three +estimators, Count-Min, Count Sketch, compact KLL and UnivMon. Every other +implemented kind has **no ASAPv1 golden and therefore no ASAPv1 drift guard**; [its README](https://github.com/ProjectASAP/sketchlib-golden-bytes#coverage) lists what is covered. The older `portable` path is guarded separately, by the `sketchlib-go` goldens of its own listed above. diff --git a/docs/tests.md b/docs/tests.md index ccb1571c..1ef8677e 100644 --- a/docs/tests.md +++ b/docs/tests.md @@ -2131,6 +2131,9 @@ Test file: [`tests/asapv1_golden.rs`](../tests/asapv1_golden.rs) | `hll_hip_p12_matches_golden` | The HIP variant's three running scalars travel on the wire beside its registers. | Builds `HllSketch::from_raw(HllVariant::Hip, 12, regs, 1.5, 2.5, 3.0)` over the shared P12 pattern, verifies `to_msgpack()` equals `asapv1_golden/hll_hip_p12.hex`, that `HyperLogLogHIPP12::deserialize_from_bytes` re-serializes the fixture identically, and that `HllSketch::from_msgpack` reports the registers, `HllVariant::Hip`, `hip_kxq0` `1.5`, `hip_kxq1` `2.5` and `hip_est` `3.0`. | | `kll_f64_k200_matches_golden` | A deterministic f64 KLL serializes to its golden bytes, which round-trip unchanged. | Verifies `KLL::::init_kll_with_seed(200, 42)` fed `1..=50` (below the level-0 capacity, so no compaction fires) serializes to `asapv1_golden/kll_f64_k200.hex`, and that decoding the fixture re-serializes to the same bytes and reports `quantile(0.0)` `1.0` and `quantile(1.0)` `50.0`. | | `kll_i64_k200_matches_golden` | A deterministic i64 KLL serializes to its golden bytes, which round-trip unchanged. | Verifies `KLL::::init_kll_with_seed(200, 42)` fed `1..=50` serializes to `asapv1_golden/kll_i64_k200.hex`, and that decoding the fixture re-serializes to the same bytes and reports `quantile(0.0)` `1.0` and `quantile(1.0)` `50.0`. | +| `univmon_empty_l2_2x4_h2_matches_golden` | A freshly constructed pyramid has one encoding, pinned by its golden. | Verifies `UnivMon::init_univmon(2, 2, 4, 2)` serializes to `asapv1_golden/univmon_empty_l2_2x4_h2.hex`; reading the envelope by hand gives kind_id `0x10 0x00`, `key_type` `u64`, 16 zero counts, 4 zero `l2` accumulators, `heap_lens` `[0, 0]`, no keys or heap counts, candidate flags `[true, true]`, `bucket_size` `0` and `update_mode` `0`; the decode matches the source's shape, weight, flags, counters, per-layer `get_l2()` and heaps, and re-serializes identically. | +| `univmon_i64_l2_2x4_h2_matches_golden` | The `i64` key type reaches the bytes, and only it differs from the string fixture. | Builds the same layers as the string fixture with heap keys `i64::MIN` and `-1` on layer 0 and `128` on layer 1, verifies the bytes equal `asapv1_golden/univmon_i64_l2_2x4_h2.hex`, that the hand-read payload carries those keys under `key_type` `i64` with the same counts, `l2`, heap counts, flags, weight and mode, that the decode matches the source and re-serializes identically, that layer 0's heap holds `HeapItem::I64(i64::MIN)`, and that the fixture differs from the string one. | +| `univmon_str_l2_2x4_h2_matches_golden` | A known 2-layer pyramid with string keys serializes to its golden bytes and decodes back to that state. | Builds `UnivMon::init_univmon(2, 2, 4, 2)`, writes layer 0's cells `[[0, 127, 128, 65536], [-1, -33, -32768, -2147483648]]` and layer 1's `[[3, -2, 0, 1], [0, 0, 5, -4]]` through `L2HH::apply_delta`, seats `alpha`=`65536` and `beta`=`300` in layer 0's heap and `gamma`=`5` in layer 1's, calls `set_total_weight(70_000)` and `mark_layer_candidates_incomplete(1)`, and verifies the bytes equal `asapv1_golden/univmon_str_l2_2x4_h2.hex`. Reading the envelope by hand gives kind_id `0x10 0x00`, shape `(2, 2, 4, 2)`, `key_type` `string`, the 16 cells in layer then row-major order, `l2` `[4294999809, 4611686019501130818, 14, 41]`, `heap_lens` `[2, 1]`, keys `[alpha, beta, gamma]`, heap counts `[65536, 300, 5]`, flags `[true, false]`, `bucket_size` `70000` and `update_mode` `1`; the decode matches the source and re-serializes identically. | ### MessagePack Envelope Compatibility @@ -2202,7 +2205,7 @@ Test file: [`tests/xtest_consumer.rs`](../tests/xtest_consumer.rs) | test_name | test_description | what_is_tested | | --- | --- | --- | -| `cross_language_proto` | Protobuf envelopes read from the cross-language fixture directory answer their point queries in band. | Asserts nothing unless fixtures are available: with no `XTEST_DIR` and no discoverable producer directory holding `tests/cross_language/xtest_producer_test.go`, it prints a skip line and returns. With fixtures it decodes nine `SketchEnvelope` files and records a pass or failure for each — `countmin.pb` min-frequency for `item:42` at least `101`, `kll.pb` p50 within 5% of `5000` and p99 within 5% of `9900`, `ddsketch.pb` the same two quantiles within 2%, `hll.pb` an ErtlMLE cardinality inside `[40_000, 65_000]`, `countsketch.pb` `cs:hot` at least `200`, `coco.pb` `coco:hot` at least `500`, `elastic.pb` `elephant` at least `900`, `univmon.pb` a cardinality inside `[1_000, 15_000]`, `hydra.pb` `hydra:42` at least `51` — using the hash seeds `0xcafe3553`, `0x6a09e667` and `0xbb67ae85`, and, when `countmin_sampled.pb` and `hll_sampled.pb` exist, requires `effective_sample_p` of `0.1` and a `rescale_count` by `1/p` within 5% of `100_000` and 6% of `200_000`. A single `assert!` at the end fails if any check failed; a missing or mistyped envelope panics on the spot. | +| `cross_language_proto` | Protobuf envelopes read from the cross-language fixture directory answer their point queries in band. | Asserts nothing unless fixtures are available: with no `XTEST_DIR` and no discoverable producer directory holding `tests/cross_language/xtest_producer_test.go`, it prints a skip line and returns. With fixtures it decodes eight `SketchEnvelope` files and records a pass or failure for each — `countmin.pb` min-frequency for `item:42` at least `101`, `kll.pb` p50 within 5% of `5000` and p99 within 5% of `9900`, `ddsketch.pb` the same two quantiles within 2%, `hll.pb` an ErtlMLE cardinality inside `[40_000, 65_000]`, `countsketch.pb` `cs:hot` at least `200`, `coco.pb` `coco:hot` at least `500`, `elastic.pb` `elephant` at least `900`, `hydra.pb` `hydra:42` at least `51` — using the hash seeds `0xcafe3553`, `0x6a09e667` and `0xbb67ae85`, and, when `countmin_sampled.pb` and `hll_sampled.pb` exist, requires `effective_sample_p` of `0.1` and a `rescale_count` by `1/p` within 5% of `100_000` and 6% of `200_000`. A single `assert!` at the end fails if any check failed; a missing or mistyped envelope panics on the spot. | ### HLL Custom Precision diff --git a/proto/sketchlib.proto b/proto/sketchlib.proto index 0a6d8193..d79e63b5 100644 --- a/proto/sketchlib.proto +++ b/proto/sketchlib.proto @@ -9,7 +9,6 @@ // hll/hll.proto — HyperLogLogState, HLLVariant // kll/kll.proto — KLLState, CoinState // ddsketch/ddsketch.proto — DDSketchState -// univmon/univmon.proto — UnivMonState, UnivMonLayer // hydra/hydra.proto — HydraState, HydraCounterType, HydraCell // // Design goals: @@ -34,7 +33,6 @@ import "countsketch/countsketch.proto"; import "hll/hll.proto"; import "kll/kll.proto"; import "ddsketch/ddsketch.proto"; -import "univmon/univmon.proto"; import "hydra/hydra.proto"; import "cocosketch/cocosketch.proto"; import "elasticsketch/elasticsketch.proto"; @@ -71,12 +69,13 @@ message SketchEnvelope { HyperLogLogState hll = 12; KLLState kll = 13; DDSketchState ddsketch = 14; - UnivMonState univmon = 15; HydraState hydra = 16; CocoSketchState coco = 17; ElasticState elastic = 18; } + reserved 15; + reserved "univmon"; reserved 19 to 31; // Reserved for future sketch types. // Geometric / NitroSketch sampling probability for the producing sketch. diff --git a/src/proto/generated/sketchlib.v1.rs b/src/proto/generated/sketchlib.v1.rs index 2c098d3f..abd34660 100644 --- a/src/proto/generated/sketchlib.v1.rs +++ b/src/proto/generated/sketchlib.v1.rs @@ -892,7 +892,7 @@ pub struct SketchEnvelope { /// The sketch payload. Exactly one field must be set. #[prost( oneof = "sketch_envelope::SketchState", - tags = "10, 11, 12, 13, 14, 15, 16, 17, 18" + tags = "10, 11, 12, 13, 14, 16, 17, 18" )] pub sketch_state: ::core::option::Option, } @@ -911,8 +911,6 @@ pub mod sketch_envelope { Kll(super::KllState), #[prost(message, tag = "14")] Ddsketch(super::DdSketchState), - #[prost(message, tag = "15")] - Univmon(super::UnivMonState), #[prost(message, tag = "16")] Hydra(super::HydraState), #[prost(message, tag = "17")] diff --git a/tests/asapv1_golden.rs b/tests/asapv1_golden.rs index 30b41651..5c4ebfb4 100644 --- a/tests/asapv1_golden.rs +++ b/tests/asapv1_golden.rs @@ -9,9 +9,12 @@ //! See `asapv1_golden/README.md`. use asap_sketchlib::{ - Classic, Count, CountMin, ErtlMLE, FastPath, HllSketch, HllVariant, HyperLogLogHIPP12, - HyperLogLogP12, KLL, MessagePackCodec, RegularPath, Vector2D, + Classic, Count, CountDelta, CountMin, DataInput, ErtlMLE, FastPath, HeapItem, HllSketch, + HllVariant, HyperLogLogHIPP12, HyperLogLogP12, KLL, L2HH, MessagePackCodec, RegularPath, + UnivMon, Vector2D, }; +use serde::Deserialize; +use serde::de::DeserializeOwned; fn decode_hex(s: &str) -> Vec { let s = s.trim(); @@ -32,6 +35,9 @@ const GOLDEN_CS_FAST: &str = include_str!("../asapv1_golden/cs_i64_fast_2x4.hex" const GOLDEN_CS_I32: &str = include_str!("../asapv1_golden/cs_i32_regular_2x4.hex"); const GOLDEN_KLL_F64: &str = include_str!("../asapv1_golden/kll_f64_k200.hex"); const GOLDEN_KLL_I64: &str = include_str!("../asapv1_golden/kll_i64_k200.hex"); +const GOLDEN_UNIVMON_STR: &str = include_str!("../asapv1_golden/univmon_str_l2_2x4_h2.hex"); +const GOLDEN_UNIVMON_I64: &str = include_str!("../asapv1_golden/univmon_i64_l2_2x4_h2.hex"); +const GOLDEN_UNIVMON_EMPTY: &str = include_str!("../asapv1_golden/univmon_empty_l2_2x4_h2.hex"); /// The known P12 register pattern shared by all three HLL fixtures. fn p12_registers() -> Vec { @@ -287,3 +293,242 @@ fn kll_i64_k200_matches_golden() { assert_eq!(decoded.quantile(0.0), 1.0); assert_eq!(decoded.quantile(1.0), 50.0); } + +// --------------------------------------------------------------------------- +// UnivMon: a 2-layer pyramid of 2x4 CountL2HH layers with heap capacity 2. +// Cells are written through `L2HH::apply_delta` (which carries each row's L2 +// accumulator) and heap entries through `HHHeap::update`; nothing is hashed. +// --------------------------------------------------------------------------- + +/// Layer 1's counters; layer 0 holds `CS_VALS`. +const UNIVMON_LAYER1: [[i64; 4]; 2] = [[3, -2, 0, 1], [0, 0, 5, -4]]; + +/// Each row's sum of squared cells: layer 0's row 1 reaches the uint64 range. +const UNIVMON_L2: [i64; 4] = [4_294_999_809, 4_611_686_019_501_130_818, 14, 41]; + +/// The heap counts in emitted order: layer 0 descending, then layer 1. +const UNIVMON_HEAP_COUNTS: [i64; 3] = [65536, 300, 5]; + +/// The i64 fixture's keys in emitted order: int64, negative fixint, uint8. +const UNIVMON_I64_KEYS: [i64; 3] = [i64::MIN, -1, 128]; + +/// A UnivMon payload `[counts, l2, heap_lens, keys, heap_counts, +/// candidate_complete, bucket_size, update_mode]`. +type UnivMonPayloadView = ( + Vec, + Vec, + Vec, + Vec, + Vec, + Vec, + u64, + u8, +); + +/// The structural keys of the UnivMon metadata map. +#[derive(Deserialize)] +struct UnivMonShape { + layer_size: u32, + sketch_row: u32, + sketch_col: u32, + heap_size: u32, + key_type: String, +} + +/// Splits an ASAPv1 envelope by hand and decodes its UnivMon metadata and payload. +fn univmon_envelope( + bytes: &[u8], +) -> (Vec, UnivMonShape, UnivMonPayloadView) { + assert_eq!(&bytes[..7], b"ASAPv1\x01", "magic and version"); + let kind_len = bytes[7] as usize; + let kind_id = bytes[8..8 + kind_len].to_vec(); + let at = 8 + kind_len; + let meta_len = u32::from_be_bytes(bytes[at..at + 4].try_into().unwrap()) as usize; + let payload_len = u32::from_be_bytes(bytes[at + 4..at + 8].try_into().unwrap()) as usize; + let metadata = &bytes[at + 8..at + 8 + meta_len]; + let payload = &bytes[at + 8 + meta_len..]; + assert_eq!(payload.len(), payload_len, "payload length prefix"); + ( + kind_id, + rmp_serde::from_slice(metadata).expect("UnivMon metadata"), + rmp_serde::from_slice(payload).expect("UnivMon payload"), + ) +} + +/// The populated pyramid, with layer 0's heap holding `keys[0]` / `keys[1]` +/// and layer 1's holding `keys[2]`, at `UNIVMON_HEAP_COUNTS`. +fn univmon_known(keys: [DataInput<'_>; 3]) -> UnivMon { + let mut um = UnivMon::init_univmon(2, 2, 4, 2); + for (layer, cells) in [CS_VALS, UNIVMON_LAYER1].iter().enumerate() { + for (row, values) in cells.iter().enumerate() { + for (col, &value) in values.iter().enumerate() { + um.l2_sketch_layers[layer].apply_delta(CountDelta { + row: row as u32, + col: col as u32, + value: value as i32, + }); + } + } + } + let [k0, k1, k2] = keys; + um.hh_layers[0].update(&k0, UNIVMON_HEAP_COUNTS[0]); + um.hh_layers[0].update(&k1, UNIVMON_HEAP_COUNTS[1]); + um.hh_layers[1].update(&k2, UNIVMON_HEAP_COUNTS[2]); + um.set_total_weight(70_000); + um.mark_layer_candidates_incomplete(1); + um +} + +/// Layer counters flattened in wire order. +fn univmon_counts(um: &UnivMon) -> Vec { + (0..um.layer_size) + .flat_map(|layer| { + let L2HH::COUNT(counter) = &um.l2_sketch_layers[layer]; + counter.as_storage().as_slice().to_vec() + }) + .collect() +} + +/// Each layer's heap as `(key, count)`, sorted. +fn univmon_heaps(um: &UnivMon) -> Vec> { + (0..um.layer_size) + .map(|layer| { + let mut entries: Vec<(String, i64)> = um.hh_layers[layer] + .heap() + .iter() + .map(|item| (format!("{:?}", item.key), item.count)) + .collect(); + entries.sort(); + entries + }) + .collect() +} + +/// Decodes `want`, checks it against `source`, and checks the re-encode. +fn assert_univmon_decodes_to(want: &[u8], source: &UnivMon) { + let decoded = UnivMon::deserialize_from_bytes(want).expect("decode"); + assert_eq!( + ( + decoded.layer_size, + decoded.sketch_row, + decoded.sketch_col, + decoded.heap_size + ), + (2, 2, 4, 2) + ); + assert_eq!(decoded.bucket_size, source.bucket_size); + assert_eq!(decoded.candidates_complete(), source.candidates_complete()); + assert_eq!(univmon_counts(&decoded), univmon_counts(source)); + for layer in 0..2 { + assert_eq!( + decoded.l2_sketch_layers[layer].get_l2(), + source.l2_sketch_layers[layer].get_l2() + ); + } + assert_eq!(univmon_heaps(&decoded), univmon_heaps(source)); + assert_eq!(decoded.serialize_to_bytes().expect("re-serialize"), want); +} + +fn univmon_known_counts() -> Vec { + CS_VALS + .iter() + .chain(UNIVMON_LAYER1.iter()) + .flatten() + .copied() + .collect() +} + +#[test] +fn univmon_str_l2_2x4_h2_matches_golden() { + let want = decode_hex(GOLDEN_UNIVMON_STR); + + let sketch = univmon_known([ + DataInput::Str("alpha"), + DataInput::Str("beta"), + DataInput::Str("gamma"), + ]); + let got = sketch.serialize_to_bytes().expect("serialize"); + assert_eq!(got, want, "UnivMon string-key bytes diverge from golden"); + + let (kind_id, shape, payload) = univmon_envelope::(&want); + assert_eq!(kind_id, [0x10, 0x00]); + assert_eq!( + ( + shape.layer_size, + shape.sketch_row, + shape.sketch_col, + shape.heap_size + ), + (2, 2, 4, 2) + ); + assert_eq!(shape.key_type, "string"); + let (counts, l2, heap_lens, keys, heap_counts, flags, bucket_size, update_mode) = payload; + assert_eq!(counts, univmon_known_counts()); + assert_eq!(l2, UNIVMON_L2); + assert_eq!(heap_lens, [2, 1]); + assert_eq!(keys, ["alpha", "beta", "gamma"]); + assert_eq!(heap_counts, UNIVMON_HEAP_COUNTS); + assert_eq!(flags, [true, false]); + assert_eq!(bucket_size, 70_000); + assert_eq!(update_mode, 1, "set_total_weight selects the standard mode"); + + assert_univmon_decodes_to(&want, &sketch); +} + +#[test] +fn univmon_i64_l2_2x4_h2_matches_golden() { + let want = decode_hex(GOLDEN_UNIVMON_I64); + + let sketch = univmon_known(UNIVMON_I64_KEYS.map(DataInput::I64)); + let got = sketch.serialize_to_bytes().expect("serialize"); + assert_eq!(got, want, "UnivMon i64-key bytes diverge from golden"); + + let (kind_id, shape, payload) = univmon_envelope::(&want); + assert_eq!(kind_id, [0x10, 0x00]); + assert_eq!(shape.key_type, "i64"); + let (counts, l2, heap_lens, keys, heap_counts, flags, bucket_size, update_mode) = payload; + assert_eq!(counts, univmon_known_counts()); + assert_eq!(l2, UNIVMON_L2); + assert_eq!(heap_lens, [2, 1]); + assert_eq!(keys, UNIVMON_I64_KEYS); + assert_eq!(heap_counts, UNIVMON_HEAP_COUNTS); + assert_eq!(flags, [true, false]); + assert_eq!((bucket_size, update_mode), (70_000, 1)); + + assert_univmon_decodes_to(&want, &sketch); + let decoded = UnivMon::deserialize_from_bytes(&want).expect("decode"); + assert!( + decoded.hh_layers[0] + .heap() + .iter() + .any(|item| item.key == HeapItem::I64(i64::MIN)) + ); + + // Same layers and heap counts, other keys: only `key_type` and `keys` differ. + assert_ne!(want, decode_hex(GOLDEN_UNIVMON_STR)); +} + +#[test] +fn univmon_empty_l2_2x4_h2_matches_golden() { + let want = decode_hex(GOLDEN_UNIVMON_EMPTY); + + let sketch = UnivMon::init_univmon(2, 2, 4, 2); + let got = sketch.serialize_to_bytes().expect("serialize"); + assert_eq!(got, want, "empty UnivMon bytes diverge from golden"); + + let (kind_id, shape, payload) = univmon_envelope::(&want); + assert_eq!(kind_id, [0x10, 0x00]); + assert_eq!( + shape.key_type, "u64", + "an empty pyramid records key_type u64" + ); + let (counts, l2, heap_lens, keys, heap_counts, flags, bucket_size, update_mode) = payload; + assert_eq!(counts, [0; 16]); + assert_eq!(l2, [0; 4]); + assert_eq!(heap_lens, [0, 0]); + assert!(keys.is_empty() && heap_counts.is_empty()); + assert_eq!(flags, [true, true]); + assert_eq!((bucket_size, update_mode), (0, 0)); + + assert_univmon_decodes_to(&want, &sketch); +} diff --git a/tests/xtest_consumer.rs b/tests/xtest_consumer.rs index 0113ff45..65d6588e 100644 --- a/tests/xtest_consumer.rs +++ b/tests/xtest_consumer.rs @@ -12,7 +12,6 @@ //! countsketch.pb — CountSketchState with float64 signed counters //! coco.pb — CocoSketchState (hash+val+hasKey buckets) //! elastic.pb — ElasticState (heavy buckets + light CountMin) -//! univmon.pb — UnivMonState (layered CountSketch + TopK heaps) //! hydra.pb — HydraState (CM-cell grid) //! //! Usage: @@ -360,41 +359,6 @@ fn cross_language_proto() { all_ok = false; } - // ----------------------------------------------------------------------- - // UnivMon - // ----------------------------------------------------------------------- - println!(); - println!("[UnivMon] Step 1/3 — Read univmon.pb"); - let bytes = read_file(in_dir.join("univmon.pb")); - let env = SketchEnvelope::decode(bytes.as_slice()).expect("decode univmon envelope"); - - println!( - "[UnivMon] Step 2/3 — Validate envelope (format_version={}, producer={})", - env.format_version, - env.producer.as_ref().map_or("?", |p| &p.library) - ); - - let um_state = match env.sketch_state { - Some(sketch_envelope::SketchState::Univmon(ref s)) => s.clone(), - other => panic!("expected UnivMon sketch_state, got {other:?}"), - }; - - println!( - "[UnivMon] layer_size={} sketch_rows={} sketch_cols={} heap_size={}", - um_state.layer_size, um_state.sketch_rows, um_state.sketch_cols, um_state.heap_size - ); - - let um_card = univmon_cardinality(&um_state); - // Note: the g-sum heuristic typically underestimates (Go itself reports ~4250 for 10k inserts). - // We verify the Rust result matches Go's algorithm rather than the true cardinality. - println!("[UnivMon] Step 3/3 — cardinality ≈ {um_card:.0} (g-sum heuristic, Go also ~4250)"); - if (1_000.0..=15_000.0).contains(&um_card) { - println!("[UnivMon] PASS"); - } else { - eprintln!("[UnivMon] FAIL: cardinality {um_card:.0} not in [1000, 15000]"); - all_ok = false; - } - // ----------------------------------------------------------------------- // HydraSketch // ----------------------------------------------------------------------- @@ -916,98 +880,6 @@ fn elastic_light_min(state: &ElasticState, hash: u64) -> i64 { } } -// --------------------------------------------------------------------------- -// UnivMon cardinality (g-sum heuristic) -// --------------------------------------------------------------------------- -// Mirrors Go's UnivSketch.calcGSumHeuristic(g=1, isCard=true): -// Y[L-1] = count of heap items at top layer with count > threshold -// for i from L-2 down to 0: -// tmp = Σ coe*1 for items with count > threshold -// coe = 1 - 2 * ((Hash64(key) >> (i+1)) & 1) -// Y[i] = 2*Y[i+1] + tmp -// return Y[0] -// l2_val = sqrt(median_of_first_3(layer.sketch.l2)) - -fn univmon_cardinality(state: &UnivMonState) -> f64 { - let nlayers = state.layers.len(); - if nlayers == 0 { - return 0.0; - } - - let mut y = vec![0.0f64; nlayers]; - - // Top layer - let top = &state.layers[nlayers - 1]; - let l2_val = cs_l2_from_state(top.sketch.as_ref()); - let threshold = (l2_val * 0.01) as i64; - let mut tmp = 0.0f64; - if let Some(heap) = &top.heap { - for entry in &heap.entries { - if entry.count as i64 > threshold { - tmp += 1.0; - } - } - } - y[nlayers - 1] = tmp; - - // Lower layers from L-2 down to 0 - for i in (0..nlayers - 1).rev() { - tmp = 0.0; - let layer = &state.layers[i]; - let l2_val = cs_l2_from_state(layer.sketch.as_ref()); - let threshold = (l2_val * 0.01) as i64; - - if let Some(heap) = &layer.heap { - for entry in &heap.entries { - if entry.count as i64 > threshold { - let h = xxh3_64_seeded(SEED_0, entry.key.as_bytes()); - let bit = (h >> (i + 1)) & 1; - let coe = 1.0 - 2.0 * bit as f64; - tmp += coe; - } - } - } - y[i] = 2.0 * y[i + 1] + tmp; - } - - y[0] -} - -/// cs_l2_from_state mirrors Go's CountSketchUniv.cs_l2(): -/// f2_value = MedianOfThree(l2[0], l2[1], l2[2]) -/// return sqrt(f2_value) -/// The portable l2 values are raw int64 cast to float64. -fn cs_l2_from_state(cs: Option<&CountSketchState>) -> f64 { - let cs = match cs { - Some(s) => s, - None => return 0.0, - }; - let l2 = &cs.l2; - if l2.len() < 3 { - return 0.0; - } - let med = median_of_three_f64(l2[0], l2[1], l2[2]); - med.abs().sqrt() -} - -fn median_of_three_f64(a: f64, b: f64, c: f64) -> f64 { - if a <= b { - if b <= c { - b - } else if a <= c { - c - } else { - a - } - } else if a <= c { - a - } else if b <= c { - c - } else { - b - } -} - // --------------------------------------------------------------------------- // HydraSketch CountMin frequency query // --------------------------------------------------------------------------- From 83d2001b409b052b04edabe24b1b0554b2ffd4c5 Mon Sep 17 00:00:00 2001 From: GordonYuanyc Date: Fri, 2 Oct 2026 01:04:23 -0600 Subject: [PATCH 19/36] docs(hydra): defer fixture coverage to the golden README State no fixture count; point at asapv1_golden/README.md's Coverage section. The cross-language proto test reads eight envelope files. The CHANGELOG entry notes that native Hydra routes subkeys differently from HydraKllSketch. Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 6 ++++-- docs/asapv1_wire_format.md | 4 ++-- docs/message_pack_format.md | 9 ++++----- docs/tests.md | 2 +- 4 files changed, 11 insertions(+), 10 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 7a5d9c9d..6c70aa1f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -102,8 +102,10 @@ signals a backwards-compatible change. `with_seed`, `merge_refs` and `aggregate_hydrakll`; the proto file `hydra/hydra.proto` with `HydraState`, `HydraCell` and `HydraCounterType` (and their generated `proto::sketchlib` types); and `SketchEnvelope`'s - `hydra` field (16, now reserved). A matrix of KLLs keyed by string is - `Hydra` with a one-column schema and `HydraCounter::KLL`. + `hydra` field (16, now reserved). The nearest replacement is `Hydra` with a + one-column schema and `HydraCounter::KLL`, but it routes `label:value` + subkeys through the matrix hash at `HYDRA_SEED`, not `xxh32(key, row) % + cols`, so `HydraKllSketch` cells do not map onto the new grid. ### Fixed diff --git a/docs/asapv1_wire_format.md b/docs/asapv1_wire_format.md index ab9b7433..974ce4b1 100644 --- a/docs/asapv1_wire_format.md +++ b/docs/asapv1_wire_format.md @@ -1339,14 +1339,14 @@ Good direction (more compact, higher fidelity, less Rust-internal duplication), 2. **Golden byte-vector fixtures** in one shared repo, [`sketchlib-golden-bytes`](https://github.com/ProjectASAP/sketchlib-golden-bytes), which each implementation mounts at `asapv1_golden/`; both languages decode and re-encode them byte-identically. These replace the `portable`-as-oracle round-trip test. 3. **This registry**, mirrored, never independently allocated. -Fixtures exist for eleven `kind_id`s — HLL's three estimators, Count-Min, Count Sketch, compact KLL and Hydra's five counter variants — and `sketchlib-go` mirrors all but Hydra's. Every other kind here has a payload and **no fixture**, so this document is its only contract; `asapv1_golden/README.md` lists the gap. `portable` carries its own Go goldens. +`asapv1_golden/README.md`'s Coverage section lists which `kind_id`s have fixtures. Every other kind here has a payload and **no fixture**, so this document is its only contract. `portable` carries its own Go goldens. **Hash profile on the Go side.** Rust derives the hash spec from a generic `HashProfile` bound on the hasher type; Go has no generic hasher type, so there is nothing to derive from. On the Go side the profile is simply **written into** the metadata on encode and **read from** it on decode. Go MUST validate the profile it reads (same fail-closed intent as Rust): a sketch is only mergeable/queryable if its `hash_profile_id` + seeds match the profile Go is prepared to reproduce. -Sequencing: (2) covers eleven `kind_id`s, and the one `native bytes == portable bytes` test is HLL-only, since no other `portable` type emits ASAPv1 bytes. +Sequencing: (2) covers the `kind_id`s that README lists, and the one `native bytes == portable bytes` test is HLL-only, since no other `portable` type emits ASAPv1 bytes. Retire `portable` once the fixtures cover the rest. --- diff --git a/docs/message_pack_format.md b/docs/message_pack_format.md index 8c3a6b05..c3cacb36 100644 --- a/docs/message_pack_format.md +++ b/docs/message_pack_format.md @@ -169,11 +169,10 @@ decode → re-encode them byte-identically. The `kind_id` registry is mirrored verbatim with Go's `wire/asapmsgpack/magic_ids.go`, never independently allocated. -Fixtures exist for eleven `kind_id`s — HLL's three estimators, Count-Min, -Count Sketch, compact KLL and Hydra's five counter variants; `sketchlib-go` -checks all but Hydra's. Every other implemented -kind has **no ASAPv1 golden and therefore no ASAPv1 drift guard**; -[its README](https://github.com/ProjectASAP/sketchlib-golden-bytes#coverage) lists what is covered. +`asapv1_golden/README.md`'s +[Coverage section](https://github.com/ProjectASAP/sketchlib-golden-bytes#coverage) +lists which `kind_id`s have fixtures. Every other implemented kind has **no +ASAPv1 golden and therefore no ASAPv1 drift guard**. The older `portable` path is guarded separately, by the `sketchlib-go` goldens of its own listed above. diff --git a/docs/tests.md b/docs/tests.md index 66077092..da4b539e 100644 --- a/docs/tests.md +++ b/docs/tests.md @@ -2189,7 +2189,7 @@ Test file: [`tests/xtest_consumer.rs`](../tests/xtest_consumer.rs) | test_name | test_description | what_is_tested | | --- | --- | --- | -| `cross_language_proto` | Protobuf envelopes read from the cross-language fixture directory answer their point queries in band. | Asserts nothing unless fixtures are available: with no `XTEST_DIR` and no discoverable producer directory holding `tests/cross_language/xtest_producer_test.go`, it prints a skip line and returns. With fixtures it decodes nine `SketchEnvelope` files and records a pass or failure for each — `countmin.pb` min-frequency for `item:42` at least `101`, `kll.pb` p50 within 5% of `5000` and p99 within 5% of `9900`, `ddsketch.pb` the same two quantiles within 2%, `hll.pb` an ErtlMLE cardinality inside `[40_000, 65_000]`, `countsketch.pb` `cs:hot` at least `200`, `coco.pb` `coco:hot` at least `500`, `elastic.pb` `elephant` at least `900`, `univmon.pb` a cardinality inside `[1_000, 15_000]` — using the hash seeds `0xcafe3553` and `0x6a09e667`, and, when `countmin_sampled.pb` and `hll_sampled.pb` exist, requires `effective_sample_p` of `0.1` and a `rescale_count` by `1/p` within 5% of `100_000` and 6% of `200_000`. A single `assert!` at the end fails if any check failed; a missing or mistyped envelope panics on the spot. | +| `cross_language_proto` | Protobuf envelopes read from the cross-language fixture directory answer their point queries in band. | Asserts nothing unless fixtures are available: with no `XTEST_DIR` and no discoverable producer directory holding `tests/cross_language/xtest_producer_test.go`, it prints a skip line and returns. With fixtures it decodes eight `SketchEnvelope` files and records a pass or failure for each — `countmin.pb` min-frequency for `item:42` at least `101`, `kll.pb` p50 within 5% of `5000` and p99 within 5% of `9900`, `ddsketch.pb` the same two quantiles within 2%, `hll.pb` an ErtlMLE cardinality inside `[40_000, 65_000]`, `countsketch.pb` `cs:hot` at least `200`, `coco.pb` `coco:hot` at least `500`, `elastic.pb` `elephant` at least `900`, `univmon.pb` a cardinality inside `[1_000, 15_000]` — using the hash seeds `0xcafe3553` and `0x6a09e667`, and, when `countmin_sampled.pb` and `hll_sampled.pb` exist, requires `effective_sample_p` of `0.1` and a `rescale_count` by `1/p` within 5% of `100_000` and 6% of `200_000`. A single `assert!` at the end fails if any check failed; a missing or mistyped envelope panics on the spot. | ### HLL Custom Precision From 3a524a5acaf2a78ed9161af57edd69bb1933f840 Mon Sep 17 00:00:00 2001 From: GordonYuanyc Date: Fri, 2 Oct 2026 01:04:42 -0600 Subject: [PATCH 20/36] refactor(countl2hh)!: drop the native MessagePackCodec shim, add an ASAPv1 golden Removes `message_pack_format::native::countsketch_topk`, CountL2HH's `MessagePackCodec` pass-through; `serialize_to_bytes` / `deserialize_from_bytes` emit the same ASAPv1 bytes. CountL2HH has no proto message and no portable type. Bumps `asapv1_golden` to the commit adding `count_l2hh_2x4_seed7`, built from the sketch's serde form with signed cells across the msgpack integer widths, `l2` accumulators set apart from the cells, and seed index 7, and adds the golden test. Docs defer the fixture coverage to the golden README. Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 10 +++ asapv1_golden | 2 +- docs/asapv1_wire_format.md | 4 +- docs/message_pack_format.md | 7 +- docs/tests.md | 1 + .../native/countsketch_topk.rs | 15 ---- src/message_pack_format/native/mod.rs | 1 - tests/asapv1_golden.rs | 80 ++++++++++++++++++- 8 files changed, 95 insertions(+), 25 deletions(-) delete mode 100644 src/message_pack_format/native/countsketch_topk.rs diff --git a/CHANGELOG.md b/CHANGELOG.md index 80d85f98..32ee6684 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -66,6 +66,9 @@ signals a backwards-compatible change. worker's seed and the aggregator's from one seed. The unseeded constructors draw from the thread generator. The seed is not serialized, and the ASAPv1 payload is unchanged. +- An ASAPv1 golden fixture for CountL2HH (`0x19 0x00`), `count_l2hh_2x4_seed7`: + signed cells across the msgpack integer widths, `l2` accumulators set apart + from the cells, and a non-zero seed index. ### Changed @@ -88,6 +91,13 @@ signals a backwards-compatible change. as an empty window like any other, and that window takes a place in `max_windows`. +### Removed + +- **CountL2HH's `MessagePackCodec` impl** and its module, + `message_pack_format::native::countsketch_topk`. Use + `CountL2HH::serialize_to_bytes` / `CountL2HH::deserialize_from_bytes`, which + emit the same ASAPv1 bytes. + ### Fixed - `KLLDynamic` panicked on every quantile query once a NaN had entered the diff --git a/asapv1_golden b/asapv1_golden index 8287a4ae..a70fa1ab 160000 --- a/asapv1_golden +++ b/asapv1_golden @@ -1 +1 @@ -Subproject commit 8287a4ae04e6cfbfb3ed33dafbf1dac77d8f7fc1 +Subproject commit a70fa1ab1e9d98cb6cc722d558e1efa5a04885ae diff --git a/docs/asapv1_wire_format.md b/docs/asapv1_wire_format.md index fd14e70a..2854be70 100644 --- a/docs/asapv1_wire_format.md +++ b/docs/asapv1_wire_format.md @@ -1341,14 +1341,14 @@ Good direction (more compact, higher fidelity, less Rust-internal duplication), 2. **Golden byte-vector fixtures** in one shared repo, [`sketchlib-golden-bytes`](https://github.com/ProjectASAP/sketchlib-golden-bytes), which each implementation mounts at `asapv1_golden/`; both languages decode and re-encode them byte-identically. These replace the `portable`-as-oracle round-trip test. 3. **This registry**, mirrored, never independently allocated. -Fixtures exist for six `kind_id`s — HLL's three estimators, Count-Min, Count Sketch and compact KLL — and `sketchlib-go` mirrors those. Every other kind here has a payload and **no fixture**, so this document is its only contract; `asapv1_golden/README.md` lists the gap. `portable` carries its own Go goldens. +`sketchlib-go` mirrors every fixture; the [Coverage section](https://github.com/ProjectASAP/sketchlib-golden-bytes#coverage) of `asapv1_golden/README.md` lists the `kind_id`s they cover. For a kind without a fixture, this document is the only contract. `portable` carries its own Go goldens. **Hash profile on the Go side.** Rust derives the hash spec from a generic `HashProfile` bound on the hasher type; Go has no generic hasher type, so there is nothing to derive from. On the Go side the profile is simply **written into** the metadata on encode and **read from** it on decode. Go MUST validate the profile it reads (same fail-closed intent as Rust): a sketch is only mergeable/queryable if its `hash_profile_id` + seeds match the profile Go is prepared to reproduce. -Sequencing: (2) covers six `kind_id`s, and the one `native bytes == portable bytes` test is HLL-only, since no other `portable` type emits ASAPv1 bytes. +Sequencing: the one `native bytes == portable bytes` test is HLL-only, since no other `portable` type emits ASAPv1 bytes. Retire `portable` once the fixtures cover the rest. --- diff --git a/docs/message_pack_format.md b/docs/message_pack_format.md index 12a2b91c..956a8180 100644 --- a/docs/message_pack_format.md +++ b/docs/message_pack_format.md @@ -169,10 +169,9 @@ decode → re-encode them byte-identically. The `kind_id` registry is mirrored verbatim with Go's `wire/asapmsgpack/magic_ids.go`, never independently allocated. -That proof covers the six `kind_id`s a fixture exists for — HLL's three -estimators, Count-Min, Count Sketch and compact KLL. Every other implemented -kind has **no ASAPv1 golden and therefore no ASAPv1 drift guard**; -[its README](https://github.com/ProjectASAP/sketchlib-golden-bytes#coverage) lists what is covered. +That proof covers only the `kind_id`s a fixture exists for; +[its README](https://github.com/ProjectASAP/sketchlib-golden-bytes#coverage) lists them. A kind +without a fixture has **no ASAPv1 drift guard**. The older `portable` path is guarded separately, by the `sketchlib-go` goldens of its own listed above. diff --git a/docs/tests.md b/docs/tests.md index ccb1571c..3a2ad540 100644 --- a/docs/tests.md +++ b/docs/tests.md @@ -2123,6 +2123,7 @@ Test file: [`tests/asapv1_golden.rs`](../tests/asapv1_golden.rs) | --- | --- | --- | | `cms_f64_fast_2x3_matches_golden` | A known f64 fast-path Count-Min matrix serializes to its checked-in golden bytes. | Verifies `CountMin::, FastPath>::from_storage` over the `2x3` matrix `[[0.0, 1.5, 2.25], [3.75, 4.125, 5.0625]]` serializes to `asapv1_golden/cms_f64_fast_2x3.hex`, and that decoding that fixture recovers the six cells in row-major order and re-serializes to the same bytes. | | `cms_i64_regular_2x3_matches_golden` | A known i64 regular-path Count-Min matrix serializes to its checked-in golden bytes. | Verifies `CountMin::, RegularPath>::from_storage` over the `2x3` matrix `[[0, 1, 127], [128, 300, 65536]]` serializes to `asapv1_golden/cms_i64_regular_2x3.hex`, and that decoding the fixture reports `rows()` `2`, `cols()` `3`, the six cells in row-major order, and a byte-identical re-serialization. | +| `count_l2hh_2x4_seed7_matches_golden` | A known CountL2HH state, its `l2` accumulators and seed index included, serializes to its checked-in golden bytes. | Builds `CountL2HH` from its named serde form with counts `[[127, 128, 65535, 4294967296], [-32, -33, -2147483648, i64::MIN]]`, `l2` `[256, i64::MAX]` and `seed_idx` `7`, verifies it serializes to `asapv1_golden/count_l2hh_2x4_seed7.hex`, and that decoding the fixture reports `rows()` `2`, `cols()` `4`, `seed_idx()` `7`, the eight cells, the two `l2` accumulators, and a byte-identical re-serialization. | | `cs_i32_regular_2x4_matches_golden` | The i32 counter type reaches the metadata, and an i32 golden is not an i64 sketch. | Verifies `Count::, RegularPath>::from_storage` over `[[0, 127, 128, 65536], [-1, -33, -32768, -2147483648]]` cast to `i32` serializes to `asapv1_golden/cs_i32_regular_2x4.hex`, that decoding it recovers the eight cells and re-serializes identically, that the i32 and i64 regular fixtures have equal length yet differ, and that the i32 fixture fails to decode as `Count, RegularPath>`. | | `cs_i64_fast_2x4_matches_golden` | The fast-path mode reaches the bytes, so the same matrix encodes differently per path. | Verifies `Count::, FastPath>::from_storage` over `[[0, 127, 128, 65536], [-1, -33, -32768, -2147483648]]` serializes to `asapv1_golden/cs_i64_fast_2x4.hex`, that decoding it recovers the eight cells and re-serializes identically, and that the fixture differs from `cs_i64_regular_2x4.hex` built from the same matrix. | | `cs_i64_regular_2x4_matches_golden` | A known signed Count-Sketch matrix serializes to its checked-in golden bytes. | Verifies `Count::, RegularPath>::from_storage` over `[[0, 127, 128, 65536], [-1, -33, -32768, -2147483648]]` — sweeping the msgpack integer widths in both directions — serializes to `asapv1_golden/cs_i64_regular_2x4.hex`, and that decoding the fixture reports `rows()` `2`, `cols()` `4`, the eight cells, and a byte-identical re-serialization. | diff --git a/src/message_pack_format/native/countsketch_topk.rs b/src/message_pack_format/native/countsketch_topk.rs deleted file mode 100644 index 9cc4e26c..00000000 --- a/src/message_pack_format/native/countsketch_topk.rs +++ /dev/null @@ -1,15 +0,0 @@ -//! Native MessagePack codec impl for [`crate::sketches::countsketch_topk::CountL2HH`]. - -use crate::message_pack_format::{Error, MessagePackCodec}; -use crate::sketches::countsketch_topk::CountL2HH; -use crate::{HashProfile, SketchHasher}; - -impl MessagePackCodec for CountL2HH { - fn to_msgpack(&self) -> Result, Error> { - Ok(self.serialize_to_bytes()?) - } - - fn from_msgpack(bytes: &[u8]) -> Result { - Ok(Self::deserialize_from_bytes(bytes)?) - } -} diff --git a/src/message_pack_format/native/mod.rs b/src/message_pack_format/native/mod.rs index 0a83ef96..cfaaae1a 100644 --- a/src/message_pack_format/native/mod.rs +++ b/src/message_pack_format/native/mod.rs @@ -9,7 +9,6 @@ pub mod countminsketch; pub mod countsketch; -pub mod countsketch_topk; pub mod ddsketch; pub mod hll; pub mod kll; diff --git a/tests/asapv1_golden.rs b/tests/asapv1_golden.rs index 30b41651..522300af 100644 --- a/tests/asapv1_golden.rs +++ b/tests/asapv1_golden.rs @@ -9,8 +9,8 @@ //! See `asapv1_golden/README.md`. use asap_sketchlib::{ - Classic, Count, CountMin, ErtlMLE, FastPath, HllSketch, HllVariant, HyperLogLogHIPP12, - HyperLogLogP12, KLL, MessagePackCodec, RegularPath, Vector2D, + Classic, Count, CountL2HH, CountMin, ErtlMLE, FastPath, HllSketch, HllVariant, + HyperLogLogHIPP12, HyperLogLogP12, KLL, MessagePackCodec, RegularPath, Vector2D, }; fn decode_hex(s: &str) -> Vec { @@ -30,6 +30,7 @@ const GOLDEN_CMS_F64: &str = include_str!("../asapv1_golden/cms_f64_fast_2x3.hex const GOLDEN_CS_REGULAR: &str = include_str!("../asapv1_golden/cs_i64_regular_2x4.hex"); const GOLDEN_CS_FAST: &str = include_str!("../asapv1_golden/cs_i64_fast_2x4.hex"); const GOLDEN_CS_I32: &str = include_str!("../asapv1_golden/cs_i32_regular_2x4.hex"); +const GOLDEN_L2HH: &str = include_str!("../asapv1_golden/count_l2hh_2x4_seed7.hex"); const GOLDEN_KLL_F64: &str = include_str!("../asapv1_golden/kll_f64_k200.hex"); const GOLDEN_KLL_I64: &str = include_str!("../asapv1_golden/kll_i64_k200.hex"); @@ -287,3 +288,78 @@ fn kll_i64_k200_matches_golden() { assert_eq!(decoded.quantile(0.0), 1.0); assert_eq!(decoded.quantile(1.0), 50.0); } + +// --------------------------------------------------------------------------- +// CountL2HH: counts, l2 accumulators and seed index set through the sketch's +// serde form -> serialize == golden, and golden round-trips. +// --------------------------------------------------------------------------- + +/// CountL2HH cells are signed: row 0 is positive fixint max / uint8 / uint16 / +/// uint64, row 1 is negative fixint min / int8 / int32 / int64. +const L2HH_COUNTS: [[i64; 4]; 2] = [ + [127, 128, 65535, 4294967296], + [-32, -33, -2147483648, i64::MIN], +]; +/// One accumulator per row, set independently of the cells: uint16 / uint64. +const L2HH_L2: [i64; 2] = [256, i64::MAX]; +const L2HH_SEED_INDEX: usize = 7; + +#[derive(serde::Serialize, serde::Deserialize)] +struct L2hhMatrixForm { + data: Vec, + rows: usize, + cols: usize, +} + +#[derive(serde::Serialize, serde::Deserialize)] +struct L2hhVectorForm { + data: Vec, +} + +/// The named serde form of `CountL2HH`. +#[derive(serde::Serialize, serde::Deserialize)] +struct L2hhForm { + counts: L2hhMatrixForm, + l2: L2hhVectorForm, + row: usize, + col: usize, + seed_idx: usize, +} + +fn l2hh_form(counts: &[[i64; 4]; 2], l2: &[i64; 2], seed_idx: usize) -> L2hhForm { + L2hhForm { + counts: L2hhMatrixForm { + data: counts.iter().flatten().copied().collect(), + rows: 2, + cols: 4, + }, + l2: L2hhVectorForm { data: l2.to_vec() }, + row: 2, + col: 4, + seed_idx, + } +} + +#[test] +fn count_l2hh_2x4_seed7_matches_golden() { + let want = decode_hex(GOLDEN_L2HH); + + let form = l2hh_form(&L2HH_COUNTS, &L2HH_L2, L2HH_SEED_INDEX); + let sketch: CountL2HH = + rmp_serde::from_slice(&rmp_serde::to_vec_named(&form).expect("serde form")) + .expect("CountL2HH from its serde form"); + let got = sketch.serialize_to_bytes().expect("serialize"); + assert_eq!(got, want, "CountL2HH bytes diverge from golden"); + + let decoded: CountL2HH = CountL2HH::deserialize_from_bytes(&want).expect("decode"); + let flat: Vec = L2HH_COUNTS.iter().flatten().copied().collect(); + assert_eq!(decoded.as_storage().as_slice(), flat.as_slice()); + assert_eq!(decoded.rows(), 2); + assert_eq!(decoded.cols(), 4); + assert_eq!(decoded.seed_idx(), L2HH_SEED_INDEX); + let state: L2hhForm = + rmp_serde::from_slice(&rmp_serde::to_vec_named(&decoded).expect("serde form")) + .expect("decoded serde form"); + assert_eq!(state.l2.data, L2HH_L2); + assert_eq!(decoded.serialize_to_bytes().expect("re-serialize"), want); +} From 1d5886de04652e463bd33eb36231cd6752937b68 Mon Sep 17 00:00:00 2001 From: GordonYuanyc Date: Fri, 2 Oct 2026 01:06:28 -0600 Subject: [PATCH 21/36] test(coco): replace the golden with a 3x7 table; defer fixture counts to the golden README A width that is not a power of two separates the spec's % cols fold from a mask fold. The fixture adds a 4-byte UTF-8 key. Docs no longer state how many kind_ids have fixtures; they point at the golden README's Coverage section. Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 4 +-- asapv1_golden | 2 +- docs/asapv1_wire_format.md | 4 +-- docs/message_pack_format.md | 7 +++--- docs/tests.md | 2 +- tests/asapv1_golden.rs | 49 +++++++++++++++++++------------------ 6 files changed, 34 insertions(+), 34 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 898d5544..e084db81 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -66,8 +66,8 @@ signals a backwards-compatible change. worker's seed and the aggregator's from one seed. The unseeded constructors draw from the thread generator. The seed is not serialized, and the ASAPv1 payload is unchanged. -- **ASAPv1 golden fixture for Coco** (`0c 00`): `asapv1_golden/coco_3x8.hex`, - a 3x8 bucket table set directly, checked by `tests/asapv1_golden.rs`. +- **ASAPv1 golden fixture for Coco** (`0c 00`): `asapv1_golden/coco_3x7.hex`, + a 3x7 bucket table set directly, checked by `tests/asapv1_golden.rs`. ### Changed diff --git a/asapv1_golden b/asapv1_golden index f9bbc3d7..e60f34b1 160000 --- a/asapv1_golden +++ b/asapv1_golden @@ -1 +1 @@ -Subproject commit f9bbc3d7f7a093e046d5389f9d3afc26b8c5285d +Subproject commit e60f34b1521a9b6550f9062275f3a60877329318 diff --git a/docs/asapv1_wire_format.md b/docs/asapv1_wire_format.md index 7f1853c7..62d45d5a 100644 --- a/docs/asapv1_wire_format.md +++ b/docs/asapv1_wire_format.md @@ -1341,14 +1341,14 @@ Good direction (more compact, higher fidelity, less Rust-internal duplication), 2. **Golden byte-vector fixtures** in one shared repo, [`sketchlib-golden-bytes`](https://github.com/ProjectASAP/sketchlib-golden-bytes), which each implementation mounts at `asapv1_golden/`; both languages decode and re-encode them byte-identically. These replace the `portable`-as-oracle round-trip test. 3. **This registry**, mirrored, never independently allocated. -Fixtures exist for seven `kind_id`s — HLL's three estimators, Count-Min, Count Sketch, compact KLL and Coco. Every other kind here has a payload and **no fixture**, so this document is its only contract; `asapv1_golden/README.md` lists the gap. `portable` carries its own Go goldens. +`asapv1_golden/README.md`'s Coverage section lists which `kind_id`s have fixtures; for every other kind this document is the only contract. `portable` carries its own Go goldens. **Hash profile on the Go side.** Rust derives the hash spec from a generic `HashProfile` bound on the hasher type; Go has no generic hasher type, so there is nothing to derive from. On the Go side the profile is simply **written into** the metadata on encode and **read from** it on decode. Go MUST validate the profile it reads (same fail-closed intent as Rust): a sketch is only mergeable/queryable if its `hash_profile_id` + seeds match the profile Go is prepared to reproduce. -Sequencing: (2) covers six `kind_id`s, and the one `native bytes == portable bytes` test is HLL-only, since no other `portable` type emits ASAPv1 bytes. +Sequencing: (2) covers the `kind_id`s `asapv1_golden/README.md`'s Coverage section lists, and the one `native bytes == portable bytes` test is HLL-only, since no other `portable` type emits ASAPv1 bytes. Retire `portable` once the fixtures cover the rest. --- diff --git a/docs/message_pack_format.md b/docs/message_pack_format.md index ae238e7e..108f133f 100644 --- a/docs/message_pack_format.md +++ b/docs/message_pack_format.md @@ -169,10 +169,9 @@ decode → re-encode them byte-identically. The `kind_id` registry is mirrored verbatim with Go's `wire/asapmsgpack/magic_ids.go`, never independently allocated. -That proof covers the seven `kind_id`s a fixture exists for — HLL's three -estimators, Count-Min, Count Sketch, compact KLL and Coco. Every other implemented -kind has **no ASAPv1 golden and therefore no ASAPv1 drift guard**; -[its README](https://github.com/ProjectASAP/sketchlib-golden-bytes#coverage) lists what is covered. +That proof covers only the `kind_id`s that have a fixture; +[its README's Coverage section](https://github.com/ProjectASAP/sketchlib-golden-bytes#coverage) +lists which. A kind without one has **no ASAPv1 drift guard**. The older `portable` path is guarded separately, by the `sketchlib-go` goldens of its own listed above. diff --git a/docs/tests.md b/docs/tests.md index 26dbd0de..88ac8982 100644 --- a/docs/tests.md +++ b/docs/tests.md @@ -2123,7 +2123,7 @@ Test file: [`tests/asapv1_golden.rs`](../tests/asapv1_golden.rs) | --- | --- | --- | | `cms_f64_fast_2x3_matches_golden` | A known f64 fast-path Count-Min matrix serializes to its checked-in golden bytes. | Verifies `CountMin::, FastPath>::from_storage` over the `2x3` matrix `[[0.0, 1.5, 2.25], [3.75, 4.125, 5.0625]]` serializes to `asapv1_golden/cms_f64_fast_2x3.hex`, and that decoding that fixture recovers the six cells in row-major order and re-serializes to the same bytes. | | `cms_i64_regular_2x3_matches_golden` | A known i64 regular-path Count-Min matrix serializes to its checked-in golden bytes. | Verifies `CountMin::, RegularPath>::from_storage` over the `2x3` matrix `[[0, 1, 127], [128, 300, 65536]]` serializes to `asapv1_golden/cms_i64_regular_2x3.hex`, and that decoding the fixture reports `rows()` `2`, `cols()` `3`, the six cells in row-major order, and a byte-identical re-serialization. | -| `coco_3x8_matches_golden` | A known Coco bucket table serializes to its checked-in golden bytes. | Sets 14 of the 24 buckets of `Coco::init_with_size(8, 3)` directly — each key in the column its row hashes it to, values spanning positive fixint through uint64 at both ends of each width, keys including `""`, a 31-byte and a 32-byte key and a multi-byte UTF-8 key, and `"zero"` holding `0` — verifies it serializes to `asapv1_golden/coco_3x8.hex`, and that decoding the fixture reports `d` `3`, `w` `8`, all 24 buckets in row-major order with the other ten unoccupied, and a byte-identical re-serialization. | +| `coco_3x7_matches_golden` | A known Coco bucket table serializes to its checked-in golden bytes. | Sets 15 of the 21 buckets of `Coco::init_with_size(7, 3)` directly — each key in the column its row hashes it to folded `% 7`, a width that is not a power of two so a mask fold misplaces the keys; values spanning positive fixint through uint64 at both ends of each width; keys including `""`, a 31-byte and a 32-byte key and UTF-8 code points of two, three and four bytes; and `"zero"` holding `0` — verifies it serializes to `asapv1_golden/coco_3x7.hex`, and that decoding the fixture reports `d` `3`, `w` `7`, all 21 buckets in row-major order with the other six unoccupied, and a byte-identical re-serialization. | | `cs_i32_regular_2x4_matches_golden` | The i32 counter type reaches the metadata, and an i32 golden is not an i64 sketch. | Verifies `Count::, RegularPath>::from_storage` over `[[0, 127, 128, 65536], [-1, -33, -32768, -2147483648]]` cast to `i32` serializes to `asapv1_golden/cs_i32_regular_2x4.hex`, that decoding it recovers the eight cells and re-serializes identically, that the i32 and i64 regular fixtures have equal length yet differ, and that the i32 fixture fails to decode as `Count, RegularPath>`. | | `cs_i64_fast_2x4_matches_golden` | The fast-path mode reaches the bytes, so the same matrix encodes differently per path. | Verifies `Count::, FastPath>::from_storage` over `[[0, 127, 128, 65536], [-1, -33, -32768, -2147483648]]` serializes to `asapv1_golden/cs_i64_fast_2x4.hex`, that decoding it recovers the eight cells and re-serializes identically, and that the fixture differs from `cs_i64_regular_2x4.hex` built from the same matrix. | | `cs_i64_regular_2x4_matches_golden` | A known signed Count-Sketch matrix serializes to its checked-in golden bytes. | Verifies `Count::, RegularPath>::from_storage` over `[[0, 127, 128, 65536], [-1, -33, -32768, -2147483648]]` — sweeping the msgpack integer widths in both directions — serializes to `asapv1_golden/cs_i64_regular_2x4.hex`, and that decoding the fixture reports `rows()` `2`, `cols()` `4`, the eight cells, and a byte-identical re-serialization. | diff --git a/tests/asapv1_golden.rs b/tests/asapv1_golden.rs index 69f9e307..31e3cb36 100644 --- a/tests/asapv1_golden.rs +++ b/tests/asapv1_golden.rs @@ -32,7 +32,7 @@ const GOLDEN_CS_FAST: &str = include_str!("../asapv1_golden/cs_i64_fast_2x4.hex" const GOLDEN_CS_I32: &str = include_str!("../asapv1_golden/cs_i32_regular_2x4.hex"); const GOLDEN_KLL_F64: &str = include_str!("../asapv1_golden/kll_f64_k200.hex"); const GOLDEN_KLL_I64: &str = include_str!("../asapv1_golden/kll_i64_k200.hex"); -const GOLDEN_COCO: &str = include_str!("../asapv1_golden/coco_3x8.hex"); +const GOLDEN_COCO: &str = include_str!("../asapv1_golden/coco_3x7.hex"); /// The known P12 register pattern shared by all three HLL fixtures. fn p12_registers() -> Vec { @@ -293,40 +293,41 @@ fn kll_i64_k200_matches_golden() { // Coco: build a known bucket table -> serialize == golden, round-trips. // --------------------------------------------------------------------------- -/// The occupied buckets of the 3x8 Coco fixture as `(row, col, key, value)`; +/// The occupied buckets of the 3x7 Coco fixture as `(row, col, key, value)`; /// every other bucket is unoccupied. Each key sits in the column its row hashes -/// it to, which the codec checks on both sides. -const COCO_CELLS: [(usize, usize, &str, u64); 14] = [ - (0, 0, "uint16-max", 65535), - (0, 1, "fixint-max", 127), - (0, 2, "clé-ünïcode-流量", 4), - (0, 4, "fixstr-max-31-bytes-0123456789a", 2), - (0, 5, "uint8-min", 128), - (0, 6, "uint64-max", u64::MAX), - (0, 7, "", 1), - (1, 0, "uint64-min", 4294967296), - (1, 1, "uint32-min", 65536), - (1, 3, "str8-min-32-bytes-0123456789abcd", 3), - (1, 5, "zero", 0), - (1, 6, "uint32-max", 4294967295), - (1, 7, "uint8-max", 255), - (2, 5, "uint16-min", 256), +/// it to, folded `% 7`, which the codec checks on both sides. +const COCO_CELLS: [(usize, usize, &str, u64); 15] = [ + (0, 0, "uint32-min", 65536), + (0, 1, "uint16-max", 65535), + (0, 2, "fixint-max", 127), + (0, 3, "", 1), + (0, 4, "uint8-max", 255), + (0, 5, "emoji-😀", 5), + (1, 0, "uint16-min", 256), + (1, 1, "str8-min-32-bytes-0123456789abcd", 3), + (1, 2, "uint8-min", 128), + (1, 3, "clé-ünïcode-流量", 4), + (1, 4, "fixstr-max-31-bytes-0123456789a", 2), + (1, 5, "uint64-min", 4294967296), + (1, 6, "zero", 0), + (2, 1, "uint64-max", u64::MAX), + (2, 5, "uint32-max", 4294967295), ]; -/// The fixture's 24 buckets in row-major order. +/// The fixture's 21 buckets in row-major order. fn coco_buckets() -> Vec<(Option, u64)> { - let mut buckets = vec![(None, 0); 24]; + let mut buckets = vec![(None, 0); 21]; for (r, c, key, val) in COCO_CELLS { - buckets[r * 8 + c] = (Some(key.to_string()), val); + buckets[r * 7 + c] = (Some(key.to_string()), val); } buckets } #[test] -fn coco_3x8_matches_golden() { +fn coco_3x7_matches_golden() { let want = decode_hex(GOLDEN_COCO); - let mut sketch: Coco = Coco::init_with_size(8, 3); + let mut sketch: Coco = Coco::init_with_size(7, 3); for (r, c, key, val) in COCO_CELLS { sketch.table[r][c] = CocoBucket { full_key: Some(key.to_string()), @@ -337,7 +338,7 @@ fn coco_3x8_matches_golden() { assert_eq!(got, want, "Coco bytes diverge from golden"); let decoded: Coco = Coco::deserialize_from_bytes(&want).expect("decode"); - assert_eq!((decoded.d, decoded.w), (3, 8)); + assert_eq!((decoded.d, decoded.w), (3, 7)); let cells: Vec<(Option, u64)> = decoded .table .as_slice() From a959573cf9f901f35b6a9c29348a1c129130ee23 Mon Sep 17 00:00:00 2001 From: GordonYuanyc Date: Fri, 2 Oct 2026 01:06:57 -0600 Subject: [PATCH 22/36] fix(elastic): flag the golden's free bucket; defer fixture coverage to the README Set the eviction flag on the Elastic fixtures' free bucket, a state a vacated bucket keeps, and bump asapv1_golden to the regenerated bytes. The spec and message_pack_format.md point to the golden README's Coverage section instead of counting fixtures. Co-Authored-By: Claude Opus 5.5 --- asapv1_golden | 2 +- docs/asapv1_wire_format.md | 4 ++-- docs/message_pack_format.md | 7 +++---- docs/tests.md | 2 +- tests/asapv1_golden.rs | 8 ++++---- 5 files changed, 11 insertions(+), 12 deletions(-) diff --git a/asapv1_golden b/asapv1_golden index 0f0d62b6..c2aed956 160000 --- a/asapv1_golden +++ b/asapv1_golden @@ -1 +1 @@ -Subproject commit 0f0d62b68e5686a3f03b6390f90f7bd2ea36c927 +Subproject commit c2aed956cebd5ae6833a0f2752b913ee916bdf9c diff --git a/docs/asapv1_wire_format.md b/docs/asapv1_wire_format.md index 82c05ae4..30ba5496 100644 --- a/docs/asapv1_wire_format.md +++ b/docs/asapv1_wire_format.md @@ -1341,14 +1341,14 @@ Good direction (more compact, higher fidelity, less Rust-internal duplication), 2. **Golden byte-vector fixtures** in one shared repo, [`sketchlib-golden-bytes`](https://github.com/ProjectASAP/sketchlib-golden-bytes), which each implementation mounts at `asapv1_golden/`; both languages decode and re-encode them byte-identically. These replace the `portable`-as-oracle round-trip test. 3. **This registry**, mirrored, never independently allocated. -Fixtures exist for seven `kind_id`s — HLL's three estimators, Count-Min, Count Sketch, compact KLL and Elastic — and `sketchlib-go` mirrors all but Elastic. Every other kind here has a payload and **no fixture**, so this document is its only contract; `asapv1_golden/README.md` lists the gap. `portable` carries its own Go goldens. +`asapv1_golden/README.md`'s Coverage section lists which `kind_id`s have fixtures; for every other kind this document is the only contract. `portable` carries its own Go goldens. **Hash profile on the Go side.** Rust derives the hash spec from a generic `HashProfile` bound on the hasher type; Go has no generic hasher type, so there is nothing to derive from. On the Go side the profile is simply **written into** the metadata on encode and **read from** it on decode. Go MUST validate the profile it reads (same fail-closed intent as Rust): a sketch is only mergeable/queryable if its `hash_profile_id` + seeds match the profile Go is prepared to reproduce. -Sequencing: (2) covers seven `kind_id`s, and the one `native bytes == portable bytes` test is HLL-only, since no other `portable` type emits ASAPv1 bytes. +Sequencing: (2) covers the kinds that README lists, and the one `native bytes == portable bytes` test is HLL-only, since no other `portable` type emits ASAPv1 bytes. Retire `portable` once the fixtures cover the rest. --- diff --git a/docs/message_pack_format.md b/docs/message_pack_format.md index 7b25c084..59b211b5 100644 --- a/docs/message_pack_format.md +++ b/docs/message_pack_format.md @@ -169,10 +169,9 @@ decode → re-encode them byte-identically. The `kind_id` registry is mirrored verbatim with Go's `wire/asapmsgpack/magic_ids.go`, never independently allocated. -That proof covers the seven `kind_id`s a fixture exists for — HLL's three -estimators, Count-Min, Count Sketch, compact KLL and Elastic. Every other implemented -kind has **no ASAPv1 golden and therefore no ASAPv1 drift guard**; -[its README](https://github.com/ProjectASAP/sketchlib-golden-bytes#coverage) lists what is covered. +That proof covers only the `kind_id`s a fixture exists for, which +[its README's Coverage section](https://github.com/ProjectASAP/sketchlib-golden-bytes#coverage) +lists; every other implemented kind has **no ASAPv1 drift guard**. The older `portable` path is guarded separately, by the `sketchlib-go` goldens of its own listed above. diff --git a/docs/tests.md b/docs/tests.md index a5f8355c..ff735ae5 100644 --- a/docs/tests.md +++ b/docs/tests.md @@ -2126,7 +2126,7 @@ Test file: [`tests/asapv1_golden.rs`](../tests/asapv1_golden.rs) | `cs_i32_regular_2x4_matches_golden` | The i32 counter type reaches the metadata, and an i32 golden is not an i64 sketch. | Verifies `Count::, RegularPath>::from_storage` over `[[0, 127, 128, 65536], [-1, -33, -32768, -2147483648]]` cast to `i32` serializes to `asapv1_golden/cs_i32_regular_2x4.hex`, that decoding it recovers the eight cells and re-serializes identically, that the i32 and i64 regular fixtures have equal length yet differ, and that the i32 fixture fails to decode as `Count, RegularPath>`. | | `cs_i64_fast_2x4_matches_golden` | The fast-path mode reaches the bytes, so the same matrix encodes differently per path. | Verifies `Count::, FastPath>::from_storage` over `[[0, 127, 128, 65536], [-1, -33, -32768, -2147483648]]` serializes to `asapv1_golden/cs_i64_fast_2x4.hex`, that decoding it recovers the eight cells and re-serializes identically, and that the fixture differs from `cs_i64_regular_2x4.hex` built from the same matrix. | | `cs_i64_regular_2x4_matches_golden` | A known signed Count-Sketch matrix serializes to its checked-in golden bytes. | Verifies `Count::, RegularPath>::from_storage` over `[[0, 127, 128, 65536], [-1, -33, -32768, -2147483648]]` — sweeping the msgpack integer widths in both directions — serializes to `asapv1_golden/cs_i64_regular_2x4.hex`, and that decoding the fixture reports `rows()` `2`, `cols()` `4`, the eight cells, and a byte-identical re-serialization. | -| `elastic_4b_2x4_matches_golden` | A known Elastic heavy table and light layer serialize to their golden bytes. | Sets a 4-bucket `2x4` sketch's buckets to a free one, a 31-byte flow id with votes `127`/`128`, an empty flow id with votes `1`/`65535` and the eviction flag, and a 32-byte flow id with votes `i32::MAX`/`256` and the flag, and its light cells to `[[0, 255, 65536, i32::MAX], [-1, -33, -32768, i32::MIN]]`; verifies the bytes equal `asapv1_golden/elastic_4b_2x4.hex`, and that decoding the fixture recovers every bucket, `bktlen` `4`, the light dimensions and cells, and a byte-identical re-serialization. | +| `elastic_4b_2x4_matches_golden` | A known Elastic heavy table and light layer serialize to their golden bytes. | Sets a 4-bucket `2x4` sketch's buckets to a free one carrying the eviction flag, a 31-byte flow id with votes `127`/`128`, an empty flow id with votes `1`/`65535` and the eviction flag, and a 32-byte flow id with votes `i32::MAX`/`256` and the flag, and its light cells to `[[0, 255, 65536, i32::MAX], [-1, -33, -32768, i32::MIN]]`; verifies the bytes equal `asapv1_golden/elastic_4b_2x4.hex`, and that decoding the fixture recovers every bucket, `bktlen` `4`, the light dimensions and cells, and a byte-identical re-serialization. | | `elastic_4b_2x4_stale_matches_golden` | `stale_copies` reaches the bytes, and nothing else differs. | Builds the same state on a 2-bucket sketch after `expand_heavy()`, verifies the bytes equal `asapv1_golden/elastic_4b_2x4_stale.hex`, that decoding recovers the same state and re-serializes identically, and that the two fixtures have equal length and differ in exactly one byte, `0xc2` against `0xc3`. | | `hll_classic_p12_matches_golden` | A known P12 Classic register pattern serializes to its golden bytes on both the portable and native paths. | Builds `HllSketch::from_raw(HllVariant::Regular, 12, regs, 0.0, 0.0, 0.0)` where `regs` is 4,096 zeros with `[0] = 1`, `[1] = 7`, `[100] = 42` and `[4095] = 3`, verifies `to_msgpack()` equals `asapv1_golden/hll_classic_p12.hex`, that `HyperLogLogP12::::deserialize_from_bytes` on the fixture recovers those registers and re-serializes identically, and that `HllSketch::from_msgpack` reports the same registers, `precision` `12` and `HllVariant::Regular`. | | `hll_ertl_mle_p12_matches_golden` | The Datafusion variant tag reaches the bytes for the same P12 register pattern. | Builds `HllSketch::from_raw(HllVariant::Datafusion, 12, regs, 0.0, 0.0, 0.0)` over the shared P12 pattern (`[0] = 1`, `[1] = 7`, `[100] = 42`, `[4095] = 3` in 4,096 registers), verifies `to_msgpack()` equals `asapv1_golden/hll_ertl_mle_p12.hex`, that `HyperLogLogP12::::deserialize_from_bytes` recovers the registers and re-serializes identically, and that `HllSketch::from_msgpack` reports `HllVariant::Datafusion`. | diff --git a/tests/asapv1_golden.rs b/tests/asapv1_golden.rs index 59576aba..f744e41d 100644 --- a/tests/asapv1_golden.rs +++ b/tests/asapv1_golden.rs @@ -295,11 +295,11 @@ fn kll_i64_k200_matches_golden() { // The two fixtures hold the same state and differ only by `stale_copies`. // --------------------------------------------------------------------------- -/// Heavy buckets `(flow_id, vote_pos, vote_neg, eviction)`: a free bucket, a -/// 31-byte fixstr id, an empty id and a 32-byte str8 id; the votes sweep -/// positive fixint / uint8 / uint16 / uint32. +/// Heavy buckets `(flow_id, vote_pos, vote_neg, eviction)`: a free bucket with +/// the eviction flag, a 31-byte fixstr id, an empty id and a 32-byte str8 id; +/// the votes sweep positive fixint / uint8 / uint16 / uint32. const ELASTIC_HEAVY: [(&str, i32, i32, bool); 4] = [ - ("", 0, 0, false), + ("", 0, 0, true), ("10.0.0.1:443>192.168.10.20:5123", 127, 128, false), ("", 1, 65535, true), ("10.0.0.1:443>192.168.10.20:51234", 2147483647, 256, true), From b7716fa954c183124ca08e9c6564c92d19d0cc65 Mon Sep 17 00:00:00 2001 From: GordonYuanyc Date: Fri, 2 Oct 2026 01:07:07 -0600 Subject: [PATCH 23/36] fix(ddsketch): address review of the ASAPv1-only change - Add the empty-state DDSketch golden (ddsketch_empty_a001) and its test; build the golden mirror state with named fields. - Restore a core-only small-n probe that separates ceil(q*n) from floor(q*(n-1)). - Replace the one-variant DdRankConvention with dd_rank_index / dd_order_statistic. - Spec: alpha is carried as given, never re-derived from gamma. - Docs state current facts only; fixture counts defer to the golden README's Coverage section. Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 7 ++-- asapv1_golden | 2 +- docs/asapv1_wire_format.md | 10 +++--- docs/ddsketch_signed_values.md | 10 ++---- docs/e2e_testing_harness.md | 4 +-- docs/message_pack_format.md | 8 ++--- docs/tests.md | 4 ++- src/sketches/ddsketch/wire.rs | 3 +- tests/README.md | 2 +- tests/asapv1_golden.rs | 41 ++++++++++++++++++--- tests/common/conformance.rs | 5 ++- tests/common/specs.rs | 66 ++++++++++------------------------ tests/e2e/quantiles.rs | 47 ++++++++++++++++++++++-- 13 files changed, 127 insertions(+), 82 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 180401bd..9639f712 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -66,9 +66,10 @@ signals a backwards-compatible change. worker's seed and the aggregator's from one seed. The unseeded constructors draw from the thread generator. The seed is not serialized, and the ASAPv1 payload is unchanged. -- **DDSketch ASAPv1 golden fixtures**: `ddsketch_positive_a001` (metadata - version 1) and `ddsketch_signed_a001` (metadata version 2, negative store and - zero count), checked by `tests/asapv1_golden.rs`. +- **DDSketch ASAPv1 golden fixtures**: `ddsketch_positive_a001` and + `ddsketch_empty_a001` (metadata version 1) and `ddsketch_signed_a001` + (metadata version 2, negative store and zero count), checked by + `tests/asapv1_golden.rs`. ### Changed diff --git a/asapv1_golden b/asapv1_golden index 1d2d0ffd..52df5d51 160000 --- a/asapv1_golden +++ b/asapv1_golden @@ -1 +1 @@ -Subproject commit 1d2d0ffd3b0f3deb0efa506ad65f6c83e418e38e +Subproject commit 52df5d51bbb588bc374fd6700bfc6cba9cb96986 diff --git a/docs/asapv1_wire_format.md b/docs/asapv1_wire_format.md index 97bdd7ae..a4cfcdb0 100644 --- a/docs/asapv1_wire_format.md +++ b/docs/asapv1_wire_format.md @@ -673,7 +673,7 @@ Wire counter types are the base Count-Min's: **`"i32"`, `"i64"` and `"f64"`**. ` | 3 | `min` | f64 | smallest value as ingested; `+inf` when empty | | 4 | `max` | f64 | largest value as ingested; `-inf` when empty | -**Metadata vs payload.** `alpha` shapes every bucket index and is chosen at construction, so per the config-to-metadata rule it is a structural param, like HLL's `precision`. `gamma`, `log_gamma` and `inv_log_gamma` are derived from it (`gamma = (1 + alpha) / (1 - alpha)`) and appear nowhere: the decoder re-derives them exactly as `DDSketch::new` does. The store's dimensions are not configuration — the store grows with the data — so there is no `rows`/`cols` analogue; `counts` is self-delimiting and `offset` positions it. The metadata is `metadata_version` then `alpha`. +**Metadata vs payload.** `alpha` shapes every bucket index and is chosen at construction, so per the config-to-metadata rule it is a structural param, like HLL's `precision`. `gamma`, `log_gamma` and `inv_log_gamma` are derived from it (`gamma = (1 + alpha) / (1 - alpha)`) and appear nowhere: the decoder re-derives them exactly as `DDSketch::new` does. The store's dimensions are not configuration — the store grows with the data — so there is no `rows`/`cols` analogue; `counts` is self-delimiting and `offset` positions it. The metadata is `metadata_version` then `alpha`. `alpha` is the constructor's argument bit for bit, never re-derived from `gamma`. **DDSketch carries no hash-spec group.** It never hashes: it maps a raw `f64` through a logarithm and carries no hasher type parameter at all. The hash spec answers "how were keys hashed", and a sketch that does not hash has no truthful answer, so the group is omitted entirely. This is the Q-KLL precedent, and the reason DDSketch's metadata schema is not `HashProfile`-derived the way HLL's and Count-Min's are. @@ -681,9 +681,9 @@ Wire counter types are the base Count-Min's: **`"i32"`, `"i64"` and `"f64"`**. ` **`sum` saturates at signed `f64::MAX`.** All three paths that advance it — `add`, `apply_delta` and `merge` — go through one guarded step: the new total is taken when it is finite and `f64::MAX` with the overflow sign is stored otherwise. So a sketch that ingests enough mass to overflow the accumulator still serializes, and only `sum` degrades, only past `f64::MAX`. `count`, the bucket store, `min` and `max` do not pass through the guard and are unaffected. This is what keeps rule 7's finiteness requirement satisfiable: an unguarded `+=` would leave a legally-ingested sketch carrying `+inf`, which the same rule refuses on both doors, so a sketch nothing did wrong to would have no encoding. -**Signed extension (metadata version 2).** Positive-only states retain the five-field payload and version 1. States with negative observations or a zero count use version 2 and append three fields: `negative_counts: array`, `negative_offset: int`, and `zero_count: u64`. Negative values use the same mapping on their magnitudes; quantile traversal visits their buckets in descending index order, then zeros, then positive buckets in ascending order. Values smaller in magnitude than the mapping minimum count as zero; nonfinite and too-large magnitudes are rejected by `try_add`. The legacy `add` wrapper ignores those errors. +**Signed extension (metadata version 2).** Positive-only states retain the five-field payload and version 1. States with negative observations or a zero count use version 2 and append three fields: `negative_counts: array`, `negative_offset: int`, and `zero_count: u64`. Negative values use the same mapping on their magnitudes; quantile traversal visits their buckets in descending index order, then zeros, then positive buckets in ascending order. Values smaller in magnitude than the mapping minimum count as zero; nonfinite and too-large magnitudes are rejected by `try_add` and ignored by `add`. -For version 2 the total is `sum(counts) + sum(negative_counts) + zero_count`, checked for overflow. Both stores obey the span checks below. A nonempty signed state requires finite scalars and `min <= max`; the positive-only `min > 0` and `sum >= min` checks do not apply. Empty-state scalars remain unchanged. Version 1 readers reject version 2. The existing positive-only golden bytes remain unchanged. +For version 2 the total is `sum(counts) + sum(negative_counts) + zero_count`, checked for overflow. Both stores obey the span checks below. A nonempty signed state requires finite scalars and `min <= max`; the positive-only `min > 0` and `sum >= min` checks do not apply. Empty-state scalars are as in version 1. A decoder that knows only version 1 rejects version 2. **Emitted order (cross-language contract).** `counts` is the dense store in its own bucket-index order, ascending from `offset`, with the `GROW_CHUNK` padding zeros left in place and `offset` left where the sketch put it. Nothing is trimmed and nothing is sorted, so a decoded sketch re-serializes byte-identically and `store_counts()` / `store_offset()` mean the same thing on both sides. @@ -1341,14 +1341,14 @@ Good direction (more compact, higher fidelity, less Rust-internal duplication), 2. **Golden byte-vector fixtures** in one shared repo, [`sketchlib-golden-bytes`](https://github.com/ProjectASAP/sketchlib-golden-bytes), which each implementation mounts at `asapv1_golden/`; both languages decode and re-encode them byte-identically. These replace the `portable`-as-oracle round-trip test. 3. **This registry**, mirrored, never independently allocated. -Fixtures exist for seven `kind_id`s — HLL's three estimators, Count-Min, Count Sketch, DDSketch and compact KLL — and `sketchlib-go` mirrors all but DDSketch's. Every other kind here has a payload and **no fixture**, so this document is its only contract; `asapv1_golden/README.md` lists the gap. `portable` carries its own Go goldens. +`asapv1_golden/README.md` ("Coverage") lists which `kind_id`s have fixtures; a kind without one has this document as its only contract. `portable` carries its own Go goldens. **Hash profile on the Go side.** Rust derives the hash spec from a generic `HashProfile` bound on the hasher type; Go has no generic hasher type, so there is nothing to derive from. On the Go side the profile is simply **written into** the metadata on encode and **read from** it on decode. Go MUST validate the profile it reads (same fail-closed intent as Rust): a sketch is only mergeable/queryable if its `hash_profile_id` + seeds match the profile Go is prepared to reproduce. -Sequencing: (2) covers seven `kind_id`s, and the one `native bytes == portable bytes` test is HLL-only, since no other `portable` type emits ASAPv1 bytes. +Sequencing: (2) covers the kinds the golden README lists, and the one `native bytes == portable bytes` test is HLL-only, since no other `portable` type emits ASAPv1 bytes. Retire `portable` once the fixtures cover the rest. --- diff --git a/docs/ddsketch_signed_values.md b/docs/ddsketch_signed_values.md index 603629fa..1aab66d9 100644 --- a/docs/ddsketch_signed_values.md +++ b/docs/ddsketch_signed_values.md @@ -6,11 +6,7 @@ Quantiles use the ceil(q*n) order-statistic convention and exact ingested extrema. For indexable nonzero order statistics the estimate has absolute error at most `alpha * abs(value)`, for either sign. -If interpolation combines endpoints x and y with weight t, the absolute error is bounded by `alpha * ((1-t)*abs(x) + t*abs(y))`, excluding zero-mapped tiny values. Opposite signs can cancel, so the interpolated result need not have relative error alpha. Dividing quantiles additionally requires a finite nonzero true denominator and valid relative-error bounds for both operands. This change does not establish those planner preconditions. +## Wire and OctoSketch -## Wire and downstream upgrade - -- Native ASAPv1: positive-only states keep metadata version 1 and their existing bytes. Signed/zero states use metadata version 2 and append negative counts, negative offset, and zero count. Old native readers reject version 2. -- Octo: `DdDelta` now requires a `store: DdStore` discriminator. Existing positive delta constructors should use `DdStore::Positive`. Worker residual keys are `(DdStore, i32)`. Both promotion and flush preserve all three categories; partial counts still need flushing before complete queries. - -Go implementations of the ASAP wire formats and downstream dependency pins are not changed by this Rust library patch. +- ASAPv1: positive-only states use metadata version 1. States with negative samples or zeros use version 2, which appends negative counts, negative offset, and zero count. +- OctoSketch: `DdDelta` carries a `store: DdStore` discriminator, and worker residual keys are `(DdStore, i32)`. Promotion and flush preserve all three categories; partial counts need flushing before complete queries. diff --git a/docs/e2e_testing_harness.md b/docs/e2e_testing_harness.md index 35cd7438..76173de9 100644 --- a/docs/e2e_testing_harness.md +++ b/docs/e2e_testing_harness.md @@ -101,7 +101,7 @@ One spec per *metric*, not per sketch, because the metric is what differs: | `CountSketchSpec` | two-sided L2, rank-independent | marginal `sqrt(3/w)·‖f₋ᵢ‖₂` at `P[Bin(d, 1/3) ≥ ⌈d/2⌉]`; simultaneous at the smallest κ with `P[Bin(d,1/κ) ≥ ⌈d/2⌉] ≤ δ/D` | | `SecondMomentSpec` | F2 from a Count Sketch matrix | `sqrt(2κ/w)`, same median amplification | | `KllRankSpec` | KLL **maximum** rank error over a q grid | `ε(k) = 2.446 / k^0.9433` — an Apache DataSketches *characterization fit*, not a theorem about this code | -| `RelativeQuantileSpec` + `DdRankConvention` | DDSketch relative value error | `α + ULP slack` vs the exact `ceil(q·n)` order statistic | +| `RelativeQuantileSpec` + `dd_rank_index` | DDSketch relative value error | `α + ULP slack` vs the exact `ceil(q·n)` order statistic | | `CardinalityConfidenceSpec` | HLL / KMV | `z · σ_rel`, σ derived exactly, the tail a normal approximation | | `SamplingConfidenceSpec` | Nitro | `z · sqrt(f(p·r(1−r) + (1−p)/p))`, `r = frac(1/p)` | | `PrioritySampleSpec` | `UniformSampling` | `len = ⌈n·rate⌉` exactly; `Var[mean] = (σ_N²/m)(N−m)/(N−1)` | @@ -142,7 +142,7 @@ Keeping these apart is deliberate. There is no shared promise the same thing: a correct KLL can return a value 100× off on a heavy-tailed stream and still be within its rank guarantee, and a correct DDSketch has no rank guarantee at all. `DDSketch::get_value_at_quantile` -answers `sorted[ceil(q·n) − 1]`, and `DdRankConvention` names that truth. +answers `sorted[ceil(q·n) − 1]`, and `dd_rank_index` computes that truth. Two distinctions are easy to lose: diff --git a/docs/message_pack_format.md b/docs/message_pack_format.md index 708a79e8..b6d36bb2 100644 --- a/docs/message_pack_format.md +++ b/docs/message_pack_format.md @@ -169,10 +169,10 @@ decode → re-encode them byte-identically. The `kind_id` registry is mirrored verbatim with Go's `wire/asapmsgpack/magic_ids.go`, never independently allocated. -That proof covers the seven `kind_id`s a fixture exists for — HLL's three -estimators, Count-Min, Count Sketch, DDSketch and compact KLL. Every other implemented -kind has **no ASAPv1 golden and therefore no ASAPv1 drift guard**; -[its README](https://github.com/ProjectASAP/sketchlib-golden-bytes#coverage) lists what is covered. +That proof covers only the `kind_id`s a fixture exists for; a kind without one +has **no ASAPv1 drift guard**. +[The golden README's Coverage section](https://github.com/ProjectASAP/sketchlib-golden-bytes#coverage) +lists them. The older `portable` path is guarded separately, by the `sketchlib-go` goldens of its own listed above. diff --git a/docs/tests.md b/docs/tests.md index 8256c369..30e21d5f 100644 --- a/docs/tests.md +++ b/docs/tests.md @@ -1675,6 +1675,7 @@ Test file: [`tests/e2e/quantiles.rs`](../tests/e2e/quantiles.rs) | test_name | test_description | what_is_tested | | --- | --- | --- | | `ddsketch_satisfies_the_relative_value_error_contract` | DDSketch meets the relative-value-error guarantee against the `ceil(q*n)` order statistic. | Crosses `alpha in {0.001, 0.01, 0.05, 0.1}` x `n in {1_000, 20_000, 100_000}` x the six `dds_streams` shapes (`adversarial-bucket-edges` log-uniform on `gamma` edges, positive-filtered `normal(1000, 250)`, `exponential(1e-3)`, `uniform` over `[1e6, 1e7)`, `zipf(1.1)` over 8192 keys in `[1e6, 1e7]`, and `wide-dynamic-range` spanning nine decades). Asserts `DDSketch::get_count()` equals `n` (no sample dropped), then per alpha tallies `RelativeQuantileSpec::core(alpha)` on `get_value_at_quantile` over the q grid `[0.0, 0.01, 0.1, 0.5, 0.9, 0.99, 1.0]` and closes with `assert_none`: every estimate must be within `alpha + 8*f64::EPSILON*(1 + \|ln v\|)` of `sorted[ceil(q*n) - 1]`. Zero violations tolerated, since the guarantee is deterministic. | +| `ddsketch_answers_the_ceil_nearest_rank_order_statistic_at_small_n` | At small `n` the answer is the `ceil(q*n)` order statistic, not the `floor(q*(n-1))` one. | Runs 8 `(n, q, idx)` probes — `(3, 0.4, 1)`, `(4, 0.34, 1)`, `(4, 0.3, 1)`, `(7, 0.2, 1)`, `(7, 0.6, 4)`, `(5, 0.5, 2)`, `(10, 0.25, 2)`, `(10, 0.15, 1)` — on values `100 * gamma^(3i)` at `alpha = 0.01`, so neighbouring ranks are three buckets apart. Asserts `dd_rank_index(n, q) == idx`, that `get_value_at_quantile(q)` passes `RelativeQuantileSpec::check` against `sorted[idx]`, and that at least `3` probes have `floor(q*(n-1)) != idx`. | | `ddsketch_endpoints_are_exact` | The endpoints are the exact extremes. | For each of `alpha in {0.001, 0.01, 0.05, 0.1}` over 5_000 `log_uniform_f64` values (seed `4_242`, exponent range `3..30`), asserts `get_value_at_quantile(0.0) == Some(truth.min())`, `(1.0) == Some(truth.max())`, and `min()`/`max()` equal the exact extremes. | | `ddsketch_satisfies_the_relative_error_contract_at_bucket_boundaries` | Values on, just below and just above a bucket edge all stay inside `alpha` on both sides. | For each of the four alphas and `k in {-40, -7, 0, 1, 13, 60, 200}` computes `edge = gamma^k` and probes `edge`, `edge*(1 - f64::EPSILON)`, `edge*(1 + f64::EPSILON)`, `edge*sqrt(gamma)` and `edge*gamma*(1 - 1e-12)`. Each probe is the single sample of a fresh sketch, so the mapping is isolated from any rank effect; `get_value_at_quantile(0.5)` must satisfy `alpha + numerical_slack` against the probe itself. Probes the mapping refuses (`get_count() == 0`) are skipped rather than asserted on. | | `ddsketch_merge_and_delta_replay_preserve_the_relative_error_contract` | Shard merge and bucket-delta replay both leave the relative-error contract intact. | Per `alpha in {0.001, 0.01, 0.05, 0.1}` over `zipf_f64(40_000, 8_192, 1.1, 1e3, 1e7)`: a 4-way round-robin shard merge must report `get_count() == 40_000` exactly and satisfy the contract over the full q grid; a sketch rebuilt by replaying every non-zero `store_counts()` entry as an `octo_delta::DdDelta` at `store_offset() + i` must reproduce `get_count()` and satisfy the contract over the interior grid `DDS_QS[1..6] = [0.01, 0.1, 0.5, 0.9, 0.99]` (its endpoints are bucket representatives, so `q=0`/`q=1` are excluded). Both tallies use `assert_none`. | @@ -1868,7 +1869,7 @@ Test file: [`tests/e2e/windows.rs`](../tests/e2e/windows.rs) | `eh_coco_variant_conserves_the_retained_mass` | The merged Coco window holds exactly the retained mass. | Same EH, stream and merge over `COCO(Coco::init_with_size_and_seed(1024, 4, COCO_SEED))`, and verifies the merged table's recorded mass equals the number of events in the retained span. | | `eh_hll_variant_satisfies_the_register_error_model_over_the_retained_window` | A merged HLL payload lands inside the register error model for the retained span's cardinality. | Feeds `10_000` uniform draws over `200_000` (seed `0x0E11_0001`) into `ExponentialHistogram::new(8, 1_000_000, EHSketchList::HLL(HyperLogLog::::new()))`, merges the full retained span, and checks `merged.query(&DataInput::Str("card"))` against the exact distinct count of that span under `CardinalityConfidenceSpec::hll(14, 4.0)` - a relative band of `4 * 1.04 / 2^7 = 0.0325` at `m = 2^14` registers. This is a single trial, aggregated by `assert_independent_binomial` at `p = 6.3e-5`, so no violation is tolerated. | | `eh_kll_variant_satisfies_the_rank_error_characterization_over_the_retained_window` | A merged KLL payload's worst rank error stays inside the DataSketches characterization across independent compaction seeds. | Builds `12` separate `ExponentialHistogram::new(8, 1_000_000, EHSketchList::KLL(KLL::init_kll_with_seed(200, seed)))` instances with `seed = 0x5EED_0200 + t * 0x9E37_79B9_7F4A_7C15`, feeds each the same `10_000` uniform values over `1_000_000` (stream seed `0x0E11_0001`) at `t = 0..9_999`, merges each one's full retained span, and reduces each trial to its worst normalized rank error over the grid `[0.1, 0.25, 0.5, 0.75, 0.9]` read through `merged.query(&DataInput::F64(q))`; `KllRankSpec::datasketches(200)` supplies `eps = 2.446/200^0.9433 = 0.0165`, and the `12` trial outcomes are accepted by `assert_independent_binomial` at `p = 0.01`. | -| `eh_ddsketch_variant_satisfies_the_relative_value_error_contract_over_the_window` | A merged DDSketch payload holds relative value error and retains every observation. | Feeds `10_000` values `1_000_000 + uniform[0, 9_000_000)` (seed `0x0E11_0001`) into `ExponentialHistogram::new(8, 1_000_000, EHSketchList::DDS(DDSketch::new(0.01)))`, merges the full retained span, and checks `merged.query(&DataInput::F64(q))` at `q` in `[0.1, 0.25, 0.5, 0.75, 0.9]` against the span's exact order statistics under `RelativeQuantileSpec::core(0.01)` (`CeilNearestRank` convention, tolerance `alpha` plus a few-ULP slack) with zero violations; then asserts `merged.query(&DataInput::Str("count"))` equals the span's observation count exactly. | +| `eh_ddsketch_variant_satisfies_the_relative_value_error_contract_over_the_window` | A merged DDSketch payload holds relative value error and retains every observation. | Feeds `10_000` values `1_000_000 + uniform[0, 9_000_000)` (seed `0x0E11_0001`) into `ExponentialHistogram::new(8, 1_000_000, EHSketchList::DDS(DDSketch::new(0.01)))`, merges the full retained span, and checks `merged.query(&DataInput::F64(q))` at `q` in `[0.1, 0.25, 0.5, 0.75, 0.9]` against the span's exact order statistics under `RelativeQuantileSpec::core(0.01)` (`ceil(q*n)` order statistic, tolerance `alpha` plus a few-ULP slack) with zero violations; then asserts `merged.query(&DataInput::Str("count"))` equals the span's observation count exactly. | | `eh_univmon_variant_reports_the_exact_l1_over_the_retained_window` | A merged UnivMon payload reports L1 exactly and L2 inside the AMS second-moment band of its own grid. | Feeds `10_000` Zipf(1.1) keys over a `2_048` domain (seed `0x0E11_0001`) into `ExponentialHistogram::new(8, 1_000_000, EHSketchList::UNIVMON(UnivMon::init_univmon(32, 5, 2_048, 8)))`, merges the full retained span, and asserts `query(&DataInput::Str("l1"))` equals the span's exact total mass; then asserts `query(&DataInput::Str("l2"))` lies in `[\|\|f\|\|_2 * sqrt(1-b), \|\|f\|\|_2 * sqrt(1+b)]` with `b = SecondMomentSpec::new(5, 2_048).relative_bound() = sqrt(2*3/2048) = 0.0541`, i.e. about `-2.7%/+2.7%` on the norm. The bound asserted is the terminal layer's, not a composed constant for the recurrence. | | `every_eh_variant_can_merge_into_its_own_kind` | Every mergeable `EHSketchList` variant has a merge arm, so bucket consolidation cannot silently drop a payload. | For `CM` and `CS` at `3x256`, `COCO(256, 4)`, `COUNTL2HH` at `3x256`, `DDS(0.01)`, `ELASTIC(256)`, `HLL`, `KLL(200, seed 0x5EED_0300)`, `UNIVMON(32, 3, 256, 4)` and - under `--features experimental` - `UNIFORM(UniformSampling::with_seed(0.5, 7))`, feeds each side `64` distinct `DataInput::String` keys and `64` distinct `DataInput::F64` values (`k0..k63` / `1.0..64.0` on the left, `k64..k127` / `100.0..163.0` on the right) and asserts `left.merge(&right).is_ok()`. Only the `Ok` status is asserted; the merged contents are not read back. | | `eh_uniform_sampling_variant_reports_exact_retention_bookkeeping` | The reservoir payload's retention counters are exact over the merged span and its samples were all observed. | Gated on `#[cfg(feature = "experimental")]`. Feeds `10_000` uniform values over `1_000_000` (seed `0x0E11_0001`) into `ExponentialHistogram::new(8, 1_000_000, EHSketchList::UNIFORM(UniformSampling::with_seed(0.1, 0x5A_9101)))`, merges the full retained span, and asserts `query(&DataInput::Str("total_seen"))` equals `hi - lo + 1` exactly, that `query(&DataInput::Str("len"))` is greater than `0` and at most `total_seen`, and that every sample index `0..len` returns a value whose bit pattern appears in the span. The retained count is not held against the `0.1` rate's budget beyond `<= total_seen`. | @@ -2100,6 +2101,7 @@ Test file: [`tests/asapv1_golden.rs`](../tests/asapv1_golden.rs) | `cs_i32_regular_2x4_matches_golden` | The i32 counter type reaches the metadata, and an i32 golden is not an i64 sketch. | Verifies `Count::, RegularPath>::from_storage` over `[[0, 127, 128, 65536], [-1, -33, -32768, -2147483648]]` cast to `i32` serializes to `asapv1_golden/cs_i32_regular_2x4.hex`, that decoding it recovers the eight cells and re-serializes identically, that the i32 and i64 regular fixtures have equal length yet differ, and that the i32 fixture fails to decode as `Count, RegularPath>`. | | `cs_i64_fast_2x4_matches_golden` | The fast-path mode reaches the bytes, so the same matrix encodes differently per path. | Verifies `Count::, FastPath>::from_storage` over `[[0, 127, 128, 65536], [-1, -33, -32768, -2147483648]]` serializes to `asapv1_golden/cs_i64_fast_2x4.hex`, that decoding it recovers the eight cells and re-serializes identically, and that the fixture differs from `cs_i64_regular_2x4.hex` built from the same matrix. | | `cs_i64_regular_2x4_matches_golden` | A known signed Count-Sketch matrix serializes to its checked-in golden bytes. | Verifies `Count::, RegularPath>::from_storage` over `[[0, 127, 128, 65536], [-1, -33, -32768, -2147483648]]` — sweeping the msgpack integer widths in both directions — serializes to `asapv1_golden/cs_i64_regular_2x4.hex`, and that decoding the fixture reports `rows()` `2`, `cols()` `4`, the eight cells, and a byte-identical re-serialization. | +| `ddsketch_empty_a001_matches_golden` | The empty DDSketch state serializes to its golden bytes as metadata version 1, and the golden decodes back to it. | Builds the state at `alpha = 0.01` with empty stores, `sum = 0.0`, `min = +inf`, `max = -inf`, and verifies both it and a fresh `DDSketch::new(0.01)` serialize to `ddsketch_empty_a001.hex`; the decode has empty stores at offset `0`, a zero count and `zero_count`, `sum` `0.0`, `min()` and `max()` of `None`, and re-encodes byte-identically. | | `ddsketch_positive_a001_matches_golden` | A positive-only DDSketch state serializes to its golden bytes as metadata version 1, and the golden decodes back to that state. | Builds `DDSketch` at `alpha = 0.01` through its serde form from the store `[1, 0, 127, 128, 300, 65536, 4294967296]` at offset `-40`, `sum = 2181071000.0`, `min = 0.453125`, `max = 0.5078125`, and verifies `serialize_to_bytes` equals `ddsketch_positive_a001.hex`; the decode reports that `alpha`, store, offset and scalars, an empty negative store, a zero `zero_count`, a `get_count()` equal to the bucket sum, and re-encodes byte-identically. | | `ddsketch_signed_a001_matches_golden` | A signed DDSketch state serializes to its golden bytes as metadata version 2, and the golden decodes back to that state. | Builds `DDSketch` at `alpha = 0.01` from the positive store `[3, 0, 2]` at offset `310`, the negative store `[5, 1]` at offset `-208`, `zero_count = 7`, `sum = 2523.90625`, `min = -0.016`, `max = 515.0`, and verifies `serialize_to_bytes` equals `ddsketch_signed_a001.hex`; the decode reports both stores, both offsets, the zero count, the scalars and a `get_count()` of `18`, and re-encodes byte-identically. | | `hll_classic_p12_matches_golden` | A known P12 Classic register pattern serializes to its golden bytes on both the portable and native paths. | Builds `HllSketch::from_raw(HllVariant::Regular, 12, regs, 0.0, 0.0, 0.0)` where `regs` is 4,096 zeros with `[0] = 1`, `[1] = 7`, `[100] = 42` and `[4095] = 3`, verifies `to_msgpack()` equals `asapv1_golden/hll_classic_p12.hex`, that `HyperLogLogP12::::deserialize_from_bytes` on the fixture recovers those registers and re-serializes identically, and that `HllSketch::from_msgpack` reports the same registers, `precision` `12` and `HllVariant::Regular`. | diff --git a/src/sketches/ddsketch/wire.rs b/src/sketches/ddsketch/wire.rs index ce83d9d3..9aa4c1f9 100644 --- a/src/sketches/ddsketch/wire.rs +++ b/src/sketches/ddsketch/wire.rs @@ -23,9 +23,8 @@ //! //! ## Signed stores //! -//! Positive-only payloads retain metadata version 1. Signed or zero-containing +//! Positive-only payloads use metadata version 1. Signed or zero-containing //! payloads use version 2 and append a negative-magnitude store and zero count. -//! Older native readers reject version 2 instead of silently dropping samples. use rmp_serde::{decode::Error as RmpDecodeError, encode::Error as RmpEncodeError, from_slice}; use serde::{Deserialize, Serialize}; diff --git a/tests/README.md b/tests/README.md index f18d2bfb..1c1baeb8 100644 --- a/tests/README.md +++ b/tests/README.md @@ -15,7 +15,7 @@ its theorem. | Path | Purpose | | --- | --- | | `common/mod.rs` | Seeded stream generators (`zipf_u64`, `uniform_u64`, `normal_f64`, `exponential_f64`, adversarial `log_uniform_f64`, `duplicate_heavy_f64`, `monotonic_f64`, `outside_in_ordering`), exact truth trackers (`FreqTruth`, `NumericTruth`), assertion helpers | -| `common/specs.rs` | One error model per metric — `CountMinSpec`, `CountSketchSpec`, `SecondMomentSpec`, `KllRankSpec`, `RelativeQuantileSpec` (+ `DdRankConvention`), `CardinalityConfidenceSpec`, `SamplingConfidenceSpec`, `PrioritySampleSpec` — plus the three acceptance rules and the trial-unit rules that say which one applies | +| `common/specs.rs` | One error model per metric — `CountMinSpec`, `CountSketchSpec`, `SecondMomentSpec`, `KllRankSpec`, `RelativeQuantileSpec` (+ `dd_rank_index`), `CardinalityConfidenceSpec`, `SamplingConfidenceSpec`, `PrioritySampleSpec` — plus the three acceptance rules and the trial-unit rules that say which one applies | | `common/conformance.rs` | Capability traits + standard conformance batteries (the floor) | | `conformance_kit.rs` | Reference adapters: established sketches running through the kit (copy these) | | `e2e/` | Every end-to-end suite, one target (`cargo test --test e2e`), with `common/` compiled once for all of them | diff --git a/tests/asapv1_golden.rs b/tests/asapv1_golden.rs index f57bf89f..43e3fc2d 100644 --- a/tests/asapv1_golden.rs +++ b/tests/asapv1_golden.rs @@ -35,6 +35,7 @@ const GOLDEN_KLL_F64: &str = include_str!("../asapv1_golden/kll_f64_k200.hex"); const GOLDEN_KLL_I64: &str = include_str!("../asapv1_golden/kll_i64_k200.hex"); const GOLDEN_DD_POSITIVE: &str = include_str!("../asapv1_golden/ddsketch_positive_a001.hex"); const GOLDEN_DD_SIGNED: &str = include_str!("../asapv1_golden/ddsketch_signed_a001.hex"); +const GOLDEN_DD_EMPTY: &str = include_str!("../asapv1_golden/ddsketch_empty_a001.hex"); /// The known P12 register pattern shared by all three HLL fixtures. fn p12_registers() -> Vec { @@ -293,8 +294,9 @@ fn kll_i64_k200_matches_golden() { // --------------------------------------------------------------------------- // DDSketch: bucket stores, offsets, zero count and scalars set directly -// (DDSketch never hashes). The positive-only state is metadata version 1; the -// signed state is version 2 and adds the negative store and zero count. +// (DDSketch never hashes). The positive-only and empty states are metadata +// version 1; the signed state is version 2 and adds the negative store and +// zero count. // --------------------------------------------------------------------------- #[derive(Serialize)] @@ -303,7 +305,7 @@ struct DdStore { offset: i32, } -/// The field order of `DDSketch`'s serde form. +/// The field names of `DDSketch`'s serde form. #[derive(Serialize)] struct DdState { alpha: f64, @@ -350,7 +352,7 @@ fn dd_sketch( }, zero_count, }; - rmp_serde::from_slice(&rmp_serde::to_vec(&state).expect("encode state")) + rmp_serde::from_slice(&rmp_serde::to_vec_named(&state).expect("encode state")) .expect("a valid DDSketch state") } @@ -417,3 +419,34 @@ fn ddsketch_signed_a001_matches_golden() { assert_eq!(decoded.get_count(), 18); assert_eq!(decoded.serialize_to_bytes().expect("re-serialize"), want); } + +#[test] +fn ddsketch_empty_a001_matches_golden() { + let want = decode_hex(GOLDEN_DD_EMPTY); + + let state = dd_sketch( + (&[], 0), + (&[], 0), + 0, + (0.0, f64::INFINITY, f64::NEG_INFINITY), + ); + let got = state.serialize_to_bytes().expect("serialize"); + assert_eq!(got, want, "DDSketch empty bytes diverge from golden"); + assert_eq!( + DDSketch::new(0.01).serialize_to_bytes().expect("serialize"), + want, + "a fresh sketch is the empty state" + ); + + let decoded = DDSketch::deserialize_from_bytes(&want).expect("decode"); + assert_eq!(decoded.alpha(), 0.01); + assert!(decoded.store_counts().is_empty()); + assert_eq!(decoded.store_offset(), 0); + assert!(decoded.negative_store_counts().is_empty()); + assert_eq!(decoded.zero_count(), 0); + assert_eq!(decoded.get_count(), 0); + assert_eq!(decoded.sum(), 0.0); + assert_eq!(decoded.min(), None); + assert_eq!(decoded.max(), None); + assert_eq!(decoded.serialize_to_bytes().expect("re-serialize"), want); +} diff --git a/tests/common/conformance.rs b/tests/common/conformance.rs index 5810a644..2b013dbd 100644 --- a/tests/common/conformance.rs +++ b/tests/common/conformance.rs @@ -429,8 +429,7 @@ where /// interchangeable: a rank band says nothing about DDSketch's promise, and a /// relative-value band says nothing about KLL's. /// -/// The order statistic compared against comes from the spec's -/// `DdRankConvention`. +/// The order statistic compared against is `specs::dd_order_statistic`. pub fn relative_quantile_battery( sketch: &str, new_sketch: F, @@ -454,7 +453,7 @@ where sk.update(*v); } for &q in qs { - let truth = spec.convention.order_statistic(&sorted, q); + let truth = super::specs::dd_order_statistic(&sorted, q); let est = sk.quantile(q); let outcome = spec.check(q, est, truth); report.record( diff --git a/tests/common/specs.rs b/tests/common/specs.rs index 54670d60..1cc4b03d 100644 --- a/tests/common/specs.rs +++ b/tests/common/specs.rs @@ -996,47 +996,26 @@ pub fn breakpoint_rank_interval_distance(estimated: &[(f64, f64)], sorted_truth: // DDSketch: relative value error // --------------------------------------------------------------------------- -/// Which order statistic a DDSketch answers a quantile query with; the truth -/// a test compares against follows it. -#[derive(Clone, Copy, Debug, PartialEq, Eq)] -pub enum DdRankConvention { - /// `DDSketch::get_value_at_quantile`: `rank = ceil(q * n)`, 1-based, so the - /// answer is `sorted[ceil(q*n) - 1]`. `q = 0` and `q = 1` short-circuit to - /// the exact stored minimum and maximum. - CeilNearestRank, +/// Zero-based index into an ascending slice of the order statistic +/// `DDSketch::get_value_at_quantile` answers `q` with: `rank = ceil(q * n)`, +/// 1-based, so `sorted[ceil(q*n) - 1]`. `q = 0` and `q = 1` are the exact +/// minimum and maximum. +pub fn dd_rank_index(n: usize, q: f64) -> usize { + assert!(n > 0, "a quantile rank is undefined on an empty stream"); + let q = q.clamp(0.0, 1.0); + (((q * n as f64).ceil() as usize).clamp(1, n)) - 1 } -impl DdRankConvention { - /// The zero-based index into an ascending `sorted` slice this convention - /// answers `q` with. - pub fn index(self, n: usize, q: f64) -> usize { - assert!( - n > 0, - "a quantile convention is undefined on an empty stream" - ); - let q = q.clamp(0.0, 1.0); - match self { - DdRankConvention::CeilNearestRank => (((q * n as f64).ceil() as usize).clamp(1, n)) - 1, - } - } - - /// The exact order statistic this convention answers `q` with. - pub fn order_statistic(self, sorted: &[f64], q: f64) -> f64 { - sorted[self.index(sorted.len(), q)] - } - - pub fn name(self) -> &'static str { - match self { - DdRankConvention::CeilNearestRank => "ceil(q*n) nearest-rank", - } - } +/// The exact order statistic `DDSketch` answers `q` with. +pub fn dd_order_statistic(sorted: &[f64], q: f64) -> f64 { + sorted[dd_rank_index(sorted.len(), q)] } /// Relative-value-error contract for a DDSketch of accuracy parameter `alpha`. /// /// DDSketch's guarantee is on the *value*, not the rank: the returned estimate /// for the `q`-quantile must be within `alpha` relative error of the exact -/// order statistic **at the same `q` under the sketch's rank convention**: +/// `ceil(q*n)` order statistic at the same `q` (see [`dd_rank_index`]): /// /// ```text /// |est - true| / |true| <= alpha + numerical_slack(true) @@ -1059,16 +1038,12 @@ impl DdRankConvention { #[derive(Clone, Copy, Debug)] pub struct RelativeQuantileSpec { pub alpha: f64, - pub convention: DdRankConvention, } impl RelativeQuantileSpec { - /// A spec for the core `DDSketch`. + /// A spec for `DDSketch` at accuracy `alpha`. pub fn core(alpha: f64) -> Self { - Self { - alpha, - convention: DdRankConvention::CeilNearestRank, - } + Self { alpha } } /// A few ULP of headroom, scaled by the magnitude of the logarithm because @@ -1098,24 +1073,21 @@ impl RelativeQuantileSpec { } else { Err(format!( "q={q}: est {estimate:.10e} vs exact {true_value:.10e} \ - -> relative error {rel:.3e} > alpha + slack = {tol:.3e} (alpha={}, {})", + -> relative error {rel:.3e} > alpha + slack = {tol:.3e} \ + (alpha={}, ceil(q*n) nearest-rank)", self.alpha, - self.convention.name(), )) } } - /// Tallies a sketch's answers over a q grid against the exact order - /// statistics **of this spec's own rank convention**. - /// - /// `sorted` must be ascending. Using one truth helper for both - /// implementations would compare each against the other's question. + /// Tallies a sketch's answers over a q grid against the exact `ceil(q*n)` + /// order statistics. `sorted` must be ascending. pub fn tally_into(&self, tally: &mut Tally, sorted: &[f64], qs: &[f64], quantile: F) where F: Fn(f64) -> Option, { for &q in qs { - let truth = self.convention.order_statistic(sorted, q); + let truth = dd_order_statistic(sorted, q); match quantile(q) { None => tally.record(false, || format!("q={q}: sketch returned None")), Some(est) => { diff --git a/tests/e2e/quantiles.rs b/tests/e2e/quantiles.rs index 197c266b..9ae1be2e 100644 --- a/tests/e2e/quantiles.rs +++ b/tests/e2e/quantiles.rs @@ -32,11 +32,11 @@ //! DDSketch's guarantee is deterministic — bucket width alone, no hash and no //! sampling — so its batteries tolerate zero violations and no statistical //! model applies. Its truth is the `ceil(q*n)` order statistic; see -//! `DdRankConvention`. +//! `dd_rank_index`. use crate::common; -use common::specs::{KllRankSpec, RelativeQuantileSpec, Tally, rank_error}; +use common::specs::{KllRankSpec, RelativeQuantileSpec, Tally, dd_rank_index, rank_error}; use common::{ NumericTruth, assert_between, duplicate_heavy_f64, exponential_f64, log_uniform_f64, monotonic_f64, normal_f64, outside_in_ordering, uniform_u64, zipf_f64, @@ -631,6 +631,49 @@ fn ddsketch_satisfies_the_relative_value_error_contract() { } } +/// At small `n` and ragged `q` the `ceil(q*n)` order statistic differs from +/// the `floor(q*(n-1))` one, and with values three buckets apart the two are +/// more than `alpha` apart, so these probes tell the conventions apart. +#[test] +fn ddsketch_answers_the_ceil_nearest_rank_order_statistic_at_small_n() { + // (n, q, expected 0-based index sorted[ceil(q*n) - 1]) + const PROBES: [(usize, f64, usize); 8] = [ + (3, 0.4, 1), + (4, 0.34, 1), + (4, 0.3, 1), + (7, 0.2, 1), + (7, 0.6, 4), + (5, 0.5, 2), + (10, 0.25, 2), + (10, 0.15, 1), + ]; + const ALPHA: f64 = 0.01; + let gamma = (1.0 + ALPHA) / (1.0 - ALPHA); + let spec = RelativeQuantileSpec::core(ALPHA); + + let mut distinguishing = 0usize; + for &(n, q, idx) in &PROBES { + assert_eq!(dd_rank_index(n, q), idx, "ceil(q*n)-1 at n={n} q={q}"); + if (q * (n - 1) as f64).floor() as usize != idx { + distinguishing += 1; + } + + let sorted: Vec = (0..n).map(|i| 100.0 * gamma.powi(3 * i as i32)).collect(); + let mut sketch = DDSketch::new(ALPHA); + for v in &sorted { + sketch.add(v); + } + let est = sketch.get_value_at_quantile(q).expect("non-empty"); + if let Err(detail) = spec.check(q, est, sorted[idx]) { + panic!("DDSketch n={n} q={q}: {detail}"); + } + } + assert!( + distinguishing >= 3, + "only {distinguishing} probes separate ceil(q*n) from floor(q*(n-1))" + ); +} + /// The sketch tracks the exact minimum and maximum beside the bucket store, and /// `get_value_at_quantile` short-circuits `q <= 0` and `q >= 1` to them, so its /// endpoints are exact. From 0c206994cfa5a853ecd01bd631cb11cb96928af9 Mon Sep 17 00:00:00 2001 From: GordonYuanyc Date: Fri, 2 Oct 2026 01:09:52 -0600 Subject: [PATCH 24/36] test(hll): check the P14 ASAPv1 goldens for all three estimators Bump asapv1_golden to the commit adding hll_classic_p14, hll_ertl_mle_p14 and hll_hip_p14, built from fixed registers holding the maximum P14 rank, and check each in both directions. Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 4 +++ asapv1_golden | 2 +- docs/tests.md | 3 ++ tests/asapv1_golden.rs | 64 ++++++++++++++++++++++++++++++++++++++++-- 4 files changed, 70 insertions(+), 3 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 5266204e..7ce2ef4d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -69,6 +69,10 @@ signals a backwards-compatible change. - **HyperLogLog `from_storage` constructors.** `HyperLogLogImpl::from_storage(registers)` and `HyperLogLogHIPImpl::from_storage(registers, kxq0, kxq1, est)` build a sketch from known register storage (and HIP running state). +- **HLL P14 golden fixtures** `hll_classic_p14`, `hll_ertl_mle_p14` and + `hll_hip_p14` in `asapv1_golden/`, checked in both directions by + `tests/asapv1_golden.rs`. Their registers touch the first, a middle and the + last index and hold 51, the largest P14 rank. ### Changed diff --git a/asapv1_golden b/asapv1_golden index 8287a4ae..43b6cd2d 160000 --- a/asapv1_golden +++ b/asapv1_golden @@ -1 +1 @@ -Subproject commit 8287a4ae04e6cfbfb3ed33dafbf1dac77d8f7fc1 +Subproject commit 43b6cd2d3f14688f56edda5e8c987c16fbd4f919 diff --git a/docs/tests.md b/docs/tests.md index 66513dab..9b3227b4 100644 --- a/docs/tests.md +++ b/docs/tests.md @@ -2095,8 +2095,11 @@ Test file: [`tests/asapv1_golden.rs`](../tests/asapv1_golden.rs) | `cs_i64_fast_2x4_matches_golden` | The fast-path mode reaches the bytes, so the same matrix encodes differently per path. | Verifies `Count::, FastPath>::from_storage` over `[[0, 127, 128, 65536], [-1, -33, -32768, -2147483648]]` serializes to `asapv1_golden/cs_i64_fast_2x4.hex`, that decoding it recovers the eight cells and re-serializes identically, and that the fixture differs from `cs_i64_regular_2x4.hex` built from the same matrix. | | `cs_i64_regular_2x4_matches_golden` | A known signed Count-Sketch matrix serializes to its checked-in golden bytes. | Verifies `Count::, RegularPath>::from_storage` over `[[0, 127, 128, 65536], [-1, -33, -32768, -2147483648]]` — sweeping the msgpack integer widths in both directions — serializes to `asapv1_golden/cs_i64_regular_2x4.hex`, and that decoding the fixture reports `rows()` `2`, `cols()` `4`, the eight cells, and a byte-identical re-serialization. | | `hll_classic_p12_matches_golden` | A known P12 Classic register pattern serializes to its golden bytes, which round-trip unchanged. | Builds `HyperLogLogP12::::from_storage` over 4,096 registers that are zero except `[0] = 1`, `[1] = 7`, `[100] = 42` and `[4095] = 3`, verifies `serialize_to_bytes()` equals `asapv1_golden/hll_classic_p12.hex`, and that `deserialize_from_bytes` on the fixture recovers those registers and re-serializes identically. | +| `hll_classic_p14_matches_golden` | A known P14 Classic register pattern, holding the maximum rank, serializes to its golden bytes, which round-trip unchanged. | Builds `HyperLogLogP14::::from_storage` over 16,384 registers that are zero except `[0] = 1`, `[1] = 7`, `[8192] = 42` and `[16383] = 51`, verifies `serialize_to_bytes()` equals `asapv1_golden/hll_classic_p14.hex`, and that `deserialize_from_bytes` on the fixture recovers those registers and re-serializes identically. | | `hll_ertl_mle_p12_matches_golden` | The ErtlMLE kind_id reaches the bytes for the same P12 register pattern. | Builds `HyperLogLogP12::::from_storage` over the shared P12 pattern, verifies `serialize_to_bytes()` equals `asapv1_golden/hll_ertl_mle_p12.hex`, and that `deserialize_from_bytes` recovers the registers and re-serializes identically. | +| `hll_ertl_mle_p14_matches_golden` | The ErtlMLE kind_id reaches the bytes for the same P14 register pattern. | Builds `HyperLogLogP14::::from_storage` over the shared P14 pattern, verifies `serialize_to_bytes()` equals `asapv1_golden/hll_ertl_mle_p14.hex`, and that `deserialize_from_bytes` recovers the registers and re-serializes identically. | | `hll_hip_p12_matches_golden` | The HIP variant's three running scalars travel on the wire beside its registers. | Builds `HyperLogLogHIPP12::from_storage` over the shared P12 pattern with `kxq0` `1.5`, `kxq1` `2.5` and `est` `3.0`, verifies `serialize_to_bytes()` equals `asapv1_golden/hll_hip_p12.hex`, that the decode's `Debug` form (registers and all three scalars) equals the built sketch's, and that it re-serializes identically. | +| `hll_hip_p14_matches_golden` | The P14 HIP registers and running scalars match their golden bytes in both directions. | Builds `HyperLogLogHIPP14::from_storage` over the shared P14 pattern with `kxq0` `16380.5`, `kxq1` `0.25` and `est` `4.125`, verifies `serialize_to_bytes()` equals `asapv1_golden/hll_hip_p14.hex`, that the decode's `Debug` form equals the built sketch's, and that it re-serializes identically. | | `kll_f64_k200_matches_golden` | A deterministic f64 KLL serializes to its golden bytes, which round-trip unchanged. | Verifies `KLL::::init_kll_with_seed(200, 42)` fed `1..=50` (below the level-0 capacity, so no compaction fires) serializes to `asapv1_golden/kll_f64_k200.hex`, and that decoding the fixture re-serializes to the same bytes and reports `quantile(0.0)` `1.0` and `quantile(1.0)` `50.0`. | | `kll_i64_k200_matches_golden` | A deterministic i64 KLL serializes to its golden bytes, which round-trip unchanged. | Verifies `KLL::::init_kll_with_seed(200, 42)` fed `1..=50` serializes to `asapv1_golden/kll_i64_k200.hex`, and that decoding the fixture re-serializes to the same bytes and reports `quantile(0.0)` `1.0` and `quantile(1.0)` `50.0`. | diff --git a/tests/asapv1_golden.rs b/tests/asapv1_golden.rs index b820c978..a1710e33 100644 --- a/tests/asapv1_golden.rs +++ b/tests/asapv1_golden.rs @@ -9,8 +9,9 @@ //! See `asapv1_golden/README.md`. use asap_sketchlib::{ - Classic, Count, CountMin, ErtlMLE, FastPath, HllBucketListP12, HllRegisterStorage, - HyperLogLogHIPP12, HyperLogLogP12, KLL, RegularPath, Vector2D, + Classic, Count, CountMin, ErtlMLE, FastPath, HllBucketListP12, HllBucketListP14, + HllRegisterStorage, HyperLogLogHIPP12, HyperLogLogHIPP14, HyperLogLogP12, HyperLogLogP14, KLL, + RegularPath, Vector2D, }; fn decode_hex(s: &str) -> Vec { @@ -25,6 +26,9 @@ fn decode_hex(s: &str) -> Vec { const GOLDEN_CLASSIC: &str = include_str!("../asapv1_golden/hll_classic_p12.hex"); const GOLDEN_ERTL: &str = include_str!("../asapv1_golden/hll_ertl_mle_p12.hex"); const GOLDEN_HIP: &str = include_str!("../asapv1_golden/hll_hip_p12.hex"); +const GOLDEN_CLASSIC_P14: &str = include_str!("../asapv1_golden/hll_classic_p14.hex"); +const GOLDEN_ERTL_P14: &str = include_str!("../asapv1_golden/hll_ertl_mle_p14.hex"); +const GOLDEN_HIP_P14: &str = include_str!("../asapv1_golden/hll_hip_p14.hex"); const GOLDEN_CMS_I64: &str = include_str!("../asapv1_golden/cms_i64_regular_2x3.hex"); const GOLDEN_CMS_F64: &str = include_str!("../asapv1_golden/cms_f64_fast_2x3.hex"); const GOLDEN_CS_REGULAR: &str = include_str!("../asapv1_golden/cs_i64_regular_2x4.hex"); @@ -49,6 +53,23 @@ fn p12_storage() -> HllBucketListP12 { storage } +/// The known P14 register pattern: the first, a middle and the last index, +/// up to 51, the largest rank a P14 register holds. +fn p14_registers() -> Vec { + let mut r = vec![0u8; 16384]; + r[0] = 1; + r[1] = 7; + r[8192] = 42; + r[16383] = 51; + r +} + +fn p14_storage() -> HllBucketListP14 { + let mut storage = HllBucketListP14::default(); + storage.as_mut_slice().copy_from_slice(&p14_registers()); + storage +} + const I64_VALS: [[i64; 3]; 2] = [[0, 1, 127], [128, 300, 65536]]; const F64_VALS: [[f64; 3]; 2] = [[0.0, 1.5, 2.25], [3.75, 4.125, 5.0625]]; @@ -102,6 +123,45 @@ fn hll_hip_p12_matches_golden() { assert_eq!(decoded.serialize_to_bytes().expect("re-serialize"), want); } +#[test] +fn hll_classic_p14_matches_golden() { + let want = decode_hex(GOLDEN_CLASSIC_P14); + + let sketch = HyperLogLogP14::::from_storage(p14_storage()); + let got = sketch.serialize_to_bytes().expect("serialize"); + assert_eq!(got, want, "Classic P14 bytes diverge from golden"); + + let decoded = HyperLogLogP14::::deserialize_from_bytes(&want).expect("decode"); + assert_eq!(decoded.registers_as_slice(), p14_registers().as_slice()); + assert_eq!(decoded.serialize_to_bytes().expect("re-serialize"), want); +} + +#[test] +fn hll_ertl_mle_p14_matches_golden() { + let want = decode_hex(GOLDEN_ERTL_P14); + + let sketch = HyperLogLogP14::::from_storage(p14_storage()); + let got = sketch.serialize_to_bytes().expect("serialize"); + assert_eq!(got, want, "Ertl-MLE P14 bytes diverge from golden"); + + let decoded = HyperLogLogP14::::deserialize_from_bytes(&want).expect("decode"); + assert_eq!(decoded.registers_as_slice(), p14_registers().as_slice()); + assert_eq!(decoded.serialize_to_bytes().expect("re-serialize"), want); +} + +#[test] +fn hll_hip_p14_matches_golden() { + let want = decode_hex(GOLDEN_HIP_P14); + + let sketch = HyperLogLogHIPP14::from_storage(p14_storage(), 16380.5, 0.25, 4.125); + let got = sketch.serialize_to_bytes().expect("serialize"); + assert_eq!(got, want, "HIP P14 bytes diverge from golden"); + + let decoded = HyperLogLogHIPP14::deserialize_from_bytes(&want).expect("decode"); + assert_eq!(format!("{decoded:?}"), format!("{sketch:?}")); + assert_eq!(decoded.serialize_to_bytes().expect("re-serialize"), want); +} + // --------------------------------------------------------------------------- // Count-Min: build known matrix state -> serialize == golden, round-trips. // --------------------------------------------------------------------------- From 62f9a951bd299c26e060059e9e3a5694a3d79f1c Mon Sep 17 00:00:00 2001 From: GordonYuanyc Date: Fri, 2 Oct 2026 01:10:36 -0600 Subject: [PATCH 25/36] fix(countl2hh): cover every msgpack integer width in the golden Bumps `asapv1_golden` to the commit regenerating `count_l2hh_2x4_seed7` with an int16 cell and a uint32 `l2` accumulator, so the cells and accumulators reach all ten integer families. Drops the claim that `sketchlib-go` mirrors every fixture from the spec. Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 5 +++-- asapv1_golden | 2 +- docs/asapv1_wire_format.md | 2 +- docs/tests.md | 2 +- tests/asapv1_golden.rs | 11 ++++------- 5 files changed, 10 insertions(+), 12 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 32ee6684..cec140ef 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -67,8 +67,9 @@ signals a backwards-compatible change. draw from the thread generator. The seed is not serialized, and the ASAPv1 payload is unchanged. - An ASAPv1 golden fixture for CountL2HH (`0x19 0x00`), `count_l2hh_2x4_seed7`: - signed cells across the msgpack integer widths, `l2` accumulators set apart - from the cells, and a non-zero seed index. + a non-zero seed index, and cells and `l2` accumulators set apart from each + other that together reach every msgpack integer width, positive fixint + through uint64 and negative fixint through int64. ### Changed diff --git a/asapv1_golden b/asapv1_golden index a70fa1ab..d6e5d63c 160000 --- a/asapv1_golden +++ b/asapv1_golden @@ -1 +1 @@ -Subproject commit a70fa1ab1e9d98cb6cc722d558e1efa5a04885ae +Subproject commit d6e5d63c39c637016fe069f27acc320983c7947d diff --git a/docs/asapv1_wire_format.md b/docs/asapv1_wire_format.md index 2854be70..acabb9ad 100644 --- a/docs/asapv1_wire_format.md +++ b/docs/asapv1_wire_format.md @@ -1341,7 +1341,7 @@ Good direction (more compact, higher fidelity, less Rust-internal duplication), 2. **Golden byte-vector fixtures** in one shared repo, [`sketchlib-golden-bytes`](https://github.com/ProjectASAP/sketchlib-golden-bytes), which each implementation mounts at `asapv1_golden/`; both languages decode and re-encode them byte-identically. These replace the `portable`-as-oracle round-trip test. 3. **This registry**, mirrored, never independently allocated. -`sketchlib-go` mirrors every fixture; the [Coverage section](https://github.com/ProjectASAP/sketchlib-golden-bytes#coverage) of `asapv1_golden/README.md` lists the `kind_id`s they cover. For a kind without a fixture, this document is the only contract. `portable` carries its own Go goldens. +The [Coverage section](https://github.com/ProjectASAP/sketchlib-golden-bytes#coverage) of `asapv1_golden/README.md` lists the `kind_id`s the fixtures cover. For a kind without a fixture, this document is the only contract. `portable` carries its own Go goldens. **Hash profile on the Go side.** Rust derives the hash spec from a generic `HashProfile` bound on the hasher type; Go has no generic hasher type, so there is nothing to derive from. diff --git a/docs/tests.md b/docs/tests.md index 3a2ad540..af3c10c7 100644 --- a/docs/tests.md +++ b/docs/tests.md @@ -2123,7 +2123,7 @@ Test file: [`tests/asapv1_golden.rs`](../tests/asapv1_golden.rs) | --- | --- | --- | | `cms_f64_fast_2x3_matches_golden` | A known f64 fast-path Count-Min matrix serializes to its checked-in golden bytes. | Verifies `CountMin::, FastPath>::from_storage` over the `2x3` matrix `[[0.0, 1.5, 2.25], [3.75, 4.125, 5.0625]]` serializes to `asapv1_golden/cms_f64_fast_2x3.hex`, and that decoding that fixture recovers the six cells in row-major order and re-serializes to the same bytes. | | `cms_i64_regular_2x3_matches_golden` | A known i64 regular-path Count-Min matrix serializes to its checked-in golden bytes. | Verifies `CountMin::, RegularPath>::from_storage` over the `2x3` matrix `[[0, 1, 127], [128, 300, 65536]]` serializes to `asapv1_golden/cms_i64_regular_2x3.hex`, and that decoding the fixture reports `rows()` `2`, `cols()` `3`, the six cells in row-major order, and a byte-identical re-serialization. | -| `count_l2hh_2x4_seed7_matches_golden` | A known CountL2HH state, its `l2` accumulators and seed index included, serializes to its checked-in golden bytes. | Builds `CountL2HH` from its named serde form with counts `[[127, 128, 65535, 4294967296], [-32, -33, -2147483648, i64::MIN]]`, `l2` `[256, i64::MAX]` and `seed_idx` `7`, verifies it serializes to `asapv1_golden/count_l2hh_2x4_seed7.hex`, and that decoding the fixture reports `rows()` `2`, `cols()` `4`, `seed_idx()` `7`, the eight cells, the two `l2` accumulators, and a byte-identical re-serialization. | +| `count_l2hh_2x4_seed7_matches_golden` | A known CountL2HH state, its `l2` accumulators and seed index included, serializes to its checked-in golden bytes. | Builds `CountL2HH` from its named serde form with counts `[[127, 128, 65535, -32768], [-32, -33, -2147483648, i64::MIN]]`, `l2` `[65536, i64::MAX]` and `seed_idx` `7`, verifies it serializes to `asapv1_golden/count_l2hh_2x4_seed7.hex`, and that decoding the fixture reports `rows()` `2`, `cols()` `4`, `seed_idx()` `7`, the eight cells, the two `l2` accumulators, and a byte-identical re-serialization. | | `cs_i32_regular_2x4_matches_golden` | The i32 counter type reaches the metadata, and an i32 golden is not an i64 sketch. | Verifies `Count::, RegularPath>::from_storage` over `[[0, 127, 128, 65536], [-1, -33, -32768, -2147483648]]` cast to `i32` serializes to `asapv1_golden/cs_i32_regular_2x4.hex`, that decoding it recovers the eight cells and re-serializes identically, that the i32 and i64 regular fixtures have equal length yet differ, and that the i32 fixture fails to decode as `Count, RegularPath>`. | | `cs_i64_fast_2x4_matches_golden` | The fast-path mode reaches the bytes, so the same matrix encodes differently per path. | Verifies `Count::, FastPath>::from_storage` over `[[0, 127, 128, 65536], [-1, -33, -32768, -2147483648]]` serializes to `asapv1_golden/cs_i64_fast_2x4.hex`, that decoding it recovers the eight cells and re-serializes identically, and that the fixture differs from `cs_i64_regular_2x4.hex` built from the same matrix. | | `cs_i64_regular_2x4_matches_golden` | A known signed Count-Sketch matrix serializes to its checked-in golden bytes. | Verifies `Count::, RegularPath>::from_storage` over `[[0, 127, 128, 65536], [-1, -33, -32768, -2147483648]]` — sweeping the msgpack integer widths in both directions — serializes to `asapv1_golden/cs_i64_regular_2x4.hex`, and that decoding the fixture reports `rows()` `2`, `cols()` `4`, the eight cells, and a byte-identical re-serialization. | diff --git a/tests/asapv1_golden.rs b/tests/asapv1_golden.rs index 522300af..5d2ff847 100644 --- a/tests/asapv1_golden.rs +++ b/tests/asapv1_golden.rs @@ -295,13 +295,10 @@ fn kll_i64_k200_matches_golden() { // --------------------------------------------------------------------------- /// CountL2HH cells are signed: row 0 is positive fixint max / uint8 / uint16 / -/// uint64, row 1 is negative fixint min / int8 / int32 / int64. -const L2HH_COUNTS: [[i64; 4]; 2] = [ - [127, 128, 65535, 4294967296], - [-32, -33, -2147483648, i64::MIN], -]; -/// One accumulator per row, set independently of the cells: uint16 / uint64. -const L2HH_L2: [i64; 2] = [256, i64::MAX]; +/// int16, row 1 is negative fixint min / int8 / int32 / int64. +const L2HH_COUNTS: [[i64; 4]; 2] = [[127, 128, 65535, -32768], [-32, -33, -2147483648, i64::MIN]]; +/// One accumulator per row, set independently of the cells: uint32 / uint64. +const L2HH_L2: [i64; 2] = [65536, i64::MAX]; const L2HH_SEED_INDEX: usize = 7; #[derive(serde::Serialize, serde::Deserialize)] From c043b6317733592c5fa2aefb8d3bf595c428534e Mon Sep 17 00:00:00 2001 From: GordonYuanyc Date: Fri, 2 Oct 2026 01:11:53 -0600 Subject: [PATCH 26/36] test(univmon): rebuild the goldens on distinct dimensions; fix fixture docs - asapv1_golden -> 946e5c5: univmon_{str,i64,empty}_l3_2x4_h5 replace the l2_2x4_h2 files. 3 layers, 2 rows, 4 columns, heap 5, heap lengths [3, 1, 2]; i64 keys add int16 and uint64. - Fixture docs point at the golden README's Coverage section instead of a count; spec 3.15 states the empty pyramid's key_type is "u64". - features.md: SketchEnvelope's sketches, with UnivMonState only in HydraCell. Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 4 +- asapv1_golden | 2 +- docs/asapv1_wire_format.md | 6 +- docs/features.md | 2 +- docs/message_pack_format.md | 6 +- docs/tests.md | 6 +- tests/asapv1_golden.rs | 129 ++++++++++++++++++++++-------------- 7 files changed, 91 insertions(+), 64 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index c10a7b16..fb95839e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -67,8 +67,8 @@ signals a backwards-compatible change. draw from the thread generator. The seed is not serialized, and the ASAPv1 payload is unchanged. - **ASAPv1 golden fixtures for `UnivMon`** (kind_id `0x10 0x00`): - `univmon_str_l2_2x4_h2`, `univmon_i64_l2_2x4_h2` and - `univmon_empty_l2_2x4_h2`, checked in `tests/asapv1_golden.rs`. + `univmon_str_l3_2x4_h5`, `univmon_i64_l3_2x4_h5` and + `univmon_empty_l3_2x4_h5`, checked in `tests/asapv1_golden.rs`. ### Changed diff --git a/asapv1_golden b/asapv1_golden index e2086f05..946e5c5f 160000 --- a/asapv1_golden +++ b/asapv1_golden @@ -1 +1 @@ -Subproject commit e2086f053724cfe894f240f8d9e8fb03a540689e +Subproject commit 946e5c5f89ebe990cf57636523076d279b498fc8 diff --git a/docs/asapv1_wire_format.md b/docs/asapv1_wire_format.md index 695cf324..4038e34a 100644 --- a/docs/asapv1_wire_format.md +++ b/docs/asapv1_wire_format.md @@ -972,7 +972,7 @@ The encode side enforces rules 3 and 4 as well, and rejects a sketch holding the **A layer's `CountL2HH` is inlined, not nested.** The layer's `counts` and `l2` are appended to the outer sketch's own positional arrays, and its `rows` / `cols` / `seed_index` come from the outer metadata plus the layer's position. No nested envelope, magic, version or metadata map is emitted per layer. -**Metadata vs payload.** `layer_size`, `sketch_row`, `sketch_col` and `heap_size` are `init_univmon`'s four arguments — configuration that shapes the payload, so the descriptor carries them and every derived length is checked rather than re-stored. `key_type` is a structural param in the §3.5 sense: it fixes the element type of `keys`, and the payload cannot be read without it. One `key_type` covers every layer; a pyramid whose layers mix `HeapItem` variants has no single one and fails to serialize, as a Space-Saving summary does. Structural-param order is `layer_size, sketch_row, sketch_col, heap_size, key_type`. +**Metadata vs payload.** `layer_size`, `sketch_row`, `sketch_col` and `heap_size` are `init_univmon`'s four arguments — configuration that shapes the payload, so the descriptor carries them and every derived length is checked rather than re-stored. `key_type` is a structural param in the §3.5 sense: it fixes the element type of `keys`, and the payload cannot be read without it. One `key_type` covers every layer; a pyramid whose layers mix `HeapItem` variants has no single one and fails to serialize, as a Space-Saving summary does. Structural-param order is `layer_size, sketch_row, sketch_col, heap_size, key_type`. A pyramid with no heap entries emits `key_type = "u64"`. UnivMon carries the hash-spec group — it hashes, and a consumer must reproduce that hash to query a key — but carries **no seed-index key**. It hashes the bottom-layer finder at `BOTTOM_LAYER_FINDER` unconditionally: a fixed part of the algorithm, not a profile choice, exactly as Space-Saving's index 0 is (§3.5). The per-layer counter seed index *is* a real value, but it equals the layer's position, so it is derived and not stored, and the encoder rejects a layer whose seed index is not its position. `UnivMon` has no hasher type parameter, so its metadata is derived from `DefaultXxHasher`'s `HashProfile` — read live, never hardcoded — and a custom-profile envelope is different bytes and is rejected on decode. @@ -1341,14 +1341,14 @@ Good direction (more compact, higher fidelity, less Rust-internal duplication), 2. **Golden byte-vector fixtures** in one shared repo, [`sketchlib-golden-bytes`](https://github.com/ProjectASAP/sketchlib-golden-bytes), which each implementation mounts at `asapv1_golden/`; both languages decode and re-encode them byte-identically. These replace the `portable`-as-oracle round-trip test. 3. **This registry**, mirrored, never independently allocated. -Fixtures exist for seven `kind_id`s — HLL's three estimators, Count-Min, Count Sketch, compact KLL and UnivMon — and `sketchlib-go` mirrors those. Every other kind here has a payload and **no fixture**, so this document is its only contract; `asapv1_golden/README.md` lists the gap. `portable` carries its own Go goldens. +`asapv1_golden/README.md`'s Coverage section lists which `kind_id`s have fixtures; for every other kind, this document is the only contract. `portable` carries its own Go goldens. **Hash profile on the Go side.** Rust derives the hash spec from a generic `HashProfile` bound on the hasher type; Go has no generic hasher type, so there is nothing to derive from. On the Go side the profile is simply **written into** the metadata on encode and **read from** it on decode. Go MUST validate the profile it reads (same fail-closed intent as Rust): a sketch is only mergeable/queryable if its `hash_profile_id` + seeds match the profile Go is prepared to reproduce. -Sequencing: (2) covers seven `kind_id`s, and the one `native bytes == portable bytes` test is HLL-only, since no other `portable` type emits ASAPv1 bytes. +Sequencing: the one `native bytes == portable bytes` test is HLL-only, since no other `portable` type emits ASAPv1 bytes. Retire `portable` once the fixtures cover the rest. --- diff --git a/docs/features.md b/docs/features.md index 5a58639d..f9cfcfac 100644 --- a/docs/features.md +++ b/docs/features.md @@ -112,7 +112,7 @@ MessagePack (`rmp-serde`) support. **serde support** means the type derives `Ser | EHSketchList | Yes | Yes | | ExponentialHistogram | No | Yes | -Protobuf (prost): `.proto` definitions exist for CountMin, Count, HLL, DDSketch, KLL, Elastic, Coco, and Hydra. Rust conversion code is in progress. +Protobuf (prost): `SketchEnvelope` carries CountMin, Count, HLL, DDSketch, KLL, Elastic, Coco, and Hydra state; `UnivMonState` exists only inside `HydraCell`. Rust conversion code is in progress. ### API Stability diff --git a/docs/message_pack_format.md b/docs/message_pack_format.md index 01556231..8559304e 100644 --- a/docs/message_pack_format.md +++ b/docs/message_pack_format.md @@ -169,10 +169,8 @@ decode → re-encode them byte-identically. The `kind_id` registry is mirrored verbatim with Go's `wire/asapmsgpack/magic_ids.go`, never independently allocated. -That proof covers the seven `kind_id`s a fixture exists for — HLL's three -estimators, Count-Min, Count Sketch, compact KLL and UnivMon. Every other -implemented kind has **no ASAPv1 golden and therefore no ASAPv1 drift guard**; -[its README](https://github.com/ProjectASAP/sketchlib-golden-bytes#coverage) lists what is covered. +That proof covers only the `kind_id`s with a fixture, which +[its README's Coverage section](https://github.com/ProjectASAP/sketchlib-golden-bytes#coverage) lists. The older `portable` path is guarded separately, by the `sketchlib-go` goldens of its own listed above. diff --git a/docs/tests.md b/docs/tests.md index 1ef8677e..62fc2941 100644 --- a/docs/tests.md +++ b/docs/tests.md @@ -2131,9 +2131,9 @@ Test file: [`tests/asapv1_golden.rs`](../tests/asapv1_golden.rs) | `hll_hip_p12_matches_golden` | The HIP variant's three running scalars travel on the wire beside its registers. | Builds `HllSketch::from_raw(HllVariant::Hip, 12, regs, 1.5, 2.5, 3.0)` over the shared P12 pattern, verifies `to_msgpack()` equals `asapv1_golden/hll_hip_p12.hex`, that `HyperLogLogHIPP12::deserialize_from_bytes` re-serializes the fixture identically, and that `HllSketch::from_msgpack` reports the registers, `HllVariant::Hip`, `hip_kxq0` `1.5`, `hip_kxq1` `2.5` and `hip_est` `3.0`. | | `kll_f64_k200_matches_golden` | A deterministic f64 KLL serializes to its golden bytes, which round-trip unchanged. | Verifies `KLL::::init_kll_with_seed(200, 42)` fed `1..=50` (below the level-0 capacity, so no compaction fires) serializes to `asapv1_golden/kll_f64_k200.hex`, and that decoding the fixture re-serializes to the same bytes and reports `quantile(0.0)` `1.0` and `quantile(1.0)` `50.0`. | | `kll_i64_k200_matches_golden` | A deterministic i64 KLL serializes to its golden bytes, which round-trip unchanged. | Verifies `KLL::::init_kll_with_seed(200, 42)` fed `1..=50` serializes to `asapv1_golden/kll_i64_k200.hex`, and that decoding the fixture re-serializes to the same bytes and reports `quantile(0.0)` `1.0` and `quantile(1.0)` `50.0`. | -| `univmon_empty_l2_2x4_h2_matches_golden` | A freshly constructed pyramid has one encoding, pinned by its golden. | Verifies `UnivMon::init_univmon(2, 2, 4, 2)` serializes to `asapv1_golden/univmon_empty_l2_2x4_h2.hex`; reading the envelope by hand gives kind_id `0x10 0x00`, `key_type` `u64`, 16 zero counts, 4 zero `l2` accumulators, `heap_lens` `[0, 0]`, no keys or heap counts, candidate flags `[true, true]`, `bucket_size` `0` and `update_mode` `0`; the decode matches the source's shape, weight, flags, counters, per-layer `get_l2()` and heaps, and re-serializes identically. | -| `univmon_i64_l2_2x4_h2_matches_golden` | The `i64` key type reaches the bytes, and only it differs from the string fixture. | Builds the same layers as the string fixture with heap keys `i64::MIN` and `-1` on layer 0 and `128` on layer 1, verifies the bytes equal `asapv1_golden/univmon_i64_l2_2x4_h2.hex`, that the hand-read payload carries those keys under `key_type` `i64` with the same counts, `l2`, heap counts, flags, weight and mode, that the decode matches the source and re-serializes identically, that layer 0's heap holds `HeapItem::I64(i64::MIN)`, and that the fixture differs from the string one. | -| `univmon_str_l2_2x4_h2_matches_golden` | A known 2-layer pyramid with string keys serializes to its golden bytes and decodes back to that state. | Builds `UnivMon::init_univmon(2, 2, 4, 2)`, writes layer 0's cells `[[0, 127, 128, 65536], [-1, -33, -32768, -2147483648]]` and layer 1's `[[3, -2, 0, 1], [0, 0, 5, -4]]` through `L2HH::apply_delta`, seats `alpha`=`65536` and `beta`=`300` in layer 0's heap and `gamma`=`5` in layer 1's, calls `set_total_weight(70_000)` and `mark_layer_candidates_incomplete(1)`, and verifies the bytes equal `asapv1_golden/univmon_str_l2_2x4_h2.hex`. Reading the envelope by hand gives kind_id `0x10 0x00`, shape `(2, 2, 4, 2)`, `key_type` `string`, the 16 cells in layer then row-major order, `l2` `[4294999809, 4611686019501130818, 14, 41]`, `heap_lens` `[2, 1]`, keys `[alpha, beta, gamma]`, heap counts `[65536, 300, 5]`, flags `[true, false]`, `bucket_size` `70000` and `update_mode` `1`; the decode matches the source and re-serializes identically. | +| `univmon_empty_l3_2x4_h5_matches_golden` | A freshly constructed pyramid has one encoding, pinned by its golden. | Verifies `UnivMon::init_univmon(5, 2, 4, 3)` serializes to `asapv1_golden/univmon_empty_l3_2x4_h5.hex`; reading the envelope by hand gives kind_id `0x10 0x00`, shape `(3, 2, 4, 5)`, `key_type` `u64`, 24 zero counts, 6 zero `l2` accumulators, `heap_lens` `[0, 0, 0]`, no keys or heap counts, candidate flags `[true, true, true]`, `bucket_size` `0` and `update_mode` `0`; the decode matches the source's shape, weight, flags, counters, per-layer `get_l2()` and heaps, and re-serializes identically. | +| `univmon_i64_l3_2x4_h5_matches_golden` | The `i64` key type reaches the bytes, and only it differs from the string fixture. | Builds the same layers as the string fixture with heap keys `i64::MIN`, `-1`, `-129` on layer 0, `128` on layer 1 and `4294967296`, `7` on layer 2, verifies the bytes equal `asapv1_golden/univmon_i64_l3_2x4_h5.hex`, that the hand-read envelope carries shape `(3, 2, 4, 5)` and those keys under `key_type` `i64` with the same counts, `l2`, heap lengths, heap counts, flags, weight and mode, that the decode matches the source and re-serializes identically, that layer 0's heap holds `HeapItem::I64(i64::MIN)`, and that the fixture differs from the string one. | +| `univmon_str_l3_2x4_h5_matches_golden` | A known 3-layer pyramid with string keys serializes to its golden bytes and decodes back to that state. | Builds `UnivMon::init_univmon(5, 2, 4, 3)` (heap 5, 2 rows, 4 columns, 3 layers: pairwise distinct), writes layer 0's cells `[[0, 127, 128, 65536], [-1, -33, -32768, -2147483648]]`, layer 1's `[[3, -2, 0, 1], [0, 0, 5, -4]]` and layer 2's `[[0, 7, 0, 0], [-6, 0, 0, 0]]` through `L2HH::apply_delta`, seats `alpha`=`65536`, `beta`=`300`, `delta`=`128` in layer 0's heap, `gamma`=`5` in layer 1's and `epsilon`=`9`, `zeta`=`2` in layer 2's, calls `set_total_weight(70_000)` and `mark_layer_candidates_incomplete(1)`, and verifies the bytes equal `asapv1_golden/univmon_str_l3_2x4_h5.hex`. Reading the envelope by hand gives kind_id `0x10 0x00`, shape `(3, 2, 4, 5)`, `key_type` `string`, the 24 cells in layer then row-major order, `l2` `[4294999809, 4611686019501130818, 14, 41, 49, 36]`, `heap_lens` `[3, 1, 2]`, the six keys and heap counts `[65536, 300, 128, 5, 9, 2]` in that order, flags `[true, false, true]`, `bucket_size` `70000` and `update_mode` `1`; the decode matches the source and re-serializes identically. | ### MessagePack Envelope Compatibility diff --git a/tests/asapv1_golden.rs b/tests/asapv1_golden.rs index 5c4ebfb4..5b7370d4 100644 --- a/tests/asapv1_golden.rs +++ b/tests/asapv1_golden.rs @@ -35,9 +35,9 @@ const GOLDEN_CS_FAST: &str = include_str!("../asapv1_golden/cs_i64_fast_2x4.hex" const GOLDEN_CS_I32: &str = include_str!("../asapv1_golden/cs_i32_regular_2x4.hex"); const GOLDEN_KLL_F64: &str = include_str!("../asapv1_golden/kll_f64_k200.hex"); const GOLDEN_KLL_I64: &str = include_str!("../asapv1_golden/kll_i64_k200.hex"); -const GOLDEN_UNIVMON_STR: &str = include_str!("../asapv1_golden/univmon_str_l2_2x4_h2.hex"); -const GOLDEN_UNIVMON_I64: &str = include_str!("../asapv1_golden/univmon_i64_l2_2x4_h2.hex"); -const GOLDEN_UNIVMON_EMPTY: &str = include_str!("../asapv1_golden/univmon_empty_l2_2x4_h2.hex"); +const GOLDEN_UNIVMON_STR: &str = include_str!("../asapv1_golden/univmon_str_l3_2x4_h5.hex"); +const GOLDEN_UNIVMON_I64: &str = include_str!("../asapv1_golden/univmon_i64_l3_2x4_h5.hex"); +const GOLDEN_UNIVMON_EMPTY: &str = include_str!("../asapv1_golden/univmon_empty_l3_2x4_h5.hex"); /// The known P12 register pattern shared by all three HLL fixtures. fn p12_registers() -> Vec { @@ -295,22 +295,37 @@ fn kll_i64_k200_matches_golden() { } // --------------------------------------------------------------------------- -// UnivMon: a 2-layer pyramid of 2x4 CountL2HH layers with heap capacity 2. -// Cells are written through `L2HH::apply_delta` (which carries each row's L2 -// accumulator) and heap entries through `HHHeap::update`; nothing is hashed. +// UnivMon: 3 layers of 2x4 CountL2HH with heap capacity 5. Cells are written +// through `L2HH::apply_delta` (which carries each row's L2 accumulator) and +// heap entries through `HHHeap::update`; no hash reaches the bytes. // --------------------------------------------------------------------------- -/// Layer 1's counters; layer 0 holds `CS_VALS`. -const UNIVMON_LAYER1: [[i64; 4]; 2] = [[3, -2, 0, 1], [0, 0, 5, -4]]; +/// `init_univmon`'s arguments `(heap_size, sketch_row, sketch_col, layer_size)`, +/// pairwise distinct so no length can stand in for another. +const UNIVMON_SHAPE: (usize, usize, usize, usize) = (5, 2, 4, 3); -/// Each row's sum of squared cells: layer 0's row 1 reaches the uint64 range. -const UNIVMON_L2: [i64; 4] = [4_294_999_809, 4_611_686_019_501_130_818, 14, 41]; +/// The layers' counters; layer 0 holds `CS_VALS`. +const UNIVMON_LAYERS: [[[i64; 4]; 2]; 3] = [ + CS_VALS, + [[3, -2, 0, 1], [0, 0, 5, -4]], + [[0, 7, 0, 0], [-6, 0, 0, 0]], +]; -/// The heap counts in emitted order: layer 0 descending, then layer 1. -const UNIVMON_HEAP_COUNTS: [i64; 3] = [65536, 300, 5]; +/// Each row's sum of squared cells; both of layer 0's are uint64. +const UNIVMON_L2: [i64; 6] = [4_294_999_809, 4_611_686_019_501_130_818, 14, 41, 49, 36]; -/// The i64 fixture's keys in emitted order: int64, negative fixint, uint8. -const UNIVMON_I64_KEYS: [i64; 3] = [i64::MIN, -1, 128]; +/// Entries per layer's heap. +const UNIVMON_HEAP_LENS: [u32; 3] = [3, 1, 2]; + +/// The heap counts in emitted order: layer by layer, each descending. +const UNIVMON_HEAP_COUNTS: [i64; 6] = [65536, 300, 128, 5, 9, 2]; + +/// The string fixture's keys in emitted order. +const UNIVMON_STR_KEYS: [&str; 6] = ["alpha", "beta", "delta", "gamma", "epsilon", "zeta"]; + +/// The i64 fixture's keys in emitted order: int64, negative fixint, int16, +/// uint8, uint64, positive fixint. +const UNIVMON_I64_KEYS: [i64; 6] = [i64::MIN, -1, -129, 128, 4_294_967_296, 7]; /// A UnivMon payload `[counts, l2, heap_lens, keys, heap_counts, /// candidate_complete, bucket_size, update_mode]`. @@ -355,11 +370,12 @@ fn univmon_envelope( ) } -/// The populated pyramid, with layer 0's heap holding `keys[0]` / `keys[1]` -/// and layer 1's holding `keys[2]`, at `UNIVMON_HEAP_COUNTS`. -fn univmon_known(keys: [DataInput<'_>; 3]) -> UnivMon { - let mut um = UnivMon::init_univmon(2, 2, 4, 2); - for (layer, cells) in [CS_VALS, UNIVMON_LAYER1].iter().enumerate() { +/// The populated pyramid: `keys` fill the heaps in emitted order, cut by +/// `UNIVMON_HEAP_LENS`, at `UNIVMON_HEAP_COUNTS`; layer 1 is incomplete. +fn univmon_known(keys: [DataInput<'_>; 6]) -> UnivMon { + let (heap_size, sketch_row, sketch_col, layer_size) = UNIVMON_SHAPE; + let mut um = UnivMon::init_univmon(heap_size, sketch_row, sketch_col, layer_size); + for (layer, cells) in UNIVMON_LAYERS.iter().enumerate() { for (row, values) in cells.iter().enumerate() { for (col, &value) in values.iter().enumerate() { um.l2_sketch_layers[layer].apply_delta(CountDelta { @@ -370,10 +386,13 @@ fn univmon_known(keys: [DataInput<'_>; 3]) -> UnivMon { } } } - let [k0, k1, k2] = keys; - um.hh_layers[0].update(&k0, UNIVMON_HEAP_COUNTS[0]); - um.hh_layers[0].update(&k1, UNIVMON_HEAP_COUNTS[1]); - um.hh_layers[1].update(&k2, UNIVMON_HEAP_COUNTS[2]); + let layers = UNIVMON_HEAP_LENS + .iter() + .enumerate() + .flat_map(|(layer, &len)| std::iter::repeat_n(layer, len as usize)); + for ((layer, key), count) in layers.zip(keys.iter()).zip(UNIVMON_HEAP_COUNTS) { + um.hh_layers[layer].update(key, count); + } um.set_total_weight(70_000); um.mark_layer_candidates_incomplete(1); um @@ -414,12 +433,12 @@ fn assert_univmon_decodes_to(want: &[u8], source: &UnivMon) { decoded.sketch_col, decoded.heap_size ), - (2, 2, 4, 2) + (3, 2, 4, 5) ); assert_eq!(decoded.bucket_size, source.bucket_size); assert_eq!(decoded.candidates_complete(), source.candidates_complete()); assert_eq!(univmon_counts(&decoded), univmon_counts(source)); - for layer in 0..2 { + for layer in 0..3 { assert_eq!( decoded.l2_sketch_layers[layer].get_l2(), source.l2_sketch_layers[layer].get_l2() @@ -430,23 +449,14 @@ fn assert_univmon_decodes_to(want: &[u8], source: &UnivMon) { } fn univmon_known_counts() -> Vec { - CS_VALS - .iter() - .chain(UNIVMON_LAYER1.iter()) - .flatten() - .copied() - .collect() + UNIVMON_LAYERS.iter().flatten().flatten().copied().collect() } #[test] -fn univmon_str_l2_2x4_h2_matches_golden() { +fn univmon_str_l3_2x4_h5_matches_golden() { let want = decode_hex(GOLDEN_UNIVMON_STR); - let sketch = univmon_known([ - DataInput::Str("alpha"), - DataInput::Str("beta"), - DataInput::Str("gamma"), - ]); + let sketch = univmon_known(UNIVMON_STR_KEYS.map(DataInput::Str)); let got = sketch.serialize_to_bytes().expect("serialize"); assert_eq!(got, want, "UnivMon string-key bytes diverge from golden"); @@ -459,16 +469,16 @@ fn univmon_str_l2_2x4_h2_matches_golden() { shape.sketch_col, shape.heap_size ), - (2, 2, 4, 2) + (3, 2, 4, 5) ); assert_eq!(shape.key_type, "string"); let (counts, l2, heap_lens, keys, heap_counts, flags, bucket_size, update_mode) = payload; assert_eq!(counts, univmon_known_counts()); assert_eq!(l2, UNIVMON_L2); - assert_eq!(heap_lens, [2, 1]); - assert_eq!(keys, ["alpha", "beta", "gamma"]); + assert_eq!(heap_lens, UNIVMON_HEAP_LENS); + assert_eq!(keys, UNIVMON_STR_KEYS); assert_eq!(heap_counts, UNIVMON_HEAP_COUNTS); - assert_eq!(flags, [true, false]); + assert_eq!(flags, [true, false, true]); assert_eq!(bucket_size, 70_000); assert_eq!(update_mode, 1, "set_total_weight selects the standard mode"); @@ -476,7 +486,7 @@ fn univmon_str_l2_2x4_h2_matches_golden() { } #[test] -fn univmon_i64_l2_2x4_h2_matches_golden() { +fn univmon_i64_l3_2x4_h5_matches_golden() { let want = decode_hex(GOLDEN_UNIVMON_I64); let sketch = univmon_known(UNIVMON_I64_KEYS.map(DataInput::I64)); @@ -485,14 +495,23 @@ fn univmon_i64_l2_2x4_h2_matches_golden() { let (kind_id, shape, payload) = univmon_envelope::(&want); assert_eq!(kind_id, [0x10, 0x00]); + assert_eq!( + ( + shape.layer_size, + shape.sketch_row, + shape.sketch_col, + shape.heap_size + ), + (3, 2, 4, 5) + ); assert_eq!(shape.key_type, "i64"); let (counts, l2, heap_lens, keys, heap_counts, flags, bucket_size, update_mode) = payload; assert_eq!(counts, univmon_known_counts()); assert_eq!(l2, UNIVMON_L2); - assert_eq!(heap_lens, [2, 1]); + assert_eq!(heap_lens, UNIVMON_HEAP_LENS); assert_eq!(keys, UNIVMON_I64_KEYS); assert_eq!(heap_counts, UNIVMON_HEAP_COUNTS); - assert_eq!(flags, [true, false]); + assert_eq!(flags, [true, false, true]); assert_eq!((bucket_size, update_mode), (70_000, 1)); assert_univmon_decodes_to(&want, &sketch); @@ -509,25 +528,35 @@ fn univmon_i64_l2_2x4_h2_matches_golden() { } #[test] -fn univmon_empty_l2_2x4_h2_matches_golden() { +fn univmon_empty_l3_2x4_h5_matches_golden() { let want = decode_hex(GOLDEN_UNIVMON_EMPTY); - let sketch = UnivMon::init_univmon(2, 2, 4, 2); + let (heap_size, sketch_row, sketch_col, layer_size) = UNIVMON_SHAPE; + let sketch = UnivMon::init_univmon(heap_size, sketch_row, sketch_col, layer_size); let got = sketch.serialize_to_bytes().expect("serialize"); assert_eq!(got, want, "empty UnivMon bytes diverge from golden"); let (kind_id, shape, payload) = univmon_envelope::(&want); assert_eq!(kind_id, [0x10, 0x00]); + assert_eq!( + ( + shape.layer_size, + shape.sketch_row, + shape.sketch_col, + shape.heap_size + ), + (3, 2, 4, 5) + ); assert_eq!( shape.key_type, "u64", "an empty pyramid records key_type u64" ); let (counts, l2, heap_lens, keys, heap_counts, flags, bucket_size, update_mode) = payload; - assert_eq!(counts, [0; 16]); - assert_eq!(l2, [0; 4]); - assert_eq!(heap_lens, [0, 0]); + assert_eq!(counts, [0; 24]); + assert_eq!(l2, [0; 6]); + assert_eq!(heap_lens, [0, 0, 0]); assert!(keys.is_empty() && heap_counts.is_empty()); - assert_eq!(flags, [true, true]); + assert_eq!(flags, [true, true, true]); assert_eq!((bucket_size, update_mode), (0, 0)); assert_univmon_decodes_to(&want, &sketch); From 1fe43a51969c2d4c141b7a211545fc939b85ef4e Mon Sep 17 00:00:00 2001 From: GordonYuanyc Date: Fri, 2 Oct 2026 01:11:56 -0600 Subject: [PATCH 27/36] feat(set_aggregator)!: ASAPv1 payloads for SetAggregator and DeltaResult; drop the portable format MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit SetAggregator (0x08 0x00) and DeltaResult (0x09 0x00) get ASAPv1 payloads (spec §3.21, §3.22): metadata is {metadata_version: 1}; the payloads are [values] and [added, removed], each an array of str in strictly ascending UTF-8 byte order. DeltaResult refuses a key in both sets on both sides. Both types move to sketches::set_aggregator with serialize_to_bytes / deserialize_from_bytes; the crate-root re-exports are kept. The portable modules, their MessagePackCodec impls and DeltaResult's serde derives are removed. Adds four golden fixtures and bumps asapv1_golden. Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 17 + asapv1_golden | 2 +- docs/asapv1_wire_format.md | 55 ++- docs/tests.md | 53 +-- src/lib.rs | 5 +- .../portable/delta_set_aggregator.rs | 65 ---- src/message_pack_format/portable/mod.rs | 2 - src/sketches/mod.rs | 4 + .../portable => sketches}/set_aggregator.rs | 78 +--- src/sketches/set_aggregator/wire.rs | 355 ++++++++++++++++++ tests/PBT_COVERAGE.md | 10 +- tests/asapv1_golden.rs | 90 ++++- tests/e2e/cardinality.rs | 11 +- tests/msgpack_compat.rs | 29 +- tests/pbt/set_aggregator.rs | 44 ++- 15 files changed, 599 insertions(+), 221 deletions(-) delete mode 100644 src/message_pack_format/portable/delta_set_aggregator.rs rename src/{message_pack_format/portable => sketches}/set_aggregator.rs (50%) create mode 100644 src/sketches/set_aggregator/wire.rs diff --git a/CHANGELOG.md b/CHANGELOG.md index 80d85f98..d2585109 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -66,6 +66,15 @@ signals a backwards-compatible change. worker's seed and the aggregator's from one seed. The unseeded constructors draw from the thread generator. The seed is not serialized, and the ASAPv1 payload is unchanged. +- **ASAPv1 payloads for `SetAggregator` (`0x08 0x00`) and `DeltaResult` + (`0x09 0x00`)**, spec §3.21 and §3.22. The metadata is `metadata_version` + alone; the payload is `[values]` and `[added, removed]`, each an array of + `str` in ascending UTF-8 byte order, and a key in both `added` and `removed` + is refused on both sides. Both types gain `serialize_to_bytes` / + `deserialize_from_bytes` and live in `asap_sketchlib::sketches::set_aggregator`; + the crate-root re-exports are unchanged. Golden fixtures + `set_aggregator_strings`, `set_aggregator_empty`, `delta_result_strings` and + `delta_result_empty` are checked by `tests/asapv1_golden.rs`. ### Changed @@ -88,6 +97,14 @@ signals a backwards-compatible change. as an empty window like any other, and that window takes a place in `max_windows`. +### Removed + +- **The portable MessagePack format for `SetAggregator` and `DeltaResult`**: + the modules `message_pack_format::portable::set_aggregator` and + `message_pack_format::portable::delta_set_aggregator`, both types' + `MessagePackCodec` impls, and `DeltaResult`'s `serde::Serialize` / + `Deserialize` derives. They serialize only as ASAPv1. + ### Fixed - `KLLDynamic` panicked on every quantile query once a NaN had entered the diff --git a/asapv1_golden b/asapv1_golden index 8287a4ae..6e31b115 160000 --- a/asapv1_golden +++ b/asapv1_golden @@ -1 +1 @@ -Subproject commit 8287a4ae04e6cfbfb3ed33dafbf1dac77d8f7fc1 +Subproject commit 6e31b11560d03b8288e13e9d7d496c8d1408e10a diff --git a/docs/asapv1_wire_format.md b/docs/asapv1_wire_format.md index fd14e70a..a3994621 100644 --- a/docs/asapv1_wire_format.md +++ b/docs/asapv1_wire_format.md @@ -26,7 +26,7 @@ If the doc feels long, these sections carry the key points: ## Status -- **Implemented (Rust).** Every kind_id the Section 1 registry marks *implemented* serializes through the shared `message_pack_format::envelope` module per this spec: HLL (three estimators), Count-Min, CMSHeap, Count Sketch, CSHeap, DDSketch, KLL (compact + dynamic), Hydra (five counter variants), Elastic, Coco, UniformSampling, KMV, UnivMon, UnivMon Optimized, ExponentialHistogram, EHSketchList, Bloom, Space-Saving, CountL2HH and UnivMon-Q. +- **Implemented (Rust).** Every kind_id the Section 1 registry marks *implemented* serializes through the shared `message_pack_format::envelope` module per this spec: HLL (three estimators), Count-Min, CMSHeap, Count Sketch, CSHeap, DDSketch, KLL (compact + dynamic), Hydra (five counter variants), Elastic, Coco, UniformSampling, KMV, UnivMon, UnivMon Optimized, ExponentialHistogram, EHSketchList, Bloom, Space-Saving, CountL2HH, UnivMon-Q, SetAggregator and DeltaResult. - **Self-describing.** The hash-spec metadata is derived from the hasher's `HashProfile` (read live, never hardcoded), so the bytes truthfully describe how a sketch was hashed; custom hash profiles are supported (Section 2). - **Byte-level encoding** is pinned in Section 4; the resolved decisions are summarized at the end. - **`sketchlib-go`** is aligned separately (see Cross-language contract). @@ -197,8 +197,8 @@ This registry is the master list of algorithms still to design payloads for, and | `0x07 0x02` | Hydra | Count Sketch counter | Section 3.9 | implemented | | `0x07 0x03` | Hydra | HyperLogLog counter | Section 3.9 | implemented | | `0x07 0x04` | Hydra | UnivMon counter | Section 3.9 | implemented | -| `0x08 0x00` | SetAggregator | - | TBD | assigned in Go / payload not designed | -| `0x09 0x00` | DeltaResult | - | TBD | assigned in Go / payload not designed | +| `0x08 0x00` | SetAggregator | Exact string set | Section 3.21 | implemented | +| `0x09 0x00` | DeltaResult | Added / removed string sets | Section 3.22 | implemented | | `0x0a 0x00` | Count-Sketch-with-heap (CSHeap) | Count Sketch + top-k heap | Section 3.10 | implemented | | `0x0b 0x00` | Elastic | Heavy/light | Section 3.11 | implemented | | `0x0c 0x00` | Coco | CocoSketch | Section 3.12 | implemented | @@ -224,7 +224,7 @@ This registry is the master list of algorithms still to design payloads for, and - **CMSHeap vs CSHeap.** Go's `MagicCountMinSketchWithHeap` (`0x03`) is the Count-*Min*-with-heap sketch (`apis.md` to CMSHeap). The Count-*Sketch*-with-heap sketch (`apis.md` to CSHeap) is a distinct family and gets a fresh byte (`0x0a`), separate from `0x03`. - **Hydra.** `apis.md` lists the "Hydra" framework; Go's only Hydra id is `MagicHydraKLLSketch` (`0x07`), so Hydra maps here to the `0x07` family. Each base sketch under Hydra has its own variant: `0x07 0x00` KLL, `0x07 0x01` Count-Min, `0x07 0x02` Count Sketch, `0x07 0x03` HyperLogLog, `0x07 0x04` UnivMon. `0x07 0x05`-`0x07 0xff` are **reserved for Hydra over further base sketches**; a concrete variant is allocated when that combination's payload is designed. -- **SetAggregator / DeltaResult** (`0x08` / `0x09`) come from Go's `magic_ids.go` and do not appear as sketches in `apis.md` (they are aggregation and delta-result envelopes, distinct from stand-alone sketches). They are kept here so the family space stays mirrored verbatim with Go. +- **SetAggregator / DeltaResult** (`0x08` / `0x09`) take their family bytes from Go's `magic_ids.go`. They are exact string sets, not sketches, so `apis.md` does not list them. - **`0x19`-`0x1c`.** CountL2HH, UnivMon-Q, FoldCMS and FoldCS have no counterpart in Go's `magic_ids.go`; their family bytes are allocated here first, and Go mirrors them from this registry. - **`Unstable`** rows mirror the `Unstable` status those sketches carry in `apis.md`. The kind_id is permanent either way; the payload and the sketch API may still change, and a change to the payload takes a new id (Q-VER). @@ -276,7 +276,7 @@ The metadata map opens with `metadata_version`, then carries **two groups** of f The two tables below are the field-by-field detail of each group. -> **Non-hashing sketches omit the hash-spec group entirely** (Q-KLL, generalized as Q-NOHASH). The hash spec answers "how were keys hashed"; a sketch that does not hash its inputs has no truthful answer. Five kinds are in this class: **KLL** (§3.3), **DDSketch** (§3.8) and **UniformSampling** (§3.13), none of which hashes its inputs at all, and **ExponentialHistogram** (§3.17) and **EHSketchList** (§3.18), whose hashing all belongs to the sketch nested inside them. Each drops the six hash-spec keys and the seed index, keeps `metadata_version`, and has a metadata schema that is not built from a `HashProfile` the way HLL's and Count-Min's are. KLL is the worked example: comparison-based — it orders raw numeric values with `total_cmp` and never invokes a hasher — so its map is `metadata_version`, `k`, `m`, `item_type`, and an optional `seed`. Note `seed` is unrelated to the hash `seed_list`: it is the KLL compaction RNG's reproducible seed, and it is the **only optional key** in v1 (present only when the sketch was built with one; the key is omitted otherwise, and `KLLDynamic` never emits it). A consumer that does not use it (including Go) MUST still preserve it verbatim on re-encode so the bytes round-trip identically. +> **Non-hashing sketches omit the hash-spec group entirely** (Q-KLL, generalized as Q-NOHASH). The hash spec answers "how were keys hashed"; a sketch that does not hash its inputs has no truthful answer. Seven kinds are in this class: **KLL** (§3.3), **DDSketch** (§3.8) and **UniformSampling** (§3.13), none of which hashes its inputs at all; **ExponentialHistogram** (§3.17) and **EHSketchList** (§3.18), whose hashing all belongs to the sketch nested inside them; and **SetAggregator** (§3.21) and **DeltaResult** (§3.22), which carry their keys verbatim. Each drops the six hash-spec keys and the seed index, keeps `metadata_version`, and has a metadata schema that is not built from a `HashProfile` the way HLL's and Count-Min's are. KLL is the worked example: comparison-based — it orders raw numeric values with `total_cmp` and never invokes a hasher — so its map is `metadata_version`, `k`, `m`, `item_type`, and an optional `seed`. Note `seed` is unrelated to the hash `seed_list`: it is the KLL compaction RNG's reproducible seed, and it is the **only optional key** in v1 (present only when the sketch was built with one; the key is omitted otherwise, and `KLLDynamic` never emits it). A consumer that does not use it (including Go) MUST still preserve it verbatim on re-encode so the bytes round-trip identically. **Hash spec** @@ -1228,14 +1228,47 @@ Two things read it, and neither is a stored threshold. A level's flag joins the The encode side refuses the same states: a level count that disagrees with the config, a level whose Count Sketch layout is not the declared one, levels mixing counter widths, an over-capacity candidate table, a candidate at the wrong terminal level, a duplicate or over-capacity occurrence, and an inconsistent `count` / `min` / `max` triple. So the format never emits bytes it would refuse to read back. -### 3.21: payloads not yet designed +### 3.21: SetAggregator payload (`0x08 0x00`) + +`SetAggregator` is an exact set of string keys. It takes no construction parameter and puts no hash on the wire, so its metadata is `metadata_version` alone (Section 2, Q-NOHASH) and the payload is the members: + +| Pos | Field | Type | Notes | +| ----- | ------- | ------ | ------- | +| 0 | `values` | array | the members, `str`; strictly ascending | + +**Emitted order (cross-language contract).** Members are written in ascending **byte order of their UTF-8 encoding**, which is code point order and not UTF-16 order: `"～"` (U+FF5E) precedes `"😀"` (U+1F600). A set has exactly one encoding, so a decoded set re-serializes byte-identically, and an empty set is `[[]]`. + +**Decode rules.** Fail **closed** on each, with an error and never a panic: + +1. `kind_id` is `0x08 0x00`. +2. The metadata map is exactly `{metadata_version: 1}`. +3. Every element of `values` is msgpack `str`; a `bin` is rejected even when it holds valid UTF-8. +4. `values` is strictly ascending in byte order. This rejects a duplicate as well as a reordering. + +### 3.22: DeltaResult payload (`0x09 0x00`) + +`DeltaResult` is the change between two `SetAggregator` snapshots: the keys the later one added and the keys it removed. Like SetAggregator, its metadata is `metadata_version` alone: + +| Pos | Field | Type | Notes | +| ----- | ------- | ------ | ------- | +| 0 | `added` | array | keys present only in the later snapshot, `str`; strictly ascending | +| 1 | `removed` | array | keys present only in the earlier snapshot, `str`; strictly ascending | + +Both arrays follow §3.21's emitted order. An empty delta is `[[], []]`. + +**Decode rules.** Rules 2 to 4 of §3.21 apply to both arrays, and: + +1. `kind_id` is `0x09 0x00`. +2. No key is in both `added` and `removed`. + +Rule 2 is enforced on the **encode** side too: a `DeltaResult` holding a key in both sets fails to serialize. + +### 3.23: payloads not yet designed The remaining `kind_id`s reserve a family byte with payload TBD (the Section 1 registry carries their status). Likely shape when designed: | kind_id | Sketch | Likely payload | | --------- | -------- | --------- | -| `0x08 0x00` | SetAggregator | aggregation envelope, distinct from a stand-alone sketch (Section 1 mapping notes) | -| `0x09 0x00` | DeltaResult | delta-result envelope, distinct from a stand-alone sketch (Section 1 mapping notes) | | `0x0f 0x00` | HashSketchEnsemble | may not need serialization | | `0x12 0x00` | NitroBatch | may want another serialization abstraction in Storage | | `0x15 0x00` | EHUnivOptimized (`Unstable`) | TBD | @@ -1283,7 +1316,7 @@ Golden byte-vectors lock it. *Other arrays, by name.* `words` (§3.4) is Bloom's bit grid as unsigned 64-bit integers, one per packed word. `hashes` (§3.14) is KMV's retained digests as unsigned integers — a digest above `2^63` is still `uint`, never the `int` family. `l2` (§3.19, §3.15, §3.16) is one `i64` accumulator per row. `heap_counts` (§3.7, §3.10, §3.15, §3.16) is `i64` and may be negative for a CSHeap. `levels` and `coin` (§3.3) are integer arrays; `heap_lens` and `candidate_lens` are `u32` counts; `sizes` / `min_times` / `max_times` (§3.17), `priorities` (§3.13), `values` (§3.12), `candidate_keys` / `candidate_scores` / `occurrence_keys` / `occurrence_priority_high` / `occurrence_priority_low` (§3.20) are all unsigned integers; `vote_pos` / `vote_neg` (§3.11) are `i32`. -*Key and item arrays.* `keys` and `items` take the element type the metadata names — `key_type` for the `HeapItem` keys of §3.5, §3.7, §3.10, §3.15, §3.16 and Hydra's `counter_key_type` (§3.9), per §3.5's table; `item_type` for KLL's `items` (§3.3). Two payloads name no such key and are fixed by their kind_id instead: Coco's `keys` (§3.12) and Elastic's `flow_ids` (§3.11) are nil-or-`str`, and Coco's `values` are unsigned integers. UniformSampling's `values` (§3.13) is float64, which is what its `item_type` names. +*Key and item arrays.* `keys` and `items` take the element type the metadata names — `key_type` for the `HeapItem` keys of §3.5, §3.7, §3.10, §3.15, §3.16 and Hydra's `counter_key_type` (§3.9), per §3.5's table; `item_type` for KLL's `items` (§3.3). Four payloads name no such key and are fixed by their kind_id instead: Coco's `keys` (§3.12) and Elastic's `flow_ids` (§3.11) are nil-or-`str`, SetAggregator's `values` (§3.21) and DeltaResult's `added` / `removed` (§3.22) are `str`, and Coco's `values` are unsigned integers. UniformSampling's `values` (§3.13) is float64, which is what its `item_type` names. *Scalars.* Floats are always full **float64** (`0xcb`): HLL HIP's `hip_kxq0` / `hip_kxq1` / `hip_est` (§3.1), DDSketch's `sum` / `min` / `max` (§3.8), and any float element of an array above. The one exception is a `"f32"` **key**, which is float32 (`0xca`) because the variant is the key's identity and widening it would change which key it is (§3.5); that rule governs counter and sample values, which have no identity to preserve. Booleans are msgpack `true` / `false` (`0xc3` / `0xc2`) — `evictions` and `stale_copies` (§3.11), `candidate_complete` (§3.15, §3.16), `ever_evicted` (§3.20). Every other scalar is an integer under the family/width rule: `inserted` (§3.4), `total` / `discarded_max` (§3.5), `offset` (§3.8, **signed**, since a value below `1.0` maps to a negative bucket index), `total_seen` / `rng_state` (§3.13), `bucket_size` / `update_mode` (§3.15, §3.16), `count` / `source_id` / `next_sequence` (§3.20). @@ -1341,14 +1374,14 @@ Good direction (more compact, higher fidelity, less Rust-internal duplication), 2. **Golden byte-vector fixtures** in one shared repo, [`sketchlib-golden-bytes`](https://github.com/ProjectASAP/sketchlib-golden-bytes), which each implementation mounts at `asapv1_golden/`; both languages decode and re-encode them byte-identically. These replace the `portable`-as-oracle round-trip test. 3. **This registry**, mirrored, never independently allocated. -Fixtures exist for six `kind_id`s — HLL's three estimators, Count-Min, Count Sketch and compact KLL — and `sketchlib-go` mirrors those. Every other kind here has a payload and **no fixture**, so this document is its only contract; `asapv1_golden/README.md` lists the gap. `portable` carries its own Go goldens. +`asapv1_golden/README.md`'s Coverage section lists which `kind_id`s have fixtures; for every other kind this document is the only contract. `portable` carries its own Go goldens. **Hash profile on the Go side.** Rust derives the hash spec from a generic `HashProfile` bound on the hasher type; Go has no generic hasher type, so there is nothing to derive from. On the Go side the profile is simply **written into** the metadata on encode and **read from** it on decode. Go MUST validate the profile it reads (same fail-closed intent as Rust): a sketch is only mergeable/queryable if its `hash_profile_id` + seeds match the profile Go is prepared to reproduce. -Sequencing: (2) covers six `kind_id`s, and the one `native bytes == portable bytes` test is HLL-only, since no other `portable` type emits ASAPv1 bytes. +Sequencing: (2) covers the `kind_id`s `asapv1_golden/README.md`'s Coverage section lists, and the one `native bytes == portable bytes` test is HLL-only, since no other `portable` type emits ASAPv1 bytes. Retire `portable` once the fixtures cover the rest. --- diff --git a/docs/tests.md b/docs/tests.md index ccb1571c..856fd832 100644 --- a/docs/tests.md +++ b/docs/tests.md @@ -751,6 +751,30 @@ Wire tests: [`src/sketches/uniform/wire.rs`](../src/sketches/uniform/wire.rs) | `uniform_sampling_rejects_crafted_bytes_without_panicking` | Truncated, foreign and garbage bytes are errors, never panics. | Verifies six truncations of a valid envelope, 64 bytes of `0xff`, and a valid envelope carrying a garbage payload all fail to decode. | | `uniform_sampling_rejects_serializing_an_over_full_sampler` | The encode side refuses a sampler its own decoder would reject. | Verifies a sampler holding two samples whose `total_seen` is set to `1` fails to serialize. | +### SetAggregator and DeltaResult + +Test file: [`src/sketches/set_aggregator.rs`](../src/sketches/set_aggregator.rs) + +Wire tests: [`src/sketches/set_aggregator/wire.rs`](../src/sketches/set_aggregator/wire.rs) + +| test_name | test_description | what_is_tested | +| --- | --- | --- | +| `test_creation` | A fresh aggregator holds nothing. | Verifies `SetAggregator::new()` has an empty `values` set. | +| `test_insert` | A repeated key is inserted once. | Feeds `"web"`, `"api"`, and `"web"` again and verifies `values` holds 2 keys, both `"web"` and `"api"`. | +| `test_merge` | Merge is set union. | Merges an aggregator holding `{"api", "db"}` into one holding `{"web", "api"}` and verifies the result holds exactly `"web"`, `"api"`, and `"db"`. | +| `set_aggregator_round_trips_and_re_encodes_byte_identically` | The envelope frames a set under kind_id `0x08 0x00`, and the set survives a round trip. | For a set holding `""`, `"web"`, `"api"`, `"é"`, `"😀"` and `"～"`, verifies the bytes carry `kind_id_len` `2` and `0x08 0x00`, that the decode holds the same set, and that it re-serializes to the same bytes. | +| `set_aggregator_emits_members_in_ascending_byte_order` | Members are written in UTF-8 byte order. | Verifies the same set serializes to a hand-built payload ordered `""`, `"api"`, `"web"`, `"é"`, `"～"`, `"😀"`, so U+FF5E precedes U+1F600. | +| `set_aggregator_empty_has_exactly_one_encoding` | An empty set has one encoding. | Verifies `SetAggregator::new()` serializes to a hand-built envelope with an empty `values` array and decodes back empty. | +| `set_aggregator_rejects_unordered_and_duplicate_members` | `values` must be strictly ascending. | Verifies payloads `["b", "a"]`, `["a", "a"]` and `["😀", "～"]` each fail to decode. | +| `set_aggregator_rejects_a_bin_member` | A member must be msgpack `str`. | Verifies a payload whose one member is the `bin` `b"a"` fails to decode. | +| `metadata_rejects_unknown_missing_and_wrong_version` | The metadata map is exactly `{metadata_version: 1}`. | Verifies envelopes whose metadata adds a `bogus_field`, is an empty map, or carries `metadata_version` `2` each fail to decode. | +| `each_kind_rejects_the_other` | Neither decoder takes the other's kind_id. | Verifies an empty `SetAggregator` envelope fails to decode as a `DeltaResult`, and the reverse. | +| `delta_result_round_trips_and_re_encodes_byte_identically` | The envelope frames a delta under kind_id `0x09 0x00`, both sets ordered. | For `added` `{"queue", "été", "😀"}` and `removed` `{"db", "cache", ""}`, verifies the bytes carry `0x09 0x00` and equal a hand-built payload of `["queue", "été", "😀"]` and `["", "cache", "db"]`, that the decode holds both sets, and that it re-serializes to the same bytes. | +| `delta_result_empty_round_trips` | An empty delta round-trips. | Verifies a hand-built envelope with two empty arrays decodes to two empty sets and re-serializes to the same bytes. | +| `delta_result_rejects_a_key_both_added_and_removed` | `added` and `removed` are disjoint on both doors. | Verifies a payload with `"b"` on both sides fails to decode, and that a `DeltaResult` holding `"a"` on both sides fails to serialize. | +| `delta_result_rejects_unordered_sides` | Both arrays must be strictly ascending. | Verifies an `added` of `["b", "a"]` and a `removed` of `["c", "c"]` each fail to decode. | +| `crafted_bytes_are_errors_not_panics` | Truncated and garbage bytes are errors, never panics. | Verifies six truncations of a valid `SetAggregator` envelope, 64 bytes of `0xff` for either type, and a valid `DeltaResult` envelope carrying a garbage payload all fail to decode. | + ## Sketch Frameworks ### Hydra @@ -1318,15 +1342,6 @@ Test file: [`src/message_pack_format/portable/ddsketch.rs`](../src/message_pack_ | `test_consecutive_windows_emit_own_state` | Each window emits its own state rather than a cross-window difference. | Builds `win1` from `1.0..=50.0` inserted twice (count `2` per bucket) and `win2` from `1.0..=50.0` once, and verifies `win2`'s delta against empty reconstructs `win2`'s `total_count` and that those bytes differ from `win2`'s delta against `win1`. | | `test_update_then_envelope_matches_go_golden_bytes` | The proto envelope for a populated store is byte-identical to the golden. | For an `alpha` `0.01` sketch fed `1.0..=50.0`, builds a `DdSketchState` from `wire_alpha()`, `store_counts`, and `store_offset`, wraps it in a `SketchEnvelope` with `format_version` `1`, and verifies the prost encoding equals a 403-byte inline golden carrying a 384-entry store and a `store_offset` of `-64` (sint32 zigzag `0x7f`); the golden constant records that it is the Rust-produced value and still to be reconciled against the Go reference. | -### Portable DeltaResult - -Test file: [`src/message_pack_format/portable/delta_set_aggregator.rs`](../src/message_pack_format/portable/delta_set_aggregator.rs) - -| test_name | test_description | what_is_tested | -| --- | --- | --- | -| `test_msgpack_round_trip` | Both key sets survive the round trip. | Round-trips a `DeltaResult` with `added` `{"web", "api"}` and `removed` `{"db"}` and verifies the decode holds 2 added keys including both names and 1 removed key. | -| `test_empty_sets` | A `DeltaResult` holding nothing round-trips. | Verifies a `DeltaResult` with both sets empty encodes and decodes back with both sets still empty. | - ### Portable HllSketch Test file: [`src/message_pack_format/portable/hll.rs`](../src/message_pack_format/portable/hll.rs) @@ -1406,18 +1421,6 @@ Test file: [`src/message_pack_format/portable/sampling.rs`](../src/message_pack_ | `rescale_with_env` | The envelope's own `sample_p` drives the rescale. | Verifies `rescale_count_with_env(5_000.0, env)` at `sample_p` `0.05` is `100_000.0` within `1e-6`, and that at `sample_p` `0.0` an input of `42.0` comes back unchanged. | | `quantiles_are_scale_invariant` | The quantile family is declared to need no rescale. | Asserts `is_quantile_scale_invariant()`, a `const fn` returning `true`; nothing about a quantile estimate is computed. | -### Portable SetAggregator - -Test file: [`src/message_pack_format/portable/set_aggregator.rs`](../src/message_pack_format/portable/set_aggregator.rs) - -| test_name | test_description | what_is_tested | -| --- | --- | --- | -| `test_creation` | A fresh aggregator holds nothing. | Verifies `SetAggregator::new()` has an empty `values` set. | -| `test_insert` | A repeated key is inserted once. | Feeds `"web"`, `"api"`, and `"web"` again and verifies `values` holds 2 keys, both `"web"` and `"api"`. | -| `test_merge` | Merge is set union. | Merges an aggregator holding `{"api", "db"}` into one holding `{"web", "api"}` and verifies the result holds exactly `"web"`, `"api"`, and `"db"`. | -| `test_msgpack_round_trip` | The set survives the round trip. | Round-trips an aggregator holding `{"web", "api"}` and verifies the decode holds 2 keys, both names present. | -| `test_msgpack_matches_wire_format` | The bytes are a named map under a `values` key. | For an aggregator holding `{"a"}`, verifies the bytes decode into a locally declared `StringSet { values: HashSet }` that still contains `"a"`. | - ## Common ### Common Hash Utilities @@ -1638,7 +1641,7 @@ Test file: [`tests/e2e/cardinality.rs`](../tests/e2e/cardinality.rs) | `set_aggregator_union_is_exact` | A `SetAggregator` union keeps every member exactly. | Feeds `uniform_u64(20_000, 500, 2002)` as `member-{k}` strings into `SetAggregator::update`, merges a second aggregator holding `extra-a` and `extra-b`, then asserts `agg.values.len()` equals the length of a parallel `HashSet` of the same members and that `agg.values` contains every one of them; the distinct count is compared against the in-test truth set rather than a pinned number. | | `custom_precision_accuracy_improves_with_precision_as_the_error_model_predicts` | A higher custom precision does not lose to a lower one on the same stream. | Feeds the same `N = 200_000` identities from `BASE = (1 << 40) * 41` into `HyperLogLogImpl`, `` and `` and asserts only the two orderings `e18 <= e10` and `e18 <= e13` on absolute relative error; no error band is asserted, p10 against p13 is not compared, and with one trial per precision no error ratio is pinned. | | `a_custom_precision_merge_reproduces_the_single_pass_registers_for_every_estimator` | A shard merge at a custom precision reproduces the single pass exactly, for both register estimators. | Over `N = 120_000` identities from `BASE = (1 << 40) * 42` split even/odd by `k % 2`, builds single/even/odd `HyperLogLogImpl` and `HyperLogLogImpl`, calls `merge`, and asserts for each estimator that the merged `registers_as_slice()` equals the single pass's and that `estimate()` is equal too. | -| `a_set_aggregator_delta_describes_the_change_and_survives_the_wire` | A `DeltaResult` over two set snapshots survives the MessagePack wire and replays into the later snapshot. | Builds `SetAggregator`s over `["web", "api", "db", "cache"]` and `["web", "api", "queue"]`, forms a `DeltaResult` from the two `HashSet::difference` results, and asserts `added == {"queue"}` and `removed == {"db", "cache"}` (these two pin the test's own set arithmetic, not a library call); then `to_msgpack` / `DeltaResult::from_msgpack` must return both sets unchanged, applying the decoded delta to the earlier aggregator must reproduce `after.values`, and an all-empty `DeltaResult` must round-trip still empty. | +| `a_set_aggregator_delta_describes_the_change_and_survives_the_wire` | A `DeltaResult` over two set snapshots survives the ASAPv1 wire and replays into the later snapshot. | Builds `SetAggregator`s over `["web", "api", "db", "cache"]` and `["web", "api", "queue"]`, forms a `DeltaResult` from the two `HashSet::difference` results, and asserts `added == {"queue"}` and `removed == {"db", "cache"}` (these two pin the test's own set arithmetic, not a library call); then `serialize_to_bytes` / `DeltaResult::deserialize_from_bytes` must return both sets unchanged, applying the decoded delta to the earlier aggregator must reproduce `after.values`, and an all-empty `DeltaResult` must round-trip still empty. | ### Cardinality: Duplicate-Heavy Zipf Streams @@ -2126,11 +2129,15 @@ Test file: [`tests/asapv1_golden.rs`](../tests/asapv1_golden.rs) | `cs_i32_regular_2x4_matches_golden` | The i32 counter type reaches the metadata, and an i32 golden is not an i64 sketch. | Verifies `Count::, RegularPath>::from_storage` over `[[0, 127, 128, 65536], [-1, -33, -32768, -2147483648]]` cast to `i32` serializes to `asapv1_golden/cs_i32_regular_2x4.hex`, that decoding it recovers the eight cells and re-serializes identically, that the i32 and i64 regular fixtures have equal length yet differ, and that the i32 fixture fails to decode as `Count, RegularPath>`. | | `cs_i64_fast_2x4_matches_golden` | The fast-path mode reaches the bytes, so the same matrix encodes differently per path. | Verifies `Count::, FastPath>::from_storage` over `[[0, 127, 128, 65536], [-1, -33, -32768, -2147483648]]` serializes to `asapv1_golden/cs_i64_fast_2x4.hex`, that decoding it recovers the eight cells and re-serializes identically, and that the fixture differs from `cs_i64_regular_2x4.hex` built from the same matrix. | | `cs_i64_regular_2x4_matches_golden` | A known signed Count-Sketch matrix serializes to its checked-in golden bytes. | Verifies `Count::, RegularPath>::from_storage` over `[[0, 127, 128, 65536], [-1, -33, -32768, -2147483648]]` — sweeping the msgpack integer widths in both directions — serializes to `asapv1_golden/cs_i64_regular_2x4.hex`, and that decoding the fixture reports `rows()` `2`, `cols()` `4`, the eight cells, and a byte-identical re-serialization. | +| `delta_result_empty_matches_golden` | An empty delta serializes to its golden bytes. | Verifies a `DeltaResult` with both sets empty serializes to `asapv1_golden/delta_result_empty.hex`, and that decoding the fixture gives two empty sets and a byte-identical re-serialization. | +| `delta_result_strings_matches_golden` | A known delta serializes to its golden bytes, each side in byte order. | Verifies `added` `{"queue", "été", "😀"}` and `removed` `{"", "cache", "db", "中"}` serialize to `asapv1_golden/delta_result_strings.hex`, and that decoding the fixture recovers both sets and re-serializes to the same bytes. | | `hll_classic_p12_matches_golden` | A known P12 Classic register pattern serializes to its golden bytes on both the portable and native paths. | Builds `HllSketch::from_raw(HllVariant::Regular, 12, regs, 0.0, 0.0, 0.0)` where `regs` is 4,096 zeros with `[0] = 1`, `[1] = 7`, `[100] = 42` and `[4095] = 3`, verifies `to_msgpack()` equals `asapv1_golden/hll_classic_p12.hex`, that `HyperLogLogP12::::deserialize_from_bytes` on the fixture recovers those registers and re-serializes identically, and that `HllSketch::from_msgpack` reports the same registers, `precision` `12` and `HllVariant::Regular`. | | `hll_ertl_mle_p12_matches_golden` | The Datafusion variant tag reaches the bytes for the same P12 register pattern. | Builds `HllSketch::from_raw(HllVariant::Datafusion, 12, regs, 0.0, 0.0, 0.0)` over the shared P12 pattern (`[0] = 1`, `[1] = 7`, `[100] = 42`, `[4095] = 3` in 4,096 registers), verifies `to_msgpack()` equals `asapv1_golden/hll_ertl_mle_p12.hex`, that `HyperLogLogP12::::deserialize_from_bytes` recovers the registers and re-serializes identically, and that `HllSketch::from_msgpack` reports `HllVariant::Datafusion`. | | `hll_hip_p12_matches_golden` | The HIP variant's three running scalars travel on the wire beside its registers. | Builds `HllSketch::from_raw(HllVariant::Hip, 12, regs, 1.5, 2.5, 3.0)` over the shared P12 pattern, verifies `to_msgpack()` equals `asapv1_golden/hll_hip_p12.hex`, that `HyperLogLogHIPP12::deserialize_from_bytes` re-serializes the fixture identically, and that `HllSketch::from_msgpack` reports the registers, `HllVariant::Hip`, `hip_kxq0` `1.5`, `hip_kxq1` `2.5` and `hip_est` `3.0`. | | `kll_f64_k200_matches_golden` | A deterministic f64 KLL serializes to its golden bytes, which round-trip unchanged. | Verifies `KLL::::init_kll_with_seed(200, 42)` fed `1..=50` (below the level-0 capacity, so no compaction fires) serializes to `asapv1_golden/kll_f64_k200.hex`, and that decoding the fixture re-serializes to the same bytes and reports `quantile(0.0)` `1.0` and `quantile(1.0)` `50.0`. | | `kll_i64_k200_matches_golden` | A deterministic i64 KLL serializes to its golden bytes, which round-trip unchanged. | Verifies `KLL::::init_kll_with_seed(200, 42)` fed `1..=50` serializes to `asapv1_golden/kll_i64_k200.hex`, and that decoding the fixture re-serializes to the same bytes and reports `quantile(0.0)` `1.0` and `quantile(1.0)` `50.0`. | +| `set_aggregator_empty_matches_golden` | An empty set serializes to its golden bytes. | Verifies `SetAggregator` with no members serializes to `asapv1_golden/set_aggregator_empty.hex`, and that decoding the fixture gives an empty set and a byte-identical re-serialization. | +| `set_aggregator_strings_matches_golden` | A known string set serializes to its golden bytes in UTF-8 byte order. | Verifies the set `{"", "abcdefghijklmnopqrstuvwxyz012345", "api", "web", "é", "中", "～", "😀"}` — the empty string, a 32-byte str8, and code points of two, three and four bytes — serializes to `asapv1_golden/set_aggregator_strings.hex`, and that decoding the fixture recovers the set and re-serializes to the same bytes. | ### MessagePack Envelope Compatibility @@ -2146,7 +2153,6 @@ Test file: [`tests/msgpack_compat.rs`](../tests/msgpack_compat.rs) | `count_sketch_round_trip` | A portable `CountSketch` survives a msgpack round trip with its geometry intact. | Updates `CountSketch::new(3, 64)` with `k1` `+1.0` and `k2` `+2.0`, then verifies the decode reports `rows` `3` and `cols` `64`; the signed counters and the estimates are not checked. | | `dd_sketch_decodes_go_bytes` | The test asserts nothing. | The test asserts nothing: the body is empty and the test is marked `#[ignore]` with the reason that no cross-language msgpack fixture exists in the repository. | | `dd_sketch_round_trip` | A portable `DdSketch` survives a msgpack round trip carrying its full count in the bucket array. | Updates `DdSketch::new(0.01)` with `1.0`, `10.0` and `100.0`, then verifies the decode's `total_count()` — recovered by summing the buckets, since the count is not on the wire — is `3`; `alpha`, the bucket offset and the quantiles are not checked. | -| `delta_result_round_trip` | A `DeltaResult` survives a msgpack round trip with both of its sets. | Serializes a `DeltaResult` whose `added` holds `a` and whose `removed` holds `b`, then verifies the decode's `added` contains `a` and its `removed` contains `b`. | | `hll_sketch_decodes_go_bytes` | The test asserts nothing. | The test asserts nothing: the body is empty and the test is marked `#[ignore]` with the reason that no cross-language msgpack fixture exists in the repository. | | `hll_sketch_round_trip` | A portable `HllSketch` survives a msgpack round trip with its register count intact. | Updates `HllSketch::new(HllVariant::Regular, 8)` with `b"a"`, `b"b"` and `b"c"`, then verifies only that the decode's `registers` has the same length as the source's; the register values, the precision and the variant are not checked. | | `hydra_kll_decodes_go_bytes` | The test asserts nothing. | The test asserts nothing: the body is empty and the test is marked `#[ignore]` with the reason that no cross-language msgpack fixture exists in the repository. | @@ -2154,7 +2160,6 @@ Test file: [`tests/msgpack_compat.rs`](../tests/msgpack_compat.rs) | `hydra_kll_wire_shape` | The `HydraKllSketchWire` DTO round-trips through `rmp_serde` with its two levels of nesting intact. | Serializes a wire struct of `rows: 2`, `cols: 3` and a `2x3` grid of `KllSketchData { k: 200, sketch_bytes: vec![] }`, then verifies the `rmp_serde` decode reports `rows` `2`, `cols` `3`, an outer length of `2` and an inner length of `3`. | | `kll_sketch_decodes_go_bytes` | The test asserts nothing. | The test asserts nothing: the body is empty and the test is marked `#[ignore]` with the reason that no cross-language msgpack fixture exists in the repository. | | `kll_sketch_round_trip` | A portable `KllSketch` survives a msgpack round trip with its `k` and retained mass intact. | Feeds `KllSketch::with_seed(200, 0x5EED_0800)` the values `0..100` as `f64`, then verifies the decode reports `k` `200` and `count()` `100`; the retained items and the quantile answers are not checked. | -| `set_aggregator_round_trip` | A `SetAggregator` survives a msgpack round trip with its members. | Updates a `SetAggregator::new()` with `web` and `api`, then verifies the decode holds two values, one of which is `web`. | ### Conformance Kit diff --git a/src/lib.rs b/src/lib.rs index b062b8b9..2e9bb74d 100644 --- a/src/lib.rs +++ b/src/lib.rs @@ -11,8 +11,7 @@ //! - [`message_pack_format`]: MessagePack/proto wire format shared with //! `sketchlib-go`. Owns the wire-DTO sketch types consumed by the ASAP //! query engine (`CountMinSketch`, `CountSketch`, `KllSketch`, `HllSketch`, -//! `DdSketch`, `HydraKllSketch`, `CountMinSketchWithHeap`, -//! `SetAggregator`, `DeltaResult`). +//! `DdSketch`, `HydraKllSketch`, `CountMinSketchWithHeap`). //! //! Most users can start with the crate-root re-exports such as [`DataInput`], //! [`CountMin`], [`HyperLogLog`], [`KLL`], and [`DDSketch`]. Reach for the @@ -48,7 +47,6 @@ pub use message_pack_format::portable::countsketch::{ }; pub use message_pack_format::portable::countsketch_topk::{CountSketchWithHeap, CsHeapItem}; pub use message_pack_format::portable::ddsketch::{DDSKETCH_GROW_CHUNK, DdSketch, DdSketchDelta}; -pub use message_pack_format::portable::delta_set_aggregator::DeltaResult; pub use message_pack_format::portable::hll::{HllSketch, HllSketchDelta, HllVariant}; pub use message_pack_format::portable::hydra_kll::HydraKllSketch; pub use message_pack_format::portable::kll::{KllSketch, KllSketchData}; @@ -56,6 +54,5 @@ pub use message_pack_format::portable::sampling::{ effective_sample_p, is_quantile_scale_invariant, rescale_count, rescale_count_with_env, sample_p_or_default, }; -pub use message_pack_format::portable::set_aggregator::SetAggregator; pub use sketch_framework::*; pub use sketches::*; diff --git a/src/message_pack_format/portable/delta_set_aggregator.rs b/src/message_pack_format/portable/delta_set_aggregator.rs deleted file mode 100644 index d26dbeb9..00000000 --- a/src/message_pack_format/portable/delta_set_aggregator.rs +++ /dev/null @@ -1,65 +0,0 @@ -//! Wire-format DTO for the delta set aggregator. -//! -//! Owns the [`DeltaResult`] wire DTO and its [`MessagePackCodec`] impl. -//! The streaming/state-tracking logic lives in the downstream consumer -//! (ASAPQuery's accumulators) — only the over-the-wire shape lives here. - -use std::collections::HashSet; - -use serde::{Deserialize, Serialize}; - -use crate::message_pack_format::{Error as MsgPackError, MessagePackCodec}; - -/// Wire DTO for the delta set aggregator: a snapshot of added/removed -/// string keys between two consecutive observations. -#[derive(Debug, Clone, Serialize, Deserialize)] -pub struct DeltaResult { - pub added: HashSet, - pub removed: HashSet, -} - -impl MessagePackCodec for DeltaResult { - fn to_msgpack(&self) -> Result, MsgPackError> { - Ok(rmp_serde::to_vec(self)?) - } - - fn from_msgpack(bytes: &[u8]) -> Result { - Ok(rmp_serde::from_slice(bytes)?) - } -} - -#[cfg(test)] -mod tests { - use super::*; - - #[test] - fn test_msgpack_round_trip() { - let mut added = HashSet::new(); - added.insert("web".to_string()); - added.insert("api".to_string()); - - let mut removed = HashSet::new(); - removed.insert("db".to_string()); - - let bytes = DeltaResult { added, removed }.to_msgpack().unwrap(); - let result = DeltaResult::from_msgpack(&bytes).unwrap(); - - assert_eq!(result.added.len(), 2); - assert!(result.added.contains("web")); - assert!(result.added.contains("api")); - assert_eq!(result.removed.len(), 1); - assert!(result.removed.contains("db")); - } - - #[test] - fn test_empty_sets() { - let dr = DeltaResult { - added: HashSet::new(), - removed: HashSet::new(), - }; - let bytes = dr.to_msgpack().unwrap(); - let result = DeltaResult::from_msgpack(&bytes).unwrap(); - assert!(result.added.is_empty()); - assert!(result.removed.is_empty()); - } -} diff --git a/src/message_pack_format/portable/mod.rs b/src/message_pack_format/portable/mod.rs index 97e0d372..38abf567 100644 --- a/src/message_pack_format/portable/mod.rs +++ b/src/message_pack_format/portable/mod.rs @@ -12,9 +12,7 @@ pub mod countminsketch_topk; pub mod countsketch; pub mod countsketch_topk; pub mod ddsketch; -pub mod delta_set_aggregator; pub mod hll; pub mod hydra_kll; pub mod kll; pub mod sampling; -pub mod set_aggregator; diff --git a/src/sketches/mod.rs b/src/sketches/mod.rs index c3dd98d6..617d4663 100644 --- a/src/sketches/mod.rs +++ b/src/sketches/mod.rs @@ -106,6 +106,10 @@ pub use uniform::UniformSampling; pub mod ddsketch; pub use ddsketch::DDSketch; +/// Exact string-key set and the delta between two of its snapshots. +pub mod set_aggregator; +pub use set_aggregator::{DeltaResult, SetAggregator}; + pub mod countminsketch_topk; pub use countminsketch_topk::CMSHeap; diff --git a/src/message_pack_format/portable/set_aggregator.rs b/src/sketches/set_aggregator.rs similarity index 50% rename from src/message_pack_format/portable/set_aggregator.rs rename to src/sketches/set_aggregator.rs index d346a9e8..5c51c17d 100644 --- a/src/message_pack_format/portable/set_aggregator.rs +++ b/src/sketches/set_aggregator.rs @@ -1,15 +1,13 @@ -//! Set aggregator: wire-format type for tracking a set of unique string keys. +//! Exact set of string keys, and the change between two of its snapshots. //! -//! `SetAggregator` is not a sketch — it's a plain `HashSet` wrapped -//! for cross-language interop. Wire format: `StringSet { values: -//! HashSet }` in MessagePack. No `sketches::*` equivalent exists, -//! so both the wire shape and the runtime API live here. +//! [`SetAggregator`] is not a sketch: it keeps every distinct key it sees, so +//! its cardinality and membership are exact. [`DeltaResult`] records the keys +//! one snapshot added and removed relative to an earlier one. Both serialize +//! to ASAPv1 (`docs/asapv1_wire_format.md` §3.21, §3.22). use std::collections::HashSet; -use serde::{Deserialize, Serialize}; - -use crate::message_pack_format::{Error as MsgPackError, MessagePackCodec}; +mod wire; /// Set aggregator for tracking a set of unique string keys. #[derive(Debug, Clone)] @@ -61,41 +59,17 @@ impl Default for SetAggregator { } } -// ----- Wire format ----- - -/// Borrowed serialize-side wire DTO. Avoids cloning the underlying set -/// on the encode path. -#[derive(Serialize)] -pub(crate) struct StringSetRef<'a> { - pub values: &'a HashSet, -} - -/// Owned deserialize-side wire DTO. -#[derive(Deserialize)] -pub(crate) struct StringSetOwned { - pub values: HashSet, -} - -impl MessagePackCodec for SetAggregator { - fn to_msgpack(&self) -> Result, MsgPackError> { - let wrapper = StringSetRef { - values: &self.values, - }; - Ok(rmp_serde::to_vec(&wrapper)?) - } - - fn from_msgpack(bytes: &[u8]) -> Result { - let wrapper: StringSetOwned = rmp_serde::from_slice(bytes)?; - Ok(Self { - values: wrapper.values, - }) - } +/// The keys a later set snapshot added and removed relative to an earlier +/// one. The two sets are disjoint. +#[derive(Debug, Clone)] +pub struct DeltaResult { + pub added: HashSet, + pub removed: HashSet, } #[cfg(test)] mod tests { use super::*; - use serde::Deserialize; #[test] fn test_creation() { @@ -130,32 +104,4 @@ mod tests { assert!(sa1.values.contains("api")); assert!(sa1.values.contains("db")); } - - #[test] - fn test_msgpack_round_trip() { - let mut sa = SetAggregator::new(); - sa.update("web"); - sa.update("api"); - - let bytes = sa.to_msgpack().unwrap(); - let deserialized = SetAggregator::from_msgpack(&bytes).unwrap(); - - assert_eq!(deserialized.values.len(), 2); - assert!(deserialized.values.contains("web")); - assert!(deserialized.values.contains("api")); - } - - #[test] - fn test_msgpack_matches_wire_format() { - #[derive(Deserialize)] - struct StringSet { - values: HashSet, - } - let mut sa = SetAggregator::new(); - sa.update("a"); - let bytes = sa.to_msgpack().unwrap(); - let decoded: StringSet = - rmp_serde::from_slice(&bytes).expect("should decode as StringSet { values: ... }"); - assert!(decoded.values.contains("a")); - } } diff --git a/src/sketches/set_aggregator/wire.rs b/src/sketches/set_aggregator/wire.rs new file mode 100644 index 00000000..e53f32c2 --- /dev/null +++ b/src/sketches/set_aggregator/wire.rs @@ -0,0 +1,355 @@ +//! ASAPv1 wire serialization for [`SetAggregator`] (`0x08 0x00`, §3.21) and +//! [`DeltaResult`] (`0x09 0x00`, §3.22). +//! +//! ## No hash spec, no structural params +//! +//! Neither type hashes a key onto the wire or takes a construction parameter, +//! so the metadata map is `metadata_version` alone. +//! +//! ## Canonical order +//! +//! Every string array is written in strictly ascending byte order of its UTF-8 +//! encoding, so a set has exactly one encoding. The decoder rejects any other +//! order, which also rejects a duplicate key. + +use std::collections::HashSet; + +use rmp_serde::{decode::Error as RmpDecodeError, encode::Error as RmpEncodeError, from_slice}; +use serde::{Deserialize, Serialize}; + +use crate::message_pack_format::envelope; +use crate::message_pack_format::wire_key::WireString; + +use super::{DeltaResult, SetAggregator}; + +/// SetAggregator kind_id: family `0x08`, single variant `0x00`. +const SET_AGGREGATOR_KIND: &[u8] = &[0x08, 0x00]; + +/// DeltaResult kind_id: family `0x09`, single variant `0x00`. +const DELTA_RESULT_KIND: &[u8] = &[0x09, 0x00]; + +/// The metadata map both kinds carry (ASAPv1 §2), written with +/// `to_vec_named`. `deny_unknown_fields` makes decode fail closed. +#[derive(Debug, PartialEq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +struct SetMetadata { + metadata_version: u8, +} + +const SET_METADATA: SetMetadata = SetMetadata { + metadata_version: 1, +}; + +/// SetAggregator payload (§3.21), positional: `[values]`. +#[derive(Debug, Serialize, Deserialize)] +struct SetAggregatorPayload { + values: Vec, +} + +/// DeltaResult payload (§3.22), positional: `[added, removed]`. +#[derive(Debug, Serialize, Deserialize)] +struct DeltaResultPayload { + added: Vec, + removed: Vec, +} + +/// The set's members in canonical (ascending byte) order. +fn canonical(set: &HashSet) -> Vec<&str> { + let mut keys: Vec<&str> = set.iter().map(String::as_str).collect(); + keys.sort_unstable(); + keys +} + +/// Rebuilds a set from a wire array, rejecting any array that is not strictly +/// ascending. The set reserves from the array actually shipped. +fn read_canonical(field: &str, keys: Vec) -> Result, RmpDecodeError> { + let mut set = HashSet::with_capacity(keys.len()); + let mut previous: Option = None; + for (idx, key) in keys.into_iter().enumerate() { + let key = key.into_string(); + if let Some(prev) = &previous + && prev.as_str() >= key.as_str() + { + return Err(RmpDecodeError::Uncategorized(format!( + "ASAPv1 {field} must be strictly ascending in byte order, broken at index {idx}" + ))); + } + set.insert(key.clone()); + previous = Some(key); + } + Ok(set) +} + +/// Splits the envelope, checks the kind_id and the metadata, and returns the +/// payload slice. +fn open<'a>(bytes: &'a [u8], kind: &[u8], name: &str) -> Result<&'a [u8], RmpDecodeError> { + let (kind_id, metadata, payload) = + envelope::split(bytes).map_err(RmpDecodeError::Uncategorized)?; + if kind_id != kind { + return Err(RmpDecodeError::Uncategorized(format!( + "{name} kind_id mismatch: stored {kind_id:?}, expected {kind:?}" + ))); + } + let meta: SetMetadata = from_slice(metadata)?; + if meta != SET_METADATA { + return Err(RmpDecodeError::Uncategorized(format!( + "ASAPv1 {name} envelope: metadata mismatch" + ))); + } + Ok(payload) +} + +fn metadata_bytes() -> Result, RmpEncodeError> { + rmp_serde::to_vec_named(&SET_METADATA) +} + +impl SetAggregator { + /// Serializes the set into an ASAPv1 MessagePack envelope, members in + /// ascending byte order. + pub fn serialize_to_bytes(&self) -> Result, RmpEncodeError> { + let payload = rmp_serde::to_vec(&SetAggregatorPayload { + values: canonical(&self.values), + })?; + Ok(envelope::encode( + SET_AGGREGATOR_KIND, + &metadata_bytes()?, + &payload, + )) + } + + /// Deserializes a set from an ASAPv1 MessagePack envelope. + pub fn deserialize_from_bytes(bytes: &[u8]) -> Result { + let payload = open(bytes, SET_AGGREGATOR_KIND, "SetAggregator")?; + let p: SetAggregatorPayload = from_slice(payload)?; + Ok(SetAggregator { + values: read_canonical("SetAggregator values", p.values)?, + }) + } +} + +impl DeltaResult { + /// Serializes the delta into an ASAPv1 MessagePack envelope, each set in + /// ascending byte order. A key in both `added` and `removed` is an error. + pub fn serialize_to_bytes(&self) -> Result, RmpEncodeError> { + if let Some(key) = self.added.intersection(&self.removed).next() { + return Err(RmpEncodeError::Syntax(format!( + "ASAPv1 DeltaResult: key {key:?} is both added and removed" + ))); + } + let payload = rmp_serde::to_vec(&DeltaResultPayload { + added: canonical(&self.added), + removed: canonical(&self.removed), + })?; + Ok(envelope::encode( + DELTA_RESULT_KIND, + &metadata_bytes()?, + &payload, + )) + } + + /// Deserializes a delta from an ASAPv1 MessagePack envelope. + pub fn deserialize_from_bytes(bytes: &[u8]) -> Result { + let payload = open(bytes, DELTA_RESULT_KIND, "DeltaResult")?; + let p: DeltaResultPayload = from_slice(payload)?; + let added = read_canonical("DeltaResult added", p.added)?; + let removed = read_canonical("DeltaResult removed", p.removed)?; + if let Some(key) = added.intersection(&removed).next() { + return Err(RmpDecodeError::Uncategorized(format!( + "ASAPv1 DeltaResult: key {key:?} is both added and removed" + ))); + } + Ok(DeltaResult { added, removed }) + } +} + +#[cfg(test)] +mod tests { + use super::*; + + fn set_of(keys: &[&str]) -> HashSet { + keys.iter().map(|k| k.to_string()).collect() + } + + fn crafted_set(values: &[&str]) -> Vec { + envelope::encode( + SET_AGGREGATOR_KIND, + &metadata_bytes().unwrap(), + &rmp_serde::to_vec(&SetAggregatorPayload { + values: values.to_vec(), + }) + .unwrap(), + ) + } + + fn crafted_delta(added: &[&str], removed: &[&str]) -> Vec { + envelope::encode( + DELTA_RESULT_KIND, + &metadata_bytes().unwrap(), + &rmp_serde::to_vec(&DeltaResultPayload { + added: added.to_vec(), + removed: removed.to_vec(), + }) + .unwrap(), + ) + } + + const AWKWARD: [&str; 6] = ["", "web", "api", "\u{e9}", "\u{1f600}", "\u{ff5e}"]; + + #[test] + fn set_aggregator_round_trips_and_re_encodes_byte_identically() { + let agg = SetAggregator { + values: set_of(&AWKWARD), + }; + let bytes = agg.serialize_to_bytes().expect("serialize"); + assert!(bytes.starts_with(b"ASAPv1")); + assert_eq!(&bytes[7..10], &[2u8, 0x08, 0x00]); + + let decoded = SetAggregator::deserialize_from_bytes(&bytes).expect("decode"); + assert_eq!(decoded.values, agg.values); + assert_eq!(decoded.serialize_to_bytes().expect("re-serialize"), bytes); + } + + /// Byte order, not UTF-16 order: U+FF5E precedes U+1F600. + #[test] + fn set_aggregator_emits_members_in_ascending_byte_order() { + let agg = SetAggregator { + values: set_of(&AWKWARD), + }; + let want = crafted_set(&["", "api", "web", "\u{e9}", "\u{ff5e}", "\u{1f600}"]); + assert_eq!(agg.serialize_to_bytes().expect("serialize"), want); + } + + #[test] + fn set_aggregator_empty_has_exactly_one_encoding() { + let bytes = SetAggregator::new() + .serialize_to_bytes() + .expect("serialize"); + assert_eq!(bytes, crafted_set(&[])); + let decoded = SetAggregator::deserialize_from_bytes(&bytes).expect("decode"); + assert!(decoded.values.is_empty()); + } + + #[test] + fn set_aggregator_rejects_unordered_and_duplicate_members() { + assert!(SetAggregator::deserialize_from_bytes(&crafted_set(&["b", "a"])).is_err()); + assert!(SetAggregator::deserialize_from_bytes(&crafted_set(&["a", "a"])).is_err()); + assert!( + SetAggregator::deserialize_from_bytes(&crafted_set(&["\u{1f600}", "\u{ff5e}"])) + .is_err() + ); + } + + /// A member must be msgpack `str`; a `bin` holding valid UTF-8 is rejected. + #[test] + fn set_aggregator_rejects_a_bin_member() { + let payload = rmp_serde::to_vec(&SetAggregatorPayload { + values: vec![serde_bytes::Bytes::new(b"a")], + }) + .unwrap(); + let bytes = envelope::encode(SET_AGGREGATOR_KIND, &metadata_bytes().unwrap(), &payload); + assert!(SetAggregator::deserialize_from_bytes(&bytes).is_err()); + } + + #[test] + fn metadata_rejects_unknown_missing_and_wrong_version() { + #[derive(Serialize)] + struct WithExtra { + metadata_version: u8, + bogus_field: u8, + } + #[derive(Serialize)] + struct Empty {} + let payload = rmp_serde::to_vec(&SetAggregatorPayload::<&str> { values: vec![] }).unwrap(); + let metas = [ + rmp_serde::to_vec_named(&WithExtra { + metadata_version: 1, + bogus_field: 0, + }) + .unwrap(), + rmp_serde::to_vec_named(&Empty {}).unwrap(), + rmp_serde::to_vec_named(&SetMetadata { + metadata_version: 2, + }) + .unwrap(), + ]; + for meta in metas { + let bytes = envelope::encode(SET_AGGREGATOR_KIND, &meta, &payload); + assert!(SetAggregator::deserialize_from_bytes(&bytes).is_err()); + } + } + + #[test] + fn each_kind_rejects_the_other() { + let set = SetAggregator::new().serialize_to_bytes().unwrap(); + let delta = DeltaResult { + added: HashSet::new(), + removed: HashSet::new(), + } + .serialize_to_bytes() + .unwrap(); + assert!(DeltaResult::deserialize_from_bytes(&set).is_err()); + assert!(SetAggregator::deserialize_from_bytes(&delta).is_err()); + } + + #[test] + fn delta_result_round_trips_and_re_encodes_byte_identically() { + let delta = DeltaResult { + added: set_of(&["queue", "\u{e9}t\u{e9}", "\u{1f600}"]), + removed: set_of(&["db", "cache", ""]), + }; + let bytes = delta.serialize_to_bytes().expect("serialize"); + assert_eq!(&bytes[7..10], &[2u8, 0x09, 0x00]); + assert_eq!( + bytes, + crafted_delta( + &["queue", "\u{e9}t\u{e9}", "\u{1f600}"], + &["", "cache", "db"] + ) + ); + + let decoded = DeltaResult::deserialize_from_bytes(&bytes).expect("decode"); + assert_eq!(decoded.added, delta.added); + assert_eq!(decoded.removed, delta.removed); + assert_eq!(decoded.serialize_to_bytes().expect("re-serialize"), bytes); + } + + #[test] + fn delta_result_empty_round_trips() { + let bytes = crafted_delta(&[], &[]); + let decoded = DeltaResult::deserialize_from_bytes(&bytes).expect("decode"); + assert!(decoded.added.is_empty() && decoded.removed.is_empty()); + assert_eq!(decoded.serialize_to_bytes().unwrap(), bytes); + } + + #[test] + fn delta_result_rejects_a_key_both_added_and_removed() { + assert!(DeltaResult::deserialize_from_bytes(&crafted_delta(&["a", "b"], &["b"])).is_err()); + let delta = DeltaResult { + added: set_of(&["a"]), + removed: set_of(&["a"]), + }; + assert!(delta.serialize_to_bytes().is_err()); + } + + #[test] + fn delta_result_rejects_unordered_sides() { + assert!(DeltaResult::deserialize_from_bytes(&crafted_delta(&["b", "a"], &[])).is_err()); + assert!(DeltaResult::deserialize_from_bytes(&crafted_delta(&[], &["c", "c"])).is_err()); + } + + #[test] + fn crafted_bytes_are_errors_not_panics() { + let set = SetAggregator { + values: set_of(&AWKWARD), + } + .serialize_to_bytes() + .unwrap(); + for cut in [0, 1, 6, 9, 14, set.len() - 1] { + assert!(SetAggregator::deserialize_from_bytes(&set[..cut]).is_err()); + } + assert!(SetAggregator::deserialize_from_bytes(&[0xff; 64]).is_err()); + assert!(DeltaResult::deserialize_from_bytes(&[0xff; 64]).is_err()); + let garbage = envelope::encode(DELTA_RESULT_KIND, &metadata_bytes().unwrap(), &[0xc1]); + assert!(DeltaResult::deserialize_from_bytes(&garbage).is_err()); + } +} diff --git a/tests/PBT_COVERAGE.md b/tests/PBT_COVERAGE.md index cddee44f..8bcc4e76 100644 --- a/tests/PBT_COVERAGE.md +++ b/tests/PBT_COVERAGE.md @@ -2,8 +2,8 @@ One `proptest` law per line. Each compares the sketch against an answer computed without it; what that answer is comes first in every section. One -file per sketch under `tests/pbt/`. `cargo test --test pbt` runs 367 laws; -with `--all-features`, 435, adding the four **experimental** modules. Every +file per sketch under `tests/pbt/`. `cargo test --test pbt` runs 369 laws; +with `--all-features`, 437, adding the four **experimental** modules. Every law was mutation-checked: break the implementation, the law goes red. Shared generators, `grid` and `round_trip!` live in `support.rs`. @@ -378,7 +378,7 @@ A worker holds counts back until one reaches `tau`, promotes that `tau` step, an - `max_hll_threshold(p) == 64 - p`; `HLL_PROMASK` below it at default precision; a worker at it holds nothing back - `MAX_PROMASK == i8::MAX`; `OctoThreshold` clamps 0 up to 1 and above down to `MAX_PROMASK`; every promotion fits an `i8` -## set_aggregator.rs — 9 +## set_aggregator.rs — 11 Exact. Compared against: a `HashSet` of the stream. Keys from `[a-c]{0,2}`, so streams repeat and intersect. @@ -393,7 +393,9 @@ Exact. Compared against: a `HashSet` of the stream. Keys from `[a-c]{0,2}`, so s - empty is a two-sided identity - `merge_refs` unions all inputs; refuses an empty list - wire - - msgpack keeps the key set, and again on a second round trip; keys include `""`, non-ASCII, embedded NUL, 300 chars, arbitrary `String` + - ASAPv1 keeps the key set and re-encodes to the same bytes; keys include `""`, non-ASCII, embedded NUL, 300 chars, arbitrary `String` + - the bytes of `S(xs)` equal the bytes of `S(perm(xs))` + - a `DeltaResult` of two such sets keeps `added` and `removed` and re-encodes to the same bytes ## space_saving.rs — 19 diff --git a/tests/asapv1_golden.rs b/tests/asapv1_golden.rs index 30b41651..e60d0c36 100644 --- a/tests/asapv1_golden.rs +++ b/tests/asapv1_golden.rs @@ -8,9 +8,11 @@ //! //! See `asapv1_golden/README.md`. +use std::collections::HashSet; + use asap_sketchlib::{ - Classic, Count, CountMin, ErtlMLE, FastPath, HllSketch, HllVariant, HyperLogLogHIPP12, - HyperLogLogP12, KLL, MessagePackCodec, RegularPath, Vector2D, + Classic, Count, CountMin, DeltaResult, ErtlMLE, FastPath, HllSketch, HllVariant, + HyperLogLogHIPP12, HyperLogLogP12, KLL, MessagePackCodec, RegularPath, SetAggregator, Vector2D, }; fn decode_hex(s: &str) -> Vec { @@ -32,6 +34,10 @@ const GOLDEN_CS_FAST: &str = include_str!("../asapv1_golden/cs_i64_fast_2x4.hex" const GOLDEN_CS_I32: &str = include_str!("../asapv1_golden/cs_i32_regular_2x4.hex"); const GOLDEN_KLL_F64: &str = include_str!("../asapv1_golden/kll_f64_k200.hex"); const GOLDEN_KLL_I64: &str = include_str!("../asapv1_golden/kll_i64_k200.hex"); +const GOLDEN_SET_STRINGS: &str = include_str!("../asapv1_golden/set_aggregator_strings.hex"); +const GOLDEN_SET_EMPTY: &str = include_str!("../asapv1_golden/set_aggregator_empty.hex"); +const GOLDEN_DELTA_STRINGS: &str = include_str!("../asapv1_golden/delta_result_strings.hex"); +const GOLDEN_DELTA_EMPTY: &str = include_str!("../asapv1_golden/delta_result_empty.hex"); /// The known P12 register pattern shared by all three HLL fixtures. fn p12_registers() -> Vec { @@ -287,3 +293,83 @@ fn kll_i64_k200_matches_golden() { assert_eq!(decoded.quantile(0.0), 1.0); assert_eq!(decoded.quantile(1.0), 50.0); } + +// --------------------------------------------------------------------------- +// SetAggregator / DeltaResult: string sets set directly -> serialize == +// golden, golden decodes to that state, and re-encode is byte-identical. +// --------------------------------------------------------------------------- + +fn string_set(keys: &[&str]) -> HashSet { + keys.iter().map(|k| k.to_string()).collect() +} + +/// The empty string, a 32-byte str8, and UTF-8 code points of two, three and +/// four bytes. U+FF5E precedes U+1F600 in byte order but not in UTF-16 order. +const SET_STRINGS: [&str; 8] = [ + "", + "abcdefghijklmnopqrstuvwxyz012345", + "api", + "web", + "\u{e9}", + "\u{4e2d}", + "\u{ff5e}", + "\u{1f600}", +]; + +fn check_set_golden(golden: &str, keys: &[&str]) { + let want = decode_hex(golden); + let agg = SetAggregator { + values: string_set(keys), + }; + assert_eq!( + agg.serialize_to_bytes().expect("serialize"), + want, + "SetAggregator bytes diverge from golden" + ); + + let decoded = SetAggregator::deserialize_from_bytes(&want).expect("decode"); + assert_eq!(decoded.values, agg.values); + assert_eq!(decoded.serialize_to_bytes().expect("re-serialize"), want); +} + +fn check_delta_golden(golden: &str, added: &[&str], removed: &[&str]) { + let want = decode_hex(golden); + let delta = DeltaResult { + added: string_set(added), + removed: string_set(removed), + }; + assert_eq!( + delta.serialize_to_bytes().expect("serialize"), + want, + "DeltaResult bytes diverge from golden" + ); + + let decoded = DeltaResult::deserialize_from_bytes(&want).expect("decode"); + assert_eq!(decoded.added, delta.added); + assert_eq!(decoded.removed, delta.removed); + assert_eq!(decoded.serialize_to_bytes().expect("re-serialize"), want); +} + +#[test] +fn set_aggregator_strings_matches_golden() { + check_set_golden(GOLDEN_SET_STRINGS, &SET_STRINGS); +} + +#[test] +fn set_aggregator_empty_matches_golden() { + check_set_golden(GOLDEN_SET_EMPTY, &[]); +} + +#[test] +fn delta_result_strings_matches_golden() { + check_delta_golden( + GOLDEN_DELTA_STRINGS, + &["queue", "\u{e9}t\u{e9}", "\u{1f600}"], + &["", "cache", "db", "\u{4e2d}"], + ); +} + +#[test] +fn delta_result_empty_matches_golden() { + check_delta_golden(GOLDEN_DELTA_EMPTY, &[], &[]); +} diff --git a/tests/e2e/cardinality.rs b/tests/e2e/cardinality.rs index bf0d9636..ceeaddc1 100644 --- a/tests/e2e/cardinality.rs +++ b/tests/e2e/cardinality.rs @@ -767,7 +767,7 @@ fn a_custom_precision_merge_reproduces_the_single_pass_registers_for_every_estim #[test] fn a_set_aggregator_delta_describes_the_change_and_survives_the_wire() { - use asap_sketchlib::{DeltaResult, MessagePackCodec}; + use asap_sketchlib::DeltaResult; use std::collections::HashSet; let mut before = SetAggregator::new(); @@ -797,8 +797,8 @@ fn a_set_aggregator_delta_describes_the_change_and_survives_the_wire() { "both departing keys are removed" ); - let bytes = delta.to_msgpack().expect("encode"); - let decoded = DeltaResult::from_msgpack(&bytes).expect("decode"); + let bytes = delta.serialize_to_bytes().expect("encode"); + let decoded = DeltaResult::deserialize_from_bytes(&bytes).expect("decode"); assert_eq!(decoded.added, added, "added set survived the wire"); assert_eq!(decoded.removed, removed, "removed set survived the wire"); @@ -818,8 +818,9 @@ fn a_set_aggregator_delta_describes_the_change_and_survives_the_wire() { added: HashSet::new(), removed: HashSet::new(), }; - let round_tripped = DeltaResult::from_msgpack(&empty.to_msgpack().expect("encode empty")) - .expect("decode empty"); + let round_tripped = + DeltaResult::deserialize_from_bytes(&empty.serialize_to_bytes().expect("encode empty")) + .expect("decode empty"); assert!( round_tripped.added.is_empty() && round_tripped.removed.is_empty(), "an empty delta must stay empty across the wire" diff --git a/tests/msgpack_compat.rs b/tests/msgpack_compat.rs index 267a51c6..69153c9c 100644 --- a/tests/msgpack_compat.rs +++ b/tests/msgpack_compat.rs @@ -19,20 +19,17 @@ //! `include_bytes!` and asserts deserialize succeeds and field values //! match the producer's expectations. -use std::collections::HashSet; - use asap_sketchlib::CmsHeapItem; use asap_sketchlib::message_pack_format::MessagePackCodec; use asap_sketchlib::message_pack_format::portable::countminsketch::CountMinSketchWire; use asap_sketchlib::message_pack_format::portable::countminsketch_topk::{ CountMinSketchInnerWire, CountMinSketchWithHeapWire, }; -use asap_sketchlib::message_pack_format::portable::delta_set_aggregator::DeltaResult; use asap_sketchlib::message_pack_format::portable::hydra_kll::HydraKllSketchWire; use asap_sketchlib::message_pack_format::portable::kll::KllSketchData; use asap_sketchlib::{ CountMinSketch, CountMinSketchWithHeap, CountSketch, DdSketch, HllSketch, HllVariant, - HydraKllSketch, KllSketch, SetAggregator, + HydraKllSketch, KllSketch, }; // ===== round-trip: every wire-format-aligned type ===== @@ -119,30 +116,6 @@ fn hydra_kll_sketch_round_trip() { assert_eq!(restored.cols, 4); } -#[test] -fn set_aggregator_round_trip() { - let mut s = SetAggregator::new(); - s.update("web"); - s.update("api"); - let bytes = s.to_msgpack().expect("encode"); - let restored = SetAggregator::from_msgpack(&bytes).expect("decode"); - assert_eq!(restored.values.len(), 2); - assert!(restored.values.contains("web")); -} - -#[test] -fn delta_result_round_trip() { - let mut added = HashSet::new(); - added.insert("a".to_string()); - let mut removed = HashSet::new(); - removed.insert("b".to_string()); - let dr = DeltaResult { added, removed }; - let bytes = dr.to_msgpack().expect("encode"); - let restored = DeltaResult::from_msgpack(&bytes).expect("decode"); - assert!(restored.added.contains("a")); - assert!(restored.removed.contains("b")); -} - // ===== DTO-level structural sanity ===== // // Verify that the DTO field shapes still match what `sketchlib-go` diff --git a/tests/pbt/set_aggregator.rs b/tests/pbt/set_aggregator.rs index ebec4f23..fc3891b7 100644 --- a/tests/pbt/set_aggregator.rs +++ b/tests/pbt/set_aggregator.rs @@ -3,8 +3,7 @@ //! There is no error band here, so every law is an exact equality. The oracle //! is a `std::collections::HashSet` built from the stream on the test side. -use asap_sketchlib::SetAggregator; -use asap_sketchlib::message_pack_format::MessagePackCodec; +use asap_sketchlib::{DeltaResult, SetAggregator}; use proptest::prelude::*; use std::collections::HashSet; @@ -170,18 +169,45 @@ proptest! { // ===== Wire ===== #[test] - fn a_msgpack_round_trip_preserves_the_key_set(stream in awkward_keys(120)) { + fn an_asapv1_round_trip_preserves_the_key_set(stream in awkward_keys(120)) { let s = aggregator_of(&stream); - let bytes = s.to_msgpack().expect("encode"); - let restored = SetAggregator::from_msgpack(&bytes).expect("decode"); + let bytes = s.serialize_to_bytes().expect("encode"); + let restored = SetAggregator::deserialize_from_bytes(&bytes).expect("decode"); prop_assert_eq!(&restored.values, &s.values, "round trip moved the set"); prop_assert_eq!(restored.values.len(), s.values.len()); + prop_assert_eq!( + restored.serialize_to_bytes().expect("re-encode"), + bytes, + "a re-encode changed the bytes" + ); + } + + #[test] + fn a_delta_between_two_snapshots_round_trips_byte_identically( + before in awkward_keys(80), + after in awkward_keys(80), + ) { + let (b, a) = (oracle(&before), oracle(&after)); + let delta = DeltaResult { + added: a.difference(&b).cloned().collect(), + removed: b.difference(&a).cloned().collect(), + }; + + let bytes = delta.serialize_to_bytes().expect("encode"); + let restored = DeltaResult::deserialize_from_bytes(&bytes).expect("decode"); + + prop_assert_eq!(&restored.added, &delta.added, "round trip moved added"); + prop_assert_eq!(&restored.removed, &delta.removed, "round trip moved removed"); + prop_assert_eq!(restored.serialize_to_bytes().expect("re-encode"), bytes); + } - let twice = SetAggregator::from_msgpack( - &restored.to_msgpack().expect("re-encode") - ).expect("decode a re-encode"); - prop_assert_eq!(&twice.values, &s.values, "the second round trip moved the set"); + #[test] + fn the_encoding_depends_on_the_key_set_alone((a, b) in stream_and_permutation(200)) { + prop_assert_eq!( + aggregator_of(&a).serialize_to_bytes().expect("encode"), + aggregator_of(&b).serialize_to_bytes().expect("encode") + ); } } From 0b7613a63ce81f34b12e4145dbd20eadc46bc0c9 Mon Sep 17 00:00:00 2001 From: GordonYuanyc Date: Fri, 2 Oct 2026 01:26:53 -0600 Subject: [PATCH 28/36] docs(set_aggregator): number DeltaResult's decode rules, require UTF-8, widen the string fixture MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adds SetAggregator and DeltaResult to the Q-NOHASH and Q-ORDER decisions, requires every member to be valid UTF-8, states the encoder's one encoding per set, and numbers §3.22's rules in full. The DeltaResult doc states the disjointness serializing enforces, and read_canonical checks neighbours before collecting. set_aggregator_strings gains a 31-byte fixstr and a proper-prefix chain; asapv1_golden is bumped. Co-Authored-By: Claude Opus 5.5 --- asapv1_golden | 2 +- docs/asapv1_wire_format.md | 19 ++++++++++++------- docs/tests.md | 2 +- src/sketches/set_aggregator.rs | 3 ++- src/sketches/set_aggregator/wire.rs | 20 ++++++-------------- tests/asapv1_golden.rs | 11 ++++++++--- 6 files changed, 30 insertions(+), 27 deletions(-) diff --git a/asapv1_golden b/asapv1_golden index 6e31b115..9114c14e 160000 --- a/asapv1_golden +++ b/asapv1_golden @@ -1 +1 @@ -Subproject commit 6e31b11560d03b8288e13e9d7d496c8d1408e10a +Subproject commit 9114c14e1e2fe144de69dc7eb60c15e80d8e566e diff --git a/docs/asapv1_wire_format.md b/docs/asapv1_wire_format.md index a3994621..54e1e4b3 100644 --- a/docs/asapv1_wire_format.md +++ b/docs/asapv1_wire_format.md @@ -1236,14 +1236,15 @@ The encode side refuses the same states: a level count that disagrees with the c | ----- | ------- | ------ | ------- | | 0 | `values` | array | the members, `str`; strictly ascending | -**Emitted order (cross-language contract).** Members are written in ascending **byte order of their UTF-8 encoding**, which is code point order and not UTF-16 order: `"～"` (U+FF5E) precedes `"😀"` (U+1F600). A set has exactly one encoding, so a decoded set re-serializes byte-identically, and an empty set is `[[]]`. +**Emitted order (cross-language contract).** Members are written in ascending **byte order of their UTF-8 encoding**, which is code point order and not UTF-16 order: `"～"` (U+FF5E) precedes `"😀"` (U+1F600). The encoder emits exactly one encoding per set, and an empty set is `[[]]`. **Decode rules.** Fail **closed** on each, with an error and never a panic: 1. `kind_id` is `0x08 0x00`. 2. The metadata map is exactly `{metadata_version: 1}`. 3. Every element of `values` is msgpack `str`; a `bin` is rejected even when it holds valid UTF-8. -4. `values` is strictly ascending in byte order. This rejects a duplicate as well as a reordering. +4. Every member is valid UTF-8. An encoder refuses a member that is not. +5. `values` is strictly ascending in byte order. This rejects a duplicate as well as a reordering. ### 3.22: DeltaResult payload (`0x09 0x00`) @@ -1256,12 +1257,16 @@ The encode side refuses the same states: a level count that disagrees with the c Both arrays follow §3.21's emitted order. An empty delta is `[[], []]`. -**Decode rules.** Rules 2 to 4 of §3.21 apply to both arrays, and: +**Decode rules.** Fail **closed** on each, with an error and never a panic: 1. `kind_id` is `0x09 0x00`. -2. No key is in both `added` and `removed`. +2. The metadata map is exactly `{metadata_version: 1}`. +3. Every element of `added` and `removed` is msgpack `str`; a `bin` is rejected even when it holds valid UTF-8. +4. Every member is valid UTF-8. An encoder refuses a member that is not. +5. `added` and `removed` are each strictly ascending in byte order. +6. No key is in both `added` and `removed`. -Rule 2 is enforced on the **encode** side too: a `DeltaResult` holding a key in both sets fails to serialize. +Rule 6 is enforced on the **encode** side too: a `DeltaResult` holding a key in both sets fails to serialize. ### 3.23: payloads not yet designed @@ -1402,9 +1407,9 @@ Retire `portable` once the fixtures cover the rest. - **Q-NEST-HET**: a **heterogeneous** nested sketch — one whose algorithm is data rather than a type — carries the variant's own registry `kind_id` plus its metadata and payload blocks with the framing stripped (§3.18). No name string, no second identity namespace beside the Section 1 registry, and no per-variant encoding to specify: the blocks are byte for byte what that variant's own section fixes, and the variant's own decoder runs on them unchanged, so its geometry checks, allocation guards and hash-profile pin all apply inside the wrapper. `EHSketchList` (`0x14 0x00`) is the triple, and `ExponentialHistogram` (`0x13 0x00`) inlines one per bucket. - **Q-NEST-GATE**: the nested-id namespace is **fixed and identical in every build**. An id is registry bytes, never an enum ordinal, so no id's meaning shifts because a variant is compiled out. A decoder that meets an id its build does not carry fails **closed**, before the blocks are assembled, with an error naming the variant and the feature; it never misparses, skips, substitutes, or falls through the unknown-id path. - **Q-HYDRA-IDS**: Hydra allocates **one `kind_id` per counter base sketch** (`0x07 0x00` KLL, `0x07 0x01` Count-Min, `0x07 0x02` Count Sketch, `0x07 0x03` HyperLogLog, `0x07 0x04` UnivMon), so the payload carries no variant tag and a grid mixing counter variants has no encoding. `0x07 0x05`-`0x07 0xff` stay reserved for Hydra over further base sketches. The five ids use four metadata schemas; Count-Min and Count Sketch share one, as the two KLL ids share one. -- **Q-NOHASH**: a sketch that never hashes its inputs omits the **hash-spec group entirely** — the group answers "how were keys hashed", and there is no truthful answer. This is Q-KLL generalized: KLL (§3.3), DDSketch (§3.8) and UniformSampling (§3.13) never hash, and `ExponentialHistogram` (§3.17) and `EHSketchList` (§3.18) do not hash either, since the sketch inside them does. Their metadata is structural-only and is not `HashProfile`-derived. A sketch that *does* hash carries the group, derived live from the hasher's profile, even when its counters do not hash (Hydra's KLL and UnivMon variants, §3.9). +- **Q-NOHASH**: a sketch that never hashes its inputs omits the **hash-spec group entirely** — the group answers "how were keys hashed", and there is no truthful answer. This is Q-KLL generalized: KLL (§3.3), DDSketch (§3.8) and UniformSampling (§3.13) never hash, `ExponentialHistogram` (§3.17) and `EHSketchList` (§3.18) do not hash either, since the sketch inside them does, and `SetAggregator` (§3.21) and `DeltaResult` (§3.22) carry their keys verbatim. Their metadata is structural-only and is not `HashProfile`-derived. A sketch that *does* hash carries the group, derived live from the hasher's profile, even when its counters do not hash (Hydra's KLL and UnivMon variants, §3.9). - **Q-SEEDIDX**: a **seed-index key is carried only for a hash whose index the sketch reads off the profile.** `HashProfile` declares two such constants, `CANONICAL_SEED_INDEX` and `MATRIX_SEED_INDEX`, and the metadata carries `canonical_seed_index` or `matrix_seed_index` accordingly. An index the algorithm fixes gets no key: Space-Saving's `0` (§3.5), Coco's per-array `i` (§3.12), Elastic's heavy table (§3.11), Hydra's subkey fan-out at `HYDRA_SEED` (§3.9) and the UnivMon family's `BOTTOM_LAYER_FINDER` (§3.15). A per-instance seed a caller chose is a **structural param**, not a hash-spec field: `seed_index` on CountL2HH (§3.19) and UnivMon-Q (§3.20) is echoed back on decode rather than pinned against the target. -- **Q-ORDER**: a payload whose in-memory container order does not survive a rebuild **pins an emitted order**, as a cross-language contract. Heaps, arenas and hash maps all have this property, so the rule reaches Space-Saving's triples (§3.5), the top-k heap entries (§3.7), KMV's digests (§3.14), UniformSampling's entries (§3.13), UnivMon-Q's candidates and occurrences (§3.20), and every layer- and bucket-indexed array. Two sketches holding the same state then emit the same bytes whatever order they were built in, and a decoded sketch re-serializes byte-identically. The heap key comparator is the same everywhere: descending count, ties by variant tag then value. +- **Q-ORDER**: a payload whose in-memory container order does not survive a rebuild **pins an emitted order**, as a cross-language contract. Heaps, arenas and hash maps all have this property, so the rule reaches Space-Saving's triples (§3.5), the top-k heap entries (§3.7), KMV's digests (§3.14), UniformSampling's entries (§3.13), UnivMon-Q's candidates and occurrences (§3.20), SetAggregator's and DeltaResult's string sets (§3.21, §3.22), and every layer- and bucket-indexed array. Two sketches holding the same state then emit the same bytes whatever order they were built in, and a decoded sketch re-serializes byte-identically. The heap key comparator is the same everywhere: descending count, ties by variant tag then value. - **Q-CAP**: **a declared capacity never sizes an allocation.** Every structure is built from the run the payload actually carries, and the declared bound is checked against that run first; a payload naming a capacity of `2^32 - 1` with two entries costs two entries. This holds for Space-Saving's `capacity`, the top-k heaps' `k`, KMV's `k`, UniformSampling's `total_seen`, UnivMon's `heap_size`, UnivMon-Q's `candidates` and `ordered_samples`, and Hydra's grid and counter dimensions. - **Q-KEYTYPE**: `key_type` names the **exact `HeapItem` variant** and is never widened, wherever heap keys reach the wire — Space-Saving (§3.5), CMSHeap and CSHeap (§3.7, §3.10), the UnivMon family (§3.15, §3.16) and Hydra's UnivMon counter (§3.9). A container whose keys mix variants has no single name and **fails to serialize**; `HeapItem::I128` / `U128` have no msgpack integer form and are not wire types. An empty container emits `"u64"`, so its `key_type` has one value rather than one per producer. That canonicalizes the key naming and nothing else: a UnivMon Optimized pyramid still has more than one encoding per state, since layouts differing only in an `elephant_layers` at or above `layer_size` build identical layers, and an all-elephant pyramid carries its unused mouse dimensions on the wire regardless (§3.16). What holds everywhere is the round trip: a decoded container re-serializes byte-identically. - **Q-DERIVED**: state a decoder can recompute exactly is **not carried**, and state it cannot is. Recomputed: DDSketch's `count` from its buckets (§3.8), `ExponentialHistogram`'s `l2_mass` and `merge_norm` (§3.17), Elastic's `bktlen` from the declared bucket count (§3.11), every heap's digest index and array order, and every summary a sketch answers from its own state. Carried: DDSketch's `sum` / `min` / `max`, Elastic's `stale_copies`, UnivMon's `update_mode`, `candidate_complete` and `bucket_size`, UnivMon-Q's `ever_evicted`, `source_id` and `next_sequence`, UniformSampling's `priorities`, `total_seen` and `rng_state`, and CountL2HH's `l2`. Where a cached derived field exists in memory, the encoder rejects a value that disagrees with what the decoder would recompute. diff --git a/docs/tests.md b/docs/tests.md index 856fd832..23eed419 100644 --- a/docs/tests.md +++ b/docs/tests.md @@ -2137,7 +2137,7 @@ Test file: [`tests/asapv1_golden.rs`](../tests/asapv1_golden.rs) | `kll_f64_k200_matches_golden` | A deterministic f64 KLL serializes to its golden bytes, which round-trip unchanged. | Verifies `KLL::::init_kll_with_seed(200, 42)` fed `1..=50` (below the level-0 capacity, so no compaction fires) serializes to `asapv1_golden/kll_f64_k200.hex`, and that decoding the fixture re-serializes to the same bytes and reports `quantile(0.0)` `1.0` and `quantile(1.0)` `50.0`. | | `kll_i64_k200_matches_golden` | A deterministic i64 KLL serializes to its golden bytes, which round-trip unchanged. | Verifies `KLL::::init_kll_with_seed(200, 42)` fed `1..=50` serializes to `asapv1_golden/kll_i64_k200.hex`, and that decoding the fixture re-serializes to the same bytes and reports `quantile(0.0)` `1.0` and `quantile(1.0)` `50.0`. | | `set_aggregator_empty_matches_golden` | An empty set serializes to its golden bytes. | Verifies `SetAggregator` with no members serializes to `asapv1_golden/set_aggregator_empty.hex`, and that decoding the fixture gives an empty set and a byte-identical re-serialization. | -| `set_aggregator_strings_matches_golden` | A known string set serializes to its golden bytes in UTF-8 byte order. | Verifies the set `{"", "abcdefghijklmnopqrstuvwxyz012345", "api", "web", "é", "中", "～", "😀"}` — the empty string, a 32-byte str8, and code points of two, three and four bytes — serializes to `asapv1_golden/set_aggregator_strings.hex`, and that decoding the fixture recovers the set and re-serializes to the same bytes. | +| `set_aggregator_strings_matches_golden` | A known string set serializes to its golden bytes in UTF-8 byte order. | Verifies the set `{"", "ab", "abc", "abcdefghijklmnopqrstuvwxyz012345", "api", "fixstr-max-31-bytes-0123456789a", "web", "é", "中", "～", "😀"}` — the empty string, a proper-prefix chain, a 31-byte fixstr and a 32-byte str8, and code points of two, three and four bytes — serializes to `asapv1_golden/set_aggregator_strings.hex`, and that decoding the fixture recovers the set and re-serializes to the same bytes. | ### MessagePack Envelope Compatibility diff --git a/src/sketches/set_aggregator.rs b/src/sketches/set_aggregator.rs index 5c51c17d..fddd6567 100644 --- a/src/sketches/set_aggregator.rs +++ b/src/sketches/set_aggregator.rs @@ -60,7 +60,8 @@ impl Default for SetAggregator { } /// The keys a later set snapshot added and removed relative to an earlier -/// one. The two sets are disjoint. +/// one. The two sets must be disjoint; `serialize_to_bytes` refuses a key +/// held in both. #[derive(Debug, Clone)] pub struct DeltaResult { pub added: HashSet, diff --git a/src/sketches/set_aggregator/wire.rs b/src/sketches/set_aggregator/wire.rs index e53f32c2..f6cd4d4e 100644 --- a/src/sketches/set_aggregator/wire.rs +++ b/src/sketches/set_aggregator/wire.rs @@ -63,21 +63,13 @@ fn canonical(set: &HashSet) -> Vec<&str> { /// Rebuilds a set from a wire array, rejecting any array that is not strictly /// ascending. The set reserves from the array actually shipped. fn read_canonical(field: &str, keys: Vec) -> Result, RmpDecodeError> { - let mut set = HashSet::with_capacity(keys.len()); - let mut previous: Option = None; - for (idx, key) in keys.into_iter().enumerate() { - let key = key.into_string(); - if let Some(prev) = &previous - && prev.as_str() >= key.as_str() - { - return Err(RmpDecodeError::Uncategorized(format!( - "ASAPv1 {field} must be strictly ascending in byte order, broken at index {idx}" - ))); - } - set.insert(key.clone()); - previous = Some(key); + if let Some(idx) = keys.windows(2).position(|pair| pair[0].0 >= pair[1].0) { + return Err(RmpDecodeError::Uncategorized(format!( + "ASAPv1 {field} must be strictly ascending in byte order, broken at index {}", + idx + 1 + ))); } - Ok(set) + Ok(keys.into_iter().map(WireString::into_string).collect()) } /// Splits the envelope, checks the kind_id and the metadata, and returns the diff --git a/tests/asapv1_golden.rs b/tests/asapv1_golden.rs index e60d0c36..bca93cd2 100644 --- a/tests/asapv1_golden.rs +++ b/tests/asapv1_golden.rs @@ -303,12 +303,17 @@ fn string_set(keys: &[&str]) -> HashSet { keys.iter().map(|k| k.to_string()).collect() } -/// The empty string, a 32-byte str8, and UTF-8 code points of two, three and -/// four bytes. U+FF5E precedes U+1F600 in byte order but not in UTF-16 order. -const SET_STRINGS: [&str; 8] = [ +/// The empty string, a proper-prefix chain (`"ab"`, `"abc"`, the 32-byte +/// key), a 31-byte fixstr and a 32-byte str8, and UTF-8 code points of two, +/// three and four bytes. U+FF5E precedes U+1F600 in byte order but not in +/// UTF-16 order. +const SET_STRINGS: [&str; 11] = [ "", + "ab", + "abc", "abcdefghijklmnopqrstuvwxyz012345", "api", + "fixstr-max-31-bytes-0123456789a", "web", "\u{e9}", "\u{4e2d}", From 4df805da1d661a23525a786d6d540a45379055d9 Mon Sep 17 00:00:00 2001 From: GordonYuanyc Date: Fri, 2 Oct 2026 01:49:19 -0600 Subject: [PATCH 29/36] fix: reconcile the merged ASAPv1-only branches Drop what no merged branch still uses: the empty portable probe in accuracy_probe, the unused MessagePackCodec import in e2e/composition, the removed sketch_state field in the sampling test, and the docs/tests.md section for the deleted parity probe. Share one envelope splitter between the KLL and UnivMon golden checks, and recount the property-test totals. Co-Authored-By: Claude Opus 5.5 --- docs/tests.md | 7 ---- examples/accuracy_probe.rs | 9 ----- src/message_pack_format/portable/sampling.rs | 1 - tests/PBT_COVERAGE.md | 4 +-- tests/asapv1_golden.rs | 37 ++++++++++---------- tests/e2e/composition.rs | 4 +-- 6 files changed, 22 insertions(+), 40 deletions(-) diff --git a/docs/tests.md b/docs/tests.md index 4d011328..096bb96c 100644 --- a/docs/tests.md +++ b/docs/tests.md @@ -2013,13 +2013,6 @@ Test file: [`tests/spec_self_tests.rs`](../tests/spec_self_tests.rs) | `simultaneous_kappa_reflects_the_corrected_threshold` | The simultaneous `kappa` search reaches its target and is wider than a `d/2 + 1` threshold demands. | For `CountSketchSpec::new(4, 2048)`, verifies `simultaneous_kappa(512, SIMULTANEOUS_LEVEL)` at a level of `1e-3` yields a `kappa` whose `key_failure_at(kappa)` is at most `1e-3 / 512`, and that this `kappa` is strictly larger than the one a geometric-then-bisection search over `binomial_tail_ge(4, 3, 1.0 / kappa)` finds for the same target. | | `two_same_direction_bad_rows_move_an_averaged_four_row_median_out_of_band` | Two same-direction bad rows break a four-row averaged median, and one does not. | With an inline median-of-four `(v[1] + v[2]) / 2`, a true value of `100.0` and an error scale of `10.0` (band `[90, 110]`), verifies `[95, 105, 100, 10_000]` stays in band, `[95, 105, 10_000, 10_000]` leaves it, and `[95, 105, -10_000, 10_000]` stays in band, asserting the bad-row count of each fixture first. | -### Proto Envelope Parity Probe - -Test file: [`tests/sketches_go_parity_probe.rs`](../tests/sketches_go_parity_probe.rs) - -| test_name | test_description | what_is_tested | -| --- | --- | --- | - ### HLL Custom Precision Test file: [`tests/hll_custom_precision.rs`](../tests/hll_custom_precision.rs) diff --git a/examples/accuracy_probe.rs b/examples/accuracy_probe.rs index 0fc54024..275332cd 100644 --- a/examples/accuracy_probe.rs +++ b/examples/accuracy_probe.rs @@ -114,7 +114,6 @@ fn main() { #[cfg(feature = "experimental")] probe_eh_univ(); probe_tumbling(); - probe_portable_wire_types(); println!("===================================================================="); println!("Probe complete. FAIL lines above are candidate wrong-query-results."); @@ -989,11 +988,3 @@ fn probe_tumbling() { ); p.finish("Tumbling"); } - -// ---------------------------------------------------- Portable wire types -fn probe_portable_wire_types() { - let mut p = Probe::new(); - println!("--- Portable wire types (Go-parity DTOs) ---"); - - p.finish("Portable wire types"); -} diff --git a/src/message_pack_format/portable/sampling.rs b/src/message_pack_format/portable/sampling.rs index e6951d18..5265d243 100644 --- a/src/message_pack_format/portable/sampling.rs +++ b/src/message_pack_format/portable/sampling.rs @@ -78,7 +78,6 @@ mod tests { producer: None, hash_spec: None, sample_p: p, - sketch_state: None, } } diff --git a/tests/PBT_COVERAGE.md b/tests/PBT_COVERAGE.md index 7aa9706d..ad07dfa6 100644 --- a/tests/PBT_COVERAGE.md +++ b/tests/PBT_COVERAGE.md @@ -2,8 +2,8 @@ One `proptest` law per line. Each compares the sketch against an answer computed without it; what that answer is comes first in every section. One -file per sketch under `tests/pbt/`. `cargo test --test pbt` runs 369 laws; -with `--all-features`, 437, adding the four **experimental** modules. Every +file per sketch under `tests/pbt/`. `cargo test --test pbt` runs 364 laws; +with `--all-features`, 432, adding the four **experimental** modules. Every law was mutation-checked: break the implementation, the law goes red. Shared generators, `grid` and `round_trip!` live in `support.rs`. diff --git a/tests/asapv1_golden.rs b/tests/asapv1_golden.rs index bd87ac41..3b0fa3f0 100644 --- a/tests/asapv1_golden.rs +++ b/tests/asapv1_golden.rs @@ -30,6 +30,20 @@ fn decode_hex(s: &str) -> Vec { .collect() } +/// Splits an ASAPv1 envelope by hand into its kind_id, metadata and payload. +fn split_envelope(bytes: &[u8]) -> (Vec, &[u8], &[u8]) { + assert_eq!(&bytes[..7], b"ASAPv1\x01", "magic and version"); + let kind_len = bytes[7] as usize; + let kind_id = bytes[8..8 + kind_len].to_vec(); + let at = 8 + kind_len; + let meta_len = u32::from_be_bytes(bytes[at..at + 4].try_into().unwrap()) as usize; + let payload_len = u32::from_be_bytes(bytes[at + 4..at + 8].try_into().unwrap()) as usize; + let metadata = &bytes[at + 8..at + 8 + meta_len]; + let payload = &bytes[at + 8 + meta_len..]; + assert_eq!(payload.len(), payload_len, "payload length prefix"); + (kind_id, metadata, payload) +} + const GOLDEN_CLASSIC: &str = include_str!("../asapv1_golden/hll_classic_p12.hex"); const GOLDEN_ERTL: &str = include_str!("../asapv1_golden/hll_ertl_mle_p12.hex"); const GOLDEN_HIP: &str = include_str!("../asapv1_golden/hll_hip_p12.hex"); @@ -576,16 +590,9 @@ const KLL_COIN_SEED_42: (u64, u64, u32) = (42, 0, 0); /// A KLL payload `[levels, items, coin]`, read straight from the envelope. type KllPayloadView = (Vec, Vec, (u64, u64, u32)); -/// Splits an ASAPv1 envelope by hand and decodes its KLL payload. +/// Decodes the KLL payload of an ASAPv1 envelope. fn kll_envelope(bytes: &[u8]) -> (Vec, KllPayloadView) { - assert_eq!(&bytes[..7], b"ASAPv1\x01", "magic and version"); - let kind_len = bytes[7] as usize; - let kind_id = bytes[8..8 + kind_len].to_vec(); - let at = 8 + kind_len; - let meta_len = u32::from_be_bytes(bytes[at..at + 4].try_into().unwrap()) as usize; - let payload_len = u32::from_be_bytes(bytes[at + 4..at + 8].try_into().unwrap()) as usize; - let payload = &bytes[at + 8 + meta_len..]; - assert_eq!(payload.len(), payload_len, "payload length prefix"); + let (kind_id, _, payload) = split_envelope(bytes); ( kind_id, rmp_serde::from_slice(payload).expect("KLL payload"), @@ -1183,19 +1190,11 @@ struct UnivMonShape { key_type: String, } -/// Splits an ASAPv1 envelope by hand and decodes its UnivMon metadata and payload. +/// Decodes the UnivMon metadata and payload of an ASAPv1 envelope. fn univmon_envelope( bytes: &[u8], ) -> (Vec, UnivMonShape, UnivMonPayloadView) { - assert_eq!(&bytes[..7], b"ASAPv1\x01", "magic and version"); - let kind_len = bytes[7] as usize; - let kind_id = bytes[8..8 + kind_len].to_vec(); - let at = 8 + kind_len; - let meta_len = u32::from_be_bytes(bytes[at..at + 4].try_into().unwrap()) as usize; - let payload_len = u32::from_be_bytes(bytes[at + 4..at + 8].try_into().unwrap()) as usize; - let metadata = &bytes[at + 8..at + 8 + meta_len]; - let payload = &bytes[at + 8 + meta_len..]; - assert_eq!(payload.len(), payload_len, "payload length prefix"); + let (kind_id, metadata, payload) = split_envelope(bytes); ( kind_id, rmp_serde::from_slice(metadata).expect("UnivMon metadata"), diff --git a/tests/e2e/composition.rs b/tests/e2e/composition.rs index 9949dbe7..3f7f321c 100644 --- a/tests/e2e/composition.rs +++ b/tests/e2e/composition.rs @@ -17,8 +17,8 @@ use common::{FreqTruth, NumericTruth, uniform_u64, zipf_u64}; use asap_sketchlib::{ CMSHeap, Classic, Count, CountMin, DataInput, EnsembleSketch, ErtlMLE, FastPath, - HashSketchEnsemble, HeapItem, HyperLogLog, HyperLogLogHIP, KLL, MessagePackCodec, RegularPath, - UnivMonQ, UnivMonQConfig, Vector2D, + HashSketchEnsemble, HeapItem, HyperLogLog, HyperLogLogHIP, KLL, RegularPath, UnivMonQ, + UnivMonQConfig, Vector2D, }; const ROWS: usize = 3; From efe6c284ce842a509fcecacbc86c62de7d24272d Mon Sep 17 00:00:00 2001 From: GordonYuanyc Date: Fri, 2 Oct 2026 01:51:02 -0600 Subject: [PATCH 30/36] refactor!: drop MessagePackCodec and the portable and native modules Every sketch serializes through serialize_to_bytes / deserialize_from_bytes, so the codec trait, its error type, the last native shim (KMV) and the portable sampling helpers have no ASAPv1 role left. Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 9 ++ docs/tests.md | 16 --- src/lib.rs | 5 - src/message_pack_format/codec.rs | 21 ---- src/message_pack_format/error.rs | 45 ------- src/message_pack_format/mod.rs | 15 --- src/message_pack_format/native/kmv.rs | 19 --- src/message_pack_format/native/mod.rs | 11 -- src/message_pack_format/portable/mod.rs | 7 -- src/message_pack_format/portable/sampling.rs | 123 ------------------- 10 files changed, 9 insertions(+), 262 deletions(-) delete mode 100644 src/message_pack_format/codec.rs delete mode 100644 src/message_pack_format/error.rs delete mode 100644 src/message_pack_format/native/kmv.rs delete mode 100644 src/message_pack_format/native/mod.rs delete mode 100644 src/message_pack_format/portable/mod.rs delete mode 100644 src/message_pack_format/portable/sampling.rs diff --git a/CHANGELOG.md b/CHANGELOG.md index 23b0fc48..3b2d8f07 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -266,6 +266,15 @@ signals a backwards-compatible change. `message_pack_format::portable::delta_set_aggregator`, both types' `MessagePackCodec` impls, and `DeltaResult`'s `serde::Serialize` / `Deserialize` derives. They serialize only as ASAPv1. +- **`MessagePackCodec` and the `portable` and `native` modules.** Removed + `message_pack_format::{MessagePackCodec, Error}` and the crate-root + `MessagePackCodec`; `message_pack_format::native`, with the `MessagePackCodec` + impl for `KMV`; and `message_pack_format::portable`, with `sampling` and its + crate-root re-exports `effective_sample_p`, `sample_p_or_default`, + `rescale_count`, `rescale_count_with_env` and `is_quantile_scale_invariant`. + Every sketch encodes through `serialize_to_bytes` / `deserialize_from_bytes`; + ASAPv1 carries no sampling probability. + ### Fixed - `KLLDynamic` panicked on every quantile query once a NaN had entered the diff --git a/docs/tests.md b/docs/tests.md index 096bb96c..678fa955 100644 --- a/docs/tests.md +++ b/docs/tests.md @@ -1237,22 +1237,6 @@ Test file: [`src/sketch_framework/tumbling.rs`](../src/sketch_framework/tumbling | `tumbling_very_small_windows` | Five hundred rotations under a fifty-window cap still answer the retained tail. | For `TumblingWindow::new(10, 50, cfg, 52)` over `FoldCMSConfig { rows: 3, full_cols: 1024, fold_level: 3, top_k: 10 }` fed a seeded (`0x77_1100`) Zipf(`domain = 500`, `exponent = 1.2`) stream of `5000` samples at timestamps `i` - `500` windows, of which only the newest `50` closed plus the active survive - verifies over `85%` of the truth built from samples `4500..5000` is within `(e / 1024) * L1` under `query_all`. | | `tumbling_skewed_load` | A window carrying most of the stream does not break the Count-Min bound. | For `TumblingWindow::new(10_000, 5, cfg, 7)` over `FoldCMSConfig { rows: 3, full_cols: 4096, fold_level: 4, top_k: 20 }` fed five seeded (`0xBE_EF00 + w`) Zipf(`domain = 5000`, `exponent = 1.1`) phases - `1_000` samples each except `36_000` for phase `2`, each phase followed by a jump to the next window boundary - verifies over `90%` of keys are within `(e / 4096) * L1`, the truth covering all `40_000` samples including the two light windows the `max_windows` cap has already evicted. | -## MessagePack Portable Wire Types - - -### Portable Sampling Rescale - -Test file: [`src/message_pack_format/portable/sampling.rs`](../src/message_pack_format/portable/sampling.rs) - -| test_name | test_description | what_is_tested | -| --- | --- | --- | -| `dual_read_zero_means_one` | An unset `sample_p` reads as no sampling. | Verifies both `effective_sample_p` over an envelope carrying `sample_p` `0.0` and `sample_p_or_default(0.0)` return `1.0`. | -| `out_of_range_falls_back_to_one` | A malformed probability falls back to no sampling. | Verifies `sample_p_or_default` returns `1.0` for `-0.5`, for `1.5`, and for `f64::NAN`. | -| `valid_p_passes_through` | A probability inside `(0, 1]` is used as given. | Verifies `effective_sample_p` over an envelope carrying `sample_p` `0.1` returns `0.1`, and that `sample_p_or_default(1.0)` returns `1.0`. | -| `rescale_count_inverts_probability` | A count-like estimate is rescaled by `1/p`. | Verifies `rescale_count(10_000.0, 0.1)` is `100_000.0` within `1e-6`, that `rescale_count(123.0, 1.0)` is `123.0`, and that a `p` of `0.0` returns `123.0` rather than dividing by zero. | -| `rescale_with_env` | The envelope's own `sample_p` drives the rescale. | Verifies `rescale_count_with_env(5_000.0, env)` at `sample_p` `0.05` is `100_000.0` within `1e-6`, and that at `sample_p` `0.0` an input of `42.0` comes back unchanged. | -| `quantiles_are_scale_invariant` | The quantile family is declared to need no rescale. | Asserts `is_quantile_scale_invariant()`, a `const fn` returning `true`; nothing about a quantile estimate is computed. | - ## Common ### Common Hash Utilities diff --git a/src/lib.rs b/src/lib.rs index 2e41a0e2..b1c13ed9 100644 --- a/src/lib.rs +++ b/src/lib.rs @@ -37,10 +37,5 @@ pub mod __private { } pub use common::*; -pub use message_pack_format::MessagePackCodec; -pub use message_pack_format::portable::sampling::{ - effective_sample_p, is_quantile_scale_invariant, rescale_count, rescale_count_with_env, - sample_p_or_default, -}; pub use sketch_framework::*; pub use sketches::*; diff --git a/src/message_pack_format/codec.rs b/src/message_pack_format/codec.rs deleted file mode 100644 index 8ab7de32..00000000 --- a/src/message_pack_format/codec.rs +++ /dev/null @@ -1,21 +0,0 @@ -//! `MessagePackCodec` trait — unified encode/decode contract for the -//! wire-format-aligned types in [`crate::message_pack_format::portable`]. - -use super::Error; - -/// Round-trippable MessagePack codec. -/// -/// All [`crate::message_pack_format::portable`] sketch / aggregator types implement this trait; -/// it is the single entry point for encode/decode against the wire format. -/// -/// There is no `dyn`-safe usage — callers always know the concrete -/// type — so impls are dispatched statically (zero-cost) via -/// monomorphization. -pub trait MessagePackCodec: Sized { - /// Serialize `self` to MessagePack bytes matching the wire format - /// shared with `sketchlib-go`. - fn to_msgpack(&self) -> Result, Error>; - - /// Deserialize a MessagePack byte slice into `Self`. - fn from_msgpack(bytes: &[u8]) -> Result; -} diff --git a/src/message_pack_format/error.rs b/src/message_pack_format/error.rs deleted file mode 100644 index a69452bb..00000000 --- a/src/message_pack_format/error.rs +++ /dev/null @@ -1,45 +0,0 @@ -//! Unified error type for MessagePack encode/decode operations. - -use std::error::Error as StdError; -use std::fmt; - -/// Error returned by [`crate::message_pack_format::MessagePackCodec`] -/// implementations and by the `dto`-level free functions in -/// [`crate::message_pack_format`]. -#[derive(Debug)] -pub enum Error { - /// MessagePack encoding failed. - Encode(rmp_serde::encode::Error), - /// MessagePack decoding failed. - Decode(rmp_serde::decode::Error), -} - -impl fmt::Display for Error { - fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { - match self { - Error::Encode(e) => write!(f, "MessagePack encode failed: {e}"), - Error::Decode(e) => write!(f, "MessagePack decode failed: {e}"), - } - } -} - -impl StdError for Error { - fn source(&self) -> Option<&(dyn StdError + 'static)> { - match self { - Error::Encode(e) => Some(e), - Error::Decode(e) => Some(e), - } - } -} - -impl From for Error { - fn from(value: rmp_serde::encode::Error) -> Self { - Error::Encode(value) - } -} - -impl From for Error { - fn from(value: rmp_serde::decode::Error) -> Self { - Error::Decode(value) - } -} diff --git a/src/message_pack_format/mod.rs b/src/message_pack_format/mod.rs index 8b485028..54d603a5 100644 --- a/src/message_pack_format/mod.rs +++ b/src/message_pack_format/mod.rs @@ -8,21 +8,6 @@ //! [`crate::sketches`] and [`crate::sketch_framework`]. `sketchlib-go` //! mirrors ASAPv1, `asapv1_golden/` guards against drift, and //! `docs/asapv1_wire_format.md` is the spec. -//! -//! Two sub-modules are the older path, being retired: -//! -//! - [`portable`] — pre-envelope types with their own Go goldens. -//! - [`native`] — thin shims over the sketches' `serialize_to_bytes` / -//! `deserialize_from_bytes`, each a pass-through to ASAPv1. -//! -//! [`MessagePackCodec`] and [`Error`] are the contract both share. -pub mod codec; pub(crate) mod envelope; -pub mod error; -pub mod native; -pub mod portable; pub(crate) mod wire_key; - -pub use codec::MessagePackCodec; -pub use error::Error; diff --git a/src/message_pack_format/native/kmv.rs b/src/message_pack_format/native/kmv.rs deleted file mode 100644 index 464fba23..00000000 --- a/src/message_pack_format/native/kmv.rs +++ /dev/null @@ -1,19 +0,0 @@ -//! Native MessagePack codec impl for [`crate::sketches::kmv::KMV`]. -//! -//! The bytes are the ASAPv1 wire envelope (kind `0x0e 0x00`); the impl is -//! bounded on `H: HashProfile`, so a sketch built with an unprofiled hasher has -//! no codec. - -use crate::message_pack_format::{Error, MessagePackCodec}; -use crate::sketches::kmv::KMV; -use crate::{HashProfile, SketchHasher}; - -impl MessagePackCodec for KMV { - fn to_msgpack(&self) -> Result, Error> { - Ok(self.serialize_to_bytes()?) - } - - fn from_msgpack(bytes: &[u8]) -> Result { - Ok(Self::deserialize_from_bytes(bytes)?) - } -} diff --git a/src/message_pack_format/native/mod.rs b/src/message_pack_format/native/mod.rs deleted file mode 100644 index 1ae645c4..00000000 --- a/src/message_pack_format/native/mod.rs +++ /dev/null @@ -1,11 +0,0 @@ -//! Older `MessagePackCodec` shims over the pure-Rust sketch types in -//! [`crate::sketches`], being retired. -//! -//! Each impl forwards to that sketch's own `serialize_to_bytes` / -//! `deserialize_from_bytes`, so the bytes are ASAPv1 — the framing in -//! `envelope.rs` plus the per-sketch `wire.rs` — and the shim adds nothing -//! to them. Call those methods directly; `docs/asapv1_wire_format.md` -//! specifies the format `sketchlib-go` mirrors. - -#[cfg(feature = "experimental")] -pub mod kmv; diff --git a/src/message_pack_format/portable/mod.rs b/src/message_pack_format/portable/mod.rs deleted file mode 100644 index 66328d4f..00000000 --- a/src/message_pack_format/portable/mod.rs +++ /dev/null @@ -1,7 +0,0 @@ -//! Older per-sketch MessagePack wire types, being retired. -//! -//! Each per-algorithm submodule holds one sketch's pre-envelope wire type. -//! New code uses ASAPv1: prefer a sketch's own `serialize_to_bytes`; do not -//! add callers here. - -pub mod sampling; diff --git a/src/message_pack_format/portable/sampling.rs b/src/message_pack_format/portable/sampling.rs deleted file mode 100644 index 5265d243..00000000 --- a/src/message_pack_format/portable/sampling.rs +++ /dev/null @@ -1,123 +0,0 @@ -//! Query-time rescaling for NitroSketch / geometric-sampled sketches. -//! -//! Sampling lives on the wire as a single `sample_p` field on the -//! [`SketchEnvelope`] (NOT inside any -//! per-sketch state struct — so downstream code that builds state structs by -//! literal is unaffected). The producer admits each stream update with -//! probability `p` and stores the RAW SAMPLED state. The backend reads `p` here -//! and applies the `× 1/p` rescale at QUERY time: -//! -//! - **count-like** estimators (HLL cardinality, CountMin / CountSketch -//! frequency, SUM / COUNT) are inverse-probability unbiased: multiply by -//! `1/p`. See [`rescale_count`]. -//! - **quantile** estimators (KLL, DDSketch) are scale-invariant under uniform -//! sampling — every retained item carries the same `1/p` weight, which -//! cancels in the rank computation — so the quantile is read unchanged. The -//! carried `p` is informational (it widens the error budget, §7). See -//! [`is_quantile_scale_invariant`]. -//! -//! Dual-read default: an absent / `0.0` `sample_p` (every pre-sampling -//! producer, and every `p = 1.0` exact producer, which leaves the field at the -//! proto3 default) is interpreted as `1.0` — i.e. no rescale, a true no-op. - -use crate::proto::sketchlib::SketchEnvelope; - -/// Read the effective sampling probability from an envelope, applying the -/// dual-read default: a `0.0` (unset / pre-sampling) `sample_p` means `1.0` -/// (exact, no sampling). Values are clamped to `(0, 1]`; a malformed negative or -/// `>1` value falls back to `1.0` rather than corrupting the rescale. -#[inline] -pub fn effective_sample_p(env: &SketchEnvelope) -> f64 { - sample_p_or_default(env.sample_p) -} - -/// Apply the dual-read default to a raw `sample_p` scalar (e.g. read directly -/// off a decoded envelope field). `0.0` → `1.0`; out-of-range → `1.0`. -#[inline] -pub fn sample_p_or_default(p: f64) -> f64 { - if p > 0.0 && p <= 1.0 { - p - } else { - // 0.0 (unset / exact), NaN, negative, or >1 all mean "no sampling". - 1.0 - } -} - -/// Rescale a count-like estimate (cardinality / frequency / sum / count) read -/// from a sampled sketch back to its unbiased full-stream estimate: `raw / p`. -/// With `p == 1.0` this returns `raw` unchanged. -#[inline] -pub fn rescale_count(raw: f64, p: f64) -> f64 { - let p = sample_p_or_default(p); - raw / p -} - -/// Rescale a count-like estimate using the `sample_p` carried by `env`. -#[inline] -pub fn rescale_count_with_env(raw: f64, env: &SketchEnvelope) -> f64 { - rescale_count(raw, effective_sample_p(env)) -} - -/// Whether a quantile estimate needs NO rescale under uniform sampling. -/// Always `true`: KLL and DDSketch quantiles are scale-invariant because every -/// retained item carries the identical `1/p` weight, which cancels in the rank. -/// Provided as a named, self-documenting guard for call sites that branch on -/// family. -#[inline] -pub const fn is_quantile_scale_invariant() -> bool { - true -} - -#[cfg(test)] -mod tests { - use super::*; - - fn env_with_p(p: f64) -> SketchEnvelope { - SketchEnvelope { - format_version: 1, - producer: None, - hash_spec: None, - sample_p: p, - } - } - - #[test] - fn dual_read_zero_means_one() { - // A pre-sampling / exact envelope leaves sample_p at the proto3 default. - assert_eq!(effective_sample_p(&env_with_p(0.0)), 1.0); - assert_eq!(sample_p_or_default(0.0), 1.0); - } - - #[test] - fn out_of_range_falls_back_to_one() { - assert_eq!(sample_p_or_default(-0.5), 1.0); - assert_eq!(sample_p_or_default(1.5), 1.0); - assert_eq!(sample_p_or_default(f64::NAN), 1.0); - } - - #[test] - fn valid_p_passes_through() { - assert_eq!(effective_sample_p(&env_with_p(0.1)), 0.1); - assert_eq!(sample_p_or_default(1.0), 1.0); - } - - #[test] - fn rescale_count_inverts_probability() { - // Sampled raw count ≈ p·n; raw/p recovers n. - assert!((rescale_count(10_000.0, 0.1) - 100_000.0).abs() < 1e-6); - assert_eq!(rescale_count(123.0, 1.0), 123.0); - // Dual-read: a 0.0 p must not divide-by-zero; it means no sampling. - assert_eq!(rescale_count(123.0, 0.0), 123.0); - } - - #[test] - fn rescale_with_env() { - assert!((rescale_count_with_env(5_000.0, &env_with_p(0.05)) - 100_000.0).abs() < 1e-6); - assert_eq!(rescale_count_with_env(42.0, &env_with_p(0.0)), 42.0); - } - - #[test] - fn quantiles_are_scale_invariant() { - assert!(is_quantile_scale_invariant()); - } -} From 7ba0890b69830433e50214e8c5d069b240b33d79 Mon Sep 17 00:00:00 2001 From: GordonYuanyc Date: Fri, 2 Oct 2026 01:52:43 -0600 Subject: [PATCH 31/36] refactor!: drop the protobuf layer No sketch has a SketchEnvelope field any more, so the proto sources, the vendored prost output, the gen-proto tool, its CI drift job and the prost and bytes dependencies go with them. Co-Authored-By: Claude Opus 5.5 --- .github/workflows/ci.yml | 28 - .github/workflows/docs.yml | 4 - .gitignore | 4 - CHANGELOG.md | 8 + Cargo.lock | 52 -- Cargo.toml | 7 - NOTICE | 2 - README.md | 25 - docs/development_guide.md | 12 +- proto/common/common.proto | 98 ---- proto/countminsketch/countminsketch.proto | 59 -- proto/countsketch/countsketch.proto | 74 --- proto/hll/hll.proto | 75 --- proto/kll/kll.proto | 105 ---- proto/sketchlib.proto | 92 --- proto/univmon/univmon.proto | 50 -- src/lib.rs | 3 - src/proto.rs | 20 - src/proto/generated/sketchlib.v1.rs | 487 ---------------- tools/gen-proto/Cargo.lock | 646 ---------------------- tools/gen-proto/Cargo.toml | 15 - tools/gen-proto/src/main.rs | 90 --- 22 files changed, 10 insertions(+), 1946 deletions(-) delete mode 100644 proto/common/common.proto delete mode 100644 proto/countminsketch/countminsketch.proto delete mode 100644 proto/countsketch/countsketch.proto delete mode 100644 proto/hll/hll.proto delete mode 100644 proto/kll/kll.proto delete mode 100644 proto/sketchlib.proto delete mode 100644 proto/univmon/univmon.proto delete mode 100644 src/proto.rs delete mode 100644 src/proto/generated/sketchlib.v1.rs delete mode 100644 tools/gen-proto/Cargo.lock delete mode 100644 tools/gen-proto/Cargo.toml delete mode 100644 tools/gen-proto/src/main.rs diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index baeb15c8..f75ceb1a 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -111,31 +111,3 @@ jobs: - name: Run tests run: cargo test --features experimental --locked --verbose - - proto-vendored-up-to-date: - name: Vendored proto code is up to date - runs-on: ubuntu-latest - - steps: - - name: Checkout repository - uses: actions/checkout@v4 - - - name: Set up Rust - uses: dtolnay/rust-toolchain@stable - - - name: Cache Cargo dependencies and build artifacts - uses: Swatinem/rust-cache@v2 - with: - workspaces: tools/gen-proto -> tools/gen-proto/target - - - name: Regenerate vendored proto code - run: cargo run --manifest-path tools/gen-proto/Cargo.toml --locked - - - name: Verify src/proto/generated/ has no drift - run: | - if ! git diff --exit-code -- src/proto/generated; then - echo "::error::src/proto/generated/ is out of date. Run:" >&2 - echo " cargo run --manifest-path tools/gen-proto/Cargo.toml" >&2 - echo "and commit the result." >&2 - exit 1 - fi diff --git a/.github/workflows/docs.yml b/.github/workflows/docs.yml index 3458961f..b31ab278 100644 --- a/.github/workflows/docs.yml +++ b/.github/workflows/docs.yml @@ -5,8 +5,6 @@ on: branches: [ "main" ] paths: - "src/**" - - "proto/**" - - "tools/gen-proto/**" - "Cargo.toml" - "Cargo.lock" - "README.md" @@ -16,8 +14,6 @@ on: branches: [ "main" ] paths: - "src/**" - - "proto/**" - - "tools/gen-proto/**" - "Cargo.toml" - "Cargo.lock" - "README.md" diff --git a/.gitignore b/.gitignore index f9f50175..43b62aa0 100644 --- a/.gitignore +++ b/.gitignore @@ -13,10 +13,6 @@ profile.json.gz binary Cargo.lock -# Track the lockfile for the maintainer-only `tools/gen-proto` so that CI's -# vendored-proto drift check uses the same `prost-build` version the -# maintainer used when committing `src/proto/generated/`. -!/tools/gen-proto/Cargo.lock # Task planning artifacts (kept in PR description, not in main history) tasks/ diff --git a/CHANGELOG.md b/CHANGELOG.md index 3b2d8f07..e14b8edb 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -274,6 +274,14 @@ signals a backwards-compatible change. `rescale_count`, `rescale_count_with_env` and `is_quantile_scale_invariant`. Every sketch encodes through `serialize_to_bytes` / `deserialize_from_bytes`; ASAPv1 carries no sampling probability. +- **The protobuf layer.** Removed the public module `proto::sketchlib` with + `SketchEnvelope`, `ProducerInfo`, `HashSpec`, `HashAlgorithm`, + `SeedDerivation`, `CounterType` and the state messages no envelope field + carries any more (`HyperLogLogState`, `KllState`, `CountMinState`, + `CountSketchState`, `UnivMonState` and their nested types); the `.proto` + sources under `proto/`, the vendored `src/proto/generated/`, the + `tools/gen-proto` generator and its CI drift check; and the `prost` and + `bytes` dependencies. ### Fixed diff --git a/Cargo.lock b/Cargo.lock index e4f6f24d..7291bf73 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -2,21 +2,13 @@ # It is not intended for manual editing. version = 4 -[[package]] -name = "anyhow" -version = "1.0.102" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7f202df86484c868dbad7eaa557ef785d5c66295e41b460ef922eca0723b842c" - [[package]] name = "asap_sketchlib" version = "0.3.0" dependencies = [ - "bytes", "core_affinity", "crossbeam-channel", "proptest", - "prost", "rand", "rmp-serde", "serde", @@ -60,12 +52,6 @@ version = "1.5.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "1fd0f2584146f6f2ef48085050886acf353beff7305ebd1ae69500e27c67f64b" -[[package]] -name = "bytes" -version = "1.11.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1e748733b7cbc798e1434b6ac524f0c1ff2ab456fe201501e6497c8417a4fc33" - [[package]] name = "cfg-if" version = "1.0.3" @@ -98,12 +84,6 @@ version = "0.8.21" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "d0a5c400df2834b80a4c3327b3aad3a4c4cd4de0629063962b03235697506a28" -[[package]] -name = "either" -version = "1.15.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "48c757948c5ede0e46177b7add2e67155f70e33c07fea8284df6576da70b3719" - [[package]] name = "errno" version = "0.3.14" @@ -144,15 +124,6 @@ version = "0.5.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "fc0fef456e4baa96da950455cd02c081ca953b141298e41db3fc7e36b1da849c" -[[package]] -name = "itertools" -version = "0.10.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b0fd2260e829bddf4cb6ea802289de2f86d6a7a690192fbe91b3f46e0f2c8473" -dependencies = [ - "either", -] - [[package]] name = "libc" version = "0.2.177" @@ -233,29 +204,6 @@ dependencies = [ "unarray", ] -[[package]] -name = "prost" -version = "0.13.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2796faa41db3ec313a31f7624d9286acf277b52de526150b7e69f3debf891ee5" -dependencies = [ - "bytes", - "prost-derive", -] - -[[package]] -name = "prost-derive" -version = "0.13.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8a56d757972c98b346a9b766e3f02746cde6dd1cd1d1d563472929fdd74bec4d" -dependencies = [ - "anyhow", - "itertools", - "proc-macro2", - "quote", - "syn", -] - [[package]] name = "quick-error" version = "1.2.3" diff --git a/Cargo.toml b/Cargo.toml index d9314dc9..fa89c4e1 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -24,8 +24,6 @@ serde_bytes = "0.11" rmp-serde = "1.3.0" twox-hash = "2.1.2" smallvec = "1.13.2" -prost = "0.13" -bytes = "1" xxhash-rust = { version = "0.8", features = ["xxh32"] } core_affinity = { version = "0.8", optional = true } crossbeam-channel = { version = "0.5", optional = true } @@ -39,11 +37,6 @@ default = [] experimental = [] octo-runtime = ["core_affinity", "crossbeam-channel"] -# Note: there are intentionally no `[build-dependencies]`. The prost output -# under `src/proto/generated/` is checked in and refreshed by the maintainer -# tool at `tools/gen-proto/`. This keeps the published crate pure-Rust with -# no requirement on `protoc` for downstream consumers. - [lib] crate-type = ["rlib"] name = "asap_sketchlib" diff --git a/NOTICE b/NOTICE index 7dc07f97..552bbf3e 100644 --- a/NOTICE +++ b/NOTICE @@ -32,8 +32,6 @@ serde-big-array - Copyright (c) 2019 est31 (MIT/Apache-2.0) rmp-serde - Copyright (c) 2017 Nikolai Vazquez (MIT) twox-hash - Copyright (c) 2015 Jake Goulding (MIT) smallvec - Copyright (c) 2018 The Servo Project Developers (MIT/Apache-2.0) -prost - Copyright (c) 2022 Dan Burkert and Tokio Contributors (Apache-2.0) -bytes - Copyright (c) 2022 Carl Lerche and Tokio Contributors (MIT) Optional (behind "octo-runtime" feature): core_affinity - Copyright (c) 2016 Eliot Courtney (MIT/Apache-2.0) diff --git a/README.md b/README.md index 703058ef..1475acac 100644 --- a/README.md +++ b/README.md @@ -220,31 +220,6 @@ cargo build --features experimental cargo test --features "experimental octo-runtime" ``` -## Protobuf code generation - -`asap_sketchlib` is a pure-Rust crate: building it does **not** require -`protoc` or any build script. The Rust types generated from -`proto/**/*.proto` are vendored into `src/proto/generated/` and refreshed -manually by maintainers using the in-repo tool at `tools/gen-proto/`. - -Downstream users can therefore simply add the crate to their `Cargo.toml` and -build it like any other pure-Rust dependency. - -### For maintainers - -After editing any `.proto` file, regenerate the vendored output and commit -the result: - -```bash -cargo run --manifest-path tools/gen-proto/Cargo.toml -git add src/proto/generated -``` - -The tool uses `prost-build` together with the `protoc-bin-vendored` binary, -so no system `protoc` is required to regenerate either. CI rejects any pull -request whose committed `src/proto/generated/` does not match the result of -running this command on its current `.proto` sources. - ## FAQ ### When is Apache DataSketches a better fit? diff --git a/docs/development_guide.md b/docs/development_guide.md index 459617e1..f4a888a3 100644 --- a/docs/development_guide.md +++ b/docs/development_guide.md @@ -36,14 +36,6 @@ cargo test --features octo-runtime --locked cargo test --features experimental --locked ``` -Vendored proto code must have no drift — regenerate, then confirm the tree is -unchanged: - -```bash -cargo run --manifest-path tools/gen-proto/Cargo.toml --locked -git diff --exit-code -- src/proto/generated -``` - Rustdoc, warnings denied: ```bash @@ -51,5 +43,5 @@ RUSTDOCFLAGS="-D warnings" cargo doc --no-deps --all-features --locked ``` The Rust CI workflow skips `**/*.md` and `docs/**`; the docs workflow runs on -`docs/**`, `README.md`, `src/**`, `proto/**`, `tools/gen-proto/**` and the -manifests. A documentation-only change is therefore gated by rustdoc alone. +`docs/**`, `README.md`, `src/**` and the manifests. A documentation-only +change is therefore gated by rustdoc alone. diff --git a/proto/common/common.proto b/proto/common/common.proto deleted file mode 100644 index 3776f520..00000000 --- a/proto/common/common.proto +++ /dev/null @@ -1,98 +0,0 @@ -// common.proto — Shared types used across all sketch-specific proto files. -// -// Contains: ProducerInfo, HashSpec, HashAlgorithm, SeedDerivation, CounterType. -// These are referenced by the per-sketch protos and the top-level envelope. - -syntax = "proto3"; - -package sketchlib.v1; - -option go_package = "github.com/ProjectASAP/sketchlib-go/proto/common;commonpb"; - -// ============================================================================ -// Producer Info -// ============================================================================ - -// ProducerInfo records which library version created this envelope. -message ProducerInfo { - // One of "sketchlib-go" or "asap_sketchlib". - string library = 1; - - // Semantic version string of the producing library, e.g. "0.1.0". - string version = 2; -} - -// ============================================================================ -// Hash Specification -// ============================================================================ - -// HashSpec fully describes the hashing configuration so both sides can confirm -// they will compute identical bucket indices. -// -// Both libraries share: -// canonical_seed_index = 5 (CanonicalHashSeed / CANONICAL_HASH_SEED) -// seed_list = the 20-entry table in hash.go / hash.rs -// seed_derivation = ADDITIVE_OFFSET for per-row seeds -message HashSpec { - // Hash algorithm used for all sketch operations. - HashAlgorithm algorithm = 1; - - // Index into seed_list used as the primary (canonical) hash seed. - // Both libraries default to 5. - uint32 canonical_seed_index = 2; - - // Complete seed table. Must be identical on both sides for correct results. - // Both libraries currently use exactly 20 seeds; producers always emit all 20. - repeated uint64 seed_list = 3 [packed = true]; - - // How per-row seeds are derived from the base seed index. - SeedDerivation seed_derivation = 4; -} - -// HashAlgorithm enumerates supported hash algorithms. -// Only XXH3_64 is cross-language stable for the current implementation. -enum HashAlgorithm { - HASH_ALGORITHM_UNSPECIFIED = 0; - - // xxHash3 64-bit with seed (zeebo/xxh3 in Go; twox-hash XxHash3_64 in Rust). - // This is the standard algorithm for both libraries. - HASH_ALGORITHM_XXH3_64 = 1; - - // xxHash3 128-bit with seed. Used for HLL register indexing and packed matrix - // hashes that exceed 64 bits. - HASH_ALGORITHM_XXH3_128 = 2; - - // xxHash 64-bit (non-XXH3). Reserved for legacy data; not produced by either - // current library. - HASH_ALGORITHM_XXHASH64 = 3; -} - -// SeedDerivation describes how per-row seed indices are produced. -enum SeedDerivation { - SEED_DERIVATION_UNSPECIFIED = 0; - - // Each row r uses seed_list[(base + r) % len(seed_list)]. - // Standard derivation for multi-row sketches (CountMin, CountSketch). - SEED_DERIVATION_ADDITIVE_OFFSET = 1; - - // All rows share a single hash (Packed64 or Packed128 fast-path). - // Applicable only when the total bit-width fits in 64 or 128 bits. - SEED_DERIVATION_PACKED = 2; -} - -// ============================================================================ -// Counter Type -// ============================================================================ - -// CounterType identifies the numeric type used to store matrix counters. -// Go defaults to FLOAT64; Rust defaults to INT32. -// Determines which counts_* array is populated in CountMinState and -// CountSketchState. -enum CounterType { - COUNTER_TYPE_UNSPECIFIED = 0; - COUNTER_TYPE_INT32 = 1; // Rust default for CountMin / CountSketch. - COUNTER_TYPE_INT64 = 2; - // INT128 is stored as pairs of int64 (hi, lo) interleaved in counts_int. - COUNTER_TYPE_INT128 = 3; - COUNTER_TYPE_FLOAT64 = 4; // Go default for CountMin / CountSketch. -} diff --git a/proto/countminsketch/countminsketch.proto b/proto/countminsketch/countminsketch.proto deleted file mode 100644 index c3be0445..00000000 --- a/proto/countminsketch/countminsketch.proto +++ /dev/null @@ -1,59 +0,0 @@ -// countminsketch.proto — Portable wire format for Count-Min Sketch state. - -syntax = "proto3"; - -package sketchlib.v1; - -import "common/common.proto"; - -option go_package = "github.com/ProjectASAP/sketchlib-go/proto/countminsketch;countminsketchpb"; - -// ============================================================================ -// Count-Min Sketch -// ============================================================================ - -// CountMinState is the portable state of a Count-Min Sketch. -// -// The counter matrix is stored flat in row-major order: -// element [r][c] is at index r * cols + c. -// -// The Go implementation tracks auxiliary sum/L1/L2 arrays that the Rust -// implementation does not produce. These fields are optional; consumers -// that need them must recompute from raw insertions if absent. -message CountMinState { - // Matrix dimensions. Both must be > 0. - uint32 rows = 1; - uint32 cols = 2; - - // Indicates which counts_* field below is populated. - CounterType counter_type = 3; - - // Integer counters in row-major order, length = rows * cols. - // Populated when counter_type ∈ {INT32, INT64}. - // Use sint64 for efficient varint encoding of near-zero values. - // For INT128 counters, values are interleaved as (hi_i, lo_i) pairs, - // making the effective length 2 * rows * cols. - repeated sint64 counts_int = 4 [packed = true]; - - // Floating-point counters in row-major order, length = rows * cols. - // Populated when counter_type = FLOAT64 (Go producer). - repeated double counts_float = 5 [packed = true]; - - // ---- Optional Go-extension fields ---------------------------------------- - // These are produced by sketchlib-go and absent from asap_sketchlib producers. - // Consumers should treat absence as "unknown; recompute if needed". - - // Sum-weighted counter matrix (same layout as counts_*), length = rows * cols. - repeated double sum_counts = 6 [packed = true]; - - // Sum-of-squares counter matrix, length = rows * cols. - repeated double sum2_counts = 7 [packed = true]; - - // Per-row L1 norms, length = rows. - repeated double l1 = 8 [packed = true]; - - // Per-row L2 norms, length = rows. Recompute if absent. - repeated double l2 = 9 [packed = true]; - - reserved 10 to 15; -} diff --git a/proto/countsketch/countsketch.proto b/proto/countsketch/countsketch.proto deleted file mode 100644 index 71e05513..00000000 --- a/proto/countsketch/countsketch.proto +++ /dev/null @@ -1,74 +0,0 @@ -// countsketch.proto — CountSketchState, TopKState and HeapEntry, the -// messages UnivMon and Hydra embed. - -syntax = "proto3"; - -package sketchlib.v1; - -import "common/common.proto"; - -option go_package = "github.com/ProjectASAP/sketchlib-go/proto/countsketch;countsketchpb"; - -// ============================================================================ -// Count Sketch -// ============================================================================ - -// CountSketchState is the portable state of a Count (±1) Sketch. -// -// The counter matrix is signed because Count Sketch uses ±1 increments. -// The matrix layout is identical to CountMinState (row-major, flat). -// -// L2 norms are derived values and must be recomputed on load: -// l2[r] = sqrt( sum_c counts[r][c]^2 ) -// Producers may include them as a convenience for consumers that only query. -message CountSketchState { - // Matrix dimensions. Both must be > 0. - uint32 rows = 1; - uint32 cols = 2; - - // Indicates which counts_* field below is populated. - CounterType counter_type = 3; - - // Signed integer counters in row-major order, length = rows * cols. - // Populated when counter_type ∈ {INT32, INT64}. - repeated sint64 counts_int = 4 [packed = true]; - - // Floating-point counters in row-major order, length = rows * cols. - // Populated when counter_type = FLOAT64 (Go producer). - repeated double counts_float = 5 [packed = true]; - - // Per-row L2 norms, length = rows. Derived — recompute on load if absent. - repeated double l2 = 6 [packed = true]; - - // Optional heavy-hitter (TopK min-heap) state. - // Absent when TopK tracking is disabled. - TopKState topk = 7; - - reserved 8 to 15; -} - -// ============================================================================ -// TopK / Heavy-Hitter State -// ============================================================================ - -// TopKState is the serialized state of a min-heap heavy-hitter tracker. -message TopKState { - // Maximum number of entries retained. - uint32 k = 1; - - // All retained entries. Order is not guaranteed (heap order is not preserved). - repeated HeapEntry entries = 2; -} - -// HeapEntry is one heavy-hitter record inside a TopKState. -message HeapEntry { - // String key identifying the heavy hitter (e.g. flow ID, IP address). - // For binary keys, producers should use base64 or hex encoding and document - // their choice; a future bytes_key field is reserved at number 3. - string key = 1; - - // Approximate frequency or weight accumulated for this key. - double count = 2; - - reserved 3; // Reserved for bytes_key alternative. -} diff --git a/proto/hll/hll.proto b/proto/hll/hll.proto deleted file mode 100644 index 6a8f0793..00000000 --- a/proto/hll/hll.proto +++ /dev/null @@ -1,75 +0,0 @@ -// hll.proto — HyperLogLog cell state for HydraCell (hydra/hydra.proto). - -syntax = "proto3"; - -package sketchlib.v1; - -option go_package = "github.com/ProjectASAP/sketchlib-go/proto/hll;hllpb"; - -// ============================================================================ -// HyperLogLog -// ============================================================================ - -// HyperLogLogState is the state of one HyperLogLog cell. -// num_registers = 2^precision; variant selects the estimator. -message HyperLogLogState { - // HLL estimator variant. Required. - HLLVariant variant = 1; - - // Log₂ of the number of registers (precision parameter p). - // Default = 14 → 16 384 registers. - uint32 precision = 2; - - // Dense registers, one byte each, length 2^precision. Exactly one of - // `registers` and `registers_sparse` is set; both empty means all zero. - bytes registers = 3; - - // ---- HIP-variant extra fields (populated only when variant = HIP) -------- - - // HIP accumulator component kxq0. - double hip_kxq0 = 4; - - // HIP accumulator component kxq1. - double hip_kxq1 = 5; - - // HIP running cardinality estimate. - double hip_est = 6; - - // Sparse registers, set instead of `registers`; see HLLSparseRegisters. - HLLSparseRegisters registers_sparse = 7; - - reserved 8 to 15; -} - -// HLLSparseRegisters is the sparse (HLL++ style) full-state encoding of the -// register array. Only non-zero registers are stored, sorted by ascending -// index and varint-packed as (index_delta, value) pairs: -// -// for each non-zero register, in ascending index order: -// uvarint(index - prev_index) // prev_index starts at 0; deltas are >= 0 -// uvarint(value) // 1..=Q+1, always 1 byte in practice -// -// Registers absent from `packed` are zero. -message HLLSparseRegisters { - // Length of the dense register array (= 2^precision). - uint32 num_registers = 1; - - // Varint-packed (index_delta, value) pairs for non-zero registers, in - // ascending index order. See message-level comment for the exact layout. - bytes packed = 2; -} - -// HLLVariant identifies which HLL estimator algorithm the registers belong to. -enum HLLVariant { - HLL_VARIANT_UNSPECIFIED = 0; - - // Classic HLL with bias-correction tables (Flajolet et al. 2007). - HLL_VARIANT_REGULAR = 1; - - // Otmar Ertl's MLE estimator (arXiv:1702.01284). - HLL_VARIANT_ERTL_MLE = 2; - - // Kevin Lang's HyperLogLog++ Improved Proposal (HIP). - // Requires hip_kxq0, hip_kxq1, hip_est to be populated. - HLL_VARIANT_HIP = 3; -} diff --git a/proto/kll/kll.proto b/proto/kll/kll.proto deleted file mode 100644 index bd307a14..00000000 --- a/proto/kll/kll.proto +++ /dev/null @@ -1,105 +0,0 @@ -// kll.proto — Portable wire format for KLL quantile sketch state. - -syntax = "proto3"; - -package sketchlib.v1; - -option go_package = "github.com/ProjectASAP/sketchlib-go/proto/kll;kllpb"; - -// ============================================================================ -// KLL Sketch -// ============================================================================ - -// KLLState is the portable state of a KLL quantile sketch. -// -// The items array is partitioned into compaction levels by the levels[] array: -// items[ levels[i] .. levels[i+1] ) are the retained items at level i. -// -// Derived state that must be recomputed on load (not serialized): -// capacity_cache — computed from k, m, and level boundaries. -// level0_capacity — computed from k and num_levels. -// top_height — equals num_levels - 1. -message KLLState { - // Target compactor capacity k. Must be >= 8. - uint32 k = 1; - - // Minimum compactor size m. Must satisfy 2 <= m <= k. - uint32 m = 2; - - // Number of levels currently in use. - uint32 num_levels = 3; - - // Level boundary indices into items[], length = num_levels + 1. - // levels[0] is always 0; levels[num_levels] = len(items). - repeated uint32 levels = 4 [packed = true]; - - // All retained samples in level order, length = levels[num_levels]. - // - // RAW-F64 representation. Populated when the value-offset fixed-point - // representation (offset/value_scale/residuals, fields 7–9) is NOT used — - // i.e. when the producer could not represent the retained set exactly at - // any candidate decimal scale and fell back to raw f64. When residuals - // (field 9) is non-empty, items MUST be empty and decoders MUST reconstruct - // from the fixed-point fields instead. - repeated double items = 5 [packed = true]; - - // Random bit generator state for deterministic compaction continuation. - // Required for sketches that will receive further updates after loading. - // Consumers that only query may ignore this field. - CoinState coin = 6; - - // ---- Value-offset fixed-point representation (additive, optional) ------- - // - // KLL retained samples are typically integer-ish or low-cardinality decimal - // values whose raw f64 encoding (8 bytes/item) dominates the wire size. - // When every retained value is exactly representable as - // - // value = residual * 10^value_scale + offset - // - // the producer stores the per-sample residual integers (field 9, zigzag - // varint) plus a single per-sketch `offset` (field 7) and decimal exponent - // `value_scale` (field 8) instead of raw f64 items. This is exact for the - // stored sample SET (the producer applies an exactness guard and falls back - // to raw `items` when any value is not exactly representable), and quantiles - // are shift-equivariant (q(X - c) = q(X) - c) so query results are - // unchanged. residuals[] is in the SAME level order as items[]. - // - // Backward compatibility: when residuals (field 9) is absent/empty, decoders - // read raw f64 from items (field 5) — the original v1 behavior. Decoders on - // BOTH languages read both forms; producers gate emission of the fixed-point - // form behind the exactness guard. - // - // NOTE: this is the self-contained per-SKETCH offset carried in the sketch - // header. The epoch-bound offset for the delta-transmission path is a - // separate, later concern and is intentionally NOT modeled here. - - // Per-sketch subtraction constant. value = residual * 10^value_scale + offset. - double offset = 7; - - // Decimal scale exponent. A value v maps to residual = round((v - offset) * - // 10^(-value_scale)); reconstruction multiplies by 10^value_scale. scale 0 - // means residuals are integers; negative scales carry fractional precision - // (e.g. value_scale = -3 -> millis). sint32 so negative scales varint-pack. - sint32 value_scale = 8; - - // Per-sample residual integers, same level order and length as a raw items[] - // would have. Zigzag varint (sint64) keeps small magnitudes compact. When - // present and non-empty, this is the authoritative item source and items[] - // (field 5) MUST be empty. - repeated sint64 residuals = 9 [packed = true]; - - reserved 10 to 15; -} - -// CoinState is the RNG used by KLL's probabilistic compaction. -// Both Go and Rust use an identical xorshift-based generator. -message CoinState { - // Xorshift generator internal state word. - uint64 state = 1; - - // Up to 64 random bits buffered for single-bit coin flips. - uint64 bit_cache = 2; - - // Number of valid bits remaining in bit_cache (0–64). - uint32 remaining_bits = 3; -} diff --git a/proto/sketchlib.proto b/proto/sketchlib.proto deleted file mode 100644 index 61c538a0..00000000 --- a/proto/sketchlib.proto +++ /dev/null @@ -1,92 +0,0 @@ -// sketchlib.proto — Top-level envelope for sketchlib portable wire format. -// -// This file defines only the SketchEnvelope message. All sketch-specific -// message types live in their per-sketch proto files: -// -// common/common.proto — HashSpec, ProducerInfo, CounterType, ... -// countminsketch/countminsketch.proto — CountMinState -// countsketch/countsketch.proto — CountSketchState, TopKState, HeapEntry -// -// Design goals: -// - Cross-language: Go (sketchlib-go) ↔ Rust (asap_sketchlib) round-trips. -// - Forward-compatible: new fields use reserved numbers; old readers skip them. -// - Logical state only: no pointers, no runtime caches, no gob/msgpack artefacts. -// - Derived values (gamma, L2 norms, capacity caches) are always recomputed on load. -// - Flat row-major arrays for all 2-D matrices. -// - Hash specification is explicit so consumers can detect configuration mismatches. -// -// Version history: -// 1 — initial portable format (CountMin, Count, HLL, KLL, DDSketch) -// 2 — extended to UnivMon and Hydra (reserved, not yet stable) - -syntax = "proto3"; - -package sketchlib.v1; - -import "common/common.proto"; -import "countsketch/countsketch.proto"; - -option go_package = "github.com/ProjectASAP/sketchlib-go/proto/sketch_envelope;sketch_envelopepb"; - -// ============================================================================ -// Top-Level Envelope -// ============================================================================ - -// SketchEnvelope is the portable container for all sketch transfers. -// Producers fill exactly one field inside sketch_state. -// Consumers check format_version first, then dispatch on the oneof. -message SketchEnvelope { - // Schema version. Consumers should reject envelopes whose version exceeds - // the highest version they understand. - // 1 = CountMin, Count, HLL, KLL, DDSketch - // 2 = UnivMon, Hydra (reserved) - // 3 = CocoSketch, ElasticSketch (reserved) - uint32 format_version = 1; - - // Library and build info of the producing side. - ProducerInfo producer = 2; - - // Complete hashing configuration. Consumers must validate this against their - // own configuration before performing updates or queries on the loaded sketch. - // Mismatches do not prevent loading but will cause diverging results. - HashSpec hash_spec = 3; - - reserved 10; - reserved "count_min"; - reserved 11; - reserved "count_sketch"; - reserved 12; - reserved "hll"; - reserved 13; - reserved "kll"; - reserved 14; - reserved "ddsketch"; - reserved 15; - reserved "univmon"; - reserved 16; - reserved "hydra"; - reserved 17; - reserved "coco"; - reserved 18; - reserved "elastic"; - reserved 19 to 31; // Reserved for future sketch types. - - // Geometric / NitroSketch sampling probability for the producing sketch. - // - // The producer admits each stream update with probability `sample_p` (using - // geometric skip-sampling, statistically identical to per-update Bernoulli(p)) - // and stores the RAW SAMPLED sketch state — never the rescaled state. The - // consumer applies the `× 1/sample_p` rescale at QUERY time on the count-like - // estimators (HLL cardinality, CountSketch frequency, SUM / COUNT); - // quantile estimators (DDSketch) are scale-invariant under uniform - // sampling so they carry `sample_p` but need no rescale. - // - // Lives on the envelope (not per-sketch state) so downstream consumers that - // construct per-sketch state structs by literal are unaffected. - // - // Dual-read default: a value of 0.0 (the proto3 default for an unset field) - // is interpreted as 1.0 (exact, no sampling) for backward compatibility with - // producers that predate this field. With sample_p = 1.0 a 0.0/unset envelope - // is byte-identical to the pre-sampling format. - double sample_p = 4; -} diff --git a/proto/univmon/univmon.proto b/proto/univmon/univmon.proto deleted file mode 100644 index f22828f7..00000000 --- a/proto/univmon/univmon.proto +++ /dev/null @@ -1,50 +0,0 @@ -// univmon.proto — Portable wire format for Universal Monitoring (UnivMon) sketch state. - -syntax = "proto3"; - -package sketchlib.v1; - -import "countsketch/countsketch.proto"; - -option go_package = "github.com/ProjectASAP/sketchlib-go/proto/univmon;univmonpb"; - -// ============================================================================ -// UnivMon -// ============================================================================ - -// UnivMonState is the portable state of a Universal Monitoring (UnivMon) sketch. -// -// UnivMon is a hierarchy of L Count Sketches paired with L heavy-hitter heaps. -// An item's hash trailing-zero count determines which layers receive it, giving -// each layer a geometrically decreasing sample of the stream. -// -// All layers share the same sketch dimensions and heap capacity. -message UnivMonState { - // Number of layers L in the hierarchy (inclusive of all levels). - uint32 layer_size = 1; - - // Count Sketch dimensions shared by all layers. - uint32 sketch_rows = 2; - uint32 sketch_cols = 3; - - // Heavy-hitter heap capacity shared by all layers. - uint32 heap_size = 4; - - // Total items inserted across all layers (the "bucket size" counter). - uint64 bucket_size = 5; - - // Per-layer state, in order from layer 0 (finest) to layer_size-1 (coarsest). - // Length must equal layer_size. - repeated UnivMonLayer layers = 6; - - reserved 7 to 15; -} - -// UnivMonLayer is the joint state of one UnivMon layer. -message UnivMonLayer { - // The Count Sketch accumulating items at this layer. - CountSketchState sketch = 1; - - // The heavy-hitter heap for this layer. - TopKState heap = 2; -} diff --git a/src/lib.rs b/src/lib.rs index b1c13ed9..f98ac5a4 100644 --- a/src/lib.rs +++ b/src/lib.rs @@ -5,7 +5,6 @@ //! KLL, and DDSketch. //! - [`common`]: shared input types, hashing abstractions, storage backends, and //! reusable utilities used across sketches. -//! - [`proto`]: portable protobuf message types for sketch interchange. //! - [`sketch_framework`]: higher-level composition layers such as Hydra, //! UnivMon, tumbling windows, and batch/parallel execution helpers. //! - [`message_pack_format`]: MessagePack/proto wire format shared with @@ -20,8 +19,6 @@ /// storage backends, and reusable utilities. pub mod common; pub mod message_pack_format; -/// Portable protobuf message types for sketch interchange. -pub mod proto; /// Higher-level composition layers such as Hydra, UnivMon, tumbling windows, /// and batch/parallel execution helpers. pub mod sketch_framework; diff --git a/src/proto.rs b/src/proto.rs deleted file mode 100644 index 933fe26d..00000000 --- a/src/proto.rs +++ /dev/null @@ -1,20 +0,0 @@ -// Proto-generated types for cross-language sketch serialization. -// -// The contents of `sketchlib` are vendored: the file under -// `src/proto/generated/sketchlib.v1.rs` is produced by `tools/gen-proto` -// and committed to the repository. End-users of this crate therefore do -// not need `protoc` or any build-script machinery to build it. -// -// To regenerate after editing any `proto/**/*.proto`: -// -// cargo run --manifest-path tools/gen-proto/Cargo.toml -// -// CI enforces that the committed file matches the result of running the -// tool against the current `.proto` sources. -#[allow(rustdoc::broken_intra_doc_links)] -#[allow(missing_docs)] -#[allow(clippy::all)] -#[rustfmt::skip] -pub mod sketchlib { - include!("proto/generated/sketchlib.v1.rs"); -} diff --git a/src/proto/generated/sketchlib.v1.rs b/src/proto/generated/sketchlib.v1.rs deleted file mode 100644 index 94bd1e4e..00000000 --- a/src/proto/generated/sketchlib.v1.rs +++ /dev/null @@ -1,487 +0,0 @@ -// @generated by tools/gen-proto -- DO NOT EDIT. -// -// Regenerate with: -// cargo run --manifest-path tools/gen-proto/Cargo.toml -// -// Source of truth: proto/**/*.proto - -// This file is @generated by prost-build. -/// ProducerInfo records which library version created this envelope. -#[derive(Clone, PartialEq, ::prost::Message)] -pub struct ProducerInfo { - /// One of "sketchlib-go" or "asap_sketchlib". - #[prost(string, tag = "1")] - pub library: ::prost::alloc::string::String, - /// Semantic version string of the producing library, e.g. "0.1.0". - #[prost(string, tag = "2")] - pub version: ::prost::alloc::string::String, -} -/// HashSpec fully describes the hashing configuration so both sides can confirm -/// they will compute identical bucket indices. -/// -/// Both libraries share: -/// canonical_seed_index = 5 (CanonicalHashSeed / CANONICAL_HASH_SEED) -/// seed_list = the 20-entry table in hash.go / hash.rs -/// seed_derivation = ADDITIVE_OFFSET for per-row seeds -#[derive(Clone, PartialEq, ::prost::Message)] -pub struct HashSpec { - /// Hash algorithm used for all sketch operations. - #[prost(enumeration = "HashAlgorithm", tag = "1")] - pub algorithm: i32, - /// Index into seed_list used as the primary (canonical) hash seed. - /// Both libraries default to 5. - #[prost(uint32, tag = "2")] - pub canonical_seed_index: u32, - /// Complete seed table. Must be identical on both sides for correct results. - /// Both libraries currently use exactly 20 seeds; producers always emit all 20. - #[prost(uint64, repeated, tag = "3")] - pub seed_list: ::prost::alloc::vec::Vec, - /// How per-row seeds are derived from the base seed index. - #[prost(enumeration = "SeedDerivation", tag = "4")] - pub seed_derivation: i32, -} -/// HashAlgorithm enumerates supported hash algorithms. -/// Only XXH3_64 is cross-language stable for the current implementation. -#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash, PartialOrd, Ord, ::prost::Enumeration)] -#[repr(i32)] -pub enum HashAlgorithm { - Unspecified = 0, - /// xxHash3 64-bit with seed (zeebo/xxh3 in Go; twox-hash XxHash3_64 in Rust). - /// This is the standard algorithm for both libraries. - Xxh364 = 1, - /// xxHash3 128-bit with seed. Used for HLL register indexing and packed matrix - /// hashes that exceed 64 bits. - Xxh3128 = 2, - /// xxHash 64-bit (non-XXH3). Reserved for legacy data; not produced by either - /// current library. - Xxhash64 = 3, -} -impl HashAlgorithm { - /// String value of the enum field names used in the ProtoBuf definition. - /// - /// The values are not transformed in any way and thus are considered stable - /// (if the ProtoBuf definition does not change) and safe for programmatic use. - pub fn as_str_name(&self) -> &'static str { - match self { - Self::Unspecified => "HASH_ALGORITHM_UNSPECIFIED", - Self::Xxh364 => "HASH_ALGORITHM_XXH3_64", - Self::Xxh3128 => "HASH_ALGORITHM_XXH3_128", - Self::Xxhash64 => "HASH_ALGORITHM_XXHASH64", - } - } - /// Creates an enum from field names used in the ProtoBuf definition. - pub fn from_str_name(value: &str) -> ::core::option::Option { - match value { - "HASH_ALGORITHM_UNSPECIFIED" => Some(Self::Unspecified), - "HASH_ALGORITHM_XXH3_64" => Some(Self::Xxh364), - "HASH_ALGORITHM_XXH3_128" => Some(Self::Xxh3128), - "HASH_ALGORITHM_XXHASH64" => Some(Self::Xxhash64), - _ => None, - } - } -} -/// SeedDerivation describes how per-row seed indices are produced. -#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash, PartialOrd, Ord, ::prost::Enumeration)] -#[repr(i32)] -pub enum SeedDerivation { - Unspecified = 0, - /// Each row r uses seed_list\[(base + r) % len(seed_list)\]. - /// Standard derivation for multi-row sketches (CountMin, CountSketch). - AdditiveOffset = 1, - /// All rows share a single hash (Packed64 or Packed128 fast-path). - /// Applicable only when the total bit-width fits in 64 or 128 bits. - Packed = 2, -} -impl SeedDerivation { - /// String value of the enum field names used in the ProtoBuf definition. - /// - /// The values are not transformed in any way and thus are considered stable - /// (if the ProtoBuf definition does not change) and safe for programmatic use. - pub fn as_str_name(&self) -> &'static str { - match self { - Self::Unspecified => "SEED_DERIVATION_UNSPECIFIED", - Self::AdditiveOffset => "SEED_DERIVATION_ADDITIVE_OFFSET", - Self::Packed => "SEED_DERIVATION_PACKED", - } - } - /// Creates an enum from field names used in the ProtoBuf definition. - pub fn from_str_name(value: &str) -> ::core::option::Option { - match value { - "SEED_DERIVATION_UNSPECIFIED" => Some(Self::Unspecified), - "SEED_DERIVATION_ADDITIVE_OFFSET" => Some(Self::AdditiveOffset), - "SEED_DERIVATION_PACKED" => Some(Self::Packed), - _ => None, - } - } -} -/// CounterType identifies the numeric type used to store matrix counters. -/// Go defaults to FLOAT64; Rust defaults to INT32. -/// Determines which counts_* array is populated in CountMinState and -/// CountSketchState. -#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash, PartialOrd, Ord, ::prost::Enumeration)] -#[repr(i32)] -pub enum CounterType { - Unspecified = 0, - /// Rust default for CountMin / CountSketch. - Int32 = 1, - Int64 = 2, - /// INT128 is stored as pairs of int64 (hi, lo) interleaved in counts_int. - Int128 = 3, - /// Go default for CountMin / CountSketch. - Float64 = 4, -} -impl CounterType { - /// String value of the enum field names used in the ProtoBuf definition. - /// - /// The values are not transformed in any way and thus are considered stable - /// (if the ProtoBuf definition does not change) and safe for programmatic use. - pub fn as_str_name(&self) -> &'static str { - match self { - Self::Unspecified => "COUNTER_TYPE_UNSPECIFIED", - Self::Int32 => "COUNTER_TYPE_INT32", - Self::Int64 => "COUNTER_TYPE_INT64", - Self::Int128 => "COUNTER_TYPE_INT128", - Self::Float64 => "COUNTER_TYPE_FLOAT64", - } - } - /// Creates an enum from field names used in the ProtoBuf definition. - pub fn from_str_name(value: &str) -> ::core::option::Option { - match value { - "COUNTER_TYPE_UNSPECIFIED" => Some(Self::Unspecified), - "COUNTER_TYPE_INT32" => Some(Self::Int32), - "COUNTER_TYPE_INT64" => Some(Self::Int64), - "COUNTER_TYPE_INT128" => Some(Self::Int128), - "COUNTER_TYPE_FLOAT64" => Some(Self::Float64), - _ => None, - } - } -} -/// CountMinState is the portable state of a Count-Min Sketch. -/// -/// The counter matrix is stored flat in row-major order: -/// element [r][c] is at index r * cols + c. -/// -/// The Go implementation tracks auxiliary sum/L1/L2 arrays that the Rust -/// implementation does not produce. These fields are optional; consumers -/// that need them must recompute from raw insertions if absent. -#[derive(Clone, PartialEq, ::prost::Message)] -pub struct CountMinState { - /// Matrix dimensions. Both must be > 0. - #[prost(uint32, tag = "1")] - pub rows: u32, - #[prost(uint32, tag = "2")] - pub cols: u32, - /// Indicates which counts_* field below is populated. - #[prost(enumeration = "CounterType", tag = "3")] - pub counter_type: i32, - /// Integer counters in row-major order, length = rows * cols. - /// Populated when counter_type ∈ {INT32, INT64}. - /// Use sint64 for efficient varint encoding of near-zero values. - /// For INT128 counters, values are interleaved as (hi_i, lo_i) pairs, - /// making the effective length 2 * rows * cols. - #[prost(sint64, repeated, tag = "4")] - pub counts_int: ::prost::alloc::vec::Vec, - /// Floating-point counters in row-major order, length = rows * cols. - /// Populated when counter_type = FLOAT64 (Go producer). - #[prost(double, repeated, tag = "5")] - pub counts_float: ::prost::alloc::vec::Vec, - /// Sum-weighted counter matrix (same layout as counts_*), length = rows * cols. - #[prost(double, repeated, tag = "6")] - pub sum_counts: ::prost::alloc::vec::Vec, - /// Sum-of-squares counter matrix, length = rows * cols. - #[prost(double, repeated, tag = "7")] - pub sum2_counts: ::prost::alloc::vec::Vec, - /// Per-row L1 norms, length = rows. - #[prost(double, repeated, tag = "8")] - pub l1: ::prost::alloc::vec::Vec, - /// Per-row L2 norms, length = rows. Recompute if absent. - #[prost(double, repeated, tag = "9")] - pub l2: ::prost::alloc::vec::Vec, -} -/// CountSketchState is the portable state of a Count (±1) Sketch. -/// -/// The counter matrix is signed because Count Sketch uses ±1 increments. -/// The matrix layout is identical to CountMinState (row-major, flat). -/// -/// L2 norms are derived values and must be recomputed on load: -/// l2\[r\] = sqrt( sum_c counts[r][c]^2 ) -/// Producers may include them as a convenience for consumers that only query. -#[derive(Clone, PartialEq, ::prost::Message)] -pub struct CountSketchState { - /// Matrix dimensions. Both must be > 0. - #[prost(uint32, tag = "1")] - pub rows: u32, - #[prost(uint32, tag = "2")] - pub cols: u32, - /// Indicates which counts_* field below is populated. - #[prost(enumeration = "CounterType", tag = "3")] - pub counter_type: i32, - /// Signed integer counters in row-major order, length = rows * cols. - /// Populated when counter_type ∈ {INT32, INT64}. - #[prost(sint64, repeated, tag = "4")] - pub counts_int: ::prost::alloc::vec::Vec, - /// Floating-point counters in row-major order, length = rows * cols. - /// Populated when counter_type = FLOAT64 (Go producer). - #[prost(double, repeated, tag = "5")] - pub counts_float: ::prost::alloc::vec::Vec, - /// Per-row L2 norms, length = rows. Derived — recompute on load if absent. - #[prost(double, repeated, tag = "6")] - pub l2: ::prost::alloc::vec::Vec, - /// Optional heavy-hitter (TopK min-heap) state. - /// Absent when TopK tracking is disabled. - #[prost(message, optional, tag = "7")] - pub topk: ::core::option::Option, -} -/// TopKState is the serialized state of a min-heap heavy-hitter tracker. -#[derive(Clone, PartialEq, ::prost::Message)] -pub struct TopKState { - /// Maximum number of entries retained. - #[prost(uint32, tag = "1")] - pub k: u32, - /// All retained entries. Order is not guaranteed (heap order is not preserved). - #[prost(message, repeated, tag = "2")] - pub entries: ::prost::alloc::vec::Vec, -} -/// HeapEntry is one heavy-hitter record inside a TopKState. -#[derive(Clone, PartialEq, ::prost::Message)] -pub struct HeapEntry { - /// String key identifying the heavy hitter (e.g. flow ID, IP address). - /// For binary keys, producers should use base64 or hex encoding and document - /// their choice; a future bytes_key field is reserved at number 3. - #[prost(string, tag = "1")] - pub key: ::prost::alloc::string::String, - /// Approximate frequency or weight accumulated for this key. - #[prost(double, tag = "2")] - pub count: f64, -} -/// HyperLogLogState is the state of one HyperLogLog cell. -/// num_registers = 2^precision; variant selects the estimator. -#[derive(Clone, PartialEq, ::prost::Message)] -pub struct HyperLogLogState { - /// HLL estimator variant. Required. - #[prost(enumeration = "HllVariant", tag = "1")] - pub variant: i32, - /// Log₂ of the number of registers (precision parameter p). - /// Default = 14 → 16 384 registers. - #[prost(uint32, tag = "2")] - pub precision: u32, - /// Dense registers, one byte each, length 2^precision. Exactly one of - /// `registers` and `registers_sparse` is set; both empty means all zero. - #[prost(bytes = "vec", tag = "3")] - pub registers: ::prost::alloc::vec::Vec, - /// HIP accumulator component kxq0. - #[prost(double, tag = "4")] - pub hip_kxq0: f64, - /// HIP accumulator component kxq1. - #[prost(double, tag = "5")] - pub hip_kxq1: f64, - /// HIP running cardinality estimate. - #[prost(double, tag = "6")] - pub hip_est: f64, - /// Sparse registers, set instead of `registers`; see HLLSparseRegisters. - #[prost(message, optional, tag = "7")] - pub registers_sparse: ::core::option::Option, -} -/// HLLSparseRegisters is the sparse (HLL++ style) full-state encoding of the -/// register array. Only non-zero registers are stored, sorted by ascending -/// index and varint-packed as (index_delta, value) pairs: -/// -/// for each non-zero register, in ascending index order: -/// uvarint(index - prev_index) // prev_index starts at 0; deltas are >= 0 -/// uvarint(value) // 1..=Q+1, always 1 byte in practice -/// -/// Registers absent from `packed` are zero. -#[derive(Clone, PartialEq, ::prost::Message)] -pub struct HllSparseRegisters { - /// Length of the dense register array (= 2^precision). - #[prost(uint32, tag = "1")] - pub num_registers: u32, - /// Varint-packed (index_delta, value) pairs for non-zero registers, in - /// ascending index order. See message-level comment for the exact layout. - #[prost(bytes = "vec", tag = "2")] - pub packed: ::prost::alloc::vec::Vec, -} -/// HLLVariant identifies which HLL estimator algorithm the registers belong to. -#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash, PartialOrd, Ord, ::prost::Enumeration)] -#[repr(i32)] -pub enum HllVariant { - Unspecified = 0, - /// Classic HLL with bias-correction tables (Flajolet et al. 2007). - Regular = 1, - /// Otmar Ertl's MLE estimator (arXiv:1702.01284). - ErtlMle = 2, - /// Kevin Lang's HyperLogLog++ Improved Proposal (HIP). - /// Requires hip_kxq0, hip_kxq1, hip_est to be populated. - Hip = 3, -} -impl HllVariant { - /// String value of the enum field names used in the ProtoBuf definition. - /// - /// The values are not transformed in any way and thus are considered stable - /// (if the ProtoBuf definition does not change) and safe for programmatic use. - pub fn as_str_name(&self) -> &'static str { - match self { - Self::Unspecified => "HLL_VARIANT_UNSPECIFIED", - Self::Regular => "HLL_VARIANT_REGULAR", - Self::ErtlMle => "HLL_VARIANT_ERTL_MLE", - Self::Hip => "HLL_VARIANT_HIP", - } - } - /// Creates an enum from field names used in the ProtoBuf definition. - pub fn from_str_name(value: &str) -> ::core::option::Option { - match value { - "HLL_VARIANT_UNSPECIFIED" => Some(Self::Unspecified), - "HLL_VARIANT_REGULAR" => Some(Self::Regular), - "HLL_VARIANT_ERTL_MLE" => Some(Self::ErtlMle), - "HLL_VARIANT_HIP" => Some(Self::Hip), - _ => None, - } - } -} -/// KLLState is the portable state of a KLL quantile sketch. -/// -/// The items array is partitioned into compaction levels by the levels\[\] array: -/// items\[ levels[i\] .. levels\[i+1\] ) are the retained items at level i. -/// -/// Derived state that must be recomputed on load (not serialized): -/// capacity_cache — computed from k, m, and level boundaries. -/// level0_capacity — computed from k and num_levels. -/// top_height — equals num_levels - 1. -#[derive(Clone, PartialEq, ::prost::Message)] -pub struct KllState { - /// Target compactor capacity k. Must be >= 8. - #[prost(uint32, tag = "1")] - pub k: u32, - /// Minimum compactor size m. Must satisfy 2 <= m <= k. - #[prost(uint32, tag = "2")] - pub m: u32, - /// Number of levels currently in use. - #[prost(uint32, tag = "3")] - pub num_levels: u32, - /// Level boundary indices into items\[\], length = num_levels + 1. - /// levels\[0\] is always 0; levels\[num_levels\] = len(items). - #[prost(uint32, repeated, tag = "4")] - pub levels: ::prost::alloc::vec::Vec, - /// All retained samples in level order, length = levels\[num_levels\]. - /// - /// RAW-F64 representation. Populated when the value-offset fixed-point - /// representation (offset/value_scale/residuals, fields 7–9) is NOT used — - /// i.e. when the producer could not represent the retained set exactly at - /// any candidate decimal scale and fell back to raw f64. When residuals - /// (field 9) is non-empty, items MUST be empty and decoders MUST reconstruct - /// from the fixed-point fields instead. - #[prost(double, repeated, tag = "5")] - pub items: ::prost::alloc::vec::Vec, - /// Random bit generator state for deterministic compaction continuation. - /// Required for sketches that will receive further updates after loading. - /// Consumers that only query may ignore this field. - #[prost(message, optional, tag = "6")] - pub coin: ::core::option::Option, - /// Per-sketch subtraction constant. value = residual * 10^value_scale + offset. - #[prost(double, tag = "7")] - pub offset: f64, - /// Decimal scale exponent. A value v maps to residual = round((v - offset) * - /// 10^(-value_scale)); reconstruction multiplies by 10^value_scale. scale 0 - /// means residuals are integers; negative scales carry fractional precision - /// (e.g. value_scale = -3 -> millis). sint32 so negative scales varint-pack. - #[prost(sint32, tag = "8")] - pub value_scale: i32, - /// Per-sample residual integers, same level order and length as a raw items\[\] - /// would have. Zigzag varint (sint64) keeps small magnitudes compact. When - /// present and non-empty, this is the authoritative item source and items\[\] - /// (field 5) MUST be empty. - #[prost(sint64, repeated, tag = "9")] - pub residuals: ::prost::alloc::vec::Vec, -} -/// CoinState is the RNG used by KLL's probabilistic compaction. -/// Both Go and Rust use an identical xorshift-based generator. -#[derive(Clone, Copy, PartialEq, ::prost::Message)] -pub struct CoinState { - /// Xorshift generator internal state word. - #[prost(uint64, tag = "1")] - pub state: u64, - /// Up to 64 random bits buffered for single-bit coin flips. - #[prost(uint64, tag = "2")] - pub bit_cache: u64, - /// Number of valid bits remaining in bit_cache (0–64). - #[prost(uint32, tag = "3")] - pub remaining_bits: u32, -} -/// UnivMonState is the portable state of a Universal Monitoring (UnivMon) sketch. -/// -/// UnivMon is a hierarchy of L Count Sketches paired with L heavy-hitter heaps. -/// An item's hash trailing-zero count determines which layers receive it, giving -/// each layer a geometrically decreasing sample of the stream. -/// -/// All layers share the same sketch dimensions and heap capacity. -#[derive(Clone, PartialEq, ::prost::Message)] -pub struct UnivMonState { - /// Number of layers L in the hierarchy (inclusive of all levels). - #[prost(uint32, tag = "1")] - pub layer_size: u32, - /// Count Sketch dimensions shared by all layers. - #[prost(uint32, tag = "2")] - pub sketch_rows: u32, - #[prost(uint32, tag = "3")] - pub sketch_cols: u32, - /// Heavy-hitter heap capacity shared by all layers. - #[prost(uint32, tag = "4")] - pub heap_size: u32, - /// Total items inserted across all layers (the "bucket size" counter). - #[prost(uint64, tag = "5")] - pub bucket_size: u64, - /// Per-layer state, in order from layer 0 (finest) to layer_size-1 (coarsest). - /// Length must equal layer_size. - #[prost(message, repeated, tag = "6")] - pub layers: ::prost::alloc::vec::Vec, -} -/// UnivMonLayer is the joint state of one UnivMon layer. -#[derive(Clone, PartialEq, ::prost::Message)] -pub struct UnivMonLayer { - /// The Count Sketch accumulating items at this layer. - #[prost(message, optional, tag = "1")] - pub sketch: ::core::option::Option, - /// The heavy-hitter heap for this layer. - #[prost(message, optional, tag = "2")] - pub heap: ::core::option::Option, -} -/// SketchEnvelope is the portable container for all sketch transfers. -/// Producers fill exactly one field inside sketch_state. -/// Consumers check format_version first, then dispatch on the oneof. -#[derive(Clone, PartialEq, ::prost::Message)] -pub struct SketchEnvelope { - /// Schema version. Consumers should reject envelopes whose version exceeds - /// the highest version they understand. - /// 1 = CountMin, Count, HLL, KLL, DDSketch - /// 2 = UnivMon, Hydra (reserved) - /// 3 = CocoSketch, ElasticSketch (reserved) - #[prost(uint32, tag = "1")] - pub format_version: u32, - /// Library and build info of the producing side. - #[prost(message, optional, tag = "2")] - pub producer: ::core::option::Option, - /// Complete hashing configuration. Consumers must validate this against their - /// own configuration before performing updates or queries on the loaded sketch. - /// Mismatches do not prevent loading but will cause diverging results. - #[prost(message, optional, tag = "3")] - pub hash_spec: ::core::option::Option, - /// Geometric / NitroSketch sampling probability for the producing sketch. - /// - /// The producer admits each stream update with probability `sample_p` (using - /// geometric skip-sampling, statistically identical to per-update Bernoulli(p)) - /// and stores the RAW SAMPLED sketch state — never the rescaled state. The - /// consumer applies the `× 1/sample_p` rescale at QUERY time on the count-like - /// estimators (HLL cardinality, CountSketch frequency, SUM / COUNT); - /// quantile estimators (DDSketch) are scale-invariant under uniform - /// sampling so they carry `sample_p` but need no rescale. - /// - /// Lives on the envelope (not per-sketch state) so downstream consumers that - /// construct per-sketch state structs by literal are unaffected. - /// - /// Dual-read default: a value of 0.0 (the proto3 default for an unset field) - /// is interpreted as 1.0 (exact, no sampling) for backward compatibility with - /// producers that predate this field. With sample_p = 1.0 a 0.0/unset envelope - /// is byte-identical to the pre-sampling format. - #[prost(double, tag = "4")] - pub sample_p: f64, -} diff --git a/tools/gen-proto/Cargo.lock b/tools/gen-proto/Cargo.lock deleted file mode 100644 index 13170e87..00000000 --- a/tools/gen-proto/Cargo.lock +++ /dev/null @@ -1,646 +0,0 @@ -# This file is automatically @generated by Cargo. -# It is not intended for manual editing. -version = 4 - -[[package]] -name = "aho-corasick" -version = "1.1.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ddd31a130427c27518df266943a5308ed92d4b226cc639f5a8f1002816174301" -dependencies = [ - "memchr", -] - -[[package]] -name = "anyhow" -version = "1.0.102" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7f202df86484c868dbad7eaa557ef785d5c66295e41b460ef922eca0723b842c" - -[[package]] -name = "bitflags" -version = "2.11.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c4512299f36f043ab09a583e57bceb5a5aab7a73db1805848e8fef3c9e8c78b3" - -[[package]] -name = "bytes" -version = "1.11.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1e748733b7cbc798e1434b6ac524f0c1ff2ab456fe201501e6497c8417a4fc33" - -[[package]] -name = "cfg-if" -version = "1.0.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9330f8b2ff13f34540b44e946ef35111825727b38d33286ef986142615121801" - -[[package]] -name = "either" -version = "1.15.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "48c757948c5ede0e46177b7add2e67155f70e33c07fea8284df6576da70b3719" - -[[package]] -name = "equivalent" -version = "1.0.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "877a4ace8713b0bcf2a4e7eec82529c029f1d0619886d18145fea96c3ffe5c0f" - -[[package]] -name = "errno" -version = "0.3.14" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "39cab71617ae0d63f51a36d69f866391735b51691dbda63cf6f96d042b63efeb" -dependencies = [ - "libc", - "windows-sys", -] - -[[package]] -name = "fastrand" -version = "2.4.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9f1f227452a390804cdb637b74a86990f2a7d7ba4b7d5693aac9b4dd6defd8d6" - -[[package]] -name = "fixedbitset" -version = "0.5.7" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1d674e81391d1e1ab681a28d99df07927c6d4aa5b027d7da16ba32d1d21ecd99" - -[[package]] -name = "foldhash" -version = "0.1.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d9c4f5dac5e15c24eb999c26181a6ca40b39fe946cbe4c263c7209467bc83af2" - -[[package]] -name = "gen-proto" -version = "0.0.0" -dependencies = [ - "prost-build", - "protoc-bin-vendored", -] - -[[package]] -name = "getrandom" -version = "0.4.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0de51e6874e94e7bf76d726fc5d13ba782deca734ff60d5bb2fb2607c7406555" -dependencies = [ - "cfg-if", - "libc", - "r-efi", - "wasip2", - "wasip3", -] - -[[package]] -name = "hashbrown" -version = "0.15.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9229cfe53dfd69f0609a49f65461bd93001ea1ef889cd5529dd176593f5338a1" -dependencies = [ - "foldhash", -] - -[[package]] -name = "hashbrown" -version = "0.17.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ed5909b6e89a2db4456e54cd5f673791d7eca6732202bbf2a9cc504fe2f9b84a" - -[[package]] -name = "heck" -version = "0.5.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2304e00983f87ffb38b55b444b5e3b60a884b5d30c0fca7d82fe33449bbe55ea" - -[[package]] -name = "id-arena" -version = "2.3.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3d3067d79b975e8844ca9eb072e16b31c3c1c36928edf9c6789548c524d0d954" - -[[package]] -name = "indexmap" -version = "2.14.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d466e9454f08e4a911e14806c24e16fba1b4c121d1ea474396f396069cf949d9" -dependencies = [ - "equivalent", - "hashbrown 0.17.1", - "serde", - "serde_core", -] - -[[package]] -name = "itertools" -version = "0.14.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2b192c782037fadd9cfa75548310488aabdbf3d2da73885b31bd0abd03351285" -dependencies = [ - "either", -] - -[[package]] -name = "itoa" -version = "1.0.18" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8f42a60cbdf9a97f5d2305f08a87dc4e09308d1276d28c869c684d7777685682" - -[[package]] -name = "leb128fmt" -version = "0.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "09edd9e8b54e49e587e4f6295a7d29c3ea94d469cb40ab8ca70b288248a81db2" - -[[package]] -name = "libc" -version = "0.2.186" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "68ab91017fe16c622486840e4c83c9a37afeff978bd239b5293d61ece587de66" - -[[package]] -name = "linux-raw-sys" -version = "0.12.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "32a66949e030da00e8c7d4434b251670a91556f4144941d37452769c25d58a53" - -[[package]] -name = "log" -version = "0.4.29" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5e5032e24019045c762d3c0f28f5b6b8bbf38563a65908389bf7978758920897" - -[[package]] -name = "memchr" -version = "2.8.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f8ca58f447f06ed17d5fc4043ce1b10dd205e060fb3ce5b979b8ed8e59ff3f79" - -[[package]] -name = "multimap" -version = "0.10.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1d87ecb2933e8aeadb3e3a02b828fed80a7528047e68b4f424523a0981a3a084" - -[[package]] -name = "once_cell" -version = "1.21.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9f7c3e4beb33f85d45ae3e3a1792185706c8e16d043238c593331cc7cd313b50" - -[[package]] -name = "petgraph" -version = "0.7.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3672b37090dbd86368a4145bc067582552b29c27377cad4e0a306c97f9bd7772" -dependencies = [ - "fixedbitset", - "indexmap", -] - -[[package]] -name = "prettyplease" -version = "0.2.37" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "479ca8adacdd7ce8f1fb39ce9ecccbfe93a3f1344b3d0d97f20bc0196208f62b" -dependencies = [ - "proc-macro2", - "syn", -] - -[[package]] -name = "proc-macro2" -version = "1.0.106" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8fd00f0bb2e90d81d1044c2b32617f68fcb9fa3bb7640c23e9c748e53fb30934" -dependencies = [ - "unicode-ident", -] - -[[package]] -name = "prost" -version = "0.13.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2796faa41db3ec313a31f7624d9286acf277b52de526150b7e69f3debf891ee5" -dependencies = [ - "bytes", - "prost-derive", -] - -[[package]] -name = "prost-build" -version = "0.13.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "be769465445e8c1474e9c5dac2018218498557af32d9ed057325ec9a41ae81bf" -dependencies = [ - "heck", - "itertools", - "log", - "multimap", - "once_cell", - "petgraph", - "prettyplease", - "prost", - "prost-types", - "regex", - "syn", - "tempfile", -] - -[[package]] -name = "prost-derive" -version = "0.13.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8a56d757972c98b346a9b766e3f02746cde6dd1cd1d1d563472929fdd74bec4d" -dependencies = [ - "anyhow", - "itertools", - "proc-macro2", - "quote", - "syn", -] - -[[package]] -name = "prost-types" -version = "0.13.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "52c2c1bf36ddb1a1c396b3601a3cec27c2462e45f07c386894ec3ccf5332bd16" -dependencies = [ - "prost", -] - -[[package]] -name = "protoc-bin-vendored" -version = "3.2.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d1c381df33c98266b5f08186583660090a4ffa0889e76c7e9a5e175f645a67fa" -dependencies = [ - "protoc-bin-vendored-linux-aarch_64", - "protoc-bin-vendored-linux-ppcle_64", - "protoc-bin-vendored-linux-s390_64", - "protoc-bin-vendored-linux-x86_32", - "protoc-bin-vendored-linux-x86_64", - "protoc-bin-vendored-macos-aarch_64", - "protoc-bin-vendored-macos-x86_64", - "protoc-bin-vendored-win32", -] - -[[package]] -name = "protoc-bin-vendored-linux-aarch_64" -version = "3.2.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c350df4d49b5b9e3ca79f7e646fde2377b199e13cfa87320308397e1f37e1a4c" - -[[package]] -name = "protoc-bin-vendored-linux-ppcle_64" -version = "3.2.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a55a63e6c7244f19b5c6393f025017eb5d793fd5467823a099740a7a4222440c" - -[[package]] -name = "protoc-bin-vendored-linux-s390_64" -version = "3.2.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1dba5565db4288e935d5330a07c264a4ee8e4a5b4a4e6f4e83fad824cc32f3b0" - -[[package]] -name = "protoc-bin-vendored-linux-x86_32" -version = "3.2.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8854774b24ee28b7868cd71dccaae8e02a2365e67a4a87a6cd11ee6cdbdf9cf5" - -[[package]] -name = "protoc-bin-vendored-linux-x86_64" -version = "3.2.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b38b07546580df720fa464ce124c4b03630a6fb83e05c336fea2a241df7e5d78" - -[[package]] -name = "protoc-bin-vendored-macos-aarch_64" -version = "3.2.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "89278a9926ce312e51f1d999fee8825d324d603213344a9a706daa009f1d8092" - -[[package]] -name = "protoc-bin-vendored-macos-x86_64" -version = "3.2.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "81745feda7ccfb9471d7a4de888f0652e806d5795b61480605d4943176299756" - -[[package]] -name = "protoc-bin-vendored-win32" -version = "3.2.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "95067976aca6421a523e491fce939a3e65249bac4b977adee0ee9771568e8aa3" - -[[package]] -name = "quote" -version = "1.0.45" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "41f2619966050689382d2b44f664f4bc593e129785a36d6ee376ddf37259b924" -dependencies = [ - "proc-macro2", -] - -[[package]] -name = "r-efi" -version = "6.0.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f8dcc9c7d52a811697d2151c701e0d08956f92b0e24136cf4cf27b57a6a0d9bf" - -[[package]] -name = "regex" -version = "1.12.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e10754a14b9137dd7b1e3e5b0493cc9171fdd105e0ab477f51b72e7f3ac0e276" -dependencies = [ - "aho-corasick", - "memchr", - "regex-automata", - "regex-syntax", -] - -[[package]] -name = "regex-automata" -version = "0.4.14" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6e1dd4122fc1595e8162618945476892eefca7b88c52820e74af6262213cae8f" -dependencies = [ - "aho-corasick", - "memchr", - "regex-syntax", -] - -[[package]] -name = "regex-syntax" -version = "0.8.10" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "dc897dd8d9e8bd1ed8cdad82b5966c3e0ecae09fb1907d58efaa013543185d0a" - -[[package]] -name = "rustix" -version = "1.1.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b6fe4565b9518b83ef4f91bb47ce29620ca828bd32cb7e408f0062e9930ba190" -dependencies = [ - "bitflags", - "errno", - "libc", - "linux-raw-sys", - "windows-sys", -] - -[[package]] -name = "semver" -version = "1.0.28" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8a7852d02fc848982e0c167ef163aaff9cd91dc640ba85e263cb1ce46fae51cd" - -[[package]] -name = "serde" -version = "1.0.228" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9a8e94ea7f378bd32cbbd37198a4a91436180c5bb472411e48b5ec2e2124ae9e" -dependencies = [ - "serde_core", -] - -[[package]] -name = "serde_core" -version = "1.0.228" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "41d385c7d4ca58e59fc732af25c3983b67ac852c1a25000afe1175de458b67ad" -dependencies = [ - "serde_derive", -] - -[[package]] -name = "serde_derive" -version = "1.0.228" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d540f220d3187173da220f885ab66608367b6574e925011a9353e4badda91d79" -dependencies = [ - "proc-macro2", - "quote", - "syn", -] - -[[package]] -name = "serde_json" -version = "1.0.149" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "83fc039473c5595ace860d8c4fafa220ff474b3fc6bfdb4293327f1a37e94d86" -dependencies = [ - "itoa", - "memchr", - "serde", - "serde_core", - "zmij", -] - -[[package]] -name = "syn" -version = "2.0.117" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e665b8803e7b1d2a727f4023456bbbbe74da67099c585258af0ad9c5013b9b99" -dependencies = [ - "proc-macro2", - "quote", - "unicode-ident", -] - -[[package]] -name = "tempfile" -version = "3.27.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "32497e9a4c7b38532efcdebeef879707aa9f794296a4f0244f6f69e9bc8574bd" -dependencies = [ - "fastrand", - "getrandom", - "once_cell", - "rustix", - "windows-sys", -] - -[[package]] -name = "unicode-ident" -version = "1.0.24" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e6e4313cd5fcd3dad5cafa179702e2b244f760991f45397d14d4ebf38247da75" - -[[package]] -name = "unicode-xid" -version = "0.2.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ebc1c04c71510c7f702b52b7c350734c9ff1295c464a03335b00bb84fc54f853" - -[[package]] -name = "wasip2" -version = "1.0.3+wasi-0.2.9" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "20064672db26d7cdc89c7798c48a0fdfac8213434a1186e5ef29fd560ae223d6" -dependencies = [ - "wit-bindgen 0.57.1", -] - -[[package]] -name = "wasip3" -version = "0.4.0+wasi-0.3.0-rc-2026-01-06" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5428f8bf88ea5ddc08faddef2ac4a67e390b88186c703ce6dbd955e1c145aca5" -dependencies = [ - "wit-bindgen 0.51.0", -] - -[[package]] -name = "wasm-encoder" -version = "0.244.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "990065f2fe63003fe337b932cfb5e3b80e0b4d0f5ff650e6985b1048f62c8319" -dependencies = [ - "leb128fmt", - "wasmparser", -] - -[[package]] -name = "wasm-metadata" -version = "0.244.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "bb0e353e6a2fbdc176932bbaab493762eb1255a7900fe0fea1a2f96c296cc909" -dependencies = [ - "anyhow", - "indexmap", - "wasm-encoder", - "wasmparser", -] - -[[package]] -name = "wasmparser" -version = "0.244.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "47b807c72e1bac69382b3a6fb3dbe8ea4c0ed87ff5629b8685ae6b9a611028fe" -dependencies = [ - "bitflags", - "hashbrown 0.15.5", - "indexmap", - "semver", -] - -[[package]] -name = "windows-link" -version = "0.2.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f0805222e57f7521d6a62e36fa9163bc891acd422f971defe97d64e70d0a4fe5" - -[[package]] -name = "windows-sys" -version = "0.61.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ae137229bcbd6cdf0f7b80a31df61766145077ddf49416a728b02cb3921ff3fc" -dependencies = [ - "windows-link", -] - -[[package]] -name = "wit-bindgen" -version = "0.51.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d7249219f66ced02969388cf2bb044a09756a083d0fab1e566056b04d9fbcaa5" -dependencies = [ - "wit-bindgen-rust-macro", -] - -[[package]] -name = "wit-bindgen" -version = "0.57.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1ebf944e87a7c253233ad6766e082e3cd714b5d03812acc24c318f549614536e" - -[[package]] -name = "wit-bindgen-core" -version = "0.51.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ea61de684c3ea68cb082b7a88508a8b27fcc8b797d738bfc99a82facf1d752dc" -dependencies = [ - "anyhow", - "heck", - "wit-parser", -] - -[[package]] -name = "wit-bindgen-rust" -version = "0.51.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b7c566e0f4b284dd6561c786d9cb0142da491f46a9fbed79ea69cdad5db17f21" -dependencies = [ - "anyhow", - "heck", - "indexmap", - "prettyplease", - "syn", - "wasm-metadata", - "wit-bindgen-core", - "wit-component", -] - -[[package]] -name = "wit-bindgen-rust-macro" -version = "0.51.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0c0f9bfd77e6a48eccf51359e3ae77140a7f50b1e2ebfe62422d8afdaffab17a" -dependencies = [ - "anyhow", - "prettyplease", - "proc-macro2", - "quote", - "syn", - "wit-bindgen-core", - "wit-bindgen-rust", -] - -[[package]] -name = "wit-component" -version = "0.244.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9d66ea20e9553b30172b5e831994e35fbde2d165325bec84fc43dbf6f4eb9cb2" -dependencies = [ - "anyhow", - "bitflags", - "indexmap", - "log", - "serde", - "serde_derive", - "serde_json", - "wasm-encoder", - "wasm-metadata", - "wasmparser", - "wit-parser", -] - -[[package]] -name = "wit-parser" -version = "0.244.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ecc8ac4bc1dc3381b7f59c34f00b67e18f910c2c0f50015669dde7def656a736" -dependencies = [ - "anyhow", - "id-arena", - "indexmap", - "log", - "semver", - "serde", - "serde_derive", - "serde_json", - "unicode-xid", - "wasmparser", -] - -[[package]] -name = "zmij" -version = "1.0.21" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b8848ee67ecc8aedbaf3e4122217aff892639231befc6a1b58d29fff4c2cabaa" diff --git a/tools/gen-proto/Cargo.toml b/tools/gen-proto/Cargo.toml deleted file mode 100644 index 0cc4ec49..00000000 --- a/tools/gen-proto/Cargo.toml +++ /dev/null @@ -1,15 +0,0 @@ -[package] -name = "gen-proto" -version = "0.0.0" -edition = "2024" -publish = false -description = "Internal maintainer tool that regenerates src/proto/generated/ from proto/*.proto." - -[dependencies] -prost-build = "0.13" -protoc-bin-vendored = "3" - -# Standalone workspace so this crate is fully isolated from the parent -# `asap_sketchlib` package. It must not appear in the published crate's -# build graph or Cargo.lock. -[workspace] diff --git a/tools/gen-proto/src/main.rs b/tools/gen-proto/src/main.rs deleted file mode 100644 index 7285b917..00000000 --- a/tools/gen-proto/src/main.rs +++ /dev/null @@ -1,90 +0,0 @@ -//! Maintainer-only tool that regenerates the vendored prost output under -//! `src/proto/generated/` from the `.proto` files in `proto/`. -//! -//! Run from the repository root with: -//! -//! ```bash -//! cargo run --manifest-path tools/gen-proto/Cargo.toml -//! ``` -//! -//! CI enforces that the committed output matches the result of running this -//! tool (see `.github/workflows/ci.yml`). - -use std::fs; -use std::path::{Path, PathBuf}; - -// Inner attributes (`#![...]`) cannot appear inside an `include!` expansion, -// so this header is restricted to comments. Module-level attributes such as -// `#[rustfmt::skip]` and `#[allow(clippy::all)]` are applied to the wrapping -// `pub mod sketchlib { ... }` in `src/proto.rs` instead. -const HEADER: &str = "\ -// @generated by tools/gen-proto -- DO NOT EDIT. -// -// Regenerate with: -// cargo run --manifest-path tools/gen-proto/Cargo.toml -// -// Source of truth: proto/**/*.proto -"; - -const PROTO_FILES: &[&str] = &[ - "common/common.proto", - "countminsketch/countminsketch.proto", - "countsketch/countsketch.proto", - "hll/hll.proto", - "kll/kll.proto", - "univmon/univmon.proto", - "sketchlib.proto", -]; - -fn main() { - let manifest_dir = PathBuf::from(env!("CARGO_MANIFEST_DIR")); - let repo_root = manifest_dir - .parent() - .and_then(Path::parent) - .expect("tools/gen-proto must live two levels below the repo root"); - - let proto_dir = repo_root.join("proto"); - let out_dir = repo_root.join("src/proto/generated"); - - fs::create_dir_all(&out_dir).expect("failed to create generated proto directory"); - - let protoc = - protoc_bin_vendored::protoc_bin_path().expect("failed to locate vendored protoc binary"); - // SAFETY: single-threaded maintainer tool; setting PROTOC before invoking - // prost-build is the documented way to override the protoc binary. - unsafe { - std::env::set_var("PROTOC", protoc); - } - - let protos: Vec = PROTO_FILES.iter().map(|p| proto_dir.join(p)).collect(); - - prost_build::Config::new() - .out_dir(&out_dir) - .compile_protos(&protos, &[proto_dir.clone()]) - .expect("prost_build failed to compile proto files"); - - prepend_header_to_generated(&out_dir); - - eprintln!("Generated proto code under {}", out_dir.display()); -} - -fn prepend_header_to_generated(out_dir: &Path) { - for entry in fs::read_dir(out_dir).expect("failed to read generated directory") { - let entry = entry.expect("failed to iterate generated directory"); - let path = entry.path(); - if path.extension().and_then(|e| e.to_str()) != Some("rs") { - continue; - } - let body = fs::read_to_string(&path) - .unwrap_or_else(|err| panic!("failed to read {}: {err}", path.display())); - if body.starts_with(HEADER) { - continue; - } - let mut with_header = String::with_capacity(HEADER.len() + body.len() + 1); - with_header.push_str(HEADER); - with_header.push('\n'); - with_header.push_str(&body); - fs::write(&path, with_header) - .unwrap_or_else(|err| panic!("failed to write {}: {err}", path.display())); - } -} From 315e2bf03ad8e33cd07e2f8c4a47836cee3eef46 Mon Sep 17 00:00:00 2001 From: GordonYuanyc Date: Fri, 2 Oct 2026 01:53:15 -0600 Subject: [PATCH 32/36] build: drop the unused xxhash-rust dependency Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 1 + Cargo.lock | 7 ------- Cargo.toml | 1 - 3 files changed, 1 insertion(+), 8 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index e14b8edb..eb047bb6 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -282,6 +282,7 @@ signals a backwards-compatible change. sources under `proto/`, the vendored `src/proto/generated/`, the `tools/gen-proto` generator and its CI drift check; and the `prost` and `bytes` dependencies. +- **The `xxhash-rust` dependency**, which nothing uses. ### Fixed diff --git a/Cargo.lock b/Cargo.lock index 7291bf73..b6682e0d 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -16,7 +16,6 @@ dependencies = [ "serde_bytes", "smallvec", "twox-hash", - "xxhash-rust", ] [[package]] @@ -486,12 +485,6 @@ version = "0.46.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "f17a85883d4e6d00e8a97c586de764dabcc06133f7f1d55dce5cdc070ad7fe59" -[[package]] -name = "xxhash-rust" -version = "0.8.15" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "fdd20c5420375476fbd4394763288da7eb0cc0b8c11deed431a91562af7335d3" - [[package]] name = "zerocopy" version = "0.8.27" diff --git a/Cargo.toml b/Cargo.toml index fa89c4e1..dbafd6ae 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -24,7 +24,6 @@ serde_bytes = "0.11" rmp-serde = "1.3.0" twox-hash = "2.1.2" smallvec = "1.13.2" -xxhash-rust = { version = "0.8", features = ["xxh32"] } core_affinity = { version = "0.8", optional = true } crossbeam-channel = { version = "0.5", optional = true } From 305cbfc61f291506f786e4c3767a1409e93bbf92 Mon Sep 17 00:00:00 2001 From: GordonYuanyc Date: Fri, 2 Oct 2026 01:54:44 -0600 Subject: [PATCH 33/36] refactor!: move the ASAPv1 framing to a private asapv1 module message_pack_format held only the crate-private envelope and wire-key code; it now lives in src/asapv1/ and the public module is gone. Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 7 ++++--- docs/asapv1_wire_format.md | 4 ++-- docs/library_map.md | 8 +++----- docs/tests.md | 2 +- src/{message_pack_format => asapv1}/envelope.rs | 0 src/asapv1/mod.rs | 13 +++++++++++++ src/{message_pack_format => asapv1}/wire_key.rs | 0 src/lib.rs | 5 ++--- src/message_pack_format/mod.rs | 13 ------------- src/sketch_framework/eh/wire.rs | 2 +- src/sketch_framework/eh_sketch_list/wire.rs | 2 +- src/sketch_framework/hydra/wire.rs | 6 +++--- src/sketch_framework/univmon/wire.rs | 4 ++-- src/sketch_framework/univmon_optimized/wire.rs | 2 +- src/sketch_framework/univmon_q/wire.rs | 2 +- src/sketches/bloom/wire.rs | 2 +- src/sketches/coco/wire.rs | 2 +- src/sketches/countminsketch/wire.rs | 2 +- src/sketches/countminsketch_topk/heap_wire.rs | 4 ++-- src/sketches/countminsketch_topk/wire.rs | 2 +- src/sketches/countsketch/wire.rs | 2 +- src/sketches/countsketch_topk/l2hh_wire.rs | 2 +- src/sketches/countsketch_topk/wire.rs | 2 +- src/sketches/ddsketch/wire.rs | 2 +- src/sketches/elastic/wire.rs | 2 +- src/sketches/hll/wire.rs | 2 +- src/sketches/kll/wire.rs | 2 +- src/sketches/kll_dynamic/wire.rs | 4 ++-- src/sketches/kmv/wire.rs | 2 +- src/sketches/set_aggregator/wire.rs | 4 ++-- src/sketches/space_saving/wire.rs | 4 ++-- src/sketches/uniform/wire.rs | 2 +- 32 files changed, 55 insertions(+), 57 deletions(-) rename src/{message_pack_format => asapv1}/envelope.rs (100%) create mode 100644 src/asapv1/mod.rs rename src/{message_pack_format => asapv1}/wire_key.rs (100%) delete mode 100644 src/message_pack_format/mod.rs diff --git a/CHANGELOG.md b/CHANGELOG.md index eb047bb6..c3b9daa9 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -266,14 +266,15 @@ signals a backwards-compatible change. `message_pack_format::portable::delta_set_aggregator`, both types' `MessagePackCodec` impls, and `DeltaResult`'s `serde::Serialize` / `Deserialize` derives. They serialize only as ASAPv1. -- **`MessagePackCodec` and the `portable` and `native` modules.** Removed +- **`MessagePackCodec` and the public `message_pack_format` module.** Removed `message_pack_format::{MessagePackCodec, Error}` and the crate-root `MessagePackCodec`; `message_pack_format::native`, with the `MessagePackCodec` impl for `KMV`; and `message_pack_format::portable`, with `sampling` and its crate-root re-exports `effective_sample_p`, `sample_p_or_default`, `rescale_count`, `rescale_count_with_env` and `is_quantile_scale_invariant`. - Every sketch encodes through `serialize_to_bytes` / `deserialize_from_bytes`; - ASAPv1 carries no sampling probability. + The ASAPv1 framing that module also held is crate-private. Every sketch + encodes through `serialize_to_bytes` / `deserialize_from_bytes`; ASAPv1 + carries no sampling probability. - **The protobuf layer.** Removed the public module `proto::sketchlib` with `SketchEnvelope`, `ProducerInfo`, `HashSpec`, `HashAlgorithm`, `SeedDerivation`, `CounterType` and the state messages no envelope field diff --git a/docs/asapv1_wire_format.md b/docs/asapv1_wire_format.md index a2c3a244..e7b1c7b5 100644 --- a/docs/asapv1_wire_format.md +++ b/docs/asapv1_wire_format.md @@ -26,7 +26,7 @@ If the doc feels long, these sections carry the key points: ## Status -- **Implemented (Rust).** Every kind_id the Section 1 registry marks *implemented* serializes through the shared `message_pack_format::envelope` module per this spec: HLL (three estimators), Count-Min, CMSHeap, Count Sketch, CSHeap, DDSketch, KLL (compact + dynamic), Hydra (five counter variants), Elastic, Coco, UniformSampling, KMV, UnivMon, UnivMon Optimized, ExponentialHistogram, EHSketchList, Bloom, Space-Saving, CountL2HH, UnivMon-Q, SetAggregator and DeltaResult. +- **Implemented (Rust).** Every kind_id the Section 1 registry marks *implemented* serializes through the shared `asapv1::envelope` module per this spec: HLL (three estimators), Count-Min, CMSHeap, Count Sketch, CSHeap, DDSketch, KLL (compact + dynamic), Hydra (five counter variants), Elastic, Coco, UniformSampling, KMV, UnivMon, UnivMon Optimized, ExponentialHistogram, EHSketchList, Bloom, Space-Saving, CountL2HH, UnivMon-Q, SetAggregator and DeltaResult. - **Self-describing.** The hash-spec metadata is derived from the hasher's `HashProfile` (read live, never hardcoded), so the bytes truthfully describe how a sketch was hashed; custom hash profiles are supported (Section 2). - **Byte-level encoding** is pinned in Section 4; the resolved decisions are summarized at the end. - **`sketchlib-go`** is aligned separately (see Cross-language contract). @@ -239,7 +239,7 @@ This registry is the master list of algorithms still to design payloads for, and 7. Read exactly `payload_len` bytes; hand to the per-sketch payload decoder. 8. Fail **closed** on any inconsistency; never merge or query a sketch whose hash spec did not validate. -> Implementation note: the shared envelope module (`src/message_pack_format/envelope.rs`) owns rules 1-3 and the byte framing (`encode` / `split`); it is sketch-agnostic and does not know the registry. +> Implementation note: the shared envelope module (`src/asapv1/envelope.rs`) owns rules 1-3 and the byte framing (`encode` / `split`); it is sketch-agnostic and does not know the registry. > Rule 4 (and metadata/kind_id validation) happens in each sketch's decoder, which checks the `kind_id` against the ones it owns. --- diff --git a/docs/library_map.md b/docs/library_map.md index 4d813faf..0a25f974 100644 --- a/docs/library_map.md +++ b/docs/library_map.md @@ -23,11 +23,9 @@ - Not in the API index: `tumbling.rs`, `sketch_catalog.rs` - The same `.rs` + `/wire.rs` split carries the ASAPv1 wire format for `hydra.rs`, `univmon.rs`, `univmon_optimized.rs`, `univmon_q.rs`, `eh.rs` and `eh_sketch_list.rs` -- **`src/message_pack_format/`** - Serialization plumbing ([message_pack_format.md](./message_pack_format.md)). The current format is **ASAPv1**, specified in [asapv1_wire_format.md](./asapv1_wire_format.md) - - `envelope.rs` — the shared, sketch-agnostic ASAPv1 framing (magic/version/`kind_id` + length prefixes, `encode`/`split`); every `wire.rs` under `src/sketches/` and `src/sketch_framework/` calls into it - - `codec.rs` — the `MessagePackCodec` trait; `error.rs` — the unified `Error` - - `portable/` — **deprecated**, being retired. The older per-sketch wire types (`CountSketch`, `DdSketch`, …); ASAPv1 (per-sketch `wire.rs`) is now what `sketchlib-go` mirrors, not these - - `native/` — **deprecated**, being retired. Older `MessagePackCodec` shims over `src/sketches/` byte serialization +- **`src/asapv1/`** - Crate-private ASAPv1 framing, specified in [asapv1_wire_format.md](./asapv1_wire_format.md) + - `envelope.rs` — the shared, sketch-agnostic framing (magic/version/`kind_id` + length prefixes, `encode`/`split`); every `wire.rs` under `src/sketches/` and `src/sketch_framework/` calls into it + - `wire_key.rs` — the msgpack `bin` / `str` form of byte and string heap keys ## Documentation diff --git a/docs/tests.md b/docs/tests.md index 678fa955..c3f46175 100644 --- a/docs/tests.md +++ b/docs/tests.md @@ -1380,7 +1380,7 @@ Test file: [`src/common/structures/matrix_storage.rs`](../src/common/structures/ ### ASAPv1 Envelope (MessagePack) -Test file: [`src/message_pack_format/envelope.rs`](../src/message_pack_format/envelope.rs) +Test file: [`src/asapv1/envelope.rs`](../src/asapv1/envelope.rs) | test_name | test_description | what_is_tested | | --- | --- | --- | diff --git a/src/message_pack_format/envelope.rs b/src/asapv1/envelope.rs similarity index 100% rename from src/message_pack_format/envelope.rs rename to src/asapv1/envelope.rs diff --git a/src/asapv1/mod.rs b/src/asapv1/mod.rs new file mode 100644 index 00000000..b631dc46 --- /dev/null +++ b/src/asapv1/mod.rs @@ -0,0 +1,13 @@ +//! ASAPv1 framing shared by every sketch's `wire.rs`. +//! +//! Each sketch serializes into one self-delimiting envelope: a sketch-agnostic +//! frame (magic, version, `kind_id`, two length prefixes) around a metadata map +//! and a payload array. `envelope` holds the frame and `wire_key` the msgpack +//! form of byte and string keys; the `kind_id`, metadata and payload are +//! per-sketch, in the `wire.rs` beside each sketch under [`crate::sketches`] +//! and [`crate::sketch_framework`]. `sketchlib-go` mirrors ASAPv1, +//! `asapv1_golden/` guards against drift, and `docs/asapv1_wire_format.md` is +//! the spec. + +pub(crate) mod envelope; +pub(crate) mod wire_key; diff --git a/src/message_pack_format/wire_key.rs b/src/asapv1/wire_key.rs similarity index 100% rename from src/message_pack_format/wire_key.rs rename to src/asapv1/wire_key.rs diff --git a/src/lib.rs b/src/lib.rs index f98ac5a4..e78ef448 100644 --- a/src/lib.rs +++ b/src/lib.rs @@ -7,18 +7,17 @@ //! reusable utilities used across sketches. //! - [`sketch_framework`]: higher-level composition layers such as Hydra, //! UnivMon, tumbling windows, and batch/parallel execution helpers. -//! - [`message_pack_format`]: MessagePack/proto wire format shared with -//! `sketchlib-go`. //! //! Most users can start with the crate-root re-exports such as [`DataInput`], //! [`CountMin`], [`HyperLogLog`], [`KLL`], and [`DDSketch`]. Reach for the //! submodules directly when you need lower-level storage, hashing, or //! framework-specific APIs. +/// ASAPv1 envelope framing shared by every sketch's serialization. +mod asapv1; /// Shared primitives used across sketches, including input wrappers, hashers, /// storage backends, and reusable utilities. pub mod common; -pub mod message_pack_format; /// Higher-level composition layers such as Hydra, UnivMon, tumbling windows, /// and batch/parallel execution helpers. pub mod sketch_framework; diff --git a/src/message_pack_format/mod.rs b/src/message_pack_format/mod.rs deleted file mode 100644 index 54d603a5..00000000 --- a/src/message_pack_format/mod.rs +++ /dev/null @@ -1,13 +0,0 @@ -//! Rust-side MessagePack serialization plumbing. -//! -//! The current format is **ASAPv1**: each sketch serializes into one -//! self-delimiting envelope — a sketch-agnostic frame (magic, version, -//! `kind_id`, two length prefixes) around a metadata map and a payload -//! array. The framing lives in `envelope.rs`; the `kind_id`, metadata and -//! payload are per-sketch, in the `wire.rs` beside each sketch under -//! [`crate::sketches`] and [`crate::sketch_framework`]. `sketchlib-go` -//! mirrors ASAPv1, `asapv1_golden/` guards against drift, and -//! `docs/asapv1_wire_format.md` is the spec. - -pub(crate) mod envelope; -pub(crate) mod wire_key; diff --git a/src/sketch_framework/eh/wire.rs b/src/sketch_framework/eh/wire.rs index 5ed566dd..2608657a 100644 --- a/src/sketch_framework/eh/wire.rs +++ b/src/sketch_framework/eh/wire.rs @@ -30,7 +30,7 @@ use rmp_serde::{decode::Error as RmpDecodeError, encode::Error as RmpEncodeError, from_slice}; use serde::{Deserialize, Serialize}; -use crate::message_pack_format::envelope; +use crate::asapv1::envelope; use crate::sketch_framework::eh_sketch_list::wire::{SketchState, rebuild_sketch, sketch_state}; use super::{EHBucket, ExponentialHistogram, compute_l2_mass, infer_merge_norm}; diff --git a/src/sketch_framework/eh_sketch_list/wire.rs b/src/sketch_framework/eh_sketch_list/wire.rs index 16b52130..a0f1f6c7 100644 --- a/src/sketch_framework/eh_sketch_list/wire.rs +++ b/src/sketch_framework/eh_sketch_list/wire.rs @@ -30,7 +30,7 @@ use rmp_serde::{decode::Error as RmpDecodeError, encode::Error as RmpEncodeError, from_slice}; use serde::{Deserialize, Serialize}; -use crate::message_pack_format::envelope; +use crate::asapv1::envelope; use super::EHSketchList; diff --git a/src/sketch_framework/hydra/wire.rs b/src/sketch_framework/hydra/wire.rs index a14951a3..c2beeb47 100644 --- a/src/sketch_framework/hydra/wire.rs +++ b/src/sketch_framework/hydra/wire.rs @@ -44,9 +44,9 @@ use rmp_serde::{decode::Error as RmpDecodeError, encode::Error as RmpEncodeError use serde::de::IgnoredAny; use serde::{Deserialize, Serialize}; +use crate::asapv1::envelope; use crate::common::hash::check_matrix_rows; use crate::input::HydraCounter; -use crate::message_pack_format::envelope; use crate::sketch_framework::univmon::wire::{PyramidPayload, UnivMonMetadata, univmon_metadata}; use crate::{ Count, CountMin, DefaultXxHasher, ErtlMLE, FastPath, HashProfile, HllBucketListP14, @@ -1118,7 +1118,7 @@ fn pack_univmon_cells(key_type: &str, payloads: &[&[u8]]) -> Result, Rmp "f32" => pack!(f32), "f64" => pack!(f64), "string" => pack!(String), - "bytes" => pack!(crate::message_pack_format::wire_key::WireBytes), + "bytes" => pack!(crate::asapv1::wire_key::WireBytes), other => Err(encode_error(format!( "ASAPv1 Hydra: key_type {other:?} is not a wire key type" ))), @@ -1153,7 +1153,7 @@ fn unpack_univmon_cells(key_type: &str, payload: &[u8]) -> Result>, "f32" => unpack!(f32), "f64" => unpack!(f64), "string" => unpack!(String), - "bytes" => unpack!(crate::message_pack_format::wire_key::WireBytes), + "bytes" => unpack!(crate::asapv1::wire_key::WireBytes), other => Err(decode_error(format!( "ASAPv1 Hydra: key_type {other:?} is not a wire key type" ))), diff --git a/src/sketch_framework/univmon/wire.rs b/src/sketch_framework/univmon/wire.rs index 1b3f3ebe..179c474c 100644 --- a/src/sketch_framework/univmon/wire.rs +++ b/src/sketch_framework/univmon/wire.rs @@ -34,8 +34,8 @@ use rmp_serde::{decode::Error as RmpDecodeError, encode::Error as RmpEncodeError, from_slice}; use serde::{Deserialize, Serialize}; -use crate::message_pack_format::envelope; -use crate::message_pack_format::wire_key::WireBytes; +use crate::asapv1::envelope; +use crate::asapv1::wire_key::WireBytes; use crate::sketches::countminsketch_topk::heap_wire::{ EMPTY_KEY_TYPE, check_distinct_keys, heap_entries, key_type_of, rebuild_heap, }; diff --git a/src/sketch_framework/univmon_optimized/wire.rs b/src/sketch_framework/univmon_optimized/wire.rs index ae50b9e8..cfb9462a 100644 --- a/src/sketch_framework/univmon_optimized/wire.rs +++ b/src/sketch_framework/univmon_optimized/wire.rs @@ -21,7 +21,7 @@ use rmp_serde::{decode::Error as RmpDecodeError, encode::Error as RmpEncodeError, from_slice}; use serde::{Deserialize, Serialize}; -use crate::message_pack_format::envelope; +use crate::asapv1::envelope; use crate::sketch_framework::univmon::wire::{ check_layer_size, decode_pyramid, encode_pyramid, pyramid_key_type, pyramid_state, rebuild_layers, update_mode_of, update_mode_tag, diff --git a/src/sketch_framework/univmon_q/wire.rs b/src/sketch_framework/univmon_q/wire.rs index 35614917..76f9cee4 100644 --- a/src/sketch_framework/univmon_q/wire.rs +++ b/src/sketch_framework/univmon_q/wire.rs @@ -34,7 +34,7 @@ use std::marker::PhantomData; use rmp_serde::{decode::Error as RmpDecodeError, encode::Error as RmpEncodeError, from_slice}; use serde::{Deserialize, Serialize}; -use crate::message_pack_format::envelope; +use crate::asapv1::envelope; use crate::{HashProfile, SketchHasher}; use super::{ diff --git a/src/sketches/bloom/wire.rs b/src/sketches/bloom/wire.rs index 3b40f570..e67e0fbc 100644 --- a/src/sketches/bloom/wire.rs +++ b/src/sketches/bloom/wire.rs @@ -25,7 +25,7 @@ use rmp_serde::{decode::Error as RmpDecodeError, encode::Error as RmpEncodeError use serde::{Deserialize, Serialize}; use std::marker::PhantomData; -use crate::message_pack_format::envelope; +use crate::asapv1::envelope; use crate::{BitMatrix, HashProfile, SketchHasher}; use super::{BLOOM_MAX_BITS, BLOOM_MAX_SLICES, Bloom, BloomMode}; diff --git a/src/sketches/coco/wire.rs b/src/sketches/coco/wire.rs index d5577de6..c4764a07 100644 --- a/src/sketches/coco/wire.rs +++ b/src/sketches/coco/wire.rs @@ -39,8 +39,8 @@ use serde::{Deserialize, Serialize}; use std::collections::HashSet; +use crate::asapv1::envelope; use crate::common::hash::check_matrix_rows; -use crate::message_pack_format::envelope; use crate::{DataInput, HashProfile, SketchHasher, Vector2D}; use super::{Coco, CocoBucket, CocoRng}; diff --git a/src/sketches/countminsketch/wire.rs b/src/sketches/countminsketch/wire.rs index 8899df5b..75819c2b 100644 --- a/src/sketches/countminsketch/wire.rs +++ b/src/sketches/countminsketch/wire.rs @@ -20,8 +20,8 @@ use rmp_serde::{decode::Error as RmpDecodeError, encode::Error as RmpEncodeError, from_slice}; use serde::{Deserialize, Serialize}; +use crate::asapv1::envelope; use crate::common::hash::check_matrix_rows; -use crate::message_pack_format::envelope; use crate::{FastPath, HashProfile, RegularPath, SketchHasher, Vector2D}; use super::CountMin; diff --git a/src/sketches/countminsketch_topk/heap_wire.rs b/src/sketches/countminsketch_topk/heap_wire.rs index 46ab0a45..91b9142b 100644 --- a/src/sketches/countminsketch_topk/heap_wire.rs +++ b/src/sketches/countminsketch_topk/heap_wire.rs @@ -45,7 +45,7 @@ use rmp_serde::{decode::Error as RmpDecodeError, encode::Error as RmpEncodeError use serde::{Deserialize, Serialize}; use std::collections::HashSet; -use crate::message_pack_format::wire_key::WireBytes; +use crate::asapv1::wire_key::WireBytes; use crate::{HHHeap, HashProfile, HeapItem}; /// Metadata `key_type` of a heap that holds nothing. A heap with no entries has @@ -401,7 +401,7 @@ pub(crate) fn rebuild_heap( #[cfg(test)] mod tests { use super::*; - use crate::message_pack_format::envelope; + use crate::asapv1::envelope; use crate::{DataInput, DefaultXxHasher, RegularPath, Vector2D}; use super::super::CMSHeap; diff --git a/src/sketches/countminsketch_topk/wire.rs b/src/sketches/countminsketch_topk/wire.rs index e282d61e..8812082f 100644 --- a/src/sketches/countminsketch_topk/wire.rs +++ b/src/sketches/countminsketch_topk/wire.rs @@ -18,8 +18,8 @@ use rmp_serde::{decode::Error as RmpDecodeError, encode::Error as RmpEncodeError, from_slice}; use serde::{Deserialize, Serialize}; +use crate::asapv1::envelope; use crate::common::hash::check_matrix_rows; -use crate::message_pack_format::envelope; use crate::sketches::countminsketch::{CmsWireCounter, CmsWireMode}; use crate::{CountMin, HashProfile, SketchHasher, Vector2D}; diff --git a/src/sketches/countsketch/wire.rs b/src/sketches/countsketch/wire.rs index dcb5f4ee..06e35f9a 100644 --- a/src/sketches/countsketch/wire.rs +++ b/src/sketches/countsketch/wire.rs @@ -27,8 +27,8 @@ use rmp_serde::{decode::Error as RmpDecodeError, encode::Error as RmpEncodeError, from_slice}; use serde::{Deserialize, Serialize}; +use crate::asapv1::envelope; use crate::common::hash::check_matrix_rows; -use crate::message_pack_format::envelope; use crate::{FastPath, HashProfile, RegularPath, SketchHasher, Vector2D}; use super::{Count, CountSketchCounter}; diff --git a/src/sketches/countsketch_topk/l2hh_wire.rs b/src/sketches/countsketch_topk/l2hh_wire.rs index 50c1acd6..cc6a4efb 100644 --- a/src/sketches/countsketch_topk/l2hh_wire.rs +++ b/src/sketches/countsketch_topk/l2hh_wire.rs @@ -31,9 +31,9 @@ use rmp_serde::{decode::Error as RmpDecodeError, encode::Error as RmpEncodeError use serde::{Deserialize, Serialize}; use std::marker::PhantomData; +use crate::asapv1::envelope; use crate::common::hash::check_matrix_rows; use crate::common::structures::matrix_storage::cols_mask_bits; -use crate::message_pack_format::envelope; use crate::{HashProfile, SketchHasher, Vector1D, Vector2D}; use super::CountL2HH; diff --git a/src/sketches/countsketch_topk/wire.rs b/src/sketches/countsketch_topk/wire.rs index dda3a388..b290d18d 100644 --- a/src/sketches/countsketch_topk/wire.rs +++ b/src/sketches/countsketch_topk/wire.rs @@ -21,8 +21,8 @@ use rmp_serde::{decode::Error as RmpDecodeError, encode::Error as RmpEncodeError, from_slice}; use serde::{Deserialize, Serialize}; +use crate::asapv1::envelope; use crate::common::hash::check_matrix_rows; -use crate::message_pack_format::envelope; use crate::sketches::countminsketch_topk::heap_wire::{ TopKMetadata, decode_payload, encode_payload, heap_entries, rebuild_heap, topk_metadata, wire_key_type, diff --git a/src/sketches/ddsketch/wire.rs b/src/sketches/ddsketch/wire.rs index 9aa4c1f9..440d3f07 100644 --- a/src/sketches/ddsketch/wire.rs +++ b/src/sketches/ddsketch/wire.rs @@ -29,8 +29,8 @@ use rmp_serde::{decode::Error as RmpDecodeError, encode::Error as RmpEncodeError, from_slice}; use serde::{Deserialize, Serialize}; +use crate::asapv1::envelope; use crate::common::structures::Vector1D; -use crate::message_pack_format::envelope; use super::{Buckets, DDSketch}; diff --git a/src/sketches/elastic/wire.rs b/src/sketches/elastic/wire.rs index d3a010f6..bbca4f56 100644 --- a/src/sketches/elastic/wire.rs +++ b/src/sketches/elastic/wire.rs @@ -36,8 +36,8 @@ use rmp_serde::{decode::Error as RmpDecodeError, encode::Error as RmpEncodeError use serde::{Deserialize, Serialize}; use std::marker::PhantomData; +use crate::asapv1::envelope; use crate::common::hash::check_matrix_rows; -use crate::message_pack_format::envelope; use crate::{CountMin, HashProfile, SketchHasher, Vector2D}; use super::{Elastic, HeavyBucket}; diff --git a/src/sketches/hll/wire.rs b/src/sketches/hll/wire.rs index 9a5f93f2..d9712fad 100644 --- a/src/sketches/hll/wire.rs +++ b/src/sketches/hll/wire.rs @@ -12,7 +12,7 @@ use rmp_serde::{decode::Error as RmpDecodeError, encode::Error as RmpEncodeError use serde::{Deserialize, Serialize}; use std::marker::PhantomData; -use crate::message_pack_format::envelope; +use crate::asapv1::envelope; use crate::structures::fixed_structure::HllRegisterStorage; use crate::{DefaultXxHasher, HashProfile, SketchHasher}; diff --git a/src/sketches/kll/wire.rs b/src/sketches/kll/wire.rs index af6c8b78..971f66ed 100644 --- a/src/sketches/kll/wire.rs +++ b/src/sketches/kll/wire.rs @@ -29,8 +29,8 @@ use rmp_serde::{decode::Error as RmpDecodeError, encode::Error as RmpEncodeError, from_slice}; use serde::{Deserialize, Serialize}; +use crate::asapv1::envelope; use crate::common::numerical::NumericalValue; -use crate::message_pack_format::envelope; use super::{ CAPACITY_CACHE_LEN, Coin, KLL, MAX_CACHEABLE_K, MAX_LEVELS, checked_weighted_count, diff --git a/src/sketches/kll_dynamic/wire.rs b/src/sketches/kll_dynamic/wire.rs index b5b56b4f..bbe736fc 100644 --- a/src/sketches/kll_dynamic/wire.rs +++ b/src/sketches/kll_dynamic/wire.rs @@ -15,7 +15,7 @@ use rmp_serde::{decode::Error as RmpDecodeError, encode::Error as RmpEncodeError, from_slice}; use crate::Vector1D; -use crate::message_pack_format::envelope; +use crate::asapv1::envelope; use crate::sketches::kll::{ Coin, KLL_KIND_DYNAMIC, KllCoinWire, KllPayload, KllWireItem, kll_metadata, split_and_validate_meta, validate_kll_payload, @@ -88,7 +88,7 @@ where #[cfg(test)] mod tests { - use crate::message_pack_format::envelope; + use crate::asapv1::envelope; use crate::sketches::kll::KLL_KIND_DYNAMIC; use crate::sketches::kll_dynamic::KLLDynamic; diff --git a/src/sketches/kmv/wire.rs b/src/sketches/kmv/wire.rs index b1b4d1d4..a7760106 100644 --- a/src/sketches/kmv/wire.rs +++ b/src/sketches/kmv/wire.rs @@ -28,7 +28,7 @@ use rmp_serde::{decode::Error as RmpDecodeError, encode::Error as RmpEncodeError use serde::{Deserialize, Serialize}; use std::marker::PhantomData; -use crate::message_pack_format::envelope; +use crate::asapv1::envelope; use crate::{CommonHeap, HashProfile, KeepLargest, SketchHasher}; use super::KMV; diff --git a/src/sketches/set_aggregator/wire.rs b/src/sketches/set_aggregator/wire.rs index f6cd4d4e..446a7ded 100644 --- a/src/sketches/set_aggregator/wire.rs +++ b/src/sketches/set_aggregator/wire.rs @@ -17,8 +17,8 @@ use std::collections::HashSet; use rmp_serde::{decode::Error as RmpDecodeError, encode::Error as RmpEncodeError, from_slice}; use serde::{Deserialize, Serialize}; -use crate::message_pack_format::envelope; -use crate::message_pack_format::wire_key::WireString; +use crate::asapv1::envelope; +use crate::asapv1::wire_key::WireString; use super::{DeltaResult, SetAggregator}; diff --git a/src/sketches/space_saving/wire.rs b/src/sketches/space_saving/wire.rs index afa43d2b..0da9f99f 100644 --- a/src/sketches/space_saving/wire.rs +++ b/src/sketches/space_saving/wire.rs @@ -46,8 +46,8 @@ use rmp_serde::{decode::Error as RmpDecodeError, encode::Error as RmpEncodeError, from_slice}; use serde::{Deserialize, Serialize}; -use crate::message_pack_format::envelope; -use crate::message_pack_format::wire_key::{WireBytes, WireString}; +use crate::asapv1::envelope; +use crate::asapv1::wire_key::{WireBytes, WireString}; use crate::{HashProfile, HeapItem, SketchHasher}; use super::{SpaceSaving, SpaceSavingState, key_order}; diff --git a/src/sketches/uniform/wire.rs b/src/sketches/uniform/wire.rs index e562f1ff..798606a9 100644 --- a/src/sketches/uniform/wire.rs +++ b/src/sketches/uniform/wire.rs @@ -25,7 +25,7 @@ use rmp_serde::{decode::Error as RmpDecodeError, encode::Error as RmpEncodeError, from_slice}; use serde::{Deserialize, Serialize}; -use crate::message_pack_format::envelope; +use crate::asapv1::envelope; use super::{SampleEntry, UniformSampling}; From 5df57f23afa74c09f5941d91fd2daadebaa4bdc4 Mon Sep 17 00:00:00 2001 From: GordonYuanyc Date: Fri, 2 Oct 2026 01:56:51 -0600 Subject: [PATCH 34/36] docs: describe ASAPv1 as the only serialization Delete the message_pack_format page (library_map covers the code layout), drop the portable and proto sections from the wire-format spec, the library map and the e2e harness notes, and keep only what the Unreleased changelog entries still remove. Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 42 ++++----- docs/asapv1_wire_format.md | 12 +-- docs/e2e_testing_harness.md | 11 +-- docs/hhheap_acceleration.md | 6 +- docs/index.md | 1 - docs/library_map.md | 16 ---- docs/message_pack_format.md | 181 ------------------------------------ docs/project_overview.md | 1 - tests/README.md | 2 +- 9 files changed, 29 insertions(+), 243 deletions(-) delete mode 100644 docs/message_pack_format.md diff --git a/CHANGELOG.md b/CHANGELOG.md index c3b9daa9..33e86971 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -142,9 +142,8 @@ signals a backwards-compatible change. - **KLL's proto and portable formats.** `KLL` and `KLLDynamic` serialize only as ASAPv1 (`serialize_to_bytes` / `deserialize_from_bytes`, kind_ids `0x06 0x00` / `0x06 0x01`). Removed: - - the `kll` field (13) of `SketchEnvelope` (now reserved) and with it - `sketch_envelope::SketchState::Kll`; `KLLState` stays in - `proto/kll/kll.proto` only as `HydraCell.kll`; + - the `kll` field of `SketchEnvelope` and with it + `sketch_envelope::SketchState::Kll`; - the crate-root `KllSketch` and `KllSketchData`, and the public module `message_pack_format::portable::kll`: `SketchlibKll`, `new_sketchlib_kll`, `new_sketchlib_kll_with_seed`, `sketchlib_kll_update`, @@ -154,9 +153,7 @@ signals a backwards-compatible change. `MessagePackCodec` impls for `KllSketch` and `KllSketchData`, the value-offset codec (`KLL_SCALE_SWEEP`, `encode_value_offset`, `decode_value_offset`, `KllProtoItems`), and the `unsafe impl Send/Sync - for KllSketch`. `HydraKllSketch`'s per-cell type stays reachable through - the public fields `HydraKllSketch.sketch` and - `HydraKllSketchWire.sketches`, but can no longer be named; + for KllSketch`; - `message_pack_format::native::{kll, kll_dynamic}`, the `MessagePackCodec` impls for `KLL` and `KLLDynamic`; - `KLL::from_portable_state`. @@ -168,13 +165,11 @@ signals a backwards-compatible change. `encode_sparse_registers`; the `MessagePackCodec` impls for `HyperLogLogImpl` and `HyperLogLogHIPImpl` (`message_pack_format::native::hll`); the `HLLDelta` proto message (`proto::sketchlib::HllDelta`); and the `hll` - field (12) of `SketchEnvelope`, now reserved. `HyperLogLogState`, - `HLLSparseRegisters` and `HLLVariant` remain only as the `HydraCell` HLL - state. + field of `SketchEnvelope`. - **Count-Min's proto and portable formats.** `CountMin` serializes to ASAPv1 only (`serialize_to_bytes` / `deserialize_from_bytes`, kind `0x02 0x00`). Removed: the `CountMinDelta` and `CountMinCell` proto messages and the - `count_min` field (10) of `SketchEnvelope`, now reserved; + `count_min` field of `SketchEnvelope`; `message_pack_format::portable::countminsketch` (`CountMinSketch`, `CountMinSketchDelta`, `CountMinSketchWire`, `SketchlibCms`, `new_sketchlib_cms`, `sketchlib_cms_from_matrix`, @@ -182,10 +177,9 @@ signals a backwards-compatible change. and the crate-root `CountMinSketch` / `CountMinSketchDelta` re-exports; the `MessagePackCodec` impl for `CountMin` in `message_pack_format::native::countminsketch`; and the - `src/sketches/testdata/cms_envelope_golden.hex` proto golden. The - `CountMinState` proto message stays as the nested Count-Min state of - `HydraCell` and `ElasticState`. Count-Min deltas (including `hh_keys`) and - the envelope `sample_p` for Count-Min have no ASAPv1 carrier. + `src/sketches/testdata/cms_envelope_golden.hex` proto golden. Count-Min + deltas (including `hh_keys`) and the envelope `sample_p` for Count-Min have + no ASAPv1 carrier. - **Count Sketch's proto and portable formats.** Count Sketch (`Count`) serializes as ASAPv1 only (`serialize_to_bytes` / `deserialize_from_bytes`, kind `0x04 0x00`). Removed: the portable `CountSketch`, `CountSketchDelta` @@ -196,8 +190,7 @@ signals a backwards-compatible change. `apply_delta_msgpack_bytes`); the `MessagePackCodec` impl for `Count` (module `message_pack_format::native::countsketch`); the proto messages `CountSketchDelta` and `CountSketchCell`; and `SketchEnvelope`'s - `count_sketch` field (11, now reserved). `CountSketchState` stays, as - UnivMon and Hydra embed it. + `count_sketch` field. - **CMSHeap's pre-envelope MessagePack format.** `CMSHeap` serializes through ASAPv1 only (`serialize_to_bytes` / `deserialize_from_bytes`). Removed: the module `message_pack_format::portable::countminsketch_topk` with @@ -223,7 +216,7 @@ signals a backwards-compatible change. ASAPv1 only (`DDSketch::serialize_to_bytes` / `deserialize_from_bytes`). Removed: `proto/ddsketch/ddsketch.proto` and the generated `proto::sketchlib::{DdSketchState, DdSketchDelta, DdSketchBucketDelta}`; - `SketchEnvelope`'s `ddsketch` field (14), now reserved; + `SketchEnvelope`'s `ddsketch` field; `message_pack_format::portable::ddsketch` (`DdSketch`, `DdSketchDelta`, `DDSKETCH_GROW_CHUNK`, `MAX_APPLY_DELTA_SPAN_BUCKETS`) and the crate-root re-exports of the first three; the `MessagePackCodec` impl for `DDSketch`. @@ -236,25 +229,22 @@ signals a backwards-compatible change. `with_seed`, `merge_refs` and `aggregate_hydrakll`; the proto file `hydra/hydra.proto` with `HydraState`, `HydraCell` and `HydraCounterType` (and their generated `proto::sketchlib` types); and `SketchEnvelope`'s - `hydra` field (16, now reserved). The nearest replacement is `Hydra` with a + `hydra` field. The nearest replacement is `Hydra` with a one-column schema and `HydraCounter::KLL`, but it routes `label:value` subkeys through the matrix hash at `HYDRA_SEED`, not `xxh32(key, row) % cols`, so `HydraKllSketch` cells do not map onto the new grid. - **UnivMon's proto format.** `UnivMon` serializes only as ASAPv1 (`serialize_to_bytes` / `deserialize_from_bytes`, kind_id `0x10 0x00`). - Removed the `univmon` field (15) of `SketchEnvelope` (now reserved) and with - it `sketch_envelope::SketchState::Univmon`; `UnivMonState` and - `UnivMonLayer` stay in `proto/univmon/univmon.proto` only as - `HydraCell.univmon`. + Removed the `univmon` field of `SketchEnvelope` and with it + `sketch_envelope::SketchState::Univmon`. - **Coco's protobuf format.** `proto/cocosketch/cocosketch.proto` and the - generated `asap_sketchlib::proto::sketchlib::CocoSketchState` are gone, and - the `SketchEnvelope` oneof field `coco` (17) is reserved, so - `sketch_envelope::SketchState::Coco` is gone too. Coco serializes only as + generated `asap_sketchlib::proto::sketchlib::CocoSketchState` are gone, with + the `SketchEnvelope` oneof field `coco` (`sketch_envelope::SketchState::Coco`). Coco serializes only as ASAPv1 (`Coco::serialize_to_bytes` / `deserialize_from_bytes`). - **BREAKING: Elastic's protobuf form.** `proto/elasticsketch/elasticsketch.proto`, its message `ElasticState` (`asap_sketchlib::proto::sketchlib::ElasticState`), and the `SketchEnvelope` oneof variant `elastic` - (`sketch_envelope::SketchState::Elastic`, field 18, now reserved) are gone. + (`sketch_envelope::SketchState::Elastic`) are gone. `Elastic::serialize_to_bytes` / `deserialize_from_bytes` (ASAPv1, `0x0b 0x00`) is Elastic's only serialization. - **CountL2HH's `MessagePackCodec` impl** and its module, diff --git a/docs/asapv1_wire_format.md b/docs/asapv1_wire_format.md index e7b1c7b5..5f781f66 100644 --- a/docs/asapv1_wire_format.md +++ b/docs/asapv1_wire_format.md @@ -917,8 +917,6 @@ Rules 2 and 5 are enforced on the **encode** side too, so the format never emits A `NaN` sample is legal: `update(f64::NAN)` is legal in memory and `total_cmp` gives it a total order, so it round-trips rather than being rejected. -**Not this: `portable/sampling.rs`.** That module is not ASAPv1 and does not touch this payload. It is query-time rescaling for NitroSketch-style geometric sampling, reading a `sample_p` scalar off a **protobuf** `SketchEnvelope` and dividing count-like estimates by it. Its policy on a bad rate is the **opposite** of rule 2's: `sample_p_or_default` **clamps** `0.0`, a negative, a value above `1` and `NaN` all to `1.0` and carries on, where ASAPv1 rejects the envelope. Neither the field nor the policy carries over; `sample_rate` here is a construction parameter of a reservoir, not a rescale factor. - ### 3.14: KMV payload (`0x0e 0x00`) `KMV` is the k-minimum-values distinct-count estimator: it hashes each key once and retains the `k` smallest 64-bit digests it has seen. The retention bound `k` is construction config and lives in the metadata, so the payload is the retained digests alone: @@ -1365,22 +1363,20 @@ It does not convert the mode (Regular to/from Fast); that would need re-insertin ## Cross-language contract -Direction: **custom per-sketch payload replaces the `portable` types, and `sketchlib-go` mirrors each payload.** -Good direction (more compact, higher fidelity, less Rust-internal duplication), but it moves the contract from shared code to discipline. To keep it safe: +Each sketch has its own payload, and `sketchlib-go` mirrors each one. +The contract is held by discipline rather than shared code, through three things: 1. **This spec**: byte-level, language-neutral, per sketch. -2. **Golden byte-vector fixtures** in one shared repo, [`sketchlib-golden-bytes`](https://github.com/ProjectASAP/sketchlib-golden-bytes), which each implementation mounts at `asapv1_golden/`; both languages decode and re-encode them byte-identically. These replace the `portable`-as-oracle round-trip test. +2. **Golden byte-vector fixtures** in one shared repo, [`sketchlib-golden-bytes`](https://github.com/ProjectASAP/sketchlib-golden-bytes), which each implementation mounts at `asapv1_golden/`; both languages decode and re-encode them byte-identically. 3. **This registry**, mirrored, never independently allocated. -`asapv1_golden/README.md`'s Coverage section lists which `kind_id`s have fixtures; for every other kind, this document is the only contract. `portable` carries its own Go goldens. +`asapv1_golden/README.md`'s Coverage section lists which `kind_id`s have fixtures; for every other kind, this document is the only contract. **Hash profile on the Go side.** Rust derives the hash spec from a generic `HashProfile` bound on the hasher type; Go has no generic hasher type, so there is nothing to derive from. On the Go side the profile is simply **written into** the metadata on encode and **read from** it on decode. Go MUST validate the profile it reads (same fail-closed intent as Rust): a sketch is only mergeable/queryable if its `hash_profile_id` + seeds match the profile Go is prepared to reproduce. -Retire `portable` once the fixtures cover the rest. - --- ## Decisions (resolved) diff --git a/docs/e2e_testing_harness.md b/docs/e2e_testing_harness.md index 5b122244..12109a72 100644 --- a/docs/e2e_testing_harness.md +++ b/docs/e2e_testing_harness.md @@ -72,7 +72,7 @@ tests/ ├── e2e_matrix_instances.rs # every (storage, path) instance of the matrix families ├── e2e_numeric_types.rs # every NumericalValue type through KLL / DDSketch ├── e2e_windows.rs # every EHSketchList variant + TumblingWindow payloads -├── e2e_composition.rs # HashSketchEnsemble, NitroBatch, UnivMonQ config, portable facade +├── e2e_composition.rs # HashSketchEnsemble, NitroBatch, UnivMonQ config, CMSHeap / KLL merge and wire ├── e2e_frameworks.rs # Hydra and UnivMon composition ├── e2e_octo.rs # …the OctoSketch promotion protocol ├── e2e_heavy_hitters.rs # …Space-Saving, CocoSketch and Elastic @@ -186,9 +186,8 @@ suite with the matching spec from Layer 2. **Layer 4 — suites.** `conformance_kit.rs` wires established sketches through the kit as reference adapters. The `e2e_*.rs` suites add depth the kit deliberately does not -attempt: serialization round trips, window semantics, framework composition -(Hydra fan-out, tumbling windows, sliding histograms), and cross-implementation -parity between core types and their portable wire twins. +attempt: serialization round trips, window semantics, and framework +composition (Hydra fan-out, tumbling windows, sliding histograms). A composition framework can also *be* a battery subject, in the same suite that already covers it. `e2e/frameworks.rs` runs a single-column Hydra — where the @@ -282,8 +281,8 @@ bugs only appeared at deployment-shaped dimensions. ### 4. Add depth where the sketch is unusual Anything not covered by a battery belongs in the matching `e2e_*.rs` suite: -window semantics, merge order-independence, wire-format parity with the -portable type, heavy-hitter recall targets, and so on. +window semantics, merge order-independence, ASAPv1 round trips, heavy-hitter +recall targets, and so on. ## Tolerance policy diff --git a/docs/hhheap_acceleration.md b/docs/hhheap_acceleration.md index 3f84cee5..402cfd3c 100644 --- a/docs/hhheap_acceleration.md +++ b/docs/hhheap_acceleration.md @@ -91,6 +91,6 @@ the structure carries no run-to-run state at all. derived rather than carried, and a decoded heap rebuilds it before first use. A named-map encoding that carries an index field still decodes, since the extra key is skipped; a positional encoding of the three-field form does not. Nothing -in-crate writes the positional form: the portable MessagePack wire for the top-k -sketches carries a `(key, value)` list and rebuilds through `update`, and no -golden covers a top-k envelope. +in-crate writes the positional form: the ASAPv1 payload of the top-k sketches +carries parallel `keys` / `heap_counts` arrays and rebuilds the heap from them +through `update_heap_item`. diff --git a/docs/index.md b/docs/index.md index c81a311f..2d0f1096 100644 --- a/docs/index.md +++ b/docs/index.md @@ -15,7 +15,6 @@ This page is the docs home for `asap_sketchlib`. - [API Pages Directory](./api/) - [Common API](./api/api_common.md) - [ASAPv1 Wire Format](./asapv1_wire_format.md) - authoritative byte-level spec for the self-describing sketch envelope mirrored by `sketchlib-go` -- [Message Pack Format](./message_pack_format.md) - how the serialization code is organized (shared envelope + per-sketch `wire.rs`; `portable`/`native` retiring) - [Advanced Use Cases](./advanced_use_cases.md) - [HHHeap Acceleration](./hhheap_acceleration.md) - what the top-k heap costs per update, how its digest-keyed position index is maintained through the sift, and the throughput it measures at - [Test Coverage Map](./tests.md) diff --git a/docs/library_map.md b/docs/library_map.md index 0a25f974..865a5e4e 100644 --- a/docs/library_map.md +++ b/docs/library_map.md @@ -40,19 +40,3 @@ `std::sync::LazyLock` in `src/common/precompute_hash.rs`, `src/common/precompute_sample.rs`, and `src/common/precompute_sample2.rs`. - -## Proto code generation - -- `proto/**/*.proto` is the cross-language wire-format source of truth shared - with `sketchlib-go`. -- The corresponding Rust types are **vendored** under - `src/proto/generated/sketchlib.v1.rs` and re-exported by `src/proto.rs` as - `crate::proto::sketchlib`. Downstream users therefore build the crate as - pure Rust without needing `protoc` or any build script. -- To regenerate after editing any `.proto` file, run from the repository root: - - ```bash - cargo run --manifest-path tools/gen-proto/Cargo.toml - ``` - - CI enforces that the committed file matches the result of regeneration. diff --git a/docs/message_pack_format.md b/docs/message_pack_format.md deleted file mode 100644 index 4b0e9396..00000000 --- a/docs/message_pack_format.md +++ /dev/null @@ -1,181 +0,0 @@ -# Message Pack Format - -`src/message_pack_format/` holds the Rust-side serialization plumbing. The -current, self-describing wire format is **ASAPv1**; its byte-level layout is the -single source of truth in -[**`asapv1_wire_format.md`**](./asapv1_wire_format.md) — read that for the -authoritative spec. This page describes how the code is organized. - -## The ASAPv1 model - -Every serialized sketch is one self-delimiting envelope: - -```md -[ magic:6 | version:u8 | kind_id_len:u8 | kind_id:bytes - | metadata_len:u32_be | payload_len:u32_be - | metadata:msgpack-map | payload:msgpack-array ] -``` - -- **Envelope** — the sketch-agnostic frame (magic `b"ASAPv1"`, version, - `kind_id`, and the two length prefixes). It answers *is this ours?*, *how do I - parse the frame?*, and *what algorithm?* with zero knowledge of any sketch. -- **Metadata** — a msgpack **map** (self-describing) carrying the hash spec plus - the structural params needed to interpret the payload (HLL `precision`; - Count-Min `rows`/`cols`/`counter_type`/`mode`). The hash-spec values are - **derived from the hasher's [`HashProfile`](../src/common/hash.rs)** trait, so - the bytes truthfully describe how the sketch was hashed and custom hash - profiles serialize self-describingly. Each sketch has its own fixed metadata - schema with `#[serde(deny_unknown_fields)]` (fail-closed on unknown/missing - keys). A sketch that never hashes its inputs omits the hash-spec group - entirely and carries structural params alone — KLL, DDSketch and - UniformSampling, plus the two wrappers `ExponentialHistogram` and - `EHSketchList`, whose hash specs live inside the sketch each one carries. -- **Payload** — a positional msgpack **array** of the raw sketch state only - (registers, counter matrix). No field names, and nothing the `kind_id` or - metadata already determines. -- **A nested sketch is inlined, not wrapped in an envelope of its own.** A - CMSHeap's base matrix, an Elastic's light Count-Min, a Hydra cell and a - UnivMon layer all write their raw state straight into the enclosing positional - array. Where the nested algorithm is data rather than a type, the nested block - is that variant's own envelope with the framing stripped: the - `[kind_id, descriptor, state]` triple `EHSketchList` carries and - `ExponentialHistogram` inlines per bucket. - -`kind_id` + metadata together fix the payload structure completely. See -[`asapv1_wire_format.md`](./asapv1_wire_format.md) for the `kind_id` registry, -the metadata field tables, and the byte-level encoding rules. - -## Code organization - -### Shared framing — `envelope.rs` - -`src/message_pack_format/envelope.rs` is the **one shared, sketch-agnostic** -module every sketch calls into. It owns the magic sentinel, the layout version, -and the byte framing (`encode` / `split`). It validates only the magic, version, -and framing — it does **not** know the `kind_id` registry or any sketch. Rule -"which `kind_id` do I own?" and all metadata validation happen in each sketch's -decoder. - -### Per-sketch serialization — `/wire.rs` - -Serialization lives **with each sketch**, split from the algorithm, under -`src/sketches/` and `src/sketch_framework/` alike: - -- `.rs` — the **algorithm** (struct, marker types, aliases, - insert/estimate/merge). It declares `mod wire;`. -- `/wire.rs` — the **serialization** (metadata/payload DTOs, - `kind_id` consts, wire-variant/counter/mode marker traits, and the - `serialize_to_bytes` / `deserialize_from_bytes` impls). Because `wire` is a - child submodule of the sketch, it reads the struct's **private** fields - (`self.registers`, `self.counts`) directly — no field is widened for - serialization. - -Two encodings are shared by more than one sketch and live in a second module -beside `wire.rs`, declared `pub(crate)` so the other user can import it by path: - -- `src/sketches/countminsketch_topk/heap_wire.rs` — the top-k heap metadata - schema, payload, `key_type` mapping, emitted order and heap rebuild that - CMSHeap (`0x03 0x00`) and CSHeap (`0x0a 0x00`) share. -- `src/sketches/countsketch_topk/l2hh_wire.rs` — CountL2HH (`0x19 0x00`), plus - the `[counts, l2]` layer sub-payload the whole UnivMon family inlines. - -Converted: - -- **HLL** (`src/sketches/hll/wire.rs`) — all variants (Classic → `0x01 0x01`, - Ertl-MLE → `0x01 0x02`, HIP → `0x01 0x03`) × precisions (P12/P14/P16) × - `H: HashProfile`. HLL is **fully wire-covered**. (HIP is a non-generic struct - hashed through the default functions, so it is wire-eligible under the standard - profile only.) -- **Count-Min** (`src/sketches/countminsketch/wire.rs`, kind `0x02 0x00`) — - restricted to wire-eligible configs: - `CountMin, Mode, H>` where `T` is `i32`, `i64` or `f64` - (`CmsWireCounter`), `Mode` is `FastPath` or `RegularPath` (`CmsWireMode`), and - `H: HashProfile`. `i32` is carried at its own width rather than widened. - `i128` and non-`Vector2D` storage must be converted first. `rows` - and `cols` live in the metadata; the payload is just `[counts]`. -- **Count Sketch** (`src/sketches/countsketch/wire.rs`, kind `0x04 0x00`) — the - same shape as Count-Min: `Count, Mode, H>` where `T` is `i32` or - `i64` (`CsWireCounter`), `Mode` is `FastPath` or `RegularPath` (`CsWireMode`), - and `H: HashProfile`. Counters must be signed and negatable, so there is no - `f64` counterpart; `i128` and non-`Vector2D` storage must be converted first. - `i32` is carried at its own width rather than widened, so a nested - `Vector2D` sketch decodes back into its own type. `rows`, `cols`, - `counter_type` and `mode` live in the metadata; the payload is just - `[counts]`, with signed cells. -- **KLL** (`src/sketches/kll/wire.rs` compact → `0x06 0x00`, - `src/sketches/kll_dynamic/wire.rs` dynamic → `0x06 0x01`) — both variants share - one payload `[levels, items, coin]` and differ only by `kind_id`. KLL never - hashes, so its metadata carries no hash-spec group. -- **Bloom** (`src/sketches/bloom/wire.rs`, kind `0x17 0x00`) — the payload is the - bit grid packed into `u64`s plus the insert count; the wire covers the - geometries `Bloom::with_capacity` produces. -- **Space-Saving** (`src/sketches/space_saving/wire.rs`, kind `0x18 0x00`) — the - payload is the `(key, count, error)` triples plus `total` and `discarded_max`; the - Stream-Summary's links and arenas are rebuilt on decode. -- **CMSHeap** (`src/sketches/countminsketch_topk/wire.rs`, `0x03 0x00`) and - **CSHeap** (`src/sketches/countsketch_topk/wire.rs`, `0x0a 0x00`) — a base - matrix inlined ahead of the shared top-k heap entries. -- **DDSketch** (`src/sketches/ddsketch/wire.rs`, `0x05 0x00`) — the bucket store - verbatim plus `offset`, `sum`, `min` and `max`; no hash-spec group. -- **Elastic** (`src/sketches/elastic/wire.rs`, `0x0b 0x00`) — the heavy table's - four parallel arrays, the `stale_copies` flag, and the inlined light Count-Min. -- **Coco** (`src/sketches/coco/wire.rs`, `0x0c 0x00`) — the dense bucket table as - parallel nullable-key and mass arrays. -- **UniformSampling** (`src/sketches/uniform/wire.rs`, `0x0d 0x00`) — the - priority-ordered sample pairs plus `total_seen` and the SplitMix64 state; no - hash-spec group. -- **KMV** (`src/sketches/kmv/wire.rs`, `0x0e 0x00`) — the retained digests, - strictly ascending. -- **CountL2HH** (`src/sketches/countsketch_topk/l2hh_wire.rs`, `0x19 0x00`) — - `[counts, l2]`. -- **Hydra** (`src/sketch_framework/hydra/wire.rs`, `0x07 0x00`-`0x07 0x04`) — one - `kind_id` per counter variant; the cells' state tiles or nests one array. -- **UnivMon** (`src/sketch_framework/univmon/wire.rs`, `0x10 0x00`) and - **UnivMon Optimized** (`src/sketch_framework/univmon_optimized/wire.rs`, - `0x11 0x00`) — one payload shape, two metadata schemas. -- **UnivMon-Q** (`src/sketch_framework/univmon_q/wire.rs`, `0x1a 0x00`) — the - levels' counters, candidates, extrema and coordinated occurrence sample. -- **ExponentialHistogram** (`src/sketch_framework/eh/wire.rs`, `0x13 0x00`) and - **EHSketchList** (`src/sketch_framework/eh_sketch_list/wire.rs`, `0x14 0x00`) — - one `[kind_id, descriptor, state]` triple, standalone and per bucket. - -NitroBatch, FoldCMS, FoldCS, HashSketchEnsemble, EHUnivOptimized and OctoSketch -have **no `wire.rs`**; their `kind_id`s are reserved with payload TBD. - -## `portable/` and `native/` are being retired - -The `portable/` and `native/` sub-modules (and the `MessagePackCodec` trait / -unified `Error` at the module root) are the **older** serialization path and are -being **phased out** in favor of the per-sketch `wire.rs` + the shared -`envelope.rs`. - -- `portable/` holds per-sketch wire types that predate the envelope. -- `native/` is a set of thin `MessagePackCodec` shims over the sketches' - `serialize_to_bytes` / `deserialize_from_bytes`. Every type it wraps emits the - ASAPv1 envelope, so each shim is a pass-through. - -Sequencing note (from the spec): `portable` is not deleted until the golden -byte-vector fixtures are the drift guard on both sides. - -## Cross-language parity - -ASAPv1 parity with `sketchlib-go` is proven by **golden byte-vectors** in -[`sketchlib-golden-bytes`](https://github.com/ProjectASAP/sketchlib-golden-bytes), mounted as the -submodule `asapv1_golden/` (exercised by -[`tests/asapv1_golden.rs`](../tests/asapv1_golden.rs)): both languages must -decode → re-encode them byte-identically. The `kind_id` registry is mirrored -verbatim with Go's `wire/asapmsgpack/magic_ids.go`, never independently -allocated. - -That proof covers only the `kind_id`s that have a fixture; -[its README's Coverage section](https://github.com/ProjectASAP/sketchlib-golden-bytes#coverage) -lists which. A kind without one has **no ASAPv1 drift guard**. -The older `portable` path is guarded separately, by the `sketchlib-go` goldens -of its own listed above. - -## Cross-Reference - -- [`asapv1_wire_format.md`](./asapv1_wire_format.md) — the authoritative - byte-level spec (envelope, metadata schema, per-sketch payloads, encoding - rules, wire coverage). -- Generated rustdoc: `cargo doc --no-deps --all-features --open`. diff --git a/docs/project_overview.md b/docs/project_overview.md index f5cb6b8b..80103582 100644 --- a/docs/project_overview.md +++ b/docs/project_overview.md @@ -17,7 +17,6 @@ implementations, and orchestration/windowing frameworks in one crate. - [Advanced Use Cases](./advanced_use_cases.md) - Subpopulation queries, sketch coordination, and sliding-window frameworks explained separately. - [Common Module API](./api/api_common.md) - Shared types, hashing, and structures. - [ASAPv1 Wire Format](./asapv1_wire_format.md) - Authoritative byte-level spec for the self-describing sketch envelope mirrored by `sketchlib-go`. -- [Message Pack Format](./message_pack_format.md) - How the serialization code is organized (shared envelope + per-sketch `wire.rs`). - [Library Map](./library_map.md) - Source-tree module breakdown. - [Feature Status](./features.md) - Implemented, in-progress, and planned work. - [Test Coverage Map](./tests.md) - Test organization and coverage notes. diff --git a/tests/README.md b/tests/README.md index 620de7dd..4e07cd10 100644 --- a/tests/README.md +++ b/tests/README.md @@ -23,7 +23,7 @@ its theorem. | `e2e/matrix_instances.rs` | Every built-in `(storage, hashing path)` instance of CountMin / Count / CMSHeap / CSHeap, plus counter-width edges | | `e2e/numeric_types.rs` | Every `NumericalValue` type through `KLL`, `KLLDynamic` and `DDSketch::add` | | `e2e/windows.rs` | Every `EHSketchList` variant in an `ExponentialHistogram`, every `TumblingWindow` payload, and `MicroCM`'s in-cell sliding window | -| `e2e/composition.rs` | `HashSketchEnsemble`, `UnivMonQ`'s config surface, `CMSHeap`, and the portable sketch+heap facade | +| `e2e/composition.rs` | `HashSketchEnsemble`, `UnivMonQ`'s config surface, and `CMSHeap` and `KLL` through merge and wire | | `e2e/nitro.rs` | Every Nitro ingestion path — row-level `CountMin`/`Count`, `NitroBatch::insert` / `insert_cached_step`, and the bare `Vector2D` target — under one sampling model, plus merge, seeding, saturation and serde/context continuation. **All Nitro E2E behaviour lives here**; the sampler's own structural tests are unit tests in `src/common/structure_utils.rs` and `src/sketch_framework/nitro.rs`, where the cursor and skip counter are visible without a public accessor | | `e2e/frameworks.rs` | Hydra's subpopulation lattice and UnivMon composition | | `e2e/heavy_hitters.rs` | Space-Saving's error sandwich, `min_count` ceiling and Stream-Summary lists under sustained eviction, plus CocoSketch's exact mass partitions and multi-seed unbiasedness, and Elastic through the batteries and the heavy-hitter properties no battery models | From e2236acbfbe31bbf11689af7b5a1c795d8797cdd Mon Sep 17 00:00:00 2001 From: GordonYuanyc Date: Fri, 2 Oct 2026 04:14:07 -0600 Subject: [PATCH 35/36] refactor!: drop the unused common::hashspec module Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 3 + docs/library_map.md | 2 +- docs/tests.md | 13 -- src/common/hashspec.rs | 302 ----------------------------------------- src/common/mod.rs | 4 - 5 files changed, 4 insertions(+), 320 deletions(-) delete mode 100644 src/common/hashspec.rs diff --git a/CHANGELOG.md b/CHANGELOG.md index 33e86971..17ea743f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -274,6 +274,9 @@ signals a backwards-compatible change. `tools/gen-proto` generator and its CI drift check; and the `prost` and `bytes` dependencies. - **The `xxhash-rust` dependency**, which nothing uses. +- **`common::hashspec`**, which nothing uses: `HashSpec`, `SeedDerivation`, + `CANONICAL_HASH_SEED_TABLE`, `CANONICAL_HASH_SEED`, `hash_with_spec`, + `derive_index` and `derive_sign`. ### Fixed diff --git a/docs/library_map.md b/docs/library_map.md index 865a5e4e..ce2d2866 100644 --- a/docs/library_map.md +++ b/docs/library_map.md @@ -7,7 +7,7 @@ - `structures/` - High-performance data structures (`Vector1D`, `Vector2D`, `Vector3D`, `BitMatrix`, `CommonHeap`, `MatrixStorage`, `FixedMatrix`) - `heap.rs` - `HHHeap` convenience wrapper for heavy hitter tracking - `hash.rs` - Hashing utilities (`hash_for_matrix`, `hash64_seeded`, `SEEDLIST`, `BOTTOM_LAYER_FINDER`) plus `SketchHasher` for custom hasher injection and `DigestHasher` / `DigestBuildHasher` for maps keyed by an already-hashed digest - - `hashspec.rs` - `HashSpec`/`derive_index`/`derive_sign` byte parity with `sketchlib-go`; `numerical.rs` - `NumericalValue`; `structure_utils.rs` - `Nitro`, `NitroContext`, `ToF64` + - `numerical.rs` - `NumericalValue`; `structure_utils.rs` - `Nitro`, `NitroContext`, `ToF64` - **`src/sketches/`** - Sketch implementations (status source: [apis.md](./apis.md)) - `Ready` in API index: `countminsketch.rs`, `countsketch.rs`, `hll.rs`, `kll.rs`, `ddsketch.rs`, `countminsketch_topk.rs`, `countsketch_topk.rs`, `space_saving.rs`, `bloom.rs`, `elastic.rs`, `coco.rs` diff --git a/docs/tests.md b/docs/tests.md index c3f46175..4127ffe7 100644 --- a/docs/tests.md +++ b/docs/tests.md @@ -1357,19 +1357,6 @@ Test file: [`src/common/structures/bit_matrix.rs`](../src/common/structures/bit_ | `a_packed_64_grid_decodes_a_distinct_window_per_row` | Each row of a packed-64 grid reads its own window of the hash. | For a `5x1024` `BitMatrix`, verifies `hash_mode_for_matrix(5, 1024)` is `MatrixHashMode::Packed64`, then `fast_insert`s 300 `U64` keys and verifies every pair of rows differs at some column; the per-row bit windows themselves are not compared. | | `dimensions_whose_word_count_overflows_are_rejected` | Dimensions whose word count overflows fail at decode. | Verifies a crafted payload declaring `usize::MAX` rows by `128` columns with an empty `words` array is refused with an error containing "dimensions overflow". | -### Common Hash Spec - -Test file: [`src/common/hashspec.rs`](../src/common/hashspec.rs) - -| test_name | test_description | what_is_tested | -| --- | --- | --- | -| `seed_table_matches_sketchlib_go` | The seed table and the canonical seed index are pinned values. | Verifies `CANONICAL_HASH_SEED_TABLE` equals the twenty listed seeds (`0xcafe3553`, `0xade3415118`, through `0xdb0c2e0d`) in order, and that `CANONICAL_HASH_SEED` is `5`. | -| `hash_with_spec_matches_sketchlib_go` | The byte-key hash of one fixed key is a pinned value. | Verifies `hash_with_spec(&HashSpec::default(), b"projectasap")` is `887548862923853302`. | -| `derive_index_matches_go_bit_slicing` | `derive_index` slices the row's own window out of one hash. | For the `HashSpec::default()` hash of `b"projectasap"`, verifies `derive_index(&spec, row, h, 512)` is `(h >> (row * 9)) & 0x1ff` and `derive_index(&spec, row, h, 1024)` is `(h >> (row * 10)) & 0x3ff`, for rows `0..3` of each width. | -| `derive_sign_matches_go_high_bit` | `derive_sign` reads bit `63 - row` of the hash. | For the same hash, verifies `derive_sign(&spec, row, h)` is `1` where bit `63 - row` is set and `-1` where it is clear, for rows `0..5`. | -| `mask_bits_for_width_matches_go` | The mask width is the column count's bit length. | Verifies `mask_bits_for_width` returns `1` for `1` and for `2`, `2` for `4`, `9` for `512`, `10` for `1024`, and `12` for `4096`. | -| `default_hashspec_has_packed_derivation` | The default spec names packed derivation, the canonical index, and the full seed table. | Verifies `HashSpec::default()` carries `seed_derivation == SeedDerivation::Packed`, `canonical_seed_index == CANONICAL_HASH_SEED`, and a `seed_list` as long as `CANONICAL_HASH_SEED_TABLE`. | - ### MatrixHashType (Common Structure) Test file: [`src/common/structures/matrix_storage.rs`](../src/common/structures/matrix_storage.rs) diff --git a/src/common/hashspec.rs b/src/common/hashspec.rs deleted file mode 100644 index ab8c5085..00000000 --- a/src/common/hashspec.rs +++ /dev/null @@ -1,302 +0,0 @@ -//! Cross-language hash layer mirroring `sketchlib-go::common`'s -//! `HashSpec` / `DeriveIndex` / `DeriveSign` decomposition for -//! matrix-backed sketches (CountSketch, CountMin). -//! -//! `sketchlib-go` and `asap_sketchlib` share three building blocks for -//! these sketches: -//! -//! 1. A 20-entry seed table (`CANONICAL_HASH_SEED_TABLE`, identical -//! bit-for-bit to Go's `seedList`). -//! 2. A canonical seed index (`CANONICAL_HASH_SEED` = 5). -//! 3. A single XXH3-64-with-seed call per (key, seed) pair, with the -//! resulting `u64` then bit-sliced into per-row column indices and -//! ±1 signs. -//! -//! This module re-exposes that pipeline as a small, byte-key API so -//! sketches that consume keys as raw bytes can match Go's emitted matrix -//! cell-for-cell. The seed table, derive_index, and derive_sign are -//! agnostic to whether ±1 signing is applied. -//! -//! # Example -//! -//! ``` -//! use asap_sketchlib::common::hashspec::{HashSpec, hash_with_spec, derive_index, derive_sign}; -//! -//! let spec = HashSpec::default(); // matches Go's portableHashSpec() -//! let key = b"flow-7"; -//! let h = hash_with_spec(&spec, key); // single XXH3-64 with seed_list[0] -//! let col = derive_index(&spec, 0, h, 512); // row 0, width 512 -//! let sign = derive_sign(&spec, 0, h); // -1 or +1 -//! assert!(col < 512); -//! assert!(sign == -1 || sign == 1); -//! ``` - -use twox_hash::XxHash3_64; - -/// 20-entry seed table shared by all matrix-backed sketches. -/// -/// Bit-for-bit identical to `sketchlib-go::common.seedList` — the order -/// MUST match because `derive_index` / `derive_sign` produce byte -/// parity only when both libraries pull the same `u64` from index `i`. -pub const CANONICAL_HASH_SEED_TABLE: [u64; 20] = [ - 0xcafe3553, - 0xade3415118, - 0x8cc70208, - 0x2f024b2b, - 0x451a3df5, - 0x6a09e667, - 0xbb67ae85, - 0x3c6ef372, - 0xa54ff53a, - 0x510e527f, - 0x9b05688c, - 0x1f83d9ab, - 0x5be0cd19, - 0xcbbb9d5d, - 0x629a292a, - 0x9159015a, - 0x152fecd8, - 0x67332667, - 0x8eb44a87, - 0xdb0c2e0d, -]; - -/// Default canonical-seed index used by single-hash matrix-sketch -/// operations. Matches `sketchlib-go::common.CanonicalHashSeed` and -/// `asap_sketchlib::common::hash::CANONICAL_HASH_SEED`. -pub const CANONICAL_HASH_SEED: usize = 5; - -/// Hash configuration shared by matrix-backed sketches. -/// -/// Mirrors `sketchlib-go::common.HashSpec` semantics: a producer -/// records the seed table, the canonical seed index, and the seed -/// derivation strategy so consumers can validate hash compatibility -/// before merging. -/// -/// The default matches Go's `portableHashSpec()`: -/// -/// - `seed_list = CANONICAL_HASH_SEED_TABLE` -/// - `canonical_seed_index = CANONICAL_HASH_SEED` (= 5) -/// - `seed_derivation = SeedDerivation::Packed` — the matrix sketches -/// used today fit in a single packed `u64` (rows × bits_per_row ≤ 64), -/// so a single hash with `seed_list[0]` covers all rows. The -/// `Additive` variant is reserved for the per-row-hash fallback used -/// by larger matrices and is not yet exercised by the wire path. -#[derive(Clone, Debug, PartialEq, Eq)] -pub struct HashSpec { - /// Full seed table, length must be > 0. - pub seed_list: Vec, - /// Index into `seed_list` for the canonical (single-hash) seed. - pub canonical_seed_index: usize, - /// Strategy for deriving per-row seeds. - pub seed_derivation: SeedDerivation, -} - -/// Strategy for deriving per-row seeds, mirroring -/// `sketchlib-go::common.SeedDerivation`. -#[derive(Clone, Copy, Debug, PartialEq, Eq)] -pub enum SeedDerivation { - /// All rows share `seed_list[0]`; per-row column index and sign are - /// extracted via bit slicing on a single `u64` hash. Used when - /// `rows × bits_per_row ≤ 64`. - Packed, - /// Each row r uses `seed_list[(base + r) % len]`; one - /// XXH3-64-with-seed call per row. Reserved for matrices that - /// exceed the packed budget. - Additive, -} - -impl Default for HashSpec { - fn default() -> Self { - Self { - seed_list: CANONICAL_HASH_SEED_TABLE.to_vec(), - canonical_seed_index: CANONICAL_HASH_SEED, - seed_derivation: SeedDerivation::Packed, - } - } -} - -impl HashSpec { - /// Returns the seed used for the single-hash packed path. Matches - /// Go's `seedList[0]` (because Go's `Hash64` calls `HashIt(0, key)`, - /// not `HashIt(canonical_seed_index, key)` — the canonical-seed - /// index governs `CanonicalHash`/`hh_keys` lookups, not the - /// matrix hot path). The matrix path always uses index 0, - /// regardless of `canonical_seed_index`. - #[inline] - pub fn matrix_seed(&self) -> u64 { - // Mirror Go's `common.Hash64`: `HashIt(0, key)` reads - // `seedList[normalizedSeedIdx(0)]` = seedList[0]. - self.seed_list[0] - } - - /// Returns the seed for row `r` under the additive-offset - /// derivation (reserved; not used by the packed wire path). - #[inline] - pub fn row_seed(&self, row: usize) -> u64 { - let idx = row % self.seed_list.len(); - self.seed_list[idx] - } -} - -/// Hashes `key` under `spec`'s matrix seed (single XXH3-64-with-seed -/// call, mirroring Go's `common.Hash64`). Producers of byte-parity -/// CountSketch / CMS update paths call this once per key, then invoke -/// [`derive_index`] / [`derive_sign`] per row to populate the matrix. -#[inline] -pub fn hash_with_spec(spec: &HashSpec, key: &[u8]) -> u64 { - XxHash3_64::oneshot_with_seed(spec.matrix_seed(), key) -} - -/// Returns `mask_bits` such that `(1 << mask_bits) - 1` is the column -/// mask for `width`. Mirrors Go's `maskBitsForWidth`. -#[inline] -fn mask_bits_for_width(width: u32) -> u32 { - if width <= 1 { - return 1; - } - // ceil(log2(width)). Wire-path widths are power-of-two, so this - // collapses to `width.trailing_zeros()`. - let mut u = width - 1; - let mut bits = 0u32; - while u > 0 { - bits += 1; - u >>= 1; - } - bits -} - -/// Derives the row-local column index from a precomputed `hash`, -/// mirroring `sketchlib-go::common.DeriveIndex`: -/// -/// ```text -/// shift = row * mask_bits(width) -/// index = (hash >> shift) & (width - 1) -/// ``` -/// -/// `width` MUST be power-of-two (the wire-format matrix sketches -/// enforce this on construction); other values produce a mask that -/// does not equal `width - 1` and the caller's matrix-cell layout -/// will diverge from Go's. -#[inline] -pub fn derive_index(_spec: &HashSpec, row: usize, hash: u64, width: u32) -> usize { - let shift = (row as u32) * mask_bits_for_width(width); - let mask = (width as u64).wrapping_sub(1); - ((hash >> shift) & mask) as usize -} - -/// Derives the ±1 sign for row `row` from a precomputed `hash`, -/// mirroring `sketchlib-go::common.DeriveSign`: -/// -/// ```text -/// bit = (hash >> (63 - row)) & 1 -/// sign = bit == 1 ? +1 : -1 -/// ``` -/// -/// `row` must satisfy `row <= 63`; callers with deeper sketches must -/// switch to the per-row hash fallback (not yet exposed here). -#[inline] -pub fn derive_sign(_spec: &HashSpec, row: usize, hash: u64) -> i64 { - debug_assert!( - row <= 63, - "derive_sign: row {row} out of range for u64 hash" - ); - let bit = (hash >> (63 - row as u32)) & 1; - if bit == 0 { -1 } else { 1 } -} - -#[cfg(test)] -mod tests { - use super::*; - - /// Locks the seed table to Go's `seedList`. Any drift here will - /// silently desync the matrix-cell layout from `sketchlib-go`'s. - #[test] - fn seed_table_matches_sketchlib_go() { - // Captured from `sketchlib-go::common.seedList` (hash.go:13-34). - let go_seed_list: [u64; 20] = [ - 0xcafe3553, - 0xade3415118, - 0x8cc70208, - 0x2f024b2b, - 0x451a3df5, - 0x6a09e667, - 0xbb67ae85, - 0x3c6ef372, - 0xa54ff53a, - 0x510e527f, - 0x9b05688c, - 0x1f83d9ab, - 0x5be0cd19, - 0xcbbb9d5d, - 0x629a292a, - 0x9159015a, - 0x152fecd8, - 0x67332667, - 0x8eb44a87, - 0xdb0c2e0d, - ]; - assert_eq!(CANONICAL_HASH_SEED_TABLE, go_seed_list); - assert_eq!(CANONICAL_HASH_SEED, 5); - } - - /// Locks `hash_with_spec` to Go's `Hash64` for a representative - /// key. Captured by running `common.Hash64([]byte("projectasap"))` - /// in sketchlib-go (see `common/hash_test.go::TestXxh3RegressionVectors`). - #[test] - fn hash_with_spec_matches_sketchlib_go() { - let spec = HashSpec::default(); - let got = hash_with_spec(&spec, b"projectasap"); - // From sketchlib-go::common::hash_test.go: Hash64(key) = 887548862923853302. - assert_eq!(got, 887548862923853302); - } - - /// derive_index slices the hash exactly the way Go's DeriveIndex - /// does. Sanity-check across rows and widths. - #[test] - fn derive_index_matches_go_bit_slicing() { - let spec = HashSpec::default(); - // Same hash Go would compute for "projectasap" with seedList[0]. - let h = hash_with_spec(&spec, b"projectasap"); - // 9-bit width (512 cols) → mask 0x1ff, shift = row*9. - for row in 0..3 { - let expected = ((h >> (row as u32 * 9)) & 0x1ff) as usize; - assert_eq!(derive_index(&spec, row, h, 512), expected); - } - // 10-bit width (1024 cols) → mask 0x3ff, shift = row*10. - for row in 0..3 { - let expected = ((h >> (row as u32 * 10)) & 0x3ff) as usize; - assert_eq!(derive_index(&spec, row, h, 1024), expected); - } - } - - /// derive_sign extracts the high-bit-minus-row exactly like Go. - #[test] - fn derive_sign_matches_go_high_bit() { - let spec = HashSpec::default(); - let h = hash_with_spec(&spec, b"projectasap"); - for row in 0..5 { - let bit = (h >> (63 - row as u32)) & 1; - let expected: i64 = if bit == 0 { -1 } else { 1 }; - assert_eq!(derive_sign(&spec, row, h), expected); - } - } - - #[test] - fn mask_bits_for_width_matches_go() { - assert_eq!(mask_bits_for_width(1), 1); - assert_eq!(mask_bits_for_width(2), 1); - assert_eq!(mask_bits_for_width(4), 2); - assert_eq!(mask_bits_for_width(512), 9); - assert_eq!(mask_bits_for_width(1024), 10); - assert_eq!(mask_bits_for_width(4096), 12); - } - - #[test] - fn default_hashspec_has_packed_derivation() { - let spec = HashSpec::default(); - assert_eq!(spec.seed_derivation, SeedDerivation::Packed); - assert_eq!(spec.canonical_seed_index, CANONICAL_HASH_SEED); - assert_eq!(spec.seed_list.len(), CANONICAL_HASH_SEED_TABLE.len()); - } -} diff --git a/src/common/mod.rs b/src/common/mod.rs index bbdce67f..7b31f178 100644 --- a/src/common/mod.rs +++ b/src/common/mod.rs @@ -19,10 +19,6 @@ /// Hashing utilities and seed definitions shared across sketches. pub mod hash; -/// Cross-language hash spec (`HashSpec`/`derive_index`/`derive_sign`) -/// shared by matrix-backed sketches that need byte parity with -/// `sketchlib-go`. -pub mod hashspec; pub mod heap; pub mod input; pub mod numerical; From a18c64c4c929dec5b7fa366e012cb74501b2e78d Mon Sep 17 00:00:00 2001 From: GordonYuanyc Date: Fri, 2 Oct 2026 04:14:07 -0600 Subject: [PATCH 36/36] docs: correct the PBT totals and tidy two entries Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 7 ++++--- docs/e2e_testing_harness.md | 2 +- 2 files changed, 5 insertions(+), 4 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 17ea743f..933e1983 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -49,7 +49,7 @@ signals a backwards-compatible change. law that sketch claims — merge algebra, path equivalence, the paper's own model, and the wire / ASAPv1 round trip. Shared generators, the `grid` reader and the `round_trip!` macro live in `tests/pbt/support.rs`; four - modules sit behind `experimental`. 433 tests under `--all-features`, 365 + modules sit behind `experimental`. 432 tests under `--all-features`, 364 under the default feature set. Every law was mutation-checked: the implementation was broken deliberately, the law confirmed red, and confirmed not to take unrelated laws down with it; a law no semantically real mutation @@ -239,8 +239,9 @@ signals a backwards-compatible change. `sketch_envelope::SketchState::Univmon`. - **Coco's protobuf format.** `proto/cocosketch/cocosketch.proto` and the generated `asap_sketchlib::proto::sketchlib::CocoSketchState` are gone, with - the `SketchEnvelope` oneof field `coco` (`sketch_envelope::SketchState::Coco`). Coco serializes only as - ASAPv1 (`Coco::serialize_to_bytes` / `deserialize_from_bytes`). + the `SketchEnvelope` oneof field `coco` (`sketch_envelope::SketchState::Coco`). + Coco serializes only as ASAPv1 (`Coco::serialize_to_bytes` / + `deserialize_from_bytes`). - **BREAKING: Elastic's protobuf form.** `proto/elasticsketch/elasticsketch.proto`, its message `ElasticState` (`asap_sketchlib::proto::sketchlib::ElasticState`), and the `SketchEnvelope` oneof variant `elastic` diff --git a/docs/e2e_testing_harness.md b/docs/e2e_testing_harness.md index 12109a72..d4fd7117 100644 --- a/docs/e2e_testing_harness.md +++ b/docs/e2e_testing_harness.md @@ -47,7 +47,7 @@ Sketch actually promises. The harness earned its keep immediately: its first run surfaced four real defects — a Nitro estimator that returned zero because inserts and queries hashed keys differently, a structural under-count from single-row updates, a -skip-draw rounding bug that halved the effective sampling rate, and a portable +skip-draw rounding bug that halved the effective sampling rate, and a DDSketch representative that violated the advertised α bound at bucket edges. None of these were caught by the existing unit tests.