diff --git a/CHANGELOG.md b/CHANGELOG.md index 80d85f98..fb95839e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -66,6 +66,9 @@ signals a backwards-compatible change. worker's seed and the aggregator's from one seed. The unseeded constructors draw from the thread generator. The seed is not serialized, and the ASAPv1 payload is unchanged. +- **ASAPv1 golden fixtures for `UnivMon`** (kind_id `0x10 0x00`): + `univmon_str_l3_2x4_h5`, `univmon_i64_l3_2x4_h5` and + `univmon_empty_l3_2x4_h5`, checked in `tests/asapv1_golden.rs`. ### Changed @@ -88,6 +91,15 @@ signals a backwards-compatible change. as an empty window like any other, and that window takes a place in `max_windows`. +### Removed + +- **UnivMon's proto format.** `UnivMon` serializes only as ASAPv1 + (`serialize_to_bytes` / `deserialize_from_bytes`, kind_id `0x10 0x00`). + Removed the `univmon` field (15) of `SketchEnvelope` (now reserved) and with + it `sketch_envelope::SketchState::Univmon`; `UnivMonState` and + `UnivMonLayer` stay in `proto/univmon/univmon.proto` only as + `HydraCell.univmon`. + ### Fixed - `KLLDynamic` panicked on every quantile query once a NaN had entered the diff --git a/asapv1_golden b/asapv1_golden index 8287a4ae..946e5c5f 160000 --- a/asapv1_golden +++ b/asapv1_golden @@ -1 +1 @@ -Subproject commit 8287a4ae04e6cfbfb3ed33dafbf1dac77d8f7fc1 +Subproject commit 946e5c5f89ebe990cf57636523076d279b498fc8 diff --git a/docs/asapv1_wire_format.md b/docs/asapv1_wire_format.md index fd14e70a..4038e34a 100644 --- a/docs/asapv1_wire_format.md +++ b/docs/asapv1_wire_format.md @@ -972,7 +972,7 @@ The encode side enforces rules 3 and 4 as well, and rejects a sketch holding the **A layer's `CountL2HH` is inlined, not nested.** The layer's `counts` and `l2` are appended to the outer sketch's own positional arrays, and its `rows` / `cols` / `seed_index` come from the outer metadata plus the layer's position. No nested envelope, magic, version or metadata map is emitted per layer. -**Metadata vs payload.** `layer_size`, `sketch_row`, `sketch_col` and `heap_size` are `init_univmon`'s four arguments — configuration that shapes the payload, so the descriptor carries them and every derived length is checked rather than re-stored. `key_type` is a structural param in the §3.5 sense: it fixes the element type of `keys`, and the payload cannot be read without it. One `key_type` covers every layer; a pyramid whose layers mix `HeapItem` variants has no single one and fails to serialize, as a Space-Saving summary does. Structural-param order is `layer_size, sketch_row, sketch_col, heap_size, key_type`. +**Metadata vs payload.** `layer_size`, `sketch_row`, `sketch_col` and `heap_size` are `init_univmon`'s four arguments — configuration that shapes the payload, so the descriptor carries them and every derived length is checked rather than re-stored. `key_type` is a structural param in the §3.5 sense: it fixes the element type of `keys`, and the payload cannot be read without it. One `key_type` covers every layer; a pyramid whose layers mix `HeapItem` variants has no single one and fails to serialize, as a Space-Saving summary does. Structural-param order is `layer_size, sketch_row, sketch_col, heap_size, key_type`. A pyramid with no heap entries emits `key_type = "u64"`. UnivMon carries the hash-spec group — it hashes, and a consumer must reproduce that hash to query a key — but carries **no seed-index key**. It hashes the bottom-layer finder at `BOTTOM_LAYER_FINDER` unconditionally: a fixed part of the algorithm, not a profile choice, exactly as Space-Saving's index 0 is (§3.5). The per-layer counter seed index *is* a real value, but it equals the layer's position, so it is derived and not stored, and the encoder rejects a layer whose seed index is not its position. `UnivMon` has no hasher type parameter, so its metadata is derived from `DefaultXxHasher`'s `HashProfile` — read live, never hardcoded — and a custom-profile envelope is different bytes and is rejected on decode. @@ -1341,14 +1341,14 @@ Good direction (more compact, higher fidelity, less Rust-internal duplication), 2. **Golden byte-vector fixtures** in one shared repo, [`sketchlib-golden-bytes`](https://github.com/ProjectASAP/sketchlib-golden-bytes), which each implementation mounts at `asapv1_golden/`; both languages decode and re-encode them byte-identically. These replace the `portable`-as-oracle round-trip test. 3. **This registry**, mirrored, never independently allocated. -Fixtures exist for six `kind_id`s — HLL's three estimators, Count-Min, Count Sketch and compact KLL — and `sketchlib-go` mirrors those. Every other kind here has a payload and **no fixture**, so this document is its only contract; `asapv1_golden/README.md` lists the gap. `portable` carries its own Go goldens. +`asapv1_golden/README.md`'s Coverage section lists which `kind_id`s have fixtures; for every other kind, this document is the only contract. `portable` carries its own Go goldens. **Hash profile on the Go side.** Rust derives the hash spec from a generic `HashProfile` bound on the hasher type; Go has no generic hasher type, so there is nothing to derive from. On the Go side the profile is simply **written into** the metadata on encode and **read from** it on decode. Go MUST validate the profile it reads (same fail-closed intent as Rust): a sketch is only mergeable/queryable if its `hash_profile_id` + seeds match the profile Go is prepared to reproduce. -Sequencing: (2) covers six `kind_id`s, and the one `native bytes == portable bytes` test is HLL-only, since no other `portable` type emits ASAPv1 bytes. +Sequencing: the one `native bytes == portable bytes` test is HLL-only, since no other `portable` type emits ASAPv1 bytes. Retire `portable` once the fixtures cover the rest. --- diff --git a/docs/features.md b/docs/features.md index 64f296a3..f9cfcfac 100644 --- a/docs/features.md +++ b/docs/features.md @@ -112,7 +112,7 @@ MessagePack (`rmp-serde`) support. **serde support** means the type derives `Ser | EHSketchList | Yes | Yes | | ExponentialHistogram | No | Yes | -Protobuf (prost): `.proto` definitions exist for CountMin, Count, HLL, DDSketch, KLL, Elastic, Coco, Hydra, and UnivMon. Rust conversion code is in progress. +Protobuf (prost): `SketchEnvelope` carries CountMin, Count, HLL, DDSketch, KLL, Elastic, Coco, and Hydra state; `UnivMonState` exists only inside `HydraCell`. Rust conversion code is in progress. ### API Stability diff --git a/docs/message_pack_format.md b/docs/message_pack_format.md index 12a2b91c..8559304e 100644 --- a/docs/message_pack_format.md +++ b/docs/message_pack_format.md @@ -169,10 +169,8 @@ decode → re-encode them byte-identically. The `kind_id` registry is mirrored verbatim with Go's `wire/asapmsgpack/magic_ids.go`, never independently allocated. -That proof covers the six `kind_id`s a fixture exists for — HLL's three -estimators, Count-Min, Count Sketch and compact KLL. Every other implemented -kind has **no ASAPv1 golden and therefore no ASAPv1 drift guard**; -[its README](https://github.com/ProjectASAP/sketchlib-golden-bytes#coverage) lists what is covered. +That proof covers only the `kind_id`s with a fixture, which +[its README's Coverage section](https://github.com/ProjectASAP/sketchlib-golden-bytes#coverage) lists. The older `portable` path is guarded separately, by the `sketchlib-go` goldens of its own listed above. diff --git a/docs/tests.md b/docs/tests.md index ccb1571c..62fc2941 100644 --- a/docs/tests.md +++ b/docs/tests.md @@ -2131,6 +2131,9 @@ Test file: [`tests/asapv1_golden.rs`](../tests/asapv1_golden.rs) | `hll_hip_p12_matches_golden` | The HIP variant's three running scalars travel on the wire beside its registers. | Builds `HllSketch::from_raw(HllVariant::Hip, 12, regs, 1.5, 2.5, 3.0)` over the shared P12 pattern, verifies `to_msgpack()` equals `asapv1_golden/hll_hip_p12.hex`, that `HyperLogLogHIPP12::deserialize_from_bytes` re-serializes the fixture identically, and that `HllSketch::from_msgpack` reports the registers, `HllVariant::Hip`, `hip_kxq0` `1.5`, `hip_kxq1` `2.5` and `hip_est` `3.0`. | | `kll_f64_k200_matches_golden` | A deterministic f64 KLL serializes to its golden bytes, which round-trip unchanged. | Verifies `KLL::::init_kll_with_seed(200, 42)` fed `1..=50` (below the level-0 capacity, so no compaction fires) serializes to `asapv1_golden/kll_f64_k200.hex`, and that decoding the fixture re-serializes to the same bytes and reports `quantile(0.0)` `1.0` and `quantile(1.0)` `50.0`. | | `kll_i64_k200_matches_golden` | A deterministic i64 KLL serializes to its golden bytes, which round-trip unchanged. | Verifies `KLL::::init_kll_with_seed(200, 42)` fed `1..=50` serializes to `asapv1_golden/kll_i64_k200.hex`, and that decoding the fixture re-serializes to the same bytes and reports `quantile(0.0)` `1.0` and `quantile(1.0)` `50.0`. | +| `univmon_empty_l3_2x4_h5_matches_golden` | A freshly constructed pyramid has one encoding, pinned by its golden. | Verifies `UnivMon::init_univmon(5, 2, 4, 3)` serializes to `asapv1_golden/univmon_empty_l3_2x4_h5.hex`; reading the envelope by hand gives kind_id `0x10 0x00`, shape `(3, 2, 4, 5)`, `key_type` `u64`, 24 zero counts, 6 zero `l2` accumulators, `heap_lens` `[0, 0, 0]`, no keys or heap counts, candidate flags `[true, true, true]`, `bucket_size` `0` and `update_mode` `0`; the decode matches the source's shape, weight, flags, counters, per-layer `get_l2()` and heaps, and re-serializes identically. | +| `univmon_i64_l3_2x4_h5_matches_golden` | The `i64` key type reaches the bytes, and only it differs from the string fixture. | Builds the same layers as the string fixture with heap keys `i64::MIN`, `-1`, `-129` on layer 0, `128` on layer 1 and `4294967296`, `7` on layer 2, verifies the bytes equal `asapv1_golden/univmon_i64_l3_2x4_h5.hex`, that the hand-read envelope carries shape `(3, 2, 4, 5)` and those keys under `key_type` `i64` with the same counts, `l2`, heap lengths, heap counts, flags, weight and mode, that the decode matches the source and re-serializes identically, that layer 0's heap holds `HeapItem::I64(i64::MIN)`, and that the fixture differs from the string one. | +| `univmon_str_l3_2x4_h5_matches_golden` | A known 3-layer pyramid with string keys serializes to its golden bytes and decodes back to that state. | Builds `UnivMon::init_univmon(5, 2, 4, 3)` (heap 5, 2 rows, 4 columns, 3 layers: pairwise distinct), writes layer 0's cells `[[0, 127, 128, 65536], [-1, -33, -32768, -2147483648]]`, layer 1's `[[3, -2, 0, 1], [0, 0, 5, -4]]` and layer 2's `[[0, 7, 0, 0], [-6, 0, 0, 0]]` through `L2HH::apply_delta`, seats `alpha`=`65536`, `beta`=`300`, `delta`=`128` in layer 0's heap, `gamma`=`5` in layer 1's and `epsilon`=`9`, `zeta`=`2` in layer 2's, calls `set_total_weight(70_000)` and `mark_layer_candidates_incomplete(1)`, and verifies the bytes equal `asapv1_golden/univmon_str_l3_2x4_h5.hex`. Reading the envelope by hand gives kind_id `0x10 0x00`, shape `(3, 2, 4, 5)`, `key_type` `string`, the 24 cells in layer then row-major order, `l2` `[4294999809, 4611686019501130818, 14, 41, 49, 36]`, `heap_lens` `[3, 1, 2]`, the six keys and heap counts `[65536, 300, 128, 5, 9, 2]` in that order, flags `[true, false, true]`, `bucket_size` `70000` and `update_mode` `1`; the decode matches the source and re-serializes identically. | ### MessagePack Envelope Compatibility @@ -2202,7 +2205,7 @@ Test file: [`tests/xtest_consumer.rs`](../tests/xtest_consumer.rs) | test_name | test_description | what_is_tested | | --- | --- | --- | -| `cross_language_proto` | Protobuf envelopes read from the cross-language fixture directory answer their point queries in band. | Asserts nothing unless fixtures are available: with no `XTEST_DIR` and no discoverable producer directory holding `tests/cross_language/xtest_producer_test.go`, it prints a skip line and returns. With fixtures it decodes nine `SketchEnvelope` files and records a pass or failure for each — `countmin.pb` min-frequency for `item:42` at least `101`, `kll.pb` p50 within 5% of `5000` and p99 within 5% of `9900`, `ddsketch.pb` the same two quantiles within 2%, `hll.pb` an ErtlMLE cardinality inside `[40_000, 65_000]`, `countsketch.pb` `cs:hot` at least `200`, `coco.pb` `coco:hot` at least `500`, `elastic.pb` `elephant` at least `900`, `univmon.pb` a cardinality inside `[1_000, 15_000]`, `hydra.pb` `hydra:42` at least `51` — using the hash seeds `0xcafe3553`, `0x6a09e667` and `0xbb67ae85`, and, when `countmin_sampled.pb` and `hll_sampled.pb` exist, requires `effective_sample_p` of `0.1` and a `rescale_count` by `1/p` within 5% of `100_000` and 6% of `200_000`. A single `assert!` at the end fails if any check failed; a missing or mistyped envelope panics on the spot. | +| `cross_language_proto` | Protobuf envelopes read from the cross-language fixture directory answer their point queries in band. | Asserts nothing unless fixtures are available: with no `XTEST_DIR` and no discoverable producer directory holding `tests/cross_language/xtest_producer_test.go`, it prints a skip line and returns. With fixtures it decodes eight `SketchEnvelope` files and records a pass or failure for each — `countmin.pb` min-frequency for `item:42` at least `101`, `kll.pb` p50 within 5% of `5000` and p99 within 5% of `9900`, `ddsketch.pb` the same two quantiles within 2%, `hll.pb` an ErtlMLE cardinality inside `[40_000, 65_000]`, `countsketch.pb` `cs:hot` at least `200`, `coco.pb` `coco:hot` at least `500`, `elastic.pb` `elephant` at least `900`, `hydra.pb` `hydra:42` at least `51` — using the hash seeds `0xcafe3553`, `0x6a09e667` and `0xbb67ae85`, and, when `countmin_sampled.pb` and `hll_sampled.pb` exist, requires `effective_sample_p` of `0.1` and a `rescale_count` by `1/p` within 5% of `100_000` and 6% of `200_000`. A single `assert!` at the end fails if any check failed; a missing or mistyped envelope panics on the spot. | ### HLL Custom Precision diff --git a/proto/sketchlib.proto b/proto/sketchlib.proto index 0a6d8193..d79e63b5 100644 --- a/proto/sketchlib.proto +++ b/proto/sketchlib.proto @@ -9,7 +9,6 @@ // hll/hll.proto — HyperLogLogState, HLLVariant // kll/kll.proto — KLLState, CoinState // ddsketch/ddsketch.proto — DDSketchState -// univmon/univmon.proto — UnivMonState, UnivMonLayer // hydra/hydra.proto — HydraState, HydraCounterType, HydraCell // // Design goals: @@ -34,7 +33,6 @@ import "countsketch/countsketch.proto"; import "hll/hll.proto"; import "kll/kll.proto"; import "ddsketch/ddsketch.proto"; -import "univmon/univmon.proto"; import "hydra/hydra.proto"; import "cocosketch/cocosketch.proto"; import "elasticsketch/elasticsketch.proto"; @@ -71,12 +69,13 @@ message SketchEnvelope { HyperLogLogState hll = 12; KLLState kll = 13; DDSketchState ddsketch = 14; - UnivMonState univmon = 15; HydraState hydra = 16; CocoSketchState coco = 17; ElasticState elastic = 18; } + reserved 15; + reserved "univmon"; reserved 19 to 31; // Reserved for future sketch types. // Geometric / NitroSketch sampling probability for the producing sketch. diff --git a/src/proto/generated/sketchlib.v1.rs b/src/proto/generated/sketchlib.v1.rs index 2c098d3f..abd34660 100644 --- a/src/proto/generated/sketchlib.v1.rs +++ b/src/proto/generated/sketchlib.v1.rs @@ -892,7 +892,7 @@ pub struct SketchEnvelope { /// The sketch payload. Exactly one field must be set. #[prost( oneof = "sketch_envelope::SketchState", - tags = "10, 11, 12, 13, 14, 15, 16, 17, 18" + tags = "10, 11, 12, 13, 14, 16, 17, 18" )] pub sketch_state: ::core::option::Option, } @@ -911,8 +911,6 @@ pub mod sketch_envelope { Kll(super::KllState), #[prost(message, tag = "14")] Ddsketch(super::DdSketchState), - #[prost(message, tag = "15")] - Univmon(super::UnivMonState), #[prost(message, tag = "16")] Hydra(super::HydraState), #[prost(message, tag = "17")] diff --git a/tests/asapv1_golden.rs b/tests/asapv1_golden.rs index 30b41651..5b7370d4 100644 --- a/tests/asapv1_golden.rs +++ b/tests/asapv1_golden.rs @@ -9,9 +9,12 @@ //! See `asapv1_golden/README.md`. use asap_sketchlib::{ - Classic, Count, CountMin, ErtlMLE, FastPath, HllSketch, HllVariant, HyperLogLogHIPP12, - HyperLogLogP12, KLL, MessagePackCodec, RegularPath, Vector2D, + Classic, Count, CountDelta, CountMin, DataInput, ErtlMLE, FastPath, HeapItem, HllSketch, + HllVariant, HyperLogLogHIPP12, HyperLogLogP12, KLL, L2HH, MessagePackCodec, RegularPath, + UnivMon, Vector2D, }; +use serde::Deserialize; +use serde::de::DeserializeOwned; fn decode_hex(s: &str) -> Vec { let s = s.trim(); @@ -32,6 +35,9 @@ const GOLDEN_CS_FAST: &str = include_str!("../asapv1_golden/cs_i64_fast_2x4.hex" const GOLDEN_CS_I32: &str = include_str!("../asapv1_golden/cs_i32_regular_2x4.hex"); const GOLDEN_KLL_F64: &str = include_str!("../asapv1_golden/kll_f64_k200.hex"); const GOLDEN_KLL_I64: &str = include_str!("../asapv1_golden/kll_i64_k200.hex"); +const GOLDEN_UNIVMON_STR: &str = include_str!("../asapv1_golden/univmon_str_l3_2x4_h5.hex"); +const GOLDEN_UNIVMON_I64: &str = include_str!("../asapv1_golden/univmon_i64_l3_2x4_h5.hex"); +const GOLDEN_UNIVMON_EMPTY: &str = include_str!("../asapv1_golden/univmon_empty_l3_2x4_h5.hex"); /// The known P12 register pattern shared by all three HLL fixtures. fn p12_registers() -> Vec { @@ -287,3 +293,271 @@ fn kll_i64_k200_matches_golden() { assert_eq!(decoded.quantile(0.0), 1.0); assert_eq!(decoded.quantile(1.0), 50.0); } + +// --------------------------------------------------------------------------- +// UnivMon: 3 layers of 2x4 CountL2HH with heap capacity 5. Cells are written +// through `L2HH::apply_delta` (which carries each row's L2 accumulator) and +// heap entries through `HHHeap::update`; no hash reaches the bytes. +// --------------------------------------------------------------------------- + +/// `init_univmon`'s arguments `(heap_size, sketch_row, sketch_col, layer_size)`, +/// pairwise distinct so no length can stand in for another. +const UNIVMON_SHAPE: (usize, usize, usize, usize) = (5, 2, 4, 3); + +/// The layers' counters; layer 0 holds `CS_VALS`. +const UNIVMON_LAYERS: [[[i64; 4]; 2]; 3] = [ + CS_VALS, + [[3, -2, 0, 1], [0, 0, 5, -4]], + [[0, 7, 0, 0], [-6, 0, 0, 0]], +]; + +/// Each row's sum of squared cells; both of layer 0's are uint64. +const UNIVMON_L2: [i64; 6] = [4_294_999_809, 4_611_686_019_501_130_818, 14, 41, 49, 36]; + +/// Entries per layer's heap. +const UNIVMON_HEAP_LENS: [u32; 3] = [3, 1, 2]; + +/// The heap counts in emitted order: layer by layer, each descending. +const UNIVMON_HEAP_COUNTS: [i64; 6] = [65536, 300, 128, 5, 9, 2]; + +/// The string fixture's keys in emitted order. +const UNIVMON_STR_KEYS: [&str; 6] = ["alpha", "beta", "delta", "gamma", "epsilon", "zeta"]; + +/// The i64 fixture's keys in emitted order: int64, negative fixint, int16, +/// uint8, uint64, positive fixint. +const UNIVMON_I64_KEYS: [i64; 6] = [i64::MIN, -1, -129, 128, 4_294_967_296, 7]; + +/// A UnivMon payload `[counts, l2, heap_lens, keys, heap_counts, +/// candidate_complete, bucket_size, update_mode]`. +type UnivMonPayloadView = ( + Vec, + Vec, + Vec, + Vec, + Vec, + Vec, + u64, + u8, +); + +/// The structural keys of the UnivMon metadata map. +#[derive(Deserialize)] +struct UnivMonShape { + layer_size: u32, + sketch_row: u32, + sketch_col: u32, + heap_size: u32, + key_type: String, +} + +/// Splits an ASAPv1 envelope by hand and decodes its UnivMon metadata and payload. +fn univmon_envelope( + bytes: &[u8], +) -> (Vec, UnivMonShape, UnivMonPayloadView) { + assert_eq!(&bytes[..7], b"ASAPv1\x01", "magic and version"); + let kind_len = bytes[7] as usize; + let kind_id = bytes[8..8 + kind_len].to_vec(); + let at = 8 + kind_len; + let meta_len = u32::from_be_bytes(bytes[at..at + 4].try_into().unwrap()) as usize; + let payload_len = u32::from_be_bytes(bytes[at + 4..at + 8].try_into().unwrap()) as usize; + let metadata = &bytes[at + 8..at + 8 + meta_len]; + let payload = &bytes[at + 8 + meta_len..]; + assert_eq!(payload.len(), payload_len, "payload length prefix"); + ( + kind_id, + rmp_serde::from_slice(metadata).expect("UnivMon metadata"), + rmp_serde::from_slice(payload).expect("UnivMon payload"), + ) +} + +/// The populated pyramid: `keys` fill the heaps in emitted order, cut by +/// `UNIVMON_HEAP_LENS`, at `UNIVMON_HEAP_COUNTS`; layer 1 is incomplete. +fn univmon_known(keys: [DataInput<'_>; 6]) -> UnivMon { + let (heap_size, sketch_row, sketch_col, layer_size) = UNIVMON_SHAPE; + let mut um = UnivMon::init_univmon(heap_size, sketch_row, sketch_col, layer_size); + for (layer, cells) in UNIVMON_LAYERS.iter().enumerate() { + for (row, values) in cells.iter().enumerate() { + for (col, &value) in values.iter().enumerate() { + um.l2_sketch_layers[layer].apply_delta(CountDelta { + row: row as u32, + col: col as u32, + value: value as i32, + }); + } + } + } + let layers = UNIVMON_HEAP_LENS + .iter() + .enumerate() + .flat_map(|(layer, &len)| std::iter::repeat_n(layer, len as usize)); + for ((layer, key), count) in layers.zip(keys.iter()).zip(UNIVMON_HEAP_COUNTS) { + um.hh_layers[layer].update(key, count); + } + um.set_total_weight(70_000); + um.mark_layer_candidates_incomplete(1); + um +} + +/// Layer counters flattened in wire order. +fn univmon_counts(um: &UnivMon) -> Vec { + (0..um.layer_size) + .flat_map(|layer| { + let L2HH::COUNT(counter) = &um.l2_sketch_layers[layer]; + counter.as_storage().as_slice().to_vec() + }) + .collect() +} + +/// Each layer's heap as `(key, count)`, sorted. +fn univmon_heaps(um: &UnivMon) -> Vec> { + (0..um.layer_size) + .map(|layer| { + let mut entries: Vec<(String, i64)> = um.hh_layers[layer] + .heap() + .iter() + .map(|item| (format!("{:?}", item.key), item.count)) + .collect(); + entries.sort(); + entries + }) + .collect() +} + +/// Decodes `want`, checks it against `source`, and checks the re-encode. +fn assert_univmon_decodes_to(want: &[u8], source: &UnivMon) { + let decoded = UnivMon::deserialize_from_bytes(want).expect("decode"); + assert_eq!( + ( + decoded.layer_size, + decoded.sketch_row, + decoded.sketch_col, + decoded.heap_size + ), + (3, 2, 4, 5) + ); + assert_eq!(decoded.bucket_size, source.bucket_size); + assert_eq!(decoded.candidates_complete(), source.candidates_complete()); + assert_eq!(univmon_counts(&decoded), univmon_counts(source)); + for layer in 0..3 { + assert_eq!( + decoded.l2_sketch_layers[layer].get_l2(), + source.l2_sketch_layers[layer].get_l2() + ); + } + assert_eq!(univmon_heaps(&decoded), univmon_heaps(source)); + assert_eq!(decoded.serialize_to_bytes().expect("re-serialize"), want); +} + +fn univmon_known_counts() -> Vec { + UNIVMON_LAYERS.iter().flatten().flatten().copied().collect() +} + +#[test] +fn univmon_str_l3_2x4_h5_matches_golden() { + let want = decode_hex(GOLDEN_UNIVMON_STR); + + let sketch = univmon_known(UNIVMON_STR_KEYS.map(DataInput::Str)); + let got = sketch.serialize_to_bytes().expect("serialize"); + assert_eq!(got, want, "UnivMon string-key bytes diverge from golden"); + + let (kind_id, shape, payload) = univmon_envelope::(&want); + assert_eq!(kind_id, [0x10, 0x00]); + assert_eq!( + ( + shape.layer_size, + shape.sketch_row, + shape.sketch_col, + shape.heap_size + ), + (3, 2, 4, 5) + ); + assert_eq!(shape.key_type, "string"); + let (counts, l2, heap_lens, keys, heap_counts, flags, bucket_size, update_mode) = payload; + assert_eq!(counts, univmon_known_counts()); + assert_eq!(l2, UNIVMON_L2); + assert_eq!(heap_lens, UNIVMON_HEAP_LENS); + assert_eq!(keys, UNIVMON_STR_KEYS); + assert_eq!(heap_counts, UNIVMON_HEAP_COUNTS); + assert_eq!(flags, [true, false, true]); + assert_eq!(bucket_size, 70_000); + assert_eq!(update_mode, 1, "set_total_weight selects the standard mode"); + + assert_univmon_decodes_to(&want, &sketch); +} + +#[test] +fn univmon_i64_l3_2x4_h5_matches_golden() { + let want = decode_hex(GOLDEN_UNIVMON_I64); + + let sketch = univmon_known(UNIVMON_I64_KEYS.map(DataInput::I64)); + let got = sketch.serialize_to_bytes().expect("serialize"); + assert_eq!(got, want, "UnivMon i64-key bytes diverge from golden"); + + let (kind_id, shape, payload) = univmon_envelope::(&want); + assert_eq!(kind_id, [0x10, 0x00]); + assert_eq!( + ( + shape.layer_size, + shape.sketch_row, + shape.sketch_col, + shape.heap_size + ), + (3, 2, 4, 5) + ); + assert_eq!(shape.key_type, "i64"); + let (counts, l2, heap_lens, keys, heap_counts, flags, bucket_size, update_mode) = payload; + assert_eq!(counts, univmon_known_counts()); + assert_eq!(l2, UNIVMON_L2); + assert_eq!(heap_lens, UNIVMON_HEAP_LENS); + assert_eq!(keys, UNIVMON_I64_KEYS); + assert_eq!(heap_counts, UNIVMON_HEAP_COUNTS); + assert_eq!(flags, [true, false, true]); + assert_eq!((bucket_size, update_mode), (70_000, 1)); + + assert_univmon_decodes_to(&want, &sketch); + let decoded = UnivMon::deserialize_from_bytes(&want).expect("decode"); + assert!( + decoded.hh_layers[0] + .heap() + .iter() + .any(|item| item.key == HeapItem::I64(i64::MIN)) + ); + + // Same layers and heap counts, other keys: only `key_type` and `keys` differ. + assert_ne!(want, decode_hex(GOLDEN_UNIVMON_STR)); +} + +#[test] +fn univmon_empty_l3_2x4_h5_matches_golden() { + let want = decode_hex(GOLDEN_UNIVMON_EMPTY); + + let (heap_size, sketch_row, sketch_col, layer_size) = UNIVMON_SHAPE; + let sketch = UnivMon::init_univmon(heap_size, sketch_row, sketch_col, layer_size); + let got = sketch.serialize_to_bytes().expect("serialize"); + assert_eq!(got, want, "empty UnivMon bytes diverge from golden"); + + let (kind_id, shape, payload) = univmon_envelope::(&want); + assert_eq!(kind_id, [0x10, 0x00]); + assert_eq!( + ( + shape.layer_size, + shape.sketch_row, + shape.sketch_col, + shape.heap_size + ), + (3, 2, 4, 5) + ); + assert_eq!( + shape.key_type, "u64", + "an empty pyramid records key_type u64" + ); + let (counts, l2, heap_lens, keys, heap_counts, flags, bucket_size, update_mode) = payload; + assert_eq!(counts, [0; 24]); + assert_eq!(l2, [0; 6]); + assert_eq!(heap_lens, [0, 0, 0]); + assert!(keys.is_empty() && heap_counts.is_empty()); + assert_eq!(flags, [true, true, true]); + assert_eq!((bucket_size, update_mode), (0, 0)); + + assert_univmon_decodes_to(&want, &sketch); +} diff --git a/tests/xtest_consumer.rs b/tests/xtest_consumer.rs index 0113ff45..65d6588e 100644 --- a/tests/xtest_consumer.rs +++ b/tests/xtest_consumer.rs @@ -12,7 +12,6 @@ //! countsketch.pb — CountSketchState with float64 signed counters //! coco.pb — CocoSketchState (hash+val+hasKey buckets) //! elastic.pb — ElasticState (heavy buckets + light CountMin) -//! univmon.pb — UnivMonState (layered CountSketch + TopK heaps) //! hydra.pb — HydraState (CM-cell grid) //! //! Usage: @@ -360,41 +359,6 @@ fn cross_language_proto() { all_ok = false; } - // ----------------------------------------------------------------------- - // UnivMon - // ----------------------------------------------------------------------- - println!(); - println!("[UnivMon] Step 1/3 — Read univmon.pb"); - let bytes = read_file(in_dir.join("univmon.pb")); - let env = SketchEnvelope::decode(bytes.as_slice()).expect("decode univmon envelope"); - - println!( - "[UnivMon] Step 2/3 — Validate envelope (format_version={}, producer={})", - env.format_version, - env.producer.as_ref().map_or("?", |p| &p.library) - ); - - let um_state = match env.sketch_state { - Some(sketch_envelope::SketchState::Univmon(ref s)) => s.clone(), - other => panic!("expected UnivMon sketch_state, got {other:?}"), - }; - - println!( - "[UnivMon] layer_size={} sketch_rows={} sketch_cols={} heap_size={}", - um_state.layer_size, um_state.sketch_rows, um_state.sketch_cols, um_state.heap_size - ); - - let um_card = univmon_cardinality(&um_state); - // Note: the g-sum heuristic typically underestimates (Go itself reports ~4250 for 10k inserts). - // We verify the Rust result matches Go's algorithm rather than the true cardinality. - println!("[UnivMon] Step 3/3 — cardinality ≈ {um_card:.0} (g-sum heuristic, Go also ~4250)"); - if (1_000.0..=15_000.0).contains(&um_card) { - println!("[UnivMon] PASS"); - } else { - eprintln!("[UnivMon] FAIL: cardinality {um_card:.0} not in [1000, 15000]"); - all_ok = false; - } - // ----------------------------------------------------------------------- // HydraSketch // ----------------------------------------------------------------------- @@ -916,98 +880,6 @@ fn elastic_light_min(state: &ElasticState, hash: u64) -> i64 { } } -// --------------------------------------------------------------------------- -// UnivMon cardinality (g-sum heuristic) -// --------------------------------------------------------------------------- -// Mirrors Go's UnivSketch.calcGSumHeuristic(g=1, isCard=true): -// Y[L-1] = count of heap items at top layer with count > threshold -// for i from L-2 down to 0: -// tmp = Σ coe*1 for items with count > threshold -// coe = 1 - 2 * ((Hash64(key) >> (i+1)) & 1) -// Y[i] = 2*Y[i+1] + tmp -// return Y[0] -// l2_val = sqrt(median_of_first_3(layer.sketch.l2)) - -fn univmon_cardinality(state: &UnivMonState) -> f64 { - let nlayers = state.layers.len(); - if nlayers == 0 { - return 0.0; - } - - let mut y = vec![0.0f64; nlayers]; - - // Top layer - let top = &state.layers[nlayers - 1]; - let l2_val = cs_l2_from_state(top.sketch.as_ref()); - let threshold = (l2_val * 0.01) as i64; - let mut tmp = 0.0f64; - if let Some(heap) = &top.heap { - for entry in &heap.entries { - if entry.count as i64 > threshold { - tmp += 1.0; - } - } - } - y[nlayers - 1] = tmp; - - // Lower layers from L-2 down to 0 - for i in (0..nlayers - 1).rev() { - tmp = 0.0; - let layer = &state.layers[i]; - let l2_val = cs_l2_from_state(layer.sketch.as_ref()); - let threshold = (l2_val * 0.01) as i64; - - if let Some(heap) = &layer.heap { - for entry in &heap.entries { - if entry.count as i64 > threshold { - let h = xxh3_64_seeded(SEED_0, entry.key.as_bytes()); - let bit = (h >> (i + 1)) & 1; - let coe = 1.0 - 2.0 * bit as f64; - tmp += coe; - } - } - } - y[i] = 2.0 * y[i + 1] + tmp; - } - - y[0] -} - -/// cs_l2_from_state mirrors Go's CountSketchUniv.cs_l2(): -/// f2_value = MedianOfThree(l2[0], l2[1], l2[2]) -/// return sqrt(f2_value) -/// The portable l2 values are raw int64 cast to float64. -fn cs_l2_from_state(cs: Option<&CountSketchState>) -> f64 { - let cs = match cs { - Some(s) => s, - None => return 0.0, - }; - let l2 = &cs.l2; - if l2.len() < 3 { - return 0.0; - } - let med = median_of_three_f64(l2[0], l2[1], l2[2]); - med.abs().sqrt() -} - -fn median_of_three_f64(a: f64, b: f64, c: f64) -> f64 { - if a <= b { - if b <= c { - b - } else if a <= c { - c - } else { - a - } - } else if a <= c { - a - } else if b <= c { - c - } else { - b - } -} - // --------------------------------------------------------------------------- // HydraSketch CountMin frequency query // ---------------------------------------------------------------------------