diff --git a/CHANGELOG.md b/CHANGELOG.md index 80d85f98..b21c27da 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -66,6 +66,8 @@ signals a backwards-compatible change. worker's seed and the aggregator's from one seed. The unseeded constructors draw from the thread generator. The seed is not serialized, and the ASAPv1 payload is unchanged. +- **ASAPv1 golden fixtures for Elastic**: `elastic_4b_2x4` and + `elastic_4b_2x4_stale` in `asapv1_golden/`, checked by `tests/asapv1_golden.rs`. ### Changed @@ -88,6 +90,15 @@ signals a backwards-compatible change. as an empty window like any other, and that window takes a place in `max_windows`. +### Removed + +- **BREAKING: Elastic's protobuf form.** `proto/elasticsketch/elasticsketch.proto`, + its message `ElasticState` (`asap_sketchlib::proto::sketchlib::ElasticState`), + and the `SketchEnvelope` oneof variant `elastic` + (`sketch_envelope::SketchState::Elastic`, field 18, now reserved) are gone. + `Elastic::serialize_to_bytes` / `deserialize_from_bytes` (ASAPv1, `0x0b 0x00`) + is Elastic's only serialization. + ### Fixed - `KLLDynamic` panicked on every quantile query once a NaN had entered the diff --git a/asapv1_golden b/asapv1_golden index 8287a4ae..c2aed956 160000 --- a/asapv1_golden +++ b/asapv1_golden @@ -1 +1 @@ -Subproject commit 8287a4ae04e6cfbfb3ed33dafbf1dac77d8f7fc1 +Subproject commit c2aed956cebd5ae6833a0f2752b913ee916bdf9c diff --git a/docs/asapv1_wire_format.md b/docs/asapv1_wire_format.md index fd14e70a..30ba5496 100644 --- a/docs/asapv1_wire_format.md +++ b/docs/asapv1_wire_format.md @@ -1341,14 +1341,14 @@ Good direction (more compact, higher fidelity, less Rust-internal duplication), 2. **Golden byte-vector fixtures** in one shared repo, [`sketchlib-golden-bytes`](https://github.com/ProjectASAP/sketchlib-golden-bytes), which each implementation mounts at `asapv1_golden/`; both languages decode and re-encode them byte-identically. These replace the `portable`-as-oracle round-trip test. 3. **This registry**, mirrored, never independently allocated. -Fixtures exist for six `kind_id`s — HLL's three estimators, Count-Min, Count Sketch and compact KLL — and `sketchlib-go` mirrors those. Every other kind here has a payload and **no fixture**, so this document is its only contract; `asapv1_golden/README.md` lists the gap. `portable` carries its own Go goldens. +`asapv1_golden/README.md`'s Coverage section lists which `kind_id`s have fixtures; for every other kind this document is the only contract. `portable` carries its own Go goldens. **Hash profile on the Go side.** Rust derives the hash spec from a generic `HashProfile` bound on the hasher type; Go has no generic hasher type, so there is nothing to derive from. On the Go side the profile is simply **written into** the metadata on encode and **read from** it on decode. Go MUST validate the profile it reads (same fail-closed intent as Rust): a sketch is only mergeable/queryable if its `hash_profile_id` + seeds match the profile Go is prepared to reproduce. -Sequencing: (2) covers six `kind_id`s, and the one `native bytes == portable bytes` test is HLL-only, since no other `portable` type emits ASAPv1 bytes. +Sequencing: (2) covers the kinds that README lists, and the one `native bytes == portable bytes` test is HLL-only, since no other `portable` type emits ASAPv1 bytes. Retire `portable` once the fixtures cover the rest. --- diff --git a/docs/features.md b/docs/features.md index 64f296a3..2049da12 100644 --- a/docs/features.md +++ b/docs/features.md @@ -112,7 +112,7 @@ MessagePack (`rmp-serde`) support. **serde support** means the type derives `Ser | EHSketchList | Yes | Yes | | ExponentialHistogram | No | Yes | -Protobuf (prost): `.proto` definitions exist for CountMin, Count, HLL, DDSketch, KLL, Elastic, Coco, Hydra, and UnivMon. Rust conversion code is in progress. +Protobuf (prost): `.proto` definitions exist for CountMin, Count, HLL, DDSketch, KLL, Coco, Hydra, and UnivMon. Rust conversion code is in progress. ### API Stability diff --git a/docs/message_pack_format.md b/docs/message_pack_format.md index 12a2b91c..59b211b5 100644 --- a/docs/message_pack_format.md +++ b/docs/message_pack_format.md @@ -169,10 +169,9 @@ decode → re-encode them byte-identically. The `kind_id` registry is mirrored verbatim with Go's `wire/asapmsgpack/magic_ids.go`, never independently allocated. -That proof covers the six `kind_id`s a fixture exists for — HLL's three -estimators, Count-Min, Count Sketch and compact KLL. Every other implemented -kind has **no ASAPv1 golden and therefore no ASAPv1 drift guard**; -[its README](https://github.com/ProjectASAP/sketchlib-golden-bytes#coverage) lists what is covered. +That proof covers only the `kind_id`s a fixture exists for, which +[its README's Coverage section](https://github.com/ProjectASAP/sketchlib-golden-bytes#coverage) +lists; every other implemented kind has **no ASAPv1 drift guard**. The older `portable` path is guarded separately, by the `sketchlib-go` goldens of its own listed above. diff --git a/docs/tests.md b/docs/tests.md index ccb1571c..ff735ae5 100644 --- a/docs/tests.md +++ b/docs/tests.md @@ -2126,6 +2126,8 @@ Test file: [`tests/asapv1_golden.rs`](../tests/asapv1_golden.rs) | `cs_i32_regular_2x4_matches_golden` | The i32 counter type reaches the metadata, and an i32 golden is not an i64 sketch. | Verifies `Count::, RegularPath>::from_storage` over `[[0, 127, 128, 65536], [-1, -33, -32768, -2147483648]]` cast to `i32` serializes to `asapv1_golden/cs_i32_regular_2x4.hex`, that decoding it recovers the eight cells and re-serializes identically, that the i32 and i64 regular fixtures have equal length yet differ, and that the i32 fixture fails to decode as `Count, RegularPath>`. | | `cs_i64_fast_2x4_matches_golden` | The fast-path mode reaches the bytes, so the same matrix encodes differently per path. | Verifies `Count::, FastPath>::from_storage` over `[[0, 127, 128, 65536], [-1, -33, -32768, -2147483648]]` serializes to `asapv1_golden/cs_i64_fast_2x4.hex`, that decoding it recovers the eight cells and re-serializes identically, and that the fixture differs from `cs_i64_regular_2x4.hex` built from the same matrix. | | `cs_i64_regular_2x4_matches_golden` | A known signed Count-Sketch matrix serializes to its checked-in golden bytes. | Verifies `Count::, RegularPath>::from_storage` over `[[0, 127, 128, 65536], [-1, -33, -32768, -2147483648]]` — sweeping the msgpack integer widths in both directions — serializes to `asapv1_golden/cs_i64_regular_2x4.hex`, and that decoding the fixture reports `rows()` `2`, `cols()` `4`, the eight cells, and a byte-identical re-serialization. | +| `elastic_4b_2x4_matches_golden` | A known Elastic heavy table and light layer serialize to their golden bytes. | Sets a 4-bucket `2x4` sketch's buckets to a free one carrying the eviction flag, a 31-byte flow id with votes `127`/`128`, an empty flow id with votes `1`/`65535` and the eviction flag, and a 32-byte flow id with votes `i32::MAX`/`256` and the flag, and its light cells to `[[0, 255, 65536, i32::MAX], [-1, -33, -32768, i32::MIN]]`; verifies the bytes equal `asapv1_golden/elastic_4b_2x4.hex`, and that decoding the fixture recovers every bucket, `bktlen` `4`, the light dimensions and cells, and a byte-identical re-serialization. | +| `elastic_4b_2x4_stale_matches_golden` | `stale_copies` reaches the bytes, and nothing else differs. | Builds the same state on a 2-bucket sketch after `expand_heavy()`, verifies the bytes equal `asapv1_golden/elastic_4b_2x4_stale.hex`, that decoding recovers the same state and re-serializes identically, and that the two fixtures have equal length and differ in exactly one byte, `0xc2` against `0xc3`. | | `hll_classic_p12_matches_golden` | A known P12 Classic register pattern serializes to its golden bytes on both the portable and native paths. | Builds `HllSketch::from_raw(HllVariant::Regular, 12, regs, 0.0, 0.0, 0.0)` where `regs` is 4,096 zeros with `[0] = 1`, `[1] = 7`, `[100] = 42` and `[4095] = 3`, verifies `to_msgpack()` equals `asapv1_golden/hll_classic_p12.hex`, that `HyperLogLogP12::::deserialize_from_bytes` on the fixture recovers those registers and re-serializes identically, and that `HllSketch::from_msgpack` reports the same registers, `precision` `12` and `HllVariant::Regular`. | | `hll_ertl_mle_p12_matches_golden` | The Datafusion variant tag reaches the bytes for the same P12 register pattern. | Builds `HllSketch::from_raw(HllVariant::Datafusion, 12, regs, 0.0, 0.0, 0.0)` over the shared P12 pattern (`[0] = 1`, `[1] = 7`, `[100] = 42`, `[4095] = 3` in 4,096 registers), verifies `to_msgpack()` equals `asapv1_golden/hll_ertl_mle_p12.hex`, that `HyperLogLogP12::::deserialize_from_bytes` recovers the registers and re-serializes identically, and that `HllSketch::from_msgpack` reports `HllVariant::Datafusion`. | | `hll_hip_p12_matches_golden` | The HIP variant's three running scalars travel on the wire beside its registers. | Builds `HllSketch::from_raw(HllVariant::Hip, 12, regs, 1.5, 2.5, 3.0)` over the shared P12 pattern, verifies `to_msgpack()` equals `asapv1_golden/hll_hip_p12.hex`, that `HyperLogLogHIPP12::deserialize_from_bytes` re-serializes the fixture identically, and that `HllSketch::from_msgpack` reports the registers, `HllVariant::Hip`, `hip_kxq0` `1.5`, `hip_kxq1` `2.5` and `hip_est` `3.0`. | @@ -2202,7 +2204,7 @@ Test file: [`tests/xtest_consumer.rs`](../tests/xtest_consumer.rs) | test_name | test_description | what_is_tested | | --- | --- | --- | -| `cross_language_proto` | Protobuf envelopes read from the cross-language fixture directory answer their point queries in band. | Asserts nothing unless fixtures are available: with no `XTEST_DIR` and no discoverable producer directory holding `tests/cross_language/xtest_producer_test.go`, it prints a skip line and returns. With fixtures it decodes nine `SketchEnvelope` files and records a pass or failure for each — `countmin.pb` min-frequency for `item:42` at least `101`, `kll.pb` p50 within 5% of `5000` and p99 within 5% of `9900`, `ddsketch.pb` the same two quantiles within 2%, `hll.pb` an ErtlMLE cardinality inside `[40_000, 65_000]`, `countsketch.pb` `cs:hot` at least `200`, `coco.pb` `coco:hot` at least `500`, `elastic.pb` `elephant` at least `900`, `univmon.pb` a cardinality inside `[1_000, 15_000]`, `hydra.pb` `hydra:42` at least `51` — using the hash seeds `0xcafe3553`, `0x6a09e667` and `0xbb67ae85`, and, when `countmin_sampled.pb` and `hll_sampled.pb` exist, requires `effective_sample_p` of `0.1` and a `rescale_count` by `1/p` within 5% of `100_000` and 6% of `200_000`. A single `assert!` at the end fails if any check failed; a missing or mistyped envelope panics on the spot. | +| `cross_language_proto` | Protobuf envelopes read from the cross-language fixture directory answer their point queries in band. | Asserts nothing unless fixtures are available: with no `XTEST_DIR` and no discoverable producer directory holding `tests/cross_language/xtest_producer_test.go`, it prints a skip line and returns. With fixtures it decodes eight `SketchEnvelope` files and records a pass or failure for each — `countmin.pb` min-frequency for `item:42` at least `101`, `kll.pb` p50 within 5% of `5000` and p99 within 5% of `9900`, `ddsketch.pb` the same two quantiles within 2%, `hll.pb` an ErtlMLE cardinality inside `[40_000, 65_000]`, `countsketch.pb` `cs:hot` at least `200`, `coco.pb` `coco:hot` at least `500`, `univmon.pb` a cardinality inside `[1_000, 15_000]`, `hydra.pb` `hydra:42` at least `51` — using the hash seeds `0xcafe3553` and `0xbb67ae85`, and, when `countmin_sampled.pb` and `hll_sampled.pb` exist, requires `effective_sample_p` of `0.1` and a `rescale_count` by `1/p` within 5% of `100_000` and 6% of `200_000`. A single `assert!` at the end fails if any check failed; a missing or mistyped envelope panics on the spot. | ### HLL Custom Precision diff --git a/proto/elasticsketch/elasticsketch.proto b/proto/elasticsketch/elasticsketch.proto deleted file mode 100644 index 94f1f465..00000000 --- a/proto/elasticsketch/elasticsketch.proto +++ /dev/null @@ -1,41 +0,0 @@ -// elasticsketch.proto — Portable wire format for Elastic Sketch state. - -syntax = "proto3"; - -package sketchlib.v1; - -import "countminsketch/countminsketch.proto"; - -option go_package = "github.com/ProjectASAP/sketchlib-go/proto/elasticsketch;elasticsketchpb"; - -// ============================================================================ -// ElasticSketch -// ============================================================================ - -// ElasticState is the portable state of an Elastic Sketch. -// -// The Elastic Sketch has a heavy part (one bucket per index) and a light -// Count-Min layer. The heavy part is serialized as parallel arrays. -// The light layer is stored as a CountMinState with FLOAT64 counters. -message ElasticState { - // Number of heavy buckets. - uint32 bucket_count = 1; - - // Per-bucket flow identifiers (strings), length = bucket_count. - // Empty string means the bucket is unoccupied. - repeated string flow_ids = 2; - - // Positive vote counts per bucket, length = bucket_count. - repeated sint32 vote_pos = 3 [packed = true]; - - // Negative vote counts per bucket, length = bucket_count. - repeated sint32 vote_neg = 4 [packed = true]; - - // Whether each bucket is in eviction state, length = bucket_count. - repeated bool evictions = 5 [packed = true]; - - // Light Count-Min layer (FLOAT64 counters). - CountMinState light = 6; - - reserved 7 to 15; -} diff --git a/proto/sketchlib.proto b/proto/sketchlib.proto index 0a6d8193..864a15d1 100644 --- a/proto/sketchlib.proto +++ b/proto/sketchlib.proto @@ -37,7 +37,6 @@ import "ddsketch/ddsketch.proto"; import "univmon/univmon.proto"; import "hydra/hydra.proto"; import "cocosketch/cocosketch.proto"; -import "elasticsketch/elasticsketch.proto"; option go_package = "github.com/ProjectASAP/sketchlib-go/proto/sketch_envelope;sketch_envelopepb"; @@ -74,9 +73,11 @@ message SketchEnvelope { UnivMonState univmon = 15; HydraState hydra = 16; CocoSketchState coco = 17; - ElasticState elastic = 18; } + reserved 18; + reserved "elastic"; + reserved 19 to 31; // Reserved for future sketch types. // Geometric / NitroSketch sampling probability for the producing sketch. diff --git a/src/proto/generated/sketchlib.v1.rs b/src/proto/generated/sketchlib.v1.rs index 2c098d3f..0f109fd8 100644 --- a/src/proto/generated/sketchlib.v1.rs +++ b/src/proto/generated/sketchlib.v1.rs @@ -823,33 +823,6 @@ pub struct CocoSketchState { #[prost(bool, repeated, tag = "5")] pub has_keys: ::prost::alloc::vec::Vec, } -/// ElasticState is the portable state of an Elastic Sketch. -/// -/// The Elastic Sketch has a heavy part (one bucket per index) and a light -/// Count-Min layer. The heavy part is serialized as parallel arrays. -/// The light layer is stored as a CountMinState with FLOAT64 counters. -#[derive(Clone, PartialEq, ::prost::Message)] -pub struct ElasticState { - /// Number of heavy buckets. - #[prost(uint32, tag = "1")] - pub bucket_count: u32, - /// Per-bucket flow identifiers (strings), length = bucket_count. - /// Empty string means the bucket is unoccupied. - #[prost(string, repeated, tag = "2")] - pub flow_ids: ::prost::alloc::vec::Vec<::prost::alloc::string::String>, - /// Positive vote counts per bucket, length = bucket_count. - #[prost(sint32, repeated, tag = "3")] - pub vote_pos: ::prost::alloc::vec::Vec, - /// Negative vote counts per bucket, length = bucket_count. - #[prost(sint32, repeated, tag = "4")] - pub vote_neg: ::prost::alloc::vec::Vec, - /// Whether each bucket is in eviction state, length = bucket_count. - #[prost(bool, repeated, tag = "5")] - pub evictions: ::prost::alloc::vec::Vec, - /// Light Count-Min layer (FLOAT64 counters). - #[prost(message, optional, tag = "6")] - pub light: ::core::option::Option, -} /// SketchEnvelope is the portable container for all sketch transfers. /// Producers fill exactly one field inside sketch_state. /// Consumers check format_version first, then dispatch on the oneof. @@ -892,7 +865,7 @@ pub struct SketchEnvelope { /// The sketch payload. Exactly one field must be set. #[prost( oneof = "sketch_envelope::SketchState", - tags = "10, 11, 12, 13, 14, 15, 16, 17, 18" + tags = "10, 11, 12, 13, 14, 15, 16, 17" )] pub sketch_state: ::core::option::Option, } @@ -917,7 +890,5 @@ pub mod sketch_envelope { Hydra(super::HydraState), #[prost(message, tag = "17")] Coco(super::CocoSketchState), - #[prost(message, tag = "18")] - Elastic(super::ElasticState), } } diff --git a/tests/asapv1_golden.rs b/tests/asapv1_golden.rs index 30b41651..f744e41d 100644 --- a/tests/asapv1_golden.rs +++ b/tests/asapv1_golden.rs @@ -9,8 +9,8 @@ //! See `asapv1_golden/README.md`. use asap_sketchlib::{ - Classic, Count, CountMin, ErtlMLE, FastPath, HllSketch, HllVariant, HyperLogLogHIPP12, - HyperLogLogP12, KLL, MessagePackCodec, RegularPath, Vector2D, + Classic, Count, CountMin, Elastic, ErtlMLE, FastPath, HeavyBucket, HllSketch, HllVariant, + HyperLogLogHIPP12, HyperLogLogP12, KLL, MessagePackCodec, RegularPath, Vector2D, }; fn decode_hex(s: &str) -> Vec { @@ -32,6 +32,8 @@ const GOLDEN_CS_FAST: &str = include_str!("../asapv1_golden/cs_i64_fast_2x4.hex" const GOLDEN_CS_I32: &str = include_str!("../asapv1_golden/cs_i32_regular_2x4.hex"); const GOLDEN_KLL_F64: &str = include_str!("../asapv1_golden/kll_f64_k200.hex"); const GOLDEN_KLL_I64: &str = include_str!("../asapv1_golden/kll_i64_k200.hex"); +const GOLDEN_ELASTIC: &str = include_str!("../asapv1_golden/elastic_4b_2x4.hex"); +const GOLDEN_ELASTIC_STALE: &str = include_str!("../asapv1_golden/elastic_4b_2x4_stale.hex"); /// The known P12 register pattern shared by all three HLL fixtures. fn p12_registers() -> Vec { @@ -287,3 +289,88 @@ fn kll_i64_k200_matches_golden() { assert_eq!(decoded.quantile(0.0), 1.0); assert_eq!(decoded.quantile(1.0), 50.0); } + +// --------------------------------------------------------------------------- +// Elastic: heavy buckets and light counters set directly, never inserted. +// The two fixtures hold the same state and differ only by `stale_copies`. +// --------------------------------------------------------------------------- + +/// Heavy buckets `(flow_id, vote_pos, vote_neg, eviction)`: a free bucket with +/// the eviction flag, a 31-byte fixstr id, an empty id and a 32-byte str8 id; +/// the votes sweep positive fixint / uint8 / uint16 / uint32. +const ELASTIC_HEAVY: [(&str, i32, i32, bool); 4] = [ + ("", 0, 0, true), + ("10.0.0.1:443>192.168.10.20:5123", 127, 128, false), + ("", 1, 65535, true), + ("10.0.0.1:443>192.168.10.20:51234", 2147483647, 256, true), +]; + +/// Light layer: row 0 sweeps the unsigned widths up to `i32::MAX`, row 1 the +/// signed widths down to `i32::MIN`. +const ELASTIC_LIGHT: [[i32; 4]; 2] = [[0, 255, 65536, 2147483647], [-1, -33, -32768, -2147483648]]; + +/// Sets the known heavy and light state on a 4-bucket, `2x4` sketch. +fn elastic_known_state(mut sketch: Elastic) -> Elastic { + assert_eq!(sketch.heavy.len(), ELASTIC_HEAVY.len()); + for (bucket, &(id, pos, neg, eviction)) in sketch.heavy.iter_mut().zip(&ELASTIC_HEAVY) { + *bucket = HeavyBucket { + flow_id: id.to_string(), + vote_pos: pos, + vote_neg: neg, + eviction, + }; + } + sketch.light = CountMin::from_storage(Vector2D::from_fn(2, 4, |r, c| ELASTIC_LIGHT[r][c])); + sketch +} + +fn assert_elastic_known_state(sketch: &Elastic) { + let buckets: Vec<(&str, i32, i32, bool)> = sketch + .heavy + .iter() + .map(|b| (b.flow_id.as_str(), b.vote_pos, b.vote_neg, b.eviction)) + .collect(); + assert_eq!(buckets, ELASTIC_HEAVY); + assert_eq!(sketch.bktlen, 4); + assert_eq!((sketch.light.rows(), sketch.light.cols()), (2, 4)); + let flat: Vec = ELASTIC_LIGHT.iter().flatten().copied().collect(); + assert_eq!(sketch.light.as_storage().as_slice(), flat.as_slice()); +} + +#[test] +fn elastic_4b_2x4_matches_golden() { + let want = decode_hex(GOLDEN_ELASTIC); + + let sketch = elastic_known_state(Elastic::init_with_dimensions(4, 2, 4)); + let got = sketch.serialize_to_bytes().expect("serialize"); + assert_eq!(got, want, "Elastic bytes diverge from golden"); + + let decoded = Elastic::deserialize_from_bytes(&want).expect("decode"); + assert_elastic_known_state(&decoded); + assert_eq!(decoded.serialize_to_bytes().expect("re-serialize"), want); +} + +#[test] +fn elastic_4b_2x4_stale_matches_golden() { + let want = decode_hex(GOLDEN_ELASTIC_STALE); + + let mut expanded: Elastic = Elastic::init_with_dimensions(2, 2, 4); + expanded.expand_heavy(); + let sketch = elastic_known_state(expanded); + let got = sketch.serialize_to_bytes().expect("serialize"); + assert_eq!(got, want, "Elastic stale-copies bytes diverge from golden"); + + let decoded = Elastic::deserialize_from_bytes(&want).expect("decode"); + assert_elastic_known_state(&decoded); + assert_eq!(decoded.serialize_to_bytes().expect("re-serialize"), want); + + let fresh = decode_hex(GOLDEN_ELASTIC); + assert_eq!(want.len(), fresh.len()); + let differing: Vec = (0..want.len()).filter(|&i| want[i] != fresh[i]).collect(); + assert_eq!( + differing.len(), + 1, + "the fixtures differ only by stale_copies" + ); + assert_eq!((fresh[differing[0]], want[differing[0]]), (0xc2, 0xc3)); +} diff --git a/tests/xtest_consumer.rs b/tests/xtest_consumer.rs index 0113ff45..1a07680f 100644 --- a/tests/xtest_consumer.rs +++ b/tests/xtest_consumer.rs @@ -11,7 +11,6 @@ //! hll.pb — HyperLogLogState (ErtlMLE variant) //! countsketch.pb — CountSketchState with float64 signed counters //! coco.pb — CocoSketchState (hash+val+hasKey buckets) -//! elastic.pb — ElasticState (heavy buckets + light CountMin) //! univmon.pb — UnivMonState (layered CountSketch + TopK heaps) //! hydra.pb — HydraState (CM-cell grid) //! @@ -322,44 +321,6 @@ fn cross_language_proto() { all_ok = false; } - // ----------------------------------------------------------------------- - // ElasticSketch - // ----------------------------------------------------------------------- - println!(); - println!("[ElasticSketch] Step 1/3 — Read elastic.pb"); - let bytes = read_file(in_dir.join("elastic.pb")); - let env = SketchEnvelope::decode(bytes.as_slice()).expect("decode elastic envelope"); - - println!( - "[ElasticSketch] Step 2/3 — Validate envelope (format_version={}, producer={})", - env.format_version, - env.producer.as_ref().map_or("?", |p| &p.library) - ); - - let elastic_state = match env.sketch_state { - Some(sketch_envelope::SketchState::Elastic(ref s)) => s.clone(), - other => panic!("expected ElasticSketch sketch_state, got {other:?}"), - }; - - println!( - "[ElasticSketch] bucket_count={} light_rows={} light_cols={}", - elastic_state.bucket_count, - elastic_state.light.as_ref().map_or(0, |l| l.rows), - elastic_state.light.as_ref().map_or(0, |l| l.cols) - ); - - // Query "elephant" — inserted 1000 times. - // Go uses CanonicalHashSeed = seedList[5] = 0x6a09e667 - let elephant_hash = xxh3_64_seeded(SEED_5, b"elephant"); - let elephant_est = elastic_query(&elastic_state, "elephant", elephant_hash); - println!("[ElasticSketch] Step 3/3 — 'elephant' est = {elephant_est} (expect ≥ 900)"); - if elephant_est >= 900 { - println!("[ElasticSketch] PASS"); - } else { - eprintln!("[ElasticSketch] FAIL: estimate {elephant_est} < 900"); - all_ok = false; - } - // ----------------------------------------------------------------------- // UnivMon // ----------------------------------------------------------------------- @@ -553,7 +514,6 @@ fn cross_language_proto() { // --------------------------------------------------------------------------- const SEED_0: u64 = 0xcafe3553; // seedList[0] — Hash64 / default hash -const SEED_5: u64 = 0x6a09e667; // seedList[5] — CanonicalHashSeed const SEED_6: u64 = 0xbb67ae85; // seedList[6] — defaultHydraSeed // --------------------------------------------------------------------------- @@ -860,62 +820,6 @@ fn coco_estimate(state: &CocoSketchState, hash: u64) -> u64 { total } -// --------------------------------------------------------------------------- -// ElasticSketch query -// --------------------------------------------------------------------------- -// Mirrors Go's ElasticSketch.queryLocked: -// hash = HashIt(CanonicalHashSeed, []byte(id)) → SEED_5 -// idx = hash % bucket_count -// if flow_ids[idx] == id: if !eviction: return vote_pos[idx] -// else: return vote_pos[idx] + light_estimate -// else: return light_estimate -// Light layer: rows=5, cols=2048, bits=11, mask=2047 -// col_r = (hash >> (r * 11)) & 2047; min across rows - -fn elastic_query(state: &ElasticState, id: &str, hash: u64) -> i64 { - let n = state.bucket_count as usize; - let idx = (hash % n as u64) as usize; - - let heavy_match = idx < state.flow_ids.len() && state.flow_ids[idx] == id; - - if heavy_match { - let vpos = state.vote_pos.get(idx).copied().unwrap_or(0) as i64; - let evicted = state.evictions.get(idx).copied().unwrap_or(false); - if !evicted { - return vpos; - } - return vpos + elastic_light_min(state, hash); - } - elastic_light_min(state, hash) -} - -fn elastic_light_min(state: &ElasticState, hash: u64) -> i64 { - let light = match &state.light { - Some(l) => l, - None => return 0, - }; - let rows = light.rows as usize; - let cols = light.cols as usize; - let bits = col_bits(cols); // trailing zeros of cols = 11 for 2048 - let mask = (cols as u64) - 1; - let counts = &light.counts_float; - - let mut min_val = f64::MAX; - for r in 0..rows { - let shift = (r as u64) * bits; - let col = ((hash >> shift) & mask) as usize; - let v = counts[r * cols + col]; - if v < min_val { - min_val = v; - } - } - if min_val == f64::MAX { - 0 - } else { - min_val as i64 - } -} - // --------------------------------------------------------------------------- // UnivMon cardinality (g-sum heuristic) // --------------------------------------------------------------------------- diff --git a/tools/gen-proto/src/main.rs b/tools/gen-proto/src/main.rs index 1de6ad15..ab74620f 100644 --- a/tools/gen-proto/src/main.rs +++ b/tools/gen-proto/src/main.rs @@ -36,7 +36,6 @@ const PROTO_FILES: &[&str] = &[ "univmon/univmon.proto", "hydra/hydra.proto", "cocosketch/cocosketch.proto", - "elasticsketch/elasticsketch.proto", "sketchlib.proto", ];