diff --git a/CHANGELOG.md b/CHANGELOG.md index 80d85f98..9639f712 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -66,6 +66,10 @@ signals a backwards-compatible change. worker's seed and the aggregator's from one seed. The unseeded constructors draw from the thread generator. The seed is not serialized, and the ASAPv1 payload is unchanged. +- **DDSketch ASAPv1 golden fixtures**: `ddsketch_positive_a001` and + `ddsketch_empty_a001` (metadata version 1) and `ddsketch_signed_a001` + (metadata version 2, negative store and zero count), checked by + `tests/asapv1_golden.rs`. ### Changed @@ -88,6 +92,18 @@ signals a backwards-compatible change. as an empty window like any other, and that window takes a place in `max_windows`. +### Removed + +- **DDSketch's protobuf and portable formats.** DDSketch serializes through + ASAPv1 only (`DDSketch::serialize_to_bytes` / `deserialize_from_bytes`). + Removed: `proto/ddsketch/ddsketch.proto` and the generated + `proto::sketchlib::{DdSketchState, DdSketchDelta, DdSketchBucketDelta}`; + `SketchEnvelope`'s `ddsketch` field (14), now reserved; + `message_pack_format::portable::ddsketch` (`DdSketch`, `DdSketchDelta`, + `DDSKETCH_GROW_CHUNK`, `MAX_APPLY_DELTA_SPAN_BUCKETS`) and the crate-root + re-exports of the first three; the `MessagePackCodec` impl for `DDSketch`. + There is no ASAPv1 DDSketch delta. + ### Fixed - `KLLDynamic` panicked on every quantile query once a NaN had entered the diff --git a/asapv1_golden b/asapv1_golden index 8287a4ae..52df5d51 160000 --- a/asapv1_golden +++ b/asapv1_golden @@ -1 +1 @@ -Subproject commit 8287a4ae04e6cfbfb3ed33dafbf1dac77d8f7fc1 +Subproject commit 52df5d51bbb588bc374fd6700bfc6cba9cb96986 diff --git a/docs/api/api_ddsketch.md b/docs/api/api_ddsketch.md index d4724210..42baf9db 100644 --- a/docs/api/api_ddsketch.md +++ b/docs/api/api_ddsketch.md @@ -95,16 +95,6 @@ assert!(p50 >= 1.0); retained minimum and maximum rather than to a bucket representative, so the endpoints carry zero error. -This differs from the portable wire twin -`message_pack_format::portable::ddsketch::DdSketch::quantile`, which uses the -lower-quantile convention `floor(q * (n - 1))` of the DDSketch paper and of the -Go reference implementation, and which retains no min/max scalars — its -endpoints are ordinary bucket representatives, accurate to `alpha` like any -other rank. The two agree at most `q` and diverge at small `n` or ragged `q` -(at `n = 3, q = 0.4` this type answers `sorted[1]` and the portable one -`sorted[0]`). Both conventions are deliberate and are pinned by -`ddsketch_core_and_portable_answer_different_order_statistics`. - ## Caveats - Inputs must be positive values. diff --git a/docs/asapv1_wire_format.md b/docs/asapv1_wire_format.md index fd14e70a..a4cfcdb0 100644 --- a/docs/asapv1_wire_format.md +++ b/docs/asapv1_wire_format.md @@ -673,7 +673,7 @@ Wire counter types are the base Count-Min's: **`"i32"`, `"i64"` and `"f64"`**. ` | 3 | `min` | f64 | smallest value as ingested; `+inf` when empty | | 4 | `max` | f64 | largest value as ingested; `-inf` when empty | -**Metadata vs payload.** `alpha` shapes every bucket index and is chosen at construction, so per the config-to-metadata rule it is a structural param, like HLL's `precision`. `gamma`, `log_gamma` and `inv_log_gamma` are derived from it (`gamma = (1 + alpha) / (1 - alpha)`) and appear nowhere: the decoder re-derives them exactly as `DDSketch::new` does. The store's dimensions are not configuration — the store grows with the data — so there is no `rows`/`cols` analogue; `counts` is self-delimiting and `offset` positions it. The metadata is `metadata_version` then `alpha`. +**Metadata vs payload.** `alpha` shapes every bucket index and is chosen at construction, so per the config-to-metadata rule it is a structural param, like HLL's `precision`. `gamma`, `log_gamma` and `inv_log_gamma` are derived from it (`gamma = (1 + alpha) / (1 - alpha)`) and appear nowhere: the decoder re-derives them exactly as `DDSketch::new` does. The store's dimensions are not configuration — the store grows with the data — so there is no `rows`/`cols` analogue; `counts` is self-delimiting and `offset` positions it. The metadata is `metadata_version` then `alpha`. `alpha` is the constructor's argument bit for bit, never re-derived from `gamma`. **DDSketch carries no hash-spec group.** It never hashes: it maps a raw `f64` through a logarithm and carries no hasher type parameter at all. The hash spec answers "how were keys hashed", and a sketch that does not hash has no truthful answer, so the group is omitted entirely. This is the Q-KLL precedent, and the reason DDSketch's metadata schema is not `HashProfile`-derived the way HLL's and Count-Min's are. @@ -681,9 +681,9 @@ Wire counter types are the base Count-Min's: **`"i32"`, `"i64"` and `"f64"`**. ` **`sum` saturates at signed `f64::MAX`.** All three paths that advance it — `add`, `apply_delta` and `merge` — go through one guarded step: the new total is taken when it is finite and `f64::MAX` with the overflow sign is stored otherwise. So a sketch that ingests enough mass to overflow the accumulator still serializes, and only `sum` degrades, only past `f64::MAX`. `count`, the bucket store, `min` and `max` do not pass through the guard and are unaffected. This is what keeps rule 7's finiteness requirement satisfiable: an unguarded `+=` would leave a legally-ingested sketch carrying `+inf`, which the same rule refuses on both doors, so a sketch nothing did wrong to would have no encoding. -**Signed extension (metadata version 2).** Positive-only states retain the five-field payload and version 1. States with negative observations or a zero count use version 2 and append three fields: `negative_counts: array`, `negative_offset: int`, and `zero_count: u64`. Negative values use the same mapping on their magnitudes; quantile traversal visits their buckets in descending index order, then zeros, then positive buckets in ascending order. Values smaller in magnitude than the mapping minimum count as zero; nonfinite and too-large magnitudes are rejected by `try_add`. The legacy `add` wrapper ignores those errors. +**Signed extension (metadata version 2).** Positive-only states retain the five-field payload and version 1. States with negative observations or a zero count use version 2 and append three fields: `negative_counts: array`, `negative_offset: int`, and `zero_count: u64`. Negative values use the same mapping on their magnitudes; quantile traversal visits their buckets in descending index order, then zeros, then positive buckets in ascending order. Values smaller in magnitude than the mapping minimum count as zero; nonfinite and too-large magnitudes are rejected by `try_add` and ignored by `add`. -For version 2 the total is `sum(counts) + sum(negative_counts) + zero_count`, checked for overflow. Both stores obey the span checks below. A nonempty signed state requires finite scalars and `min <= max`; the positive-only `min > 0` and `sum >= min` checks do not apply. Empty-state scalars remain unchanged. Version 1 readers reject version 2. The existing positive-only golden bytes remain unchanged. +For version 2 the total is `sum(counts) + sum(negative_counts) + zero_count`, checked for overflow. Both stores obey the span checks below. A nonempty signed state requires finite scalars and `min <= max`; the positive-only `min > 0` and `sum >= min` checks do not apply. Empty-state scalars are as in version 1. A decoder that knows only version 1 rejects version 2. **Emitted order (cross-language contract).** `counts` is the dense store in its own bucket-index order, ascending from `offset`, with the `GROW_CHUNK` padding zeros left in place and `offset` left where the sketch put it. Nothing is trimmed and nothing is sorted, so a decoded sketch re-serializes byte-identically and `store_counts()` / `store_offset()` mean the same thing on both sides. @@ -1341,14 +1341,14 @@ Good direction (more compact, higher fidelity, less Rust-internal duplication), 2. **Golden byte-vector fixtures** in one shared repo, [`sketchlib-golden-bytes`](https://github.com/ProjectASAP/sketchlib-golden-bytes), which each implementation mounts at `asapv1_golden/`; both languages decode and re-encode them byte-identically. These replace the `portable`-as-oracle round-trip test. 3. **This registry**, mirrored, never independently allocated. -Fixtures exist for six `kind_id`s — HLL's three estimators, Count-Min, Count Sketch and compact KLL — and `sketchlib-go` mirrors those. Every other kind here has a payload and **no fixture**, so this document is its only contract; `asapv1_golden/README.md` lists the gap. `portable` carries its own Go goldens. +`asapv1_golden/README.md` ("Coverage") lists which `kind_id`s have fixtures; a kind without one has this document as its only contract. `portable` carries its own Go goldens. **Hash profile on the Go side.** Rust derives the hash spec from a generic `HashProfile` bound on the hasher type; Go has no generic hasher type, so there is nothing to derive from. On the Go side the profile is simply **written into** the metadata on encode and **read from** it on decode. Go MUST validate the profile it reads (same fail-closed intent as Rust): a sketch is only mergeable/queryable if its `hash_profile_id` + seeds match the profile Go is prepared to reproduce. -Sequencing: (2) covers six `kind_id`s, and the one `native bytes == portable bytes` test is HLL-only, since no other `portable` type emits ASAPv1 bytes. +Sequencing: (2) covers the kinds the golden README lists, and the one `native bytes == portable bytes` test is HLL-only, since no other `portable` type emits ASAPv1 bytes. Retire `portable` once the fixtures cover the rest. --- diff --git a/docs/ddsketch_signed_values.md b/docs/ddsketch_signed_values.md index 923eddaa..1aab66d9 100644 --- a/docs/ddsketch_signed_values.md +++ b/docs/ddsketch_signed_values.md @@ -1,18 +1,12 @@ # Signed DDSketch -Both `DDSketch` (core) and `DdSketch` (portable) retain positive samples, negative samples, and zeros. The design follows [DataDog DDSketch](https://github.com/DataDog/sketches-go/blob/master/ddsketch/ddsketch.go): separate positive and negative magnitude stores, plus a zero count. Bucket indices can be negative in either store; an index's sign does not encode an observation's sign. +`DDSketch` retains positive samples, negative samples, and zeros. The design follows [DataDog DDSketch](https://github.com/DataDog/sketches-go/blob/master/ddsketch/ddsketch.go): separate positive and negative magnitude stores, plus a zero count. Bucket indices can be negative in either store; an index's sign does not encode an observation's sign. -`try_add` / `try_update` return errors for NaN, infinities, or magnitudes above the mapping's maximum. Existing `add` / `update` wrappers keep their return types and ignore those errors. Magnitudes below the mapping's minimum are counted as zero, including subnormal inputs. Such nonzero tiny observations are outside the relative-error guarantee. Original sample values are still compressed into buckets, not retained individually. +`try_add` returns an error for NaN, infinities, or magnitudes above the mapping's maximum. `add` ignores those errors. Magnitudes below the mapping's minimum are counted as zero, including subnormal inputs. Such nonzero tiny observations are outside the relative-error guarantee. Original sample values are still compressed into buckets, not retained individually. -Core preserves its existing ceil(q*n) order-statistic convention and exact ingested extrema. Portable preserves floor(q*(n-1)); `quantile_interpolated` preserves the PromQL interpolation convention. For indexable nonzero order statistics the estimate has absolute error at most `alpha * abs(value)`, for either sign. +Quantiles use the ceil(q*n) order-statistic convention and exact ingested extrema. For indexable nonzero order statistics the estimate has absolute error at most `alpha * abs(value)`, for either sign. -If interpolation combines endpoints x and y with weight t, the absolute error is bounded by `alpha * ((1-t)*abs(x) + t*abs(y))`, excluding zero-mapped tiny values. Opposite signs can cancel, so the interpolated result need not have relative error alpha. Dividing quantiles additionally requires a finite nonzero true denominator and valid relative-error bounds for both operands. This change does not establish those planner preconditions. +## Wire and OctoSketch -## Wire and downstream upgrade - -- Native ASAPv1: positive-only states keep metadata version 1 and their existing bytes. Signed/zero states use metadata version 2 and append negative counts, negative offset, and zero count. Old native readers reject version 2. -- Portable MessagePack: positive-only snapshots keep their three fields; signed snapshots append the same three fields. Positive-only deltas keep their two arrays; signed deltas append negative indices, negative counts, and zero-count increment. New readers accept both layouts. -- Protobuf: `DdSketchState` adds fields 16–18; `DdSketchDelta` adds fields 16–17. `DdSketch::to_proto` / `from_proto` preserve all stores. Old protobuf readers silently ignore unknown fields: upgrade Collector, backend, and Go producer schema/mapping together before emitting signed protobuf. This is not DataDog's binary wire format. -- Octo: `DdDelta` now requires a `store: DdStore` discriminator. Existing positive delta constructors should use `DdStore::Positive`. Worker residual keys are `(DdStore, i32)`. Both promotion and flush preserve all three categories; partial counts still need flushing before complete queries. - -Go implementations of the ASAP wire formats and downstream dependency pins are not changed by this Rust library patch. +- ASAPv1: positive-only states use metadata version 1. States with negative samples or zeros use version 2, which appends negative counts, negative offset, and zero count. +- OctoSketch: `DdDelta` carries a `store: DdStore` discriminator, and worker residual keys are `(DdStore, i32)`. Promotion and flush preserve all three categories; partial counts need flushing before complete queries. diff --git a/docs/e2e_testing_harness.md b/docs/e2e_testing_harness.md index 373f6040..76173de9 100644 --- a/docs/e2e_testing_harness.md +++ b/docs/e2e_testing_harness.md @@ -101,7 +101,7 @@ One spec per *metric*, not per sketch, because the metric is what differs: | `CountSketchSpec` | two-sided L2, rank-independent | marginal `sqrt(3/w)·‖f₋ᵢ‖₂` at `P[Bin(d, 1/3) ≥ ⌈d/2⌉]`; simultaneous at the smallest κ with `P[Bin(d,1/κ) ≥ ⌈d/2⌉] ≤ δ/D` | | `SecondMomentSpec` | F2 from a Count Sketch matrix | `sqrt(2κ/w)`, same median amplification | | `KllRankSpec` | KLL **maximum** rank error over a q grid | `ε(k) = 2.446 / k^0.9433` — an Apache DataSketches *characterization fit*, not a theorem about this code | -| `RelativeQuantileSpec` + `DdRankConvention` | DDSketch relative value error | `α + ULP slack` vs the exact order statistic **of that implementation's own rank convention** | +| `RelativeQuantileSpec` + `dd_rank_index` | DDSketch relative value error | `α + ULP slack` vs the exact `ceil(q·n)` order statistic | | `CardinalityConfidenceSpec` | HLL / KMV | `z · σ_rel`, σ derived exactly, the tail a normal approximation | | `SamplingConfidenceSpec` | Nitro | `z · sqrt(f(p·r(1−r) + (1−p)/p))`, `r = frac(1/p)` | | `PrioritySampleSpec` | `UniformSampling` | `len = ⌈n·rate⌉` exactly; `Var[mean] = (σ_N²/m)(N−m)/(N−1)` | @@ -141,11 +141,8 @@ Keeping these apart is deliberate. There is no shared `QuantileSpec { rank_tol }` that KLL and DDSketch both use, because they do not promise the same thing: a correct KLL can return a value 100× off on a heavy-tailed stream and still be within its rank guarantee, and a correct -DDSketch has no rank guarantee at all. For the same reason the two DDSketch -implementations do not share a truth helper: `DDSketch::get_value_at_quantile` -answers `sorted[ceil(q·n) − 1]` while the portable `DdSketch::quantile` answers -`sorted[floor(q·(n−1))]`, so `DdRankConvention` carries the choice and each is -scored on the question it actually answers. +DDSketch has no rank guarantee at all. `DDSketch::get_value_at_quantile` +answers `sorted[ceil(q·n) − 1]`, and `dd_rank_index` computes that truth. Two distinctions are easy to lose: diff --git a/docs/features.md b/docs/features.md index 64f296a3..ee5c93b1 100644 --- a/docs/features.md +++ b/docs/features.md @@ -112,7 +112,7 @@ MessagePack (`rmp-serde`) support. **serde support** means the type derives `Ser | EHSketchList | Yes | Yes | | ExponentialHistogram | No | Yes | -Protobuf (prost): `.proto` definitions exist for CountMin, Count, HLL, DDSketch, KLL, Elastic, Coco, Hydra, and UnivMon. Rust conversion code is in progress. +Protobuf (prost): `.proto` definitions exist for CountMin, Count, HLL, KLL, Elastic, Coco, Hydra, and UnivMon. Rust conversion code is in progress. ### API Stability diff --git a/docs/message_pack_format.md b/docs/message_pack_format.md index 12a2b91c..b6d36bb2 100644 --- a/docs/message_pack_format.md +++ b/docs/message_pack_format.md @@ -169,10 +169,10 @@ decode → re-encode them byte-identically. The `kind_id` registry is mirrored verbatim with Go's `wire/asapmsgpack/magic_ids.go`, never independently allocated. -That proof covers the six `kind_id`s a fixture exists for — HLL's three -estimators, Count-Min, Count Sketch and compact KLL. Every other implemented -kind has **no ASAPv1 golden and therefore no ASAPv1 drift guard**; -[its README](https://github.com/ProjectASAP/sketchlib-golden-bytes#coverage) lists what is covered. +That proof covers only the `kind_id`s a fixture exists for; a kind without one +has **no ASAPv1 drift guard**. +[The golden README's Coverage section](https://github.com/ProjectASAP/sketchlib-golden-bytes#coverage) +lists them. The older `portable` path is guarded separately, by the `sketchlib-go` goldens of its own listed above. diff --git a/docs/tests.md b/docs/tests.md index ccb1571c..30e21d5f 100644 --- a/docs/tests.md +++ b/docs/tests.md @@ -1294,30 +1294,6 @@ Test file: [`src/message_pack_format/portable/countsketch_topk.rs`](../src/messa | `test_aggregate_topk_empty` | An empty stream produces nothing. | Verifies `aggregate_topk(4, 100, 10, &[], &[])` returns `None`. | | `test_median_estimator_differs_from_cms_style_min` | The median-over-rows estimator tracks the true count rather than inflating. | For a `5x64` composite with a 10-slot heap fed `"heavy"` 50 times at weight `1.0` and 20 keys `light-0`..`light-19` once each, verifies `estimate("heavy")` falls in `40.0..=60.0`. | -### Portable DdSketch - -Test file: [`src/message_pack_format/portable/ddsketch.rs`](../src/message_pack_format/portable/ddsketch.rs) - -| test_name | test_description | what_is_tested | -| --- | --- | --- | -| `msgpack_delta_against_empty_round_trips` | A msgpack delta against an empty snapshot carries the bucket store. | For an `alpha` `0.01` sketch fed `1.0..=200.0`, verifies the delta against an empty sketch at threshold `1` applied to a fresh sketch gives the same `total_count` and quantiles at `0.5`, `0.9`, and `0.99` within 1%. | -| `msgpack_delta_wire_layout_is_parallel_arrays` | The delta is a two-element array of parallel index and count arrays. | For an `alpha` `0.01` sketch fed `1.0`, `1.0`, and `1000.0`, verifies the delta bytes equal `rmp_serde::to_vec(&(idx: Vec, d_count: Vec))` recomputed from the non-zero store cells at absolute index, and that decoding reports two buckets. | -| `test_new_empty` | A fresh sketch holds nothing. | For `DdSketch::new(0.01)`, verifies `total_count` is `0` and `store_counts` is empty. | -| `test_merge_aligned_same_offset` | Aligned stores add cell by cell. | Merges `[10, 20, 30]` at offset `-1` into `[1, 2, 3]` at offset `-1` and verifies the store becomes `[11, 22, 33]`, the offset stays `-1`, and `total_count` is `66`. | -| `test_merge_overlapping_offsets` | Merge widens the window across overlapping stores. | Merges `[10, 10, 10]` at offset `0` into `[1, 1, 1]` at offset `-1` and verifies the store becomes `[1, 11, 11, 10]` at offset `-1` with `total_count` `33`. | -| `test_merge_disjoint_offsets` | Merge zero-fills the gap between disjoint stores. | Merges `[3, 4]` at offset `5` into `[1, 2]` at offset `0` and verifies the store becomes `[1, 2, 0, 0, 0, 3, 4]` at offset `0`. | -| `test_apply_delta_additive_inside_store` | Delta buckets inside the store add to their cells. | For a base `[1, 2, 3]` at offset `-1`, applies a delta with buckets `(-1,4)`, `(0,8)`, `(1,12)`, `d_count` `24`, `d_sum` `120.0`, and `new_max` `9.0`, and verifies the store becomes `[5, 10, 15]` with `total_count` `30`. | -| `test_apply_delta_expands_store_on_new_bucket` | A delta bucket past the store grows it. | For a base `[1, 2]` at offset `0`, applies a delta with the single bucket `(4, 7)`, `d_count` `7`, and `d_sum` `35.0`, and verifies the store becomes `[1, 2, 0, 0, 7]` at offset `0` with `total_count` `10`. | -| `test_apply_delta_matches_full_merge` | Applying a delta lands where merging the same addition lands. | For a base `[1, 2, 3]` at offset `0`, verifies merging `[10, 0, 20]` at offset `0` and applying the delta buckets `(0,10)` and `(2,20)` give the same store and `total_count`. | -| `test_merge_alpha_mismatch` | Sketches of different relative accuracy are not mergeable. | Verifies merging an `alpha` `0.02` sketch into an `alpha` `0.01` one fails. | -| `test_msgpack_round_trip` | The msgpack round trip preserves alpha, store, and offset. | Round-trips `from_raw(0.01, [1, 2, 3], -2)` and verifies `alpha`, `store_counts`, `store_offset`, and `total_count` are unchanged. | -| `test_msgpack_is_three_element_array` | The wire form is a three-element array, not a map. | For `from_raw(0.01, [1, 2, 3], -2)`, verifies the first byte of `to_msgpack` is `0x93`, the msgpack fixarray marker for length `3`. | -| `test_insert_and_quantile_lognormal` | Quantiles track a known log-normal distribution. | Feeds 100,000 xorshift64\*/Box-Muller log-normal samples with `mu` `3` and `sigma` `0.7` into an `alpha` `0.01` sketch and verifies `\|ln(P50 / 20.09)\|` and `\|ln(P99 / 102.4)\|` are each below `0.05`; P90 is not asserted. | -| `test_delta_chain_preserves_quantile_accuracy` | A chain of applied deltas stays inside the alpha bound of the direct path. | Over five batches of 10,000 log-normal samples at `alpha` `0.01`, seeds one sketch from the first batch and applies four deltas built by the test's own `compute_dd_delta` helper, then verifies its `total_count` equals the directly fed sketch's exactly and that quantiles at `0.5`, `0.9`, and `0.99` agree within `alpha`; the shipped `compute_delta` is not the path exercised here. | -| `test_compute_delta_against_empty_round_trips` | A delta against an empty snapshot carries the whole store. | For an `alpha` `0.01` sketch fed `1.0..=200.0`, verifies the delta against empty at threshold `1`, applied to a fresh sketch, gives the exact `total_count`, the same count for every non-zero bucket compared at absolute index, and quantiles at `0.5`, `0.9`, and `0.99` within 1%. | -| `test_consecutive_windows_emit_own_state` | Each window emits its own state rather than a cross-window difference. | Builds `win1` from `1.0..=50.0` inserted twice (count `2` per bucket) and `win2` from `1.0..=50.0` once, and verifies `win2`'s delta against empty reconstructs `win2`'s `total_count` and that those bytes differ from `win2`'s delta against `win1`. | -| `test_update_then_envelope_matches_go_golden_bytes` | The proto envelope for a populated store is byte-identical to the golden. | For an `alpha` `0.01` sketch fed `1.0..=50.0`, builds a `DdSketchState` from `wire_alpha()`, `store_counts`, and `store_offset`, wraps it in a `SketchEnvelope` with `format_version` `1`, and verifies the prost encoding equals a 403-byte inline golden carrying a 384-entry store and a `store_offset` of `-64` (sint32 zigzag `0x7f`); the golden constant records that it is the Rust-produced value and still to be reconciled against the Go reference. | - ### Portable DeltaResult Test file: [`src/message_pack_format/portable/delta_set_aggregator.rs`](../src/message_pack_format/portable/delta_set_aggregator.rs) @@ -1698,12 +1674,12 @@ Test file: [`tests/e2e/quantiles.rs`](../tests/e2e/quantiles.rs) | test_name | test_description | what_is_tested | | --- | --- | --- | -| `ddsketch_core_and_portable_satisfy_the_relative_value_error_contract` | Both DDSketch implementations meet the relative-value-error guarantee against the order statistic of their own rank convention. | Crosses `alpha in {0.001, 0.01, 0.05, 0.1}` x `n in {1_000, 20_000, 100_000}` x the six `dds_streams` shapes (`adversarial-bucket-edges` log-uniform on `gamma` edges, positive-filtered `normal(1000, 250)`, `exponential(1e-3)`, `uniform` over `[1e6, 1e7)`, `zipf(1.1)` over 8192 keys in `[1e6, 1e7]`, and `wide-dynamic-range` spanning nine decades). Asserts `DDSketch::get_count()` and `PortableDds::total_count()` both equal `n` (no sample dropped), then per alpha tallies `RelativeQuantileSpec::core(alpha)` on `get_value_at_quantile` and `RelativeQuantileSpec::portable(alpha)` on `quantile` over the q grid `[0.0, 0.01, 0.1, 0.5, 0.9, 0.99, 1.0]` and closes both with `assert_none`: every estimate must be within `alpha + 8*f64::EPSILON*(1 + \|ln v\|)` of `sorted[ceil(q*n) - 1]` for the core and `sorted[floor(q*(n-1))]` for the portable twin. Zero violations tolerated, since the guarantee is deterministic. | -| `ddsketch_core_and_portable_answer_different_order_statistics` | The core and portable quantile conventions differ, and the divergence is pinned rather than absorbed into `alpha`. | Runs 8 `(n, q, core_idx, port_idx)` probes — `(3, 0.4, 1, 0)`, `(4, 0.34, 1, 1)`, `(4, 0.3, 1, 0)`, `(7, 0.2, 1, 1)`, `(7, 0.6, 4, 3)`, `(5, 0.5, 2, 2)`, `(10, 0.25, 2, 2)`, `(10, 0.15, 1, 1)` — on values `100 * gamma^(3i)` at `ALPHA = 0.01`, so one rank of movement is a whole bucket. Asserts `DdRankConvention::CeilNearestRank.index(n, q)` equals `core_idx` and `DdRankConvention::LowerFloor.index(n, q)` equals `port_idx`, that each implementation satisfies its own `RelativeQuantileSpec::check` against `sorted[its own index]`, that the two answers are unequal on every probe whose indices disagree, and that at least `3` probes disagree. | -| `ddsketch_core_endpoints_are_exact_and_portable_endpoints_are_alpha_relative` | The core sketch's endpoints are exact while the portable twin's are only `alpha`-relative. | For each of `alpha in {0.001, 0.01, 0.05, 0.1}` over 5_000 `log_uniform_f64` values (seed `4_242`, exponent range `3..30`), asserts `core.get_value_at_quantile(0.0) == Some(truth.min())`, `(1.0) == Some(truth.max())`, and `core.min()`/`core.max()` equal the exact extremes; the portable `quantile(0.0)` and `quantile(1.0)` are only checked through `RelativeQuantileSpec::portable(alpha).check` against the true min/max, i.e. within `alpha + a few ULP`, with no exactness claimed. | -| `ddsketch_satisfies_the_relative_error_contract_at_bucket_boundaries` | Values on, just below and just above a bucket edge all stay inside `alpha` on both sides. | For each of the four alphas and `k in {-40, -7, 0, 1, 13, 60, 200}` computes `edge = gamma^k` and probes `edge`, `edge*(1 - f64::EPSILON)`, `edge*(1 + f64::EPSILON)`, `edge*sqrt(gamma)` and `edge*gamma*(1 - 1e-12)`. Each probe is the single sample of a fresh core and portable sketch, so both rank conventions land on index 0 and the mapping is isolated; `quantile(0.5)` must satisfy each spec's `alpha + numerical_slack` tolerance against the probe itself. Probes the mapping refuses (`get_count() == 0`) are skipped rather than asserted on. | -| `ddsketch_merge_and_delta_replay_preserve_the_relative_error_contract` | Shard merge and bucket-delta replay both leave the relative-error contract intact on both implementations. | Per `alpha in {0.001, 0.01, 0.05, 0.1}` over `zipf_f64(40_000, 8_192, 1.1, 1e3, 1e7)`: a 4-way round-robin core shard merge must report `get_count() == 40_000` exactly and satisfy the contract over the full q grid; a core sketch rebuilt by replaying every non-zero `store_counts()` entry as an `octo_delta::DdDelta` at `store_offset() + i` must reproduce `get_count()` and satisfy the contract over the interior grid `DDS_QS[1..6] = [0.01, 0.1, 0.5, 0.9, 0.99]` (its endpoints are bucket representatives, so `q=0`/`q=1` are excluded). On the portable side, an even/odd split merged with `merge` is scored over the full grid, and a `DdSketchDelta` carrying the merged sketch's buckets plus `d_count` replayed into an empty sketch is scored over the interior grid. All four tallies use `assert_none`. | -| `ddsketch_satisfies_the_relative_value_error_contract_at_extreme_accuracy_parameters` | The relative-value contract still holds at accuracy parameters three orders of magnitude either side of the usual range. | For `alpha in {1e-5, 1e-4, 0.3, 0.5, 0.9}` (seed `0x0DDA_0000 + i*7919`), takes the six `dds_streams(alpha, 20_000, seed)` shapes filtered to `[min, max]` from `ddsketch::ddsketch_indexable_bounds(alpha)`, skipping any shape left with fewer than 100 values. Asserts core `get_count()` and portable `total_count()` both equal the surviving count, then `assert_none` on `RelativeQuantileSpec::core(alpha)` and `::portable(alpha)` over the 7-point `DDS_QS` grid. | +| `ddsketch_satisfies_the_relative_value_error_contract` | DDSketch meets the relative-value-error guarantee against the `ceil(q*n)` order statistic. | Crosses `alpha in {0.001, 0.01, 0.05, 0.1}` x `n in {1_000, 20_000, 100_000}` x the six `dds_streams` shapes (`adversarial-bucket-edges` log-uniform on `gamma` edges, positive-filtered `normal(1000, 250)`, `exponential(1e-3)`, `uniform` over `[1e6, 1e7)`, `zipf(1.1)` over 8192 keys in `[1e6, 1e7]`, and `wide-dynamic-range` spanning nine decades). Asserts `DDSketch::get_count()` equals `n` (no sample dropped), then per alpha tallies `RelativeQuantileSpec::core(alpha)` on `get_value_at_quantile` over the q grid `[0.0, 0.01, 0.1, 0.5, 0.9, 0.99, 1.0]` and closes with `assert_none`: every estimate must be within `alpha + 8*f64::EPSILON*(1 + \|ln v\|)` of `sorted[ceil(q*n) - 1]`. Zero violations tolerated, since the guarantee is deterministic. | +| `ddsketch_answers_the_ceil_nearest_rank_order_statistic_at_small_n` | At small `n` the answer is the `ceil(q*n)` order statistic, not the `floor(q*(n-1))` one. | Runs 8 `(n, q, idx)` probes — `(3, 0.4, 1)`, `(4, 0.34, 1)`, `(4, 0.3, 1)`, `(7, 0.2, 1)`, `(7, 0.6, 4)`, `(5, 0.5, 2)`, `(10, 0.25, 2)`, `(10, 0.15, 1)` — on values `100 * gamma^(3i)` at `alpha = 0.01`, so neighbouring ranks are three buckets apart. Asserts `dd_rank_index(n, q) == idx`, that `get_value_at_quantile(q)` passes `RelativeQuantileSpec::check` against `sorted[idx]`, and that at least `3` probes have `floor(q*(n-1)) != idx`. | +| `ddsketch_endpoints_are_exact` | The endpoints are the exact extremes. | For each of `alpha in {0.001, 0.01, 0.05, 0.1}` over 5_000 `log_uniform_f64` values (seed `4_242`, exponent range `3..30`), asserts `get_value_at_quantile(0.0) == Some(truth.min())`, `(1.0) == Some(truth.max())`, and `min()`/`max()` equal the exact extremes. | +| `ddsketch_satisfies_the_relative_error_contract_at_bucket_boundaries` | Values on, just below and just above a bucket edge all stay inside `alpha` on both sides. | For each of the four alphas and `k in {-40, -7, 0, 1, 13, 60, 200}` computes `edge = gamma^k` and probes `edge`, `edge*(1 - f64::EPSILON)`, `edge*(1 + f64::EPSILON)`, `edge*sqrt(gamma)` and `edge*gamma*(1 - 1e-12)`. Each probe is the single sample of a fresh sketch, so the mapping is isolated from any rank effect; `get_value_at_quantile(0.5)` must satisfy `alpha + numerical_slack` against the probe itself. Probes the mapping refuses (`get_count() == 0`) are skipped rather than asserted on. | +| `ddsketch_merge_and_delta_replay_preserve_the_relative_error_contract` | Shard merge and bucket-delta replay both leave the relative-error contract intact. | Per `alpha in {0.001, 0.01, 0.05, 0.1}` over `zipf_f64(40_000, 8_192, 1.1, 1e3, 1e7)`: a 4-way round-robin shard merge must report `get_count() == 40_000` exactly and satisfy the contract over the full q grid; a sketch rebuilt by replaying every non-zero `store_counts()` entry as an `octo_delta::DdDelta` at `store_offset() + i` must reproduce `get_count()` and satisfy the contract over the interior grid `DDS_QS[1..6] = [0.01, 0.1, 0.5, 0.9, 0.99]` (its endpoints are bucket representatives, so `q=0`/`q=1` are excluded). Both tallies use `assert_none`. | +| `ddsketch_satisfies_the_relative_value_error_contract_at_extreme_accuracy_parameters` | The relative-value contract still holds at accuracy parameters three orders of magnitude either side of the usual range. | For `alpha in {1e-5, 1e-4, 0.3, 0.5, 0.9}` (seed `0x0DDA_0000 + i*7919`), takes the six `dds_streams(alpha, 20_000, seed)` shapes filtered to `[min, max]` from `ddsketch::ddsketch_indexable_bounds(alpha)`, skipping any shape left with fewer than 100 values. Asserts `get_count()` equals the surviving count, then `assert_none` on `RelativeQuantileSpec::core(alpha)` over the 7-point `DDS_QS` grid. | | `ddsketch_bucket_width_widens_monotonically_with_the_accuracy_parameter` | A looser accuracy parameter never needs more buckets. | Feeds 20_000 values `1.0 + uniform_u64(.., 1_000_000, 0x0DDA_9001)` into a `DDSketch` at each `alpha in {1e-5, 1e-4, 0.3, 0.5, 0.9}` in that order, asserts nothing was dropped (`get_count() == 20_000`), and asserts the occupied-bucket count (`store_counts()` entries above zero) is no larger than the previous alpha's. The check is non-strict, so equal bucket counts pass and no bucket width is measured directly. | ### Quantiles: DDSketch Input Rejection @@ -1712,8 +1688,7 @@ Test file: [`tests/e2e/quantiles.rs`](../tests/e2e/quantiles.rs) | test_name | test_description | what_is_tested | | --- | --- | --- | -| `ddsketch_rejects_untrackable_values_and_mapping_mismatches` | Untrackable values are dropped without allocating or corrupting state, and a mismatched mapping is a runtime error on both types. | At `alpha = 0.01` feeds `NAN`, `-INFINITY`, `INFINITY`, `-5.0`, `0.0`, `f64::MIN_POSITIVE`, `5e-324`, `f64::MAX` and `1e308` into both implementations and asserts `get_count() == 0`, `total_count() == 0`, `port.store_counts.len() < 10_000`, and `port.quantile(0.5) == None`. Then `[1.0, 2.0, 4.0, 8.0, 16.0]` must give `get_count() == 5` / `total_count() == 5` with the store between `1` and `512` buckets. Using the shared `ddsketch_indexable_bounds(0.01)`, `min*(1 + 1e-9)` and `max*(1 - 1e-9)` are both accepted (count `2` in both, and the two counts equal), while `min*0.5` and `max*(1 + 1e-6)` leave the counts at `2`. `DDSketch::merge` and `PortableDds::merge` across `alpha = 0.01` vs `0.05` must both return `Err`. Finally at `alpha = 1e-9`, `1e-300` is dropped by both (`total_count() == 0`, `get_count() == 0`) with `store_counts.len() == 0`, so no allocation occurs for a rejected value. | -| `portable_ddsketch_rejects_hostile_delta_spans` | A delta spanning an implausible bucket range is rejected before any allocation, and a benign one still applies. | Seeds a portable sketch at `alpha = 0.01` with `1.0`, then asserts `apply_delta` on `buckets: [(i32::MAX - 1, 1), (7, 3)]` returns `Err` and leaves both `store_counts.len()` and `store_offset` at their prior values. A benign delta `[(6, 5)]` then applies successfully and `quantile(0.9)` must equal exactly `Some(gamma.powf(6.0) * (1.0 + alpha))`, so the applied count is visible through queries. | +| `ddsketch_rejects_untrackable_values_and_mapping_mismatches` | Untrackable values are dropped without allocating or corrupting state, and a mismatched mapping is a runtime error. | At `alpha = 0.01` feeds `NAN`, `-INFINITY`, `INFINITY`, `f64::MAX` and `1e308` and asserts `get_count() == 0`, an empty store, and `get_value_at_quantile(0.5) == None`. Then `[1.0, 2.0, 4.0, 8.0, 16.0]` must give `get_count() == 5` with the store between `1` and `512` buckets. Using the shared `ddsketch_indexable_bounds(0.01)`, `min*(1 + 1e-9)` and `max*(1 - 1e-9)` are both kept (count `2`); `min*0.5` enters the zero bucket and `max*(1 + 1e-6)` is dropped (count `3`, `zero_count() == 1`). `merge` across `alpha = 0.01` vs `0.05` returns `Err`. At `alpha = 1e-9`, `1e-300` is counted (`get_count() == 1`) with an empty store, so a zero-mapped value allocates nothing. | ### Quantiles: UnivMon-Q Estimators @@ -1894,7 +1869,7 @@ Test file: [`tests/e2e/windows.rs`](../tests/e2e/windows.rs) | `eh_coco_variant_conserves_the_retained_mass` | The merged Coco window holds exactly the retained mass. | Same EH, stream and merge over `COCO(Coco::init_with_size_and_seed(1024, 4, COCO_SEED))`, and verifies the merged table's recorded mass equals the number of events in the retained span. | | `eh_hll_variant_satisfies_the_register_error_model_over_the_retained_window` | A merged HLL payload lands inside the register error model for the retained span's cardinality. | Feeds `10_000` uniform draws over `200_000` (seed `0x0E11_0001`) into `ExponentialHistogram::new(8, 1_000_000, EHSketchList::HLL(HyperLogLog::::new()))`, merges the full retained span, and checks `merged.query(&DataInput::Str("card"))` against the exact distinct count of that span under `CardinalityConfidenceSpec::hll(14, 4.0)` - a relative band of `4 * 1.04 / 2^7 = 0.0325` at `m = 2^14` registers. This is a single trial, aggregated by `assert_independent_binomial` at `p = 6.3e-5`, so no violation is tolerated. | | `eh_kll_variant_satisfies_the_rank_error_characterization_over_the_retained_window` | A merged KLL payload's worst rank error stays inside the DataSketches characterization across independent compaction seeds. | Builds `12` separate `ExponentialHistogram::new(8, 1_000_000, EHSketchList::KLL(KLL::init_kll_with_seed(200, seed)))` instances with `seed = 0x5EED_0200 + t * 0x9E37_79B9_7F4A_7C15`, feeds each the same `10_000` uniform values over `1_000_000` (stream seed `0x0E11_0001`) at `t = 0..9_999`, merges each one's full retained span, and reduces each trial to its worst normalized rank error over the grid `[0.1, 0.25, 0.5, 0.75, 0.9]` read through `merged.query(&DataInput::F64(q))`; `KllRankSpec::datasketches(200)` supplies `eps = 2.446/200^0.9433 = 0.0165`, and the `12` trial outcomes are accepted by `assert_independent_binomial` at `p = 0.01`. | -| `eh_ddsketch_variant_satisfies_the_relative_value_error_contract_over_the_window` | A merged DDSketch payload holds relative value error and retains every observation. | Feeds `10_000` values `1_000_000 + uniform[0, 9_000_000)` (seed `0x0E11_0001`) into `ExponentialHistogram::new(8, 1_000_000, EHSketchList::DDS(DDSketch::new(0.01)))`, merges the full retained span, and checks `merged.query(&DataInput::F64(q))` at `q` in `[0.1, 0.25, 0.5, 0.75, 0.9]` against the span's exact order statistics under `RelativeQuantileSpec::core(0.01)` (`CeilNearestRank` convention, tolerance `alpha` plus a few-ULP slack) with zero violations; then asserts `merged.query(&DataInput::Str("count"))` equals the span's observation count exactly. | +| `eh_ddsketch_variant_satisfies_the_relative_value_error_contract_over_the_window` | A merged DDSketch payload holds relative value error and retains every observation. | Feeds `10_000` values `1_000_000 + uniform[0, 9_000_000)` (seed `0x0E11_0001`) into `ExponentialHistogram::new(8, 1_000_000, EHSketchList::DDS(DDSketch::new(0.01)))`, merges the full retained span, and checks `merged.query(&DataInput::F64(q))` at `q` in `[0.1, 0.25, 0.5, 0.75, 0.9]` against the span's exact order statistics under `RelativeQuantileSpec::core(0.01)` (`ceil(q*n)` order statistic, tolerance `alpha` plus a few-ULP slack) with zero violations; then asserts `merged.query(&DataInput::Str("count"))` equals the span's observation count exactly. | | `eh_univmon_variant_reports_the_exact_l1_over_the_retained_window` | A merged UnivMon payload reports L1 exactly and L2 inside the AMS second-moment band of its own grid. | Feeds `10_000` Zipf(1.1) keys over a `2_048` domain (seed `0x0E11_0001`) into `ExponentialHistogram::new(8, 1_000_000, EHSketchList::UNIVMON(UnivMon::init_univmon(32, 5, 2_048, 8)))`, merges the full retained span, and asserts `query(&DataInput::Str("l1"))` equals the span's exact total mass; then asserts `query(&DataInput::Str("l2"))` lies in `[\|\|f\|\|_2 * sqrt(1-b), \|\|f\|\|_2 * sqrt(1+b)]` with `b = SecondMomentSpec::new(5, 2_048).relative_bound() = sqrt(2*3/2048) = 0.0541`, i.e. about `-2.7%/+2.7%` on the norm. The bound asserted is the terminal layer's, not a composed constant for the recurrence. | | `every_eh_variant_can_merge_into_its_own_kind` | Every mergeable `EHSketchList` variant has a merge arm, so bucket consolidation cannot silently drop a payload. | For `CM` and `CS` at `3x256`, `COCO(256, 4)`, `COUNTL2HH` at `3x256`, `DDS(0.01)`, `ELASTIC(256)`, `HLL`, `KLL(200, seed 0x5EED_0300)`, `UNIVMON(32, 3, 256, 4)` and - under `--features experimental` - `UNIFORM(UniformSampling::with_seed(0.5, 7))`, feeds each side `64` distinct `DataInput::String` keys and `64` distinct `DataInput::F64` values (`k0..k63` / `1.0..64.0` on the left, `k64..k127` / `100.0..163.0` on the right) and asserts `left.merge(&right).is_ok()`. Only the `Ok` status is asserted; the merged contents are not read back. | | `eh_uniform_sampling_variant_reports_exact_retention_bookkeeping` | The reservoir payload's retention counters are exact over the merged span and its samples were all observed. | Gated on `#[cfg(feature = "experimental")]`. Feeds `10_000` uniform values over `1_000_000` (seed `0x0E11_0001`) into `ExponentialHistogram::new(8, 1_000_000, EHSketchList::UNIFORM(UniformSampling::with_seed(0.1, 0x5A_9101)))`, merges the full retained span, and asserts `query(&DataInput::Str("total_seen"))` equals `hi - lo + 1` exactly, that `query(&DataInput::Str("len"))` is greater than `0` and at most `total_seen`, and that every sample index `0..len` returns a value whose bit pattern appears in the span. The retained count is not held against the `0.1` rate's budget beyond `<= total_seen`. | @@ -2126,6 +2101,9 @@ Test file: [`tests/asapv1_golden.rs`](../tests/asapv1_golden.rs) | `cs_i32_regular_2x4_matches_golden` | The i32 counter type reaches the metadata, and an i32 golden is not an i64 sketch. | Verifies `Count::, RegularPath>::from_storage` over `[[0, 127, 128, 65536], [-1, -33, -32768, -2147483648]]` cast to `i32` serializes to `asapv1_golden/cs_i32_regular_2x4.hex`, that decoding it recovers the eight cells and re-serializes identically, that the i32 and i64 regular fixtures have equal length yet differ, and that the i32 fixture fails to decode as `Count, RegularPath>`. | | `cs_i64_fast_2x4_matches_golden` | The fast-path mode reaches the bytes, so the same matrix encodes differently per path. | Verifies `Count::, FastPath>::from_storage` over `[[0, 127, 128, 65536], [-1, -33, -32768, -2147483648]]` serializes to `asapv1_golden/cs_i64_fast_2x4.hex`, that decoding it recovers the eight cells and re-serializes identically, and that the fixture differs from `cs_i64_regular_2x4.hex` built from the same matrix. | | `cs_i64_regular_2x4_matches_golden` | A known signed Count-Sketch matrix serializes to its checked-in golden bytes. | Verifies `Count::, RegularPath>::from_storage` over `[[0, 127, 128, 65536], [-1, -33, -32768, -2147483648]]` — sweeping the msgpack integer widths in both directions — serializes to `asapv1_golden/cs_i64_regular_2x4.hex`, and that decoding the fixture reports `rows()` `2`, `cols()` `4`, the eight cells, and a byte-identical re-serialization. | +| `ddsketch_empty_a001_matches_golden` | The empty DDSketch state serializes to its golden bytes as metadata version 1, and the golden decodes back to it. | Builds the state at `alpha = 0.01` with empty stores, `sum = 0.0`, `min = +inf`, `max = -inf`, and verifies both it and a fresh `DDSketch::new(0.01)` serialize to `ddsketch_empty_a001.hex`; the decode has empty stores at offset `0`, a zero count and `zero_count`, `sum` `0.0`, `min()` and `max()` of `None`, and re-encodes byte-identically. | +| `ddsketch_positive_a001_matches_golden` | A positive-only DDSketch state serializes to its golden bytes as metadata version 1, and the golden decodes back to that state. | Builds `DDSketch` at `alpha = 0.01` through its serde form from the store `[1, 0, 127, 128, 300, 65536, 4294967296]` at offset `-40`, `sum = 2181071000.0`, `min = 0.453125`, `max = 0.5078125`, and verifies `serialize_to_bytes` equals `ddsketch_positive_a001.hex`; the decode reports that `alpha`, store, offset and scalars, an empty negative store, a zero `zero_count`, a `get_count()` equal to the bucket sum, and re-encodes byte-identically. | +| `ddsketch_signed_a001_matches_golden` | A signed DDSketch state serializes to its golden bytes as metadata version 2, and the golden decodes back to that state. | Builds `DDSketch` at `alpha = 0.01` from the positive store `[3, 0, 2]` at offset `310`, the negative store `[5, 1]` at offset `-208`, `zero_count = 7`, `sum = 2523.90625`, `min = -0.016`, `max = 515.0`, and verifies `serialize_to_bytes` equals `ddsketch_signed_a001.hex`; the decode reports both stores, both offsets, the zero count, the scalars and a `get_count()` of `18`, and re-encodes byte-identically. | | `hll_classic_p12_matches_golden` | A known P12 Classic register pattern serializes to its golden bytes on both the portable and native paths. | Builds `HllSketch::from_raw(HllVariant::Regular, 12, regs, 0.0, 0.0, 0.0)` where `regs` is 4,096 zeros with `[0] = 1`, `[1] = 7`, `[100] = 42` and `[4095] = 3`, verifies `to_msgpack()` equals `asapv1_golden/hll_classic_p12.hex`, that `HyperLogLogP12::::deserialize_from_bytes` on the fixture recovers those registers and re-serializes identically, and that `HllSketch::from_msgpack` reports the same registers, `precision` `12` and `HllVariant::Regular`. | | `hll_ertl_mle_p12_matches_golden` | The Datafusion variant tag reaches the bytes for the same P12 register pattern. | Builds `HllSketch::from_raw(HllVariant::Datafusion, 12, regs, 0.0, 0.0, 0.0)` over the shared P12 pattern (`[0] = 1`, `[1] = 7`, `[100] = 42`, `[4095] = 3` in 4,096 registers), verifies `to_msgpack()` equals `asapv1_golden/hll_ertl_mle_p12.hex`, that `HyperLogLogP12::::deserialize_from_bytes` recovers the registers and re-serializes identically, and that `HllSketch::from_msgpack` reports `HllVariant::Datafusion`. | | `hll_hip_p12_matches_golden` | The HIP variant's three running scalars travel on the wire beside its registers. | Builds `HllSketch::from_raw(HllVariant::Hip, 12, regs, 1.5, 2.5, 3.0)` over the shared P12 pattern, verifies `to_msgpack()` equals `asapv1_golden/hll_hip_p12.hex`, that `HyperLogLogHIPP12::deserialize_from_bytes` re-serializes the fixture identically, and that `HllSketch::from_msgpack` reports the registers, `HllVariant::Hip`, `hip_kxq0` `1.5`, `hip_kxq1` `2.5` and `hip_est` `3.0`. | @@ -2144,8 +2122,6 @@ Test file: [`tests/msgpack_compat.rs`](../tests/msgpack_compat.rs) | `count_min_wire_shape` | The `CountMinSketchWire` DTO round-trips through `rmp_serde` with its field shape intact. | Serializes `CountMinSketchWire { sketch: vec![vec![1.0, 2.0]; 3], rows: 3, cols: 2 }` with `rmp_serde::to_vec` and reads it back with `rmp_serde::from_slice`, verifying `rows` `3`, `cols` `2` and a `sketch` of length `3`; the cell values are not checked. | | `count_min_with_heap_wire_shape` | The nested `CountMinSketchWithHeapWire` DTO round-trips through `rmp_serde` with its nesting intact. | Serializes a wire struct holding a `CountMinSketchInnerWire` of `2x4` zeros, one `CmsHeapItem { key: "hot", value: 42.0 }` and `heap_size: 8`, then verifies the `rmp_serde` decode reports `heap_size` `8`, one heap entry and the key `hot`; the item's value and the inner matrix are not re-checked. | | `count_sketch_round_trip` | A portable `CountSketch` survives a msgpack round trip with its geometry intact. | Updates `CountSketch::new(3, 64)` with `k1` `+1.0` and `k2` `+2.0`, then verifies the decode reports `rows` `3` and `cols` `64`; the signed counters and the estimates are not checked. | -| `dd_sketch_decodes_go_bytes` | The test asserts nothing. | The test asserts nothing: the body is empty and the test is marked `#[ignore]` with the reason that no cross-language msgpack fixture exists in the repository. | -| `dd_sketch_round_trip` | A portable `DdSketch` survives a msgpack round trip carrying its full count in the bucket array. | Updates `DdSketch::new(0.01)` with `1.0`, `10.0` and `100.0`, then verifies the decode's `total_count()` — recovered by summing the buckets, since the count is not on the wire — is `3`; `alpha`, the bucket offset and the quantiles are not checked. | | `delta_result_round_trip` | A `DeltaResult` survives a msgpack round trip with both of its sets. | Serializes a `DeltaResult` whose `added` holds `a` and whose `removed` holds `b`, then verifies the decode's `added` contains `a` and its `removed` contains `b`. | | `hll_sketch_decodes_go_bytes` | The test asserts nothing. | The test asserts nothing: the body is empty and the test is marked `#[ignore]` with the reason that no cross-language msgpack fixture exists in the repository. | | `hll_sketch_round_trip` | A portable `HllSketch` survives a msgpack round trip with its register count intact. | Updates `HllSketch::new(HllVariant::Regular, 8)` with `b"a"`, `b"b"` and `b"c"`, then verifies only that the decode's `registers` has the same length as the source's; the register values, the precision and the variant are not checked. | @@ -2166,7 +2142,6 @@ Test file: [`tests/conformance_kit.rs`](../tests/conformance_kit.rs) | `count_l2hh_passes_signed_frequency_and_merge_conformance` | `CountL2HH` passes the two-sided frequency, turnstile and merge-equivalence batteries. | Drives `CountL2HH::with_dimensions(5, 4_096)` over a 60,000-draw Zipf stream (domain 2,048, exponent 1.1, seed `9001`) through `frequency_battery` two-sided at `rel_tol` `0.06` and `abs_tol` `25.0` on every key of true count at least `25`, through `turnstile_battery` on key `42` (`+500` then `-200` must estimate `300` within `1.0`, then `-300` must cancel to within `1e-6`), and through `merge_equivalence_battery`, which shards the stream by parity, merges, and requires agreement with a single-pass sketch within `25.0 + 0.06 * count`. | | `countmin_passes_frequency_and_merge_conformance` | `CountMin` passes the one-sided frequency and merge-equivalence batteries. | Drives `CountMin::, FastPath>::with_dimensions(4, 4096)` over the 60,000-draw Zipf stream (domain 2,048, exponent 1.1, seed `9001`) through `frequency_battery` one-sided at `rel_tol` `0.01` and `abs_tol` `4.0`, which requires the absent sentinel key `i64::MIN` to estimate within `±4.0` and each key of true count at least `25` to land in `[count, count * 1.01 + 4.0]`, then through `merge_equivalence_battery` at the same spec. | | `countsketch_passes_signed_frequency_conformance` | `Count` passes the two-sided frequency, turnstile and merge-equivalence batteries. | Drives `Count::, RegularPath>::with_dimensions(5, 4096)` over the 60,000-draw Zipf stream through `frequency_battery` two-sided at `rel_tol` `0.06` and `abs_tol` `25.0`, `turnstile_battery` on key `42` via `insert_many` (`+500 - 200` must read `300` within `1.0`, then cancel to within `1e-6`), and `merge_equivalence_battery` at the same spec. | -| `ddsketch_passes_relative_quantile_conformance` | The portable `DdSketch` holds its relative-value-error promise under its own rank convention. | Runs `relative_quantile_battery` on `PortableDds::new(0.01)` over the positive samples of `normal_f64(40_000, 500.0, 80.0, 7001)`, requiring at q `0.1`, `0.25`, `0.5`, `0.75` and `0.9` a relative error of at most `0.01` plus a few-ULP numerical slack against the exact `floor(q*(n-1))` order statistic. | | `heap_backed_frequency_sketches_pass_frequency_and_merge_conformance` | Both heap-backed matrix sketches pass the frequency and merge-equivalence batteries. | Over the 60,000-draw Zipf stream (domain 2,048, exponent 1.1, seed `9001`), drives `CMSHeap::, FastPath>::new(4, 4_096, 64)` through `frequency_battery` and `merge_equivalence_battery` one-sided at `rel_tol` `0.01` and `abs_tol` `8.0`, and `CSHeap::, RegularPath>::new(5, 4_096, 64)` through both two-sided at `rel_tol` `0.06` and `abs_tol` `25.0`. | | `hll_variants_pass_cardinality_conformance` | All three native HyperLogLog estimators and the portable one estimate within 3%. | Runs `cardinality_battery` at `rel_tol` `0.03` over 100,000 distinct keys `i.wrapping_mul(0x9E37_79B9_7F4A_7C15)` on `HyperLogLog`, `HyperLogLog`, `HyperLogLogHIP` and the portable `HllSketch::new(HllVariant::Regular, 14)`; each must estimate `100_000` within 3% and stay in band after the whole stream is replayed as duplicates. | | `kll_family_passes_quantile_conformance` | The three KLL adapters hold the shared rank-error band. | Runs `quantile_battery` at the default `rank_tol` of `0.03` over q `0.1`, `0.25`, `0.5`, `0.75` and `0.9` on the positive samples of `normal_f64(40_000, 500.0, 80.0, 7001)` for `KLL::init_kll_with_seed(200, 0x4017_0001)`, `KLLDynamic::::init_kll_with_seed(200, 0x4017_0001)`, and a `KLL` at the same `k` and seed queried through `quantile_cached`. | @@ -2193,7 +2168,6 @@ Test file: [`tests/sketches_go_parity_probe.rs`](../tests/sketches_go_parity_pro | --- | --- | --- | | `sketches_count_fastpath_matches_go_count_sketch_envelope` | A `sketches::Count` fast-path matrix encodes to the checked-in Count-Sketch envelope bytes. | Fills `Count, FastPath, DefaultXxHasher>` at `3x512` with 25 `insert_many` calls over the five keys `k-a`..`k-e`, packs the row-major cells and per-row L2 sums into a `CountSketchState` with `counter_type` `INT64` inside a `SketchEnvelope { format_version: 1 }` with no producer or hash_spec, and verifies the prost encoding equals the 1,577-byte fixture `tests/cs_envelope_golden.hex`. | | `sketches_countmin_fastpath_matches_go_count_min_envelope` | A `sketches::CountMin` fast-path matrix encodes to the checked-in Count-Min envelope bytes. | Fills `CountMin, FastPath, DefaultXxHasher>` at `4x2048` with 50 `insert_many` calls over the ten keys `flow-0`..`flow-9`, packs the cells cast to `i64` plus per-row L1 and L2 sums into a `CountMinState` with `counter_type` `INT64`, and verifies the prost encoding matches the 8,275-byte fixture `src/sketches/testdata/cms_envelope_golden.hex` in length and then byte for byte. | -| `sketches_ddsketch_matches_go_envelope` | A `sketches::DDSketch` encodes to the DDSketch envelope hex constant held in the test file. | Fills `DDSketch::new(0.01)` with `1..=50` as `f64`, encodes a `DdSketchState` carrying `store_counts`, `store_offset` and an alpha round-tripped through gamma (`gamma = (1 + a) / (1 - a)`, `alpha_wire = (gamma - 1) / (gamma + 1)`), and verifies the envelope matches the 403-byte inline hex constant in length and then byte for byte; the comment on that constant records it as this crate's own output rather than a fixture from another implementation, so no cross-language parity is pinned. | | `sketches_hll_classic_matches_go_envelope` | An ErtlMLE P14 register set encodes to the checked-in HyperLogLog envelope bytes. | Feeds `HyperLogLogImpl` the `hash64_seeded(CANONICAL_HASH_SEED, ...)` digests of the IEEE-754 little-endian bytes of `1.0..=50.0` through `insert_with_hash`, encodes a `HyperLogLogState` with `variant` `2`, `precision` `14`, the dense registers, zeroed HIP scalars and no sparse form, and verifies the envelope matches the 16,398-byte fixture `src/sketches/testdata/hll_envelope_golden.hex` in length and then byte for byte; despite the name, the variant under test is `ErtlMLE`, not `Classic`. | ### Cross-Language Proto Consumer @@ -2202,7 +2176,7 @@ Test file: [`tests/xtest_consumer.rs`](../tests/xtest_consumer.rs) | test_name | test_description | what_is_tested | | --- | --- | --- | -| `cross_language_proto` | Protobuf envelopes read from the cross-language fixture directory answer their point queries in band. | Asserts nothing unless fixtures are available: with no `XTEST_DIR` and no discoverable producer directory holding `tests/cross_language/xtest_producer_test.go`, it prints a skip line and returns. With fixtures it decodes nine `SketchEnvelope` files and records a pass or failure for each — `countmin.pb` min-frequency for `item:42` at least `101`, `kll.pb` p50 within 5% of `5000` and p99 within 5% of `9900`, `ddsketch.pb` the same two quantiles within 2%, `hll.pb` an ErtlMLE cardinality inside `[40_000, 65_000]`, `countsketch.pb` `cs:hot` at least `200`, `coco.pb` `coco:hot` at least `500`, `elastic.pb` `elephant` at least `900`, `univmon.pb` a cardinality inside `[1_000, 15_000]`, `hydra.pb` `hydra:42` at least `51` — using the hash seeds `0xcafe3553`, `0x6a09e667` and `0xbb67ae85`, and, when `countmin_sampled.pb` and `hll_sampled.pb` exist, requires `effective_sample_p` of `0.1` and a `rescale_count` by `1/p` within 5% of `100_000` and 6% of `200_000`. A single `assert!` at the end fails if any check failed; a missing or mistyped envelope panics on the spot. | +| `cross_language_proto` | Protobuf envelopes read from the cross-language fixture directory answer their point queries in band. | Asserts nothing unless fixtures are available: with no `XTEST_DIR` and no discoverable producer directory holding `tests/cross_language/xtest_producer_test.go`, it prints a skip line and returns. With fixtures it decodes eight `SketchEnvelope` files and records a pass or failure for each — `countmin.pb` min-frequency for `item:42` at least `101`, `kll.pb` p50 within 5% of `5000` and p99 within 5% of `9900`, `hll.pb` an ErtlMLE cardinality inside `[40_000, 65_000]`, `countsketch.pb` `cs:hot` at least `200`, `coco.pb` `coco:hot` at least `500`, `elastic.pb` `elephant` at least `900`, `univmon.pb` a cardinality inside `[1_000, 15_000]`, `hydra.pb` `hydra:42` at least `51` — using the hash seeds `0xcafe3553`, `0x6a09e667` and `0xbb67ae85`, and, when `countmin_sampled.pb` and `hll_sampled.pb` exist, requires `effective_sample_p` of `0.1` and a `rescale_count` by `1/p` within 5% of `100_000` and 6% of `200_000`. A single `assert!` at the end fails if any check failed; a missing or mistyped envelope panics on the spot. | ### HLL Custom Precision @@ -2223,5 +2197,4 @@ Test file: [`tests/bug_verification.rs`](../tests/bug_verification.rs) | test_name | test_description | what_is_tested | | --- | --- | --- | -| `portable_ddsketch_respects_alpha_at_bucket_edges` | Both DDSketch implementations hold the advertised relative accuracy on a value sitting at a bucket's lower edge. | Sets `alpha = 0.05`, `gamma = (1+alpha)/(1-alpha)`, and feeds the bucket-edge value `gamma^20 * (1 + 1e-6)` 10,000 times into both `message_pack_format::portable::ddsketch::DdSketch::new(0.05)` and `DDSketch::new(0.05)`, then asserts the portable `quantile(0.5)` and the core `get_value_at_quantile(0.5)` are each within `alpha * (1 + 1e-6)` relative error of that value — a representative reported as the bucket log-midpoint `gamma^(k+0.5)` gives about `5.13%` here and fails. Exact agreement between the two is not asserted, because the core sketch clamps representatives to its observed min/max and the portable wire format does not carry them. | | `countl2hh_f2_survives_beyond_i64_max` | `CountL2HH`'s hot-path L2 accumulation stays exact inside `i64` and saturates beyond it. | Builds `CountL2HH::::with_dimensions_and_seed(4, 2048, 7)` and calls `fast_insert_with_count(&DataInput::U32(1), 3_000_000_000)`, asserting `get_l2_sqr()` matches the exact `9e18` to a relative `1e-12`; a second key `U32(2)` at the same count puts the true F2 at `1.8e19`, past `i64::MAX`, and `get_l2_sqr()` must then equal `i64::MAX` (`9.223e18`) to a relative `1e-12` rather than wrapping to `0` or panicking in debug. | diff --git a/examples/accuracy_probe.rs b/examples/accuracy_probe.rs index 643bfc3e..2c2efd94 100644 --- a/examples/accuracy_probe.rs +++ b/examples/accuracy_probe.rs @@ -8,7 +8,6 @@ use asap_sketchlib::common::input::{HydraCounter, HydraQuery}; use asap_sketchlib::message_pack_format::portable::countminsketch::CountMinSketch; -use asap_sketchlib::message_pack_format::portable::ddsketch::DdSketch as PortableDds; use asap_sketchlib::message_pack_format::portable::hll::{HllSketch, HllVariant}; use asap_sketchlib::message_pack_format::portable::hydra_kll::HydraKllSketch; use asap_sketchlib::message_pack_format::portable::kll::KllSketch as PortableKll; @@ -512,66 +511,48 @@ fn quantile_report(qf: impl Fn(f64) -> f64, data: &[f64], qs: &[f64]) -> (f64, S // ---------------------------------------------------------------- DDSketch fn probe_ddsketch() { let mut p = Probe::new(); - println!("--- DDSketch core vs portable (alpha=0.05) ---"); + println!("--- DDSketch (alpha=0.05) ---"); let alpha = 0.05; let gamma = (1.0f64 + alpha) / (1.0 - alpha); // Isolated adversarial probe: one distinct value sitting just above a // bucket's lower edge gamma^k, repeated N times. The only correct answer // for every quantile is v itself. let mut worst_core = 0.0f64; - let mut worst_port = 0.0f64; for k in [10i32, 20, 30] { let v = gamma.powi(k) * (1.0 + 1e-6); // just inside bucket k let n = 10_000u64; let mut core = DDSketch::new(alpha); - let mut port = PortableDds::new(alpha); for _ in 0..n { core.add(&v); - port.update(v); } let rc = rel_err(core.get_value_at_quantile(0.5).unwrap(), v); - let rp = rel_err(port.quantile(0.5).unwrap(), v); worst_core = worst_core.max(rc); - worst_port = worst_port.max(rp); } p.check( "core DDSketch honors alpha=0.05 at bucket edges", format!("max rel err {worst_core:.5}"), worst_core <= alpha * (1.0 + 1e-6), ); - p.check( - "portable DdSketch honors alpha=0.05 at bucket edges", - format!("max rel err {worst_port:.5} (shared gamma^k*(1+alpha) representative)"), - worst_port <= alpha * (1.0 + 1e-6), - ); - // Mixed-stream sanity on both. + // Mixed-stream sanity. let mut rng = StdRng::seed_from_u64(47); let mut core = DDSketch::new(alpha); - let mut port = PortableDds::new(alpha); let mut truth: Vec = Vec::new(); for _ in 0..20_000 { let k = rng.random_range(5..40); let frac = rng.random::(); let v = gamma.powi(k) * (1.0 + frac * (gamma - 1.0)); core.add(&v); - port.update(v); truth.push(v); } truth.sort_by(|a, b| a.partial_cmp(b).unwrap()); let mc = qs_max_rel(&truth, |q| core.get_value_at_quantile(q).unwrap(), &qs_dd()); - let mp = qs_max_rel(&truth, |q| port.quantile(q).unwrap(), &qs_dd()); p.check( "core mixed stream <= alpha", format!("max rel {mc:.5}"), mc <= alpha, ); - p.check( - "portable mixed stream <= alpha", - format!("max rel {mp:.5}"), - mp <= alpha, - ); p.finish("DDSketch"); } diff --git a/proto/ddsketch/ddsketch.proto b/proto/ddsketch/ddsketch.proto deleted file mode 100644 index 02b8beff..00000000 --- a/proto/ddsketch/ddsketch.proto +++ /dev/null @@ -1,75 +0,0 @@ -// ddsketch.proto — Portable wire format for DDSketch quantile sketch state. - -syntax = "proto3"; - -package sketchlib.v1; - -option go_package = "github.com/ProjectASAP/sketchlib-go/proto/ddsketch;ddsketchpb"; - -// ============================================================================ -// DDSketch -// ============================================================================ - -// DDSketchState is the portable state of a DDSketch quantile sketch. -// -// Only alpha is required to reconstruct the index mapping. Consumers must -// recompute all derived mapping values on load: -// gamma = (1.0 + alpha) / (1.0 - alpha) -// log_gamma = ln(gamma) -// inv_log_gamma = 1.0 / log_gamma -// -// Bucket indices: a value v maps to bucket floor(ln(v) * inv_log_gamma). -// Stored bucket absolute_index = array_index + store_offset. -message DDSketchState { - // Relative accuracy guarantee. Required. Must satisfy 0 < alpha < 1. - double alpha = 1; - - // Bucket count array in index order (ascending absolute bucket index). - // Absolute index of store_counts[i] is i + store_offset. - repeated uint64 store_counts = 2 [packed = true]; - - // Absolute bucket index corresponding to store_counts[0]. - // May be negative (values < 1.0 map to negative bucket indices). - sint32 store_offset = 3; - - // Tags 4-7 previously carried the DataPoint-level METRIC scalars - // (count, sum, min, max). They were dropped from the wire format: - // the count is recoverable by summing store_counts and the remaining - // aggregates are carried elsewhere. - reserved 4, 5, 6, 7; - - reserved 8 to 15; - // Signed extension: negative magnitudes use the same mapping as positives. - repeated uint64 negative_store_counts = 16 [packed = true]; - sint32 negative_store_offset = 17; - uint64 zero_count = 18; -} - -// ============================================================================ -// DDSketch Delta -// ============================================================================ - -// DDSketchDelta carries only the buckets that changed above threshold T. -// -// DataPoint-level metric scalars (d_count/d_sum/new_min/new_max/min_changed/ -// max_changed) are intentionally NOT carried on the wire: the count delta is -// recoverable by summing bucket deltas, and min/max/quantiles are derived from -// the bucket distribution within the relative-accuracy guarantee. This message -// is byte-identical to the Go reference implementation's DDSketchDelta so the -// Go and Rust runtimes emit byte-identical delta frames for identical window -// state (cross-language byte parity). -message DDSketchDelta { - repeated DDSketchBucketDelta buckets = 1; - repeated DDSketchBucketDelta negative_buckets = 16; - uint64 zero_count = 17; - - // Fields 2-7 (d_count/d_sum/new_min/new_max/min_changed/max_changed) - // removed: DataPoint-level metric scalars are no longer carried on the wire. - reserved 2, 3, 4, 5, 6, 7; -} - -// DDSketchBucketDelta is the delta for one log-scale bucket. -message DDSketchBucketDelta { - sint32 index = 1; // signed bucket index (negative for values < 1) - uint64 d_count = 2; // Δcount ≥ threshold T -} diff --git a/proto/sketchlib.proto b/proto/sketchlib.proto index 0a6d8193..602da7fc 100644 --- a/proto/sketchlib.proto +++ b/proto/sketchlib.proto @@ -8,7 +8,6 @@ // countsketch/countsketch.proto — CountSketchState, TopKState, HeapEntry // hll/hll.proto — HyperLogLogState, HLLVariant // kll/kll.proto — KLLState, CoinState -// ddsketch/ddsketch.proto — DDSketchState // univmon/univmon.proto — UnivMonState, UnivMonLayer // hydra/hydra.proto — HydraState, HydraCounterType, HydraCell // @@ -33,7 +32,6 @@ import "countminsketch/countminsketch.proto"; import "countsketch/countsketch.proto"; import "hll/hll.proto"; import "kll/kll.proto"; -import "ddsketch/ddsketch.proto"; import "univmon/univmon.proto"; import "hydra/hydra.proto"; import "cocosketch/cocosketch.proto"; @@ -70,13 +68,14 @@ message SketchEnvelope { CountSketchState count_sketch = 11; HyperLogLogState hll = 12; KLLState kll = 13; - DDSketchState ddsketch = 14; UnivMonState univmon = 15; HydraState hydra = 16; CocoSketchState coco = 17; ElasticState elastic = 18; } + reserved 14; + reserved "ddsketch"; reserved 19 to 31; // Reserved for future sketch types. // Geometric / NitroSketch sampling probability for the producing sketch. diff --git a/src/lib.rs b/src/lib.rs index b062b8b9..dd542eb6 100644 --- a/src/lib.rs +++ b/src/lib.rs @@ -11,8 +11,8 @@ //! - [`message_pack_format`]: MessagePack/proto wire format shared with //! `sketchlib-go`. Owns the wire-DTO sketch types consumed by the ASAP //! query engine (`CountMinSketch`, `CountSketch`, `KllSketch`, `HllSketch`, -//! `DdSketch`, `HydraKllSketch`, `CountMinSketchWithHeap`, -//! `SetAggregator`, `DeltaResult`). +//! `HydraKllSketch`, `CountMinSketchWithHeap`, `SetAggregator`, +//! `DeltaResult`). //! //! Most users can start with the crate-root re-exports such as [`DataInput`], //! [`CountMin`], [`HyperLogLog`], [`KLL`], and [`DDSketch`]. Reach for the @@ -47,7 +47,6 @@ pub use message_pack_format::portable::countsketch::{ COUNT_SKETCH_TOPK_CAPACITY, CountSketch, CountSketchDelta, }; pub use message_pack_format::portable::countsketch_topk::{CountSketchWithHeap, CsHeapItem}; -pub use message_pack_format::portable::ddsketch::{DDSKETCH_GROW_CHUNK, DdSketch, DdSketchDelta}; pub use message_pack_format::portable::delta_set_aggregator::DeltaResult; pub use message_pack_format::portable::hll::{HllSketch, HllSketchDelta, HllVariant}; pub use message_pack_format::portable::hydra_kll::HydraKllSketch; diff --git a/src/message_pack_format/native/ddsketch.rs b/src/message_pack_format/native/ddsketch.rs deleted file mode 100644 index 9a88bd74..00000000 --- a/src/message_pack_format/native/ddsketch.rs +++ /dev/null @@ -1,17 +0,0 @@ -//! Native MessagePack codec impl for [`crate::sketches::ddsketch::DDSketch`]. -//! -//! Produces and consumes the ASAPv1 envelope (kind_id `0x05 0x00`): `alpha` in -//! the metadata, `[counts, offset, sum, min, max]` in the payload. - -use crate::message_pack_format::{Error, MessagePackCodec}; -use crate::sketches::ddsketch::DDSketch; - -impl MessagePackCodec for DDSketch { - fn to_msgpack(&self) -> Result, Error> { - Ok(self.serialize_to_bytes()?) - } - - fn from_msgpack(bytes: &[u8]) -> Result { - Ok(Self::deserialize_from_bytes(bytes)?) - } -} diff --git a/src/message_pack_format/native/mod.rs b/src/message_pack_format/native/mod.rs index 0a83ef96..f612a145 100644 --- a/src/message_pack_format/native/mod.rs +++ b/src/message_pack_format/native/mod.rs @@ -10,7 +10,6 @@ pub mod countminsketch; pub mod countsketch; pub mod countsketch_topk; -pub mod ddsketch; pub mod hll; pub mod kll; pub mod kll_dynamic; diff --git a/src/message_pack_format/portable/ddsketch.rs b/src/message_pack_format/portable/ddsketch.rs deleted file mode 100644 index e8961d6f..00000000 --- a/src/message_pack_format/portable/ddsketch.rs +++ /dev/null @@ -1,1364 +0,0 @@ -//! Wire-format-aligned DDSketch types. The wire DTO + runtime ops -//! live together here. - -use serde::{Deserialize, Serialize}; - -use crate::message_pack_format::{Error as MsgPackError, MessagePackCodec}; - -/// Bucket-store growth chunk for the wire-format-aligned [`DdSketch`] -/// variant. Matches the Go reference implementation's `DDSketch.GrowChunk` -/// so the `store_counts` / `store_offset` layout written by -/// [`DdSketch::update`] is byte-identical to the Go `SerializePortable` -/// output for the same input stream. -pub const DDSKETCH_GROW_CHUNK: usize = 128; - -/// Safety limit for a single [`DdSketchDelta`] application: deltas arrive -/// over the wire untrusted, so a span beyond this many buckets is rejected -/// with an error instead of padding the dense store toward a multi-gigabyte -/// allocation. Generous relative to any legitimate producer: at α = 0.01 the -/// entire indexable range spans ~35k buckets. -pub const MAX_APPLY_DELTA_SPAN_BUCKETS: i64 = 1 << 22; - -// ===================================================================== -// Wire-format-aligned variant. -// -// `DdSketch` and `DdSketchDelta` below are the public-field, -// proto-decode-friendly types consumed by the query-engine -// accumulators. The high-throughput in-process variant is -// `crate::sketches::ddsketch::DDSketch`, which keeps its own design. -// ===================================================================== - -/// Sparse delta between two consecutive DDSketch snapshots — the -/// input shape for [`DdSketch::apply_delta`]. Mirrors the -/// `DDSketchDelta` proto (and its Rust bindings). Kept as a plain -/// struct so this crate doesn't need a tonic/prost dependency; proto -/// decode lives in the accumulator. -#[derive(Debug, Clone, Default)] -pub struct DdSketchDelta { - /// `(absolute_bucket_index, Δcount)` pairs, additive. - pub buckets: Vec<(i32, u64)>, - pub negative_buckets: Vec<(i32, u64)>, - pub zero_count: u64, - /// Δ total count. May be negative (signed on the wire). - pub d_count: i64, - /// Δ sum. - pub d_sum: f64, - /// Whether `new_min` carries a meaningful value. Min can only - /// decrease; a delta that didn't lower min sends `false`. - pub min_changed: bool, - pub new_min: f64, - /// Whether `new_max` carries a meaningful value. Max can only - /// increase. - pub max_changed: bool, - pub new_max: f64, -} - -/// Signed DDSketch state: separate magnitude stores and a zero count. -/// Positive-only MessagePack retains `[alpha, counts, offset]`; signed state -/// appends `[negative_counts, negative_offset, zero_count]` to that array. -#[derive(Debug, Clone, Deserialize)] -pub struct DdSketch { - /// Relative accuracy parameter; must satisfy `0 < alpha < 1`. - pub alpha: f64, - /// Bucket counts in absolute-index order. The absolute index of - /// `store_counts[i]` is `i + store_offset`. - pub store_counts: Vec, - /// Absolute bucket index corresponding to `store_counts[0]`. May - /// be negative. - pub store_offset: i32, - /// Negative-value magnitudes, indexed with the same logarithmic mapping. - #[serde(default)] - pub negative_store_counts: Vec, - #[serde(default)] - pub negative_store_offset: i32, - #[serde(default)] - pub zero_count: u64, -} - -impl Serialize for DdSketch { - fn serialize(&self, serializer: S) -> Result { - use serde::ser::SerializeStruct; - let signed = !self.negative_store_counts.is_empty() || self.zero_count != 0; - let mut state = serializer.serialize_struct("DdSketch", if signed { 6 } else { 3 })?; - state.serialize_field("alpha", &self.alpha)?; - state.serialize_field("store_counts", &self.store_counts)?; - state.serialize_field("store_offset", &self.store_offset)?; - if signed { - state.serialize_field("negative_store_counts", &self.negative_store_counts)?; - state.serialize_field("negative_store_offset", &self.negative_store_offset)?; - state.serialize_field("zero_count", &self.zero_count)?; - } - state.end() - } -} - -impl DdSketch { - /// Construct an empty sketch. - pub fn new(alpha: f64) -> Self { - assert!( - alpha > 0.0 && alpha < 1.0, - "alpha must be in (0,1); alpha=0 makes ln(gamma)=0 and every guard degenerate" - ); - Self { - alpha, - store_counts: Vec::new(), - store_offset: 0, - negative_store_counts: Vec::new(), - negative_store_offset: 0, - zero_count: 0, - } - } - - /// Construct from the decoded wire fields. - pub fn from_raw(alpha: f64, store_counts: Vec, store_offset: i32) -> Self { - Self { - alpha, - store_counts, - store_offset, - negative_store_counts: Vec::new(), - negative_store_offset: 0, - zero_count: 0, - } - } - - /// Convert every store to protobuf, including negative and zero observations. - pub fn to_proto(&self) -> crate::proto::sketchlib::DdSketchState { - crate::proto::sketchlib::DdSketchState { - alpha: self.wire_alpha(), - store_counts: self.store_counts.clone(), - store_offset: self.store_offset, - negative_store_counts: self.negative_store_counts.clone(), - negative_store_offset: self.negative_store_offset, - zero_count: self.zero_count, - } - } - - /// Restore all stores from a decoded protobuf state. - pub fn from_proto(state: crate::proto::sketchlib::DdSketchState) -> Self { - Self { - alpha: state.alpha, - store_counts: state.store_counts, - store_offset: state.store_offset, - negative_store_counts: state.negative_store_counts, - negative_store_offset: state.negative_store_offset, - zero_count: state.zero_count, - } - } - - /// Total number of values added, recovered by summing the bucket - /// counts. The wire format carries no `count` scalar, so this is the - /// authoritative count for quantile-rank computation. - pub fn total_count(&self) -> u64 { - self.store_counts - .iter() - .chain(&self.negative_store_counts) - .copied() - .fold(self.zero_count, u64::saturating_add) - } - - /// Merge one other sketch into self by aligning bucket arrays on - /// absolute indices. Both operands must share the same `alpha`. - /// - /// Like [`Self::apply_delta`], the operand is treated as untrusted wire - /// state: if the union span would exceed - /// [`MAX_APPLY_DELTA_SPAN_BUCKETS`] *and* grow beyond both operands' - /// existing stores, `Err` is returned before any allocation. Stores that - /// already legitimately span more than the cap (exotic small-α - /// configurations) merge normally as long as the union stays within - /// what either side already holds. - pub fn merge( - &mut self, - other: &DdSketch, - ) -> Result<(), Box> { - // Validate both store unions before publishing any of the merged state. - let mut merged = self.clone(); - merged.merge_positive(other)?; - let mut negative = Self::from_raw( - self.alpha, - self.negative_store_counts.clone(), - self.negative_store_offset, - ); - negative.merge_positive(&Self::from_raw( - other.alpha, - other.negative_store_counts.clone(), - other.negative_store_offset, - ))?; - merged.negative_store_counts = negative.store_counts; - merged.negative_store_offset = negative.store_offset; - merged.zero_count = self - .zero_count - .checked_add(other.zero_count) - .ok_or("DDSketch zero count overflow")?; - *self = merged; - Ok(()) - } - - fn merge_positive( - &mut self, - other: &DdSketch, - ) -> Result<(), Box> { - if (self.alpha - other.alpha).abs() > f64::EPSILON { - return Err(format!( - "DdSketch alpha mismatch: self={}, other={}", - self.alpha, other.alpha - ) - .into()); - } - - if other.store_counts.is_empty() { - return Ok(()); - } - if self.store_counts.is_empty() { - self.store_counts = other.store_counts.clone(); - self.store_offset = other.store_offset; - } else { - let self_start = self.store_offset as i64; - let self_end = self_start + self.store_counts.len() as i64; - let other_start = other.store_offset as i64; - let other_end = other_start + other.store_counts.len() as i64; - let new_start = self_start.min(other_start); - let new_end = self_end.max(other_end); - let new_len = new_end - new_start; - // Hostile-span guard: a decoded snapshot with store_offset near - // i32::MIN and a short count vector must not allocate a union - // spanning ~2^31 buckets (~17 GiB) in one call. Growth beyond - // both operands' existing spans is capped; stores that already - // legitimately exceed the cap pass through untouched. - let max_existing = - (self.store_counts.len() as i64).max(other.store_counts.len() as i64); - if new_len > MAX_APPLY_DELTA_SPAN_BUCKETS && new_len > max_existing { - return Err(format!( - "DdSketch merge spans {} buckets, exceeding the {}-bucket safety limit", - new_len, MAX_APPLY_DELTA_SPAN_BUCKETS - ) - .into()); - } - let new_len = new_len as usize; - let mut merged = vec![0u64; new_len]; - for (start, counts) in [ - (self_start, &self.store_counts), - (other_start, &other.store_counts), - ] { - for (i, &c) in counts.iter().enumerate() { - let idx = (start + i as i64 - new_start) as usize; - merged[idx] = merged[idx].saturating_add(c); - } - } - self.store_counts = merged; - self.store_offset = new_start as i32; - } - Ok(()) - } - - /// Apply a sparse delta to this sketch in place. Matches the - /// `ApplyDelta` logic in the Go reference implementation: - /// bucket counts add, total count + sum add, min can only decrease - /// and max can only increase. Used by the backend ingest path to - /// reconstitute a full sketch from a base snapshot + subsequent - /// delta-transmission frames. - /// - /// Deltas arrive over the wire and are treated as untrusted: before - /// mutating anything, the union span the delta would require is - /// checked against [`MAX_APPLY_DELTA_SPAN_BUCKETS`]. A hostile or - /// corrupt delta carrying an index near `i32::MAX` would otherwise - /// pad the dense store by ~2·10⁹ buckets (~17 GiB) in one call — - /// independent of α, unlike the `update()` path whose worst case is - /// bounded by the indexable-range guard. - pub fn apply_delta( - &mut self, - delta: &DdSketchDelta, - ) -> Result<(), Box> { - let mut merged = self.clone(); - merged.apply_positive_delta(delta)?; - let mut negative = Self::from_raw( - self.alpha, - self.negative_store_counts.clone(), - self.negative_store_offset, - ); - negative.apply_positive_delta(&DdSketchDelta { - buckets: delta.negative_buckets.clone(), - ..Default::default() - })?; - merged.negative_store_counts = negative.store_counts; - merged.negative_store_offset = negative.store_offset; - merged.zero_count = self - .zero_count - .checked_add(delta.zero_count) - .ok_or("DDSketch zero count overflow")?; - *self = merged; - Ok(()) - } - - fn apply_positive_delta( - &mut self, - delta: &DdSketchDelta, - ) -> Result<(), Box> { - // Pre-validate span before any mutation so a rejected delta - // leaves state untouched. - if !delta.buckets.is_empty() { - let mut min_d = i64::MAX; - let mut max_d = i64::MIN; - for (abs_idx, _) in &delta.buckets { - min_d = min_d.min(*abs_idx as i64); - max_d = max_d.max(*abs_idx as i64); - } - let cur_start = if self.store_counts.is_empty() { - min_d - } else { - self.store_offset as i64 - }; - let cur_end = cur_start + self.store_counts.len() as i64; - let new_start = cur_start.min(min_d); - let new_end = cur_end.max(max_d + 1); - if new_end - new_start > MAX_APPLY_DELTA_SPAN_BUCKETS { - return Err(format!( - "DdSketch delta spans {} buckets, exceeding the {}-bucket safety limit", - new_end - new_start, - MAX_APPLY_DELTA_SPAN_BUCKETS - ) - .into()); - } - } - - for (abs_idx, d_count) in &delta.buckets { - if self.store_counts.is_empty() { - self.store_counts = vec![0u64; 1]; - self.store_offset = *abs_idx; - } - let cur_start = self.store_offset as i64; - let cur_end = cur_start + self.store_counts.len() as i64; - let k = *abs_idx as i64; - if k < cur_start { - // Prepend zeros. - let pad = (cur_start - k) as usize; - let mut buf = vec![0u64; pad]; - buf.append(&mut self.store_counts); - self.store_counts = buf; - self.store_offset = *abs_idx; - } else if k >= cur_end { - let pad = (k - cur_end + 1) as usize; - self.store_counts.extend(std::iter::repeat_n(0u64, pad)); - } - let arr_idx = (k - self.store_offset as i64) as usize; - self.store_counts[arr_idx] = self.store_counts[arr_idx].saturating_add(*d_count); - } - Ok(()) - } - - /// Compute a sparse, proto-marshalled `DDSketchDelta` of `self` - /// against a `snapshot`. A bucket is included when its `Δcount` - /// (self − snapshot, clamped at 0) is `>= threshold`. - /// - /// This is the Rust twin of the Go reference implementation's - /// `ComputeDelta`: it iterates every non-empty bucket in `self`, - /// subtracts the snapshot's count for the same absolute index, and - /// emits the surviving bucket deltas. The returned bytes are a - /// `prost`-encoded [`crate::proto::sketchlib::DdSketchDelta`], - /// byte-identical to the Go `proto.Marshal(DDSketchDelta)` output - /// for the same inputs (cross-language byte parity). - /// - /// Delta-against-empty: when `snapshot` is the empty sketch, every - /// surviving bucket delta equals the window's own bucket count, so - /// the result is this window's full state encoded as a delta (no - /// cross-window subtraction). The DataPoint-level metric scalars are - /// not carried — the count delta is recoverable by summing bucket - /// deltas. - pub fn compute_delta(&self, snapshot: &DdSketch, threshold: u64) -> Vec { - use crate::proto::sketchlib::{DdSketchBucketDelta, DdSketchDelta as ProtoDelta}; - use prost::Message; - - let mut delta = ProtoDelta::default(); - if !self.store_counts.is_empty() { - for (i, &c) in self.store_counts.iter().enumerate() { - if c == 0 { - continue; - } - let k = self.store_offset + i as i32; - let snap_count: u64 = if !snapshot.store_counts.is_empty() { - let idx = k as i64 - snapshot.store_offset as i64; - if idx >= 0 && (idx as usize) < snapshot.store_counts.len() { - snapshot.store_counts[idx as usize] - } else { - 0 - } - } else { - 0 - }; - let dc = c.saturating_sub(snap_count); - if dc >= threshold { - delta.buckets.push(DdSketchBucketDelta { - index: k, - d_count: dc, - }); - } - } - } - let negative = Self::from_raw( - self.alpha, - self.negative_store_counts.clone(), - self.negative_store_offset, - ); - let previous = Self::from_raw( - snapshot.alpha, - snapshot.negative_store_counts.clone(), - snapshot.negative_store_offset, - ); - delta.negative_buckets = sparse_difference(&negative, &previous, threshold) - .into_iter() - .map(|(index, d_count)| DdSketchBucketDelta { index, d_count }) - .collect(); - delta.zero_count = self.zero_count.saturating_sub(snapshot.zero_count); - if delta.zero_count < threshold { - delta.zero_count = 0; - } - delta.encode_to_vec() - } - - /// Apply a `prost`-encoded [`crate::proto::sketchlib::DdSketchDelta`] - /// to this sketch in place (additive bucket merge). The Rust twin of - /// the Go reference implementation's `ApplyDelta`. - /// - /// Returns `Err` if `bytes` is not a valid `DDSketchDelta` proto. - pub fn apply_delta_bytes( - &mut self, - bytes: &[u8], - ) -> Result<(), Box> { - use crate::proto::sketchlib::DdSketchDelta as ProtoDelta; - use prost::Message; - - let proto = ProtoDelta::decode(bytes)?; - let delta = DdSketchDelta { - negative_buckets: proto - .negative_buckets - .into_iter() - .map(|b| (b.index, b.d_count)) - .collect(), - zero_count: proto.zero_count, - buckets: proto - .buckets - .into_iter() - .map(|b| (b.index, b.d_count)) - .collect(), - ..DdSketchDelta::default() - }; - self.apply_delta(&delta) - } - - /// MessagePack twin of [`Self::compute_delta`]. Computes the same - /// sparse bucket delta of `self` against `snapshot` (a bucket is - /// carried when its `Δcount = self − snapshot` clamped at 0 is - /// `>= threshold`), but serializes it as the parallel-array - /// MessagePack layout via [`DdSketchDelta`]'s [`MessagePackCodec`] - /// instead of proto. Same delta-against-empty semantics: against the - /// empty sketch every surviving delta equals this window's own bucket - /// count. - pub fn compute_delta_msgpack(&self, snapshot: &DdSketch, threshold: u64) -> Vec { - let mut delta = DdSketchDelta::default(); - if !self.store_counts.is_empty() { - for (i, &c) in self.store_counts.iter().enumerate() { - if c == 0 { - continue; - } - let k = self.store_offset + i as i32; - let snap_count: u64 = if !snapshot.store_counts.is_empty() { - let idx = k as i64 - snapshot.store_offset as i64; - if idx >= 0 && (idx as usize) < snapshot.store_counts.len() { - snapshot.store_counts[idx as usize] - } else { - 0 - } - } else { - 0 - }; - let dc = c.saturating_sub(snap_count); - if dc >= threshold { - delta.buckets.push((k, dc)); - } - } - } - delta.negative_buckets = sparse_difference( - &Self::from_raw( - self.alpha, - self.negative_store_counts.clone(), - self.negative_store_offset, - ), - &Self::from_raw( - snapshot.alpha, - snapshot.negative_store_counts.clone(), - snapshot.negative_store_offset, - ), - threshold, - ); - delta.zero_count = self.zero_count.saturating_sub(snapshot.zero_count); - if delta.zero_count < threshold { - delta.zero_count = 0; - } - delta - .to_msgpack() - .expect("DdSketchDelta msgpack encode is infallible for owned bucket arrays") - } - - /// MessagePack twin of [`Self::apply_delta_bytes`]. Decodes the - /// parallel-array MessagePack [`DdSketchDelta`] and applies it in - /// place (additive bucket merge). - /// - /// Returns `Err` if `bytes` is not a valid MessagePack `DdSketchDelta`. - pub fn apply_delta_msgpack_bytes( - &mut self, - bytes: &[u8], - ) -> Result<(), Box> { - let delta = DdSketchDelta::from_msgpack(bytes)?; - self.apply_delta(&delta) - } - - /// Merge a slice of references into a single new sketch. Returns - /// `Err` on alpha mismatch or an empty input. - pub fn merge_refs( - inputs: &[&DdSketch], - ) -> Result> { - let first = inputs - .first() - .ok_or("DdSketch::merge_refs called with empty input")?; - let mut merged = DdSketch::new(first.alpha); - for d in inputs { - merged.merge(d)?; - } - Ok(merged) - } - - /// Insert a single positive value. Updates count, sum, min/max - /// and increments the bucket for `floor(ln(v) / ln(gamma))` - /// where `gamma = (1+α)/(1-α)`. Provided primarily so tests can - /// build a ground-truth sketch to compare delta-apply output - /// against. - /// - /// Bucket-store growth mirrors the Go reference implementation's - /// `Buckets.ensure`: the first allocation is a half-chunk-centered - /// `GROW_CHUNK` of zeros, and subsequent expansions extend by - /// `max(needed, GROW_CHUNK)` so the on-the-wire `store_counts` / - /// `store_offset` layout is byte-identical to the Go - /// `SerializePortable` output. Without this chunked layout the - /// `DDSketchState` proto bytes would diverge from the Go producer's - /// payload (cross-language byte parity). - pub fn update(&mut self, value: f64) { - let _ = self.try_update(value); - } - - /// Reject NaN/infinity and values above the mapping range explicitly. - /// Like DataDog, magnitudes below the mapping range enter the zero bucket; - /// the relative-error guarantee does not cover those nonzero tiny values. - pub fn try_update(&mut self, value: f64) -> Result<(), &'static str> { - if !value.is_finite() { - return Err("DDSketch requires finite values"); - } - let (min_v, max_v) = crate::sketches::ddsketch::ddsketch_indexable_bounds(self.alpha); - if value.abs() > max_v { - return Err("DDSketch value exceeds indexable range"); - } - if value.abs() < min_v { - self.zero_count = self - .zero_count - .checked_add(1) - .ok_or("DDSketch zero count overflow")?; - return Ok(()); - } - let negative = value < 0.0; - let value = value.abs(); - let gamma = (1.0 + self.alpha) / (1.0 - self.alpha); - let ln_gamma = gamma.ln(); - // Reuse the positive-store growth routine for the magnitude store. - if negative { - std::mem::swap(&mut self.store_counts, &mut self.negative_store_counts); - std::mem::swap(&mut self.store_offset, &mut self.negative_store_offset); - } - let idx = (value.ln() / ln_gamma).floor() as i32; - self.ensure_bucket(idx); - let arr_idx = (idx as i64 - self.store_offset as i64) as usize; - self.store_counts[arr_idx] = self.store_counts[arr_idx].saturating_add(1); - if negative { - std::mem::swap(&mut self.store_counts, &mut self.negative_store_counts); - std::mem::swap(&mut self.store_offset, &mut self.negative_store_offset); - } - Ok(()) - } - - /// Ensure bucket `k` is addressable in `store_counts`, growing in - /// chunks of [`DDSKETCH_GROW_CHUNK`] to match the Go reference - /// implementation's `Buckets.ensure`. Empty stores are seeded with - /// a half-chunk of zeros centered on `k` (`store_offset = k - - /// GROW_CHUNK/2`); out-of-range expansions extend by `max(needed, - /// GROW_CHUNK)` on either side. The on-the-wire byte layout is - /// therefore identical between the Go and Rust producers fed the - /// same input stream — required for cross-language byte parity. - fn ensure_bucket(&mut self, k: i32) { - if self.store_counts.is_empty() { - self.store_counts = vec![0u64; DDSKETCH_GROW_CHUNK]; - self.store_offset = k - (DDSKETCH_GROW_CHUNK as i32 / 2); - return; - } - let cur_start = self.store_offset as i64; - let cur_end = cur_start + self.store_counts.len() as i64; - let kk = k as i64; - if kk < cur_start { - let needed = (cur_start - kk) as usize; - let grow = needed.max(DDSKETCH_GROW_CHUNK); - let mut buf = vec![0u64; grow]; - buf.extend_from_slice(&self.store_counts); - self.store_counts = buf; - self.store_offset -= grow as i32; - } else if kk >= cur_end { - let needed = (kk - cur_end + 1) as usize; - let grow = needed.max(DDSKETCH_GROW_CHUNK); - self.store_counts.extend(std::iter::repeat_n(0u64, grow)); - } - } - - /// Estimate the quantile at rank `q` ∈ [0, 1]. Walks the bucket - /// array in ascending absolute-index order, accumulating counts - /// until the target rank; returns the bucket's representative - /// value `gamma^k * (1 + alpha)` where `k` is the bucket's absolute - /// index. Returns `None` if the sketch is empty. - /// - /// For indexable nonzero order statistics, absolute error is at most - /// `alpha * abs(true_value)`. Tiny values mapped to zero are excluded. - pub fn quantile(&self, q: f64) -> Option { - let count = self.total_count(); - if count == 0 { - return None; - } - let target = (q * (count.saturating_sub(1)) as f64).floor() as u64; - self.value_at_rank(target) - } - - /// Estimate the linearly interpolated quantile at `q * (n - 1)`. - /// - /// Interpolation across opposite signs can cancel: its absolute error is - /// bounded by alpha times the weighted absolute endpoints, but its relative - /// error need not be bounded by alpha. Zero-mapped tiny values are excluded. - /// This interpolates the two adjacent order-statistic bucket estimates, - /// as required by PromQL quantile and continuous percentiles. The existing - /// `quantile` method retains its lower-order-statistic convention. - /// Returns `None` for invalid q/alpha, empty or overflowing counts, counts - /// above exact Float64 integer precision, or nonfinite bucket estimates. - pub fn quantile_interpolated(&self, q: f64) -> Option { - if !q.is_finite() - || !(0.0..=1.0).contains(&q) - || !self.alpha.is_finite() - || !(0.0..1.0).contains(&self.alpha) - || self.alpha == 0.0 - { - return None; - } - let count = self - .store_counts - .iter() - .chain(&self.negative_store_counts) - .try_fold(self.zero_count, |sum, count| sum.checked_add(*count))?; - if count == 0 || count > (1u64 << 53) { - return None; - } - let rank = q * (count - 1) as f64; - let lower_rank = rank.floor() as u64; - let upper_rank = rank.ceil() as u64; - let lower = self.value_at_rank(lower_rank)?; - let upper = self.value_at_rank(upper_rank)?; - if !lower.is_finite() || !upper.is_finite() { - return None; - } - let fraction = rank - lower_rank as f64; - let result = lower * (1.0 - fraction) + upper * fraction; - result.is_finite().then_some(result) - } - - fn value_at_rank(&self, target: u64) -> Option { - let mut target = target; - let gamma = (1.0 + self.alpha) / (1.0 - self.alpha); - for (i, &count) in self.negative_store_counts.iter().enumerate().rev() { - if target < count { - return Some( - -gamma.powf((self.negative_store_offset as i64 + i as i64) as f64) - * (1.0 + self.alpha), - ); - } - target = target.checked_sub(count)?; - } - if target < self.zero_count { - return Some(0.0); - } - target = target.checked_sub(self.zero_count)?; - let mut cumulative: u64 = 0; - let gamma = (1.0 + self.alpha) / (1.0 - self.alpha); - let mut last_nonempty: Option = None; - for (i, &c) in self.store_counts.iter().enumerate() { - if c > 0 { - last_nonempty = Some(i); - } - cumulative = cumulative.saturating_add(c); - if cumulative > target { - let k = self.store_offset as i64 + i as i64; - return Some(gamma.powf(k as f64) * (1.0 + self.alpha)); - } - } - // Numerical edge case: if we fall off the end (e.g. q == 1.0 and - // rounding lands past the final increment), estimate from the - // highest non-empty bucket. The wire format carries no `max` - // scalar; the representative is within DDSketch's α - // relative-accuracy bound of the true max. - last_nonempty.map(|i| { - let k = (self.store_offset as i64 + i as i64) as f64; - gamma.powf(k) * (1.0 + self.alpha) - }) - } - - /// Return the alpha value as it appears on the wire — round-tripped - /// through gamma exactly the way the Go reference implementation's - /// `DDSketch.SerializePortable` does it (`alpha = (gamma - 1) / - /// (gamma + 1)`, where `gamma = (1+α)/(1-α)`). The roundtrip - /// introduces a small floating-point drift; without applying it the - /// Rust producer's `DDSketchState.alpha` field bytes would diverge - /// from the Go producer's, and cross-language byte parity would fail - /// on the very first proto field. - #[inline] - pub fn wire_alpha(&self) -> f64 { - let gamma = (1.0 + self.alpha) / (1.0 - self.alpha); - (gamma - 1.0) / (gamma + 1.0) - } -} - -fn sparse_difference(current: &DdSketch, previous: &DdSketch, threshold: u64) -> Vec<(i32, u64)> { - current - .store_counts - .iter() - .enumerate() - .filter_map(|(i, count)| { - let key = i64::from(current.store_offset) + i as i64; - let old = key - i64::from(previous.store_offset); - let old_count = usize::try_from(old) - .ok() - .and_then(|i| previous.store_counts.get(i)) - .copied() - .unwrap_or(0); - let delta = count.saturating_sub(old_count); - (delta > 0 && delta >= threshold).then_some((key as i32, delta)) - }) - .collect() -} - -#[derive(Deserialize)] -struct DeltaWire( - Vec, - Vec, - #[serde(default)] Vec, - #[serde(default)] Vec, - #[serde(default)] u64, -); - -impl MessagePackCodec for DdSketch { - fn to_msgpack(&self) -> Result, MsgPackError> { - Ok(rmp_serde::to_vec(self)?) - } - - fn from_msgpack(bytes: &[u8]) -> Result { - Ok(rmp_serde::from_slice(bytes)?) - } -} - -impl MessagePackCodec for DdSketchDelta { - /// Encodes the sparse bucket delta as the 2-element MessagePack array - /// `[ idx:[]i32, d_count:[]u64 ]` — parallel arrays of absolute bucket - /// index → Δcount — byte-identical to Rust - /// `rmp_serde::to_vec(&(Vec, Vec))` (compact mode), the layout - /// the `sketchlib-go` `asapmsgpack` DDSketch delta encoder mirrors. Only - /// the bucket cells cross the wire; the DataPoint-level metric scalars - /// (`d_count`/`d_sum`/`new_min`/`new_max`) are reconstructable and are - /// not carried, matching the proto delta. - fn to_msgpack(&self) -> Result, MsgPackError> { - let idx: Vec = self.buckets.iter().map(|(i, _)| *i).collect(); - let d_count: Vec = self.buckets.iter().map(|(_, c)| *c).collect(); - if self.negative_buckets.is_empty() && self.zero_count == 0 { - Ok(rmp_serde::to_vec(&(idx, d_count))?) - } else { - let ni: Vec<_> = self.negative_buckets.iter().map(|b| b.0).collect(); - let nc: Vec<_> = self.negative_buckets.iter().map(|b| b.1).collect(); - Ok(rmp_serde::to_vec(&(idx, d_count, ni, nc, self.zero_count))?) - } - } - - fn from_msgpack(bytes: &[u8]) -> Result { - let DeltaWire(idx, d_count, ni, nc, zero_count) = rmp_serde::from_slice(bytes)?; - if idx.len() != d_count.len() || ni.len() != nc.len() { - return Err(rmp_serde::decode::Error::Syntax( - "DDSketch delta array lengths differ".into(), - ) - .into()); - } - Ok(DdSketchDelta { - buckets: idx.into_iter().zip(d_count).collect(), - negative_buckets: ni.into_iter().zip(nc).collect(), - zero_count, - ..DdSketchDelta::default() - }) - } -} - -#[cfg(test)] -mod tests { - use super::*; - use crate::message_pack_format::MessagePackCodec; - - #[test] - fn msgpack_delta_against_empty_round_trips() { - let mut w = DdSketch::new(0.01); - for i in 1..=200 { - w.update(i as f64); - } - let empty = DdSketch::new(0.01); - let bytes = w.compute_delta_msgpack(&empty, 1); - let mut recon = DdSketch::new(0.01); - recon.apply_delta_msgpack_bytes(&bytes).unwrap(); - assert_eq!(recon.total_count(), w.total_count()); - for q in [0.5, 0.9, 0.99] { - let got = recon.quantile(q).unwrap(); - let want = w.quantile(q).unwrap(); - assert!( - (got / want - 1.0).abs() <= 0.01, - "q={q}: got={got} want={want}" - ); - } - } - - #[test] - fn msgpack_delta_wire_layout_is_parallel_arrays() { - // Lock the cross-language wire shape: a 2-element array of parallel - // (idx, d_count) arrays, byte-identical to the tuple encoding the Go - // asapmsgpack DDSketch delta mirrors. - let mut w = DdSketch::new(0.01); - w.update(1.0); - w.update(1.0); - w.update(1000.0); - let empty = DdSketch::new(0.01); - let bytes = w.compute_delta_msgpack(&empty, 1); - let mut idx: Vec = Vec::new(); - let mut d_count: Vec = Vec::new(); - for (i, &c) in w.store_counts.iter().enumerate() { - if c != 0 { - idx.push(w.store_offset + i as i32); - d_count.push(c); - } - } - let expected = rmp_serde::to_vec(&(idx, d_count)).unwrap(); - assert_eq!(bytes, expected); - let d = DdSketchDelta::from_msgpack(&bytes).unwrap(); - assert_eq!(d.buckets.len(), 2, "two distinct buckets expected"); - } - - #[test] - fn test_new_empty() { - let d = DdSketch::new(0.01); - assert_eq!(d.total_count(), 0); - assert!(d.store_counts.is_empty()); - } - - #[test] - fn test_merge_aligned_same_offset() { - let mut a = DdSketch::from_raw(0.01, vec![1, 2, 3], -1); - let b = DdSketch::from_raw(0.01, vec![10, 20, 30], -1); - a.merge(&b).unwrap(); - assert_eq!(a.store_counts, vec![11, 22, 33]); - assert_eq!(a.store_offset, -1); - assert_eq!(a.total_count(), 66); - } - - #[test] - fn test_merge_overlapping_offsets() { - // a covers indices [-1, 0, 1]; b covers indices [0, 1, 2] - let mut a = DdSketch::from_raw(0.01, vec![1, 1, 1], -1); - let b = DdSketch::from_raw(0.01, vec![10, 10, 10], 0); - a.merge(&b).unwrap(); - // Merged window is [-1, 0, 1, 2] → [1, 11, 11, 10] - assert_eq!(a.store_counts, vec![1, 11, 11, 10]); - assert_eq!(a.store_offset, -1); - assert_eq!(a.total_count(), 33); - } - - #[test] - fn test_merge_disjoint_offsets() { - let mut a = DdSketch::from_raw(0.01, vec![1, 2], 0); - let b = DdSketch::from_raw(0.01, vec![3, 4], 5); - a.merge(&b).unwrap(); - // Window [0..7) → [1,2,0,0,0,3,4] - assert_eq!(a.store_counts, vec![1, 2, 0, 0, 0, 3, 4]); - assert_eq!(a.store_offset, 0); - } - - #[test] - fn test_apply_delta_additive_inside_store() { - let mut base = DdSketch::from_raw(0.01, vec![1, 2, 3], -1); - let delta = DdSketchDelta { - buckets: vec![(-1, 4), (0, 8), (1, 12)], - d_count: 24, - d_sum: 120.0, - min_changed: false, - new_min: 0.0, - max_changed: true, - new_max: 9.0, - ..Default::default() - }; - base.apply_delta(&delta).unwrap(); - assert_eq!(base.store_counts, vec![5, 10, 15]); - assert_eq!(base.total_count(), 30); - } - - #[test] - fn test_apply_delta_expands_store_on_new_bucket() { - // Base covers [0..2]; delta adds a bucket at absolute index 4. - let mut base = DdSketch::from_raw(0.01, vec![1, 2], 0); - let delta = DdSketchDelta { - buckets: vec![(4, 7)], - d_count: 7, - d_sum: 35.0, - min_changed: false, - new_min: 0.0, - max_changed: true, - new_max: 6.0, - ..Default::default() - }; - base.apply_delta(&delta).unwrap(); - assert_eq!(base.store_counts, vec![1, 2, 0, 0, 7]); - assert_eq!(base.store_offset, 0); - assert_eq!(base.total_count(), 10); - } - - #[test] - fn test_apply_delta_matches_full_merge() { - // Snapshot the sketch, add more samples via a merge, and confirm - // the delta+apply path lands at the same state. - let base = DdSketch::from_raw(0.01, vec![1, 2, 3], 0); - let addition = DdSketch::from_raw(0.01, vec![10, 0, 20], 0); - let mut via_merge = base.clone(); - via_merge.merge(&addition).unwrap(); - - let delta = DdSketchDelta { - buckets: vec![(0, 10), (2, 20)], - d_count: 30, - d_sum: 70.0, - min_changed: true, - new_min: 0.5, - max_changed: true, - new_max: 5.0, - ..Default::default() - }; - let mut via_delta = base; - via_delta.apply_delta(&delta).unwrap(); - - assert_eq!(via_delta.store_counts, via_merge.store_counts); - assert_eq!(via_delta.total_count(), via_merge.total_count()); - } - - #[test] - fn test_merge_alpha_mismatch() { - let mut a = DdSketch::new(0.01); - let b = DdSketch::new(0.02); - assert!(a.merge(&b).is_err()); - } - - #[test] - fn test_msgpack_round_trip() { - let original = DdSketch::from_raw(0.01, vec![1, 2, 3], -2); - let bytes = original.to_msgpack().unwrap(); - let decoded = DdSketch::from_msgpack(&bytes).unwrap(); - assert_eq!(decoded.alpha, original.alpha); - assert_eq!(decoded.store_counts, original.store_counts); - assert_eq!(decoded.store_offset, original.store_offset); - assert_eq!(decoded.total_count(), original.total_count()); - } - - /// The msgpack wire layout MUST be a 3-element array - /// `[alpha, store_counts, store_offset]` after dropping the - /// DataPoint-level METRIC scalars (count/sum/min/max). This pins the - /// element count so the bytes stay parity-aligned with the Go - /// reference implementation. - #[test] - fn test_msgpack_is_three_element_array() { - let sk = DdSketch::from_raw(0.01, vec![1, 2, 3], -2); - let bytes = sk.to_msgpack().unwrap(); - // rmp compact encoding leads with an array marker. A fixarray of - // length 3 is the single byte 0x93 (0b1001_0011). - assert_eq!( - bytes[0], 0x93, - "expected a 3-element msgpack fixarray (0x93), got {:#04x}", - bytes[0] - ); - } - - #[test] - fn test_insert_and_quantile_lognormal() { - // Ground-truth: insert a large i.i.d. log-normal sample into - // a full sketch and sanity-check P50 / P90 / P99. - let mut gt = DdSketch::new(0.01); - let mut rng = 0xdead_beefu64; - let mut next = || { - // xorshift64* - rng ^= rng << 13; - rng ^= rng >> 7; - rng ^= rng << 17; - rng - }; - // Box-Muller normal → log-normal(mu=3, sigma=0.7). - let lognormal = |u: u64, v: u64| -> f64 { - let r1 = (u as f64) / (u64::MAX as f64).max(1.0); - let r2 = (v as f64) / (u64::MAX as f64).max(1.0); - let z = (-2.0 * r1.max(1e-12).ln()).sqrt() * (2.0 * std::f64::consts::PI * r2).cos(); - (3.0 + 0.7 * z).exp() - }; - for _ in 0..100_000 { - gt.update(lognormal(next(), next())); - } - let p50 = gt.quantile(0.5).unwrap(); - let p99 = gt.quantile(0.99).unwrap(); - // Analytical P50 = exp(mu) = e^3 ≈ 20.09; - // P99 ≈ exp(mu + sigma * Φ⁻¹(0.99)) = e^(3 + 0.7×2.326) ≈ 102.4. - assert!( - (p50 / 20.09).ln().abs() < 0.05, - "P50 {p50} not close to 20.09" - ); - assert!( - (p99 / 102.4).ln().abs() < 0.05, - "P99 {p99} not close to 102.4" - ); - } - - /// Building a sketch via `base + apply_delta()` produces quantile - /// estimates within DDSketch's α bound of the ground-truth - /// full-sketch path. - #[test] - fn test_delta_chain_preserves_quantile_accuracy() { - let alpha = 0.01; - let mut rng = 0xcafe_babeu64; - let mut next = || { - rng ^= rng << 13; - rng ^= rng >> 7; - rng ^= rng << 17; - rng - }; - let lognormal = |u: u64, v: u64| -> f64 { - let r1 = (u as f64) / (u64::MAX as f64).max(1.0); - let r2 = (v as f64) / (u64::MAX as f64).max(1.0); - let z = (-2.0 * r1.max(1e-12).ln()).sqrt() * (2.0 * std::f64::consts::PI * r2).cos(); - (3.0 + 0.7 * z).exp() - }; - - // Path A (ground truth): one sketch, 50k samples inserted - // directly. - let mut full = DdSketch::new(alpha); - // Path B (delta chain): a base sketch from the first 10k - // samples, then 4 incremental "flushes" of 10k samples each, - // each transmitted as a delta computed against the previous - // snapshot. - let mut reconstituted = DdSketch::new(alpha); - let mut prev_snapshot = DdSketch::new(alpha); // what the "receiver" has cached - - let batch = 10_000; - let batches = 5; - for b in 0..batches { - let mut this_batch = prev_snapshot.clone(); - for _ in 0..batch { - let v = lognormal(next(), next()); - full.update(v); - this_batch.update(v); - } - // Compute a "delta" = diff of this_batch vs prev_snapshot - // in our in-memory struct shape. Matches what the Go - // reference implementation's `ComputeDelta` would put on - // the wire. - let delta = compute_dd_delta(&prev_snapshot, &this_batch); - if b == 0 { - // First batch seeds the reconstituted sketch. - reconstituted = this_batch.clone(); - } else { - reconstituted.apply_delta(&delta).unwrap(); - } - prev_snapshot = this_batch; - } - - // Both paths should see the same total count (exact), recovered - // by summing bucket counts now that the `count` scalar is gone. - assert_eq!( - reconstituted.total_count(), - full.total_count(), - "count diverged" - ); - - // And P50 / P90 / P99 should agree within α bound. - for q in [0.5, 0.9, 0.99] { - let got = reconstituted.quantile(q).unwrap(); - let want = full.quantile(q).unwrap(); - let rel_err = (got / want - 1.0).abs(); - assert!( - rel_err <= alpha, - "q={q} rel_err={rel_err:.4} exceeds α={alpha}: reconstituted={got}, full={want}", - ); - } - } - - /// Helper used only in the delta-chain test: computes a - /// `DdSketchDelta` from two snapshots. Mirrors the Go reference - /// implementation's `ComputeDelta` logic so the test exercises the - /// wire-format path end-to-end. - fn compute_dd_delta(snapshot: &DdSketch, current: &DdSketch) -> DdSketchDelta { - let mut cells = Vec::new(); - if !current.store_counts.is_empty() { - for (i, &c) in current.store_counts.iter().enumerate() { - if c == 0 { - continue; - } - let k = current.store_offset + i as i32; - let snap_count: u64 = if !snapshot.store_counts.is_empty() { - let idx = k as i64 - snapshot.store_offset as i64; - if idx >= 0 && (idx as usize) < snapshot.store_counts.len() { - snapshot.store_counts[idx as usize] - } else { - 0 - } - } else { - 0 - }; - let dc = c.saturating_sub(snap_count); - if dc > 0 { - cells.push((k, dc)); - } - } - } - let d_count = current.total_count() as i64 - snapshot.total_count() as i64; - DdSketchDelta { - buckets: cells, - d_count, - d_sum: 0.0, - min_changed: false, - new_min: 0.0, - max_changed: false, - new_max: 0.0, - ..Default::default() - } - } - - /// Delta-against-empty: a `compute_delta` against an EMPTY base - /// reconstructs the window's full state when applied (round-trip). - /// With `threshold = 1` every non-empty bucket survives, so applying - /// the delta to a fresh empty sketch yields the same bucket store as - /// the original window. - #[test] - fn test_compute_delta_against_empty_round_trips() { - let mut window = DdSketch::new(0.01); - for i in 1..=200 { - window.update(i as f64); - } - let empty = DdSketch::new(0.01); - - // Delta against empty = the window's full state in delta form. - let delta_bytes = window.compute_delta(&empty, 1); - - // Apply to a fresh empty base. - let mut reconstructed = DdSketch::new(0.01); - reconstructed.apply_delta_bytes(&delta_bytes).unwrap(); - - // Total count recovered exactly. - assert_eq!(reconstructed.total_count(), window.total_count()); - // Every non-empty bucket count matches (compare on absolute index). - for (i, &c) in window.store_counts.iter().enumerate() { - if c == 0 { - continue; - } - let k = window.store_offset + i as i32; - let idx = (k - reconstructed.store_offset) as usize; - assert_eq!(reconstructed.store_counts[idx], c, "bucket k={k}"); - } - // Quantiles agree within the α relative-accuracy bound. - for q in [0.5, 0.9, 0.99] { - let got = reconstructed.quantile(q).unwrap(); - let want = window.quantile(q).unwrap(); - assert!( - (got / want - 1.0).abs() <= 0.01, - "q={q}: reconstructed={got} window={want}" - ); - } - } - - /// Two consecutive windows each emit their OWN state — the second - /// window's delta-against-empty is NOT diffed against the first - /// window (no cross-window subtraction). - #[test] - fn test_consecutive_windows_emit_own_state() { - let empty = DdSketch::new(0.01); - - // Overlapping value ranges so a cross-window subtraction (win2 − - // win1) would genuinely differ from win2's own state — making the - // "no subtraction" assertion meaningful. - let mut win1 = DdSketch::new(0.01); - for i in 1..=50 { - win1.update(i as f64); - win1.update(i as f64); // win1 buckets carry count 2 - } - let mut win2 = DdSketch::new(0.01); - for i in 1..=50 { - win2.update(i as f64); // same buckets, count 1 - } - - // Each window diffs against EMPTY (delta-against-empty), not against - // the prior window. - let d2_against_empty = win2.compute_delta(&empty, 1); - let d2_against_win1 = win2.compute_delta(&win1, 1); - - // The delta-against-empty reconstructs win2 exactly. - let mut recon = DdSketch::new(0.01); - recon.apply_delta_bytes(&d2_against_empty).unwrap(); - assert_eq!(recon.total_count(), win2.total_count()); - - // Sanity: diffing against win1 would have produced a DIFFERENT - // (cross-window-subtracted) frame, proving the two are not the same - // and that diffing against empty avoids the cross-window subtraction. - assert_ne!( - d2_against_empty, d2_against_win1, - "delta-against-empty must differ from cross-window delta" - ); - } - - /// Cross-language byte-parity guard against the Go reference - /// implementation's `DDSketch.SerializePortable` output for the - /// deterministic input `(1..=50)` with `α = 0.01`. The hex blob - /// below was captured from a `proto.Marshal` of the Go envelope - /// (with `Producer` and `HashSpec` cleared). Any change to - /// [`DdSketch::update`]'s bucket-store growth that breaks parity - /// will surface here. - #[test] - fn test_update_then_envelope_matches_go_golden_bytes() { - use crate::proto::sketchlib::{ - DdSketchState, SketchEnvelope, sketch_envelope::SketchState, - }; - use prost::Message; - - let mut sk = DdSketch::new(0.01); - for i in 1..=50 { - sk.update(i as f64); - } - - let state = DdSketchState { - alpha: sk.wire_alpha(), - store_counts: sk.store_counts.clone(), - store_offset: sk.store_offset, - ..Default::default() - }; - let envelope = SketchEnvelope { - format_version: 1, - producer: None, - hash_spec: None, - sample_p: 0.0, - sketch_state: Some(SketchState::Ddsketch(state)), - }; - let mut got = Vec::with_capacity(envelope.encoded_len()); - envelope.encode(&mut got).expect("prost encode"); - - // Byte string for the same `(1..=50)`, α=0.01 input AFTER dropping - // the DataPoint-level METRIC scalars (count/sum/min/max → proto - // tags 4-7 reserved). 403 bytes total: a `SketchEnvelope` proto - // wrapping a `DDSketchState` carrying only `alpha`, `store_counts` - // (Go chunk-128 padded layout, offset = -64, len = 384) and - // `store_offset` (sint32 zigzag 0x7f = -64). - // - // NOTE: this golden is currently the RUST-produced value. It MUST be - // reconciled against the Go reference implementation's regenerated - // golden before declaring cross-language byte parity. - const GOLDEN_HEX: &str = "0801728e03096214ae47e17a843f128003000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000000000001000000000000000000000000000000000000000100000000000000000000000000000100000000000000000000010000000000000000010000000000000001000000000001000000000001000000000001000000010000000001000000010000010000000100000100000100000100000100010000010001000100010001000100010001000100010100010100010100010101000101010100010101010101010100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000187f"; - let want = decode_hex(GOLDEN_HEX); - assert_eq!( - got, - want, - "DDSketch envelope bytes diverge from golden \ - ({} bytes got vs {} bytes want)", - got.len(), - want.len(), - ); - } - - fn decode_hex(s: &str) -> Vec { - let bytes: Vec = s - .as_bytes() - .chunks(2) - .map(|pair| { - let high = hex_nibble(pair[0]); - let low = hex_nibble(pair[1]); - (high << 4) | low - }) - .collect(); - bytes - } - - #[test] - fn interpolated_quantile_preserves_adjacent_ranks_and_legacy_convention() { - let mut sketch = DdSketch::new(0.01); - sketch.update(20.0); - sketch.update(40.0); - let lower = sketch.quantile(0.0).unwrap(); - let upper = sketch.quantile(1.0).unwrap(); - assert_eq!(sketch.quantile(0.9), Some(lower)); - for (q, expected) in [(0.0, 20.0), (0.5, 30.0), (0.9, 38.0), (1.0, 40.0)] { - let actual = sketch.quantile_interpolated(q).unwrap(); - assert!((actual - expected).abs() <= expected * 0.01); - assert_eq!(actual, lower * (1.0 - q) + upper * q); - } - for q in [-0.1, 1.1, f64::NAN, f64::INFINITY] { - assert!(sketch.quantile_interpolated(q).is_none()); - } - assert!(DdSketch::new(0.01).quantile_interpolated(0.5).is_none()); - let mut singleton = DdSketch::new(0.01); - singleton.update(20.0); - assert_eq!( - singleton.quantile_interpolated(0.9), - singleton.quantile(0.0) - ); - for values in [vec![20.0, 20.0], vec![10.0, 10.0, 20.0, 40.0]] { - let mut repeated = DdSketch::new(0.01); - for &value in &values { - repeated.update(value); - } - for q in [0.0, 0.5, 0.9, 1.0] { - let rank = q * (values.len() - 1) as f64; - let fraction = rank - rank.floor(); - let expected = values[rank.floor() as usize] * (1.0 - fraction) - + values[rank.ceil() as usize] * fraction; - assert!( - (repeated.quantile_interpolated(q).unwrap() - expected).abs() - <= 0.01 * expected - ); - } - } - for alpha in [0.0, -0.1, 1.0, f64::NAN] { - assert!( - DdSketch::from_raw(alpha, vec![1], 0) - .quantile_interpolated(0.5) - .is_none() - ); - } - assert!( - DdSketch::from_raw(0.01, vec![(1u64 << 53) + 1], 0) - .quantile_interpolated(0.5) - .is_none() - ); - // Negative and zero inputs retain their ranks during interpolation. - let mut unsupported = DdSketch::new(0.01); - unsupported.update(-20.0); - unsupported.update(0.0); - assert!((unsupported.quantile_interpolated(0.9).unwrap() + 2.0).abs() <= 0.02); - let overflow = DdSketch::from_raw(0.01, vec![u64::MAX, 1], 0); - assert!(overflow.quantile_interpolated(0.5).is_none()); - } - - fn hex_nibble(c: u8) -> u8 { - match c { - b'0'..=b'9' => c - b'0', - b'a'..=b'f' => c - b'a' + 10, - b'A'..=b'F' => c - b'A' + 10, - _ => panic!("non-hex byte {}", c as char), - } - } -} diff --git a/src/message_pack_format/portable/mod.rs b/src/message_pack_format/portable/mod.rs index 97e0d372..9f79df41 100644 --- a/src/message_pack_format/portable/mod.rs +++ b/src/message_pack_format/portable/mod.rs @@ -3,15 +3,14 @@ //! Each per-algorithm submodule holds one sketch's pre-envelope wire type. //! These carry a cross-language contract of their own, on goldens apart //! from ASAPv1's `asapv1_golden/`: HLL, Count-Min, Count Sketch and KLL pin -//! byte parity against Go's serializers, KLL in both directions; DDSketch's -//! golden is a Rust self-pin, not yet reconciled. New code uses ASAPv1: -//! prefer a sketch's own `serialize_to_bytes`; do not add callers here. +//! byte parity against Go's serializers, KLL in both directions. New code +//! uses ASAPv1: prefer a sketch's own `serialize_to_bytes`; do not add +//! callers here. pub mod countminsketch; pub mod countminsketch_topk; pub mod countsketch; pub mod countsketch_topk; -pub mod ddsketch; pub mod delta_set_aggregator; pub mod hll; pub mod hydra_kll; diff --git a/src/proto/generated/sketchlib.v1.rs b/src/proto/generated/sketchlib.v1.rs index 2c098d3f..f4ccaf4b 100644 --- a/src/proto/generated/sketchlib.v1.rs +++ b/src/proto/generated/sketchlib.v1.rs @@ -601,65 +601,6 @@ pub struct CoinState { #[prost(uint32, tag = "3")] pub remaining_bits: u32, } -/// DDSketchState is the portable state of a DDSketch quantile sketch. -/// -/// Only alpha is required to reconstruct the index mapping. Consumers must -/// recompute all derived mapping values on load: -/// gamma = (1.0 + alpha) / (1.0 - alpha) -/// log_gamma = ln(gamma) -/// inv_log_gamma = 1.0 / log_gamma -/// -/// Bucket indices: a value v maps to bucket floor(ln(v) * inv_log_gamma). -/// Stored bucket absolute_index = array_index + store_offset. -#[derive(Clone, PartialEq, ::prost::Message)] -pub struct DdSketchState { - /// Relative accuracy guarantee. Required. Must satisfy 0 < alpha < 1. - #[prost(double, tag = "1")] - pub alpha: f64, - /// Bucket count array in index order (ascending absolute bucket index). - /// Absolute index of store_counts\[i\] is i + store_offset. - #[prost(uint64, repeated, tag = "2")] - pub store_counts: ::prost::alloc::vec::Vec, - /// Absolute bucket index corresponding to store_counts\[0\]. - /// May be negative (values < 1.0 map to negative bucket indices). - #[prost(sint32, tag = "3")] - pub store_offset: i32, - /// Signed extension: negative magnitudes use the same mapping as positives. - #[prost(uint64, repeated, tag = "16")] - pub negative_store_counts: ::prost::alloc::vec::Vec, - #[prost(sint32, tag = "17")] - pub negative_store_offset: i32, - #[prost(uint64, tag = "18")] - pub zero_count: u64, -} -/// DDSketchDelta carries only the buckets that changed above threshold T. -/// -/// DataPoint-level metric scalars (d_count/d_sum/new_min/new_max/min_changed/ -/// max_changed) are intentionally NOT carried on the wire: the count delta is -/// recoverable by summing bucket deltas, and min/max/quantiles are derived from -/// the bucket distribution within the relative-accuracy guarantee. This message -/// is byte-identical to the Go reference implementation's DDSketchDelta so the -/// Go and Rust runtimes emit byte-identical delta frames for identical window -/// state (cross-language byte parity). -#[derive(Clone, PartialEq, ::prost::Message)] -pub struct DdSketchDelta { - #[prost(message, repeated, tag = "1")] - pub buckets: ::prost::alloc::vec::Vec, - #[prost(message, repeated, tag = "16")] - pub negative_buckets: ::prost::alloc::vec::Vec, - #[prost(uint64, tag = "17")] - pub zero_count: u64, -} -/// DDSketchBucketDelta is the delta for one log-scale bucket. -#[derive(Clone, Copy, PartialEq, ::prost::Message)] -pub struct DdSketchBucketDelta { - /// signed bucket index (negative for values < 1) - #[prost(sint32, tag = "1")] - pub index: i32, - /// Δcount ≥ threshold T - #[prost(uint64, tag = "2")] - pub d_count: u64, -} /// UnivMonState is the portable state of a Universal Monitoring (UnivMon) sketch. /// /// UnivMon is a hierarchy of L Count Sketches paired with L heavy-hitter heaps. @@ -892,7 +833,7 @@ pub struct SketchEnvelope { /// The sketch payload. Exactly one field must be set. #[prost( oneof = "sketch_envelope::SketchState", - tags = "10, 11, 12, 13, 14, 15, 16, 17, 18" + tags = "10, 11, 12, 13, 15, 16, 17, 18" )] pub sketch_state: ::core::option::Option, } @@ -909,8 +850,6 @@ pub mod sketch_envelope { Hll(super::HyperLogLogState), #[prost(message, tag = "13")] Kll(super::KllState), - #[prost(message, tag = "14")] - Ddsketch(super::DdSketchState), #[prost(message, tag = "15")] Univmon(super::UnivMonState), #[prost(message, tag = "16")] diff --git a/src/sketches/ddsketch.rs b/src/sketches/ddsketch.rs index bc6dc56c..993c0959 100644 --- a/src/sketches/ddsketch.rs +++ b/src/sketches/ddsketch.rs @@ -212,9 +212,6 @@ impl TryFrom for DDSketch { /// `minIndexableValue`/`maxIndexableValue`. Values outside this range are /// handled without growing the dense store: smaller magnitudes enter the /// zero bucket, while larger magnitudes are rejected. -/// -/// Single source of truth shared by core `DDSketch`, the portable wire twin, -/// and tests, so the two implementations cannot drift algebraically again. pub fn ddsketch_indexable_bounds(alpha: f64) -> (f64, f64) { let gamma = (1.0 + alpha) / (1.0 - alpha); let inv_log_gamma = 1.0 / gamma.ln(); @@ -520,8 +517,7 @@ impl DDSketch { /// This is a REAL runtime check, not a `debug_assert!`: a `debug_assert!` /// compiles out in release builds, leaving a mismatched merge to corrupt /// results with no signal at all. DataDog's `MergeWith` and sketchlib-go's - /// Go `Merge` both return an error here; the portable `DdSketch::merge` in - /// this same crate does too. + /// Go `Merge` both return an error here. pub fn merge(&mut self, other: &DDSketch) -> Result<(), String> { if (self.alpha - other.alpha).abs() >= 1e-12 || (self.gamma - other.gamma).abs() >= 1e-12 { return Err(format!( diff --git a/src/sketches/ddsketch/wire.rs b/src/sketches/ddsketch/wire.rs index ce83d9d3..9aa4c1f9 100644 --- a/src/sketches/ddsketch/wire.rs +++ b/src/sketches/ddsketch/wire.rs @@ -23,9 +23,8 @@ //! //! ## Signed stores //! -//! Positive-only payloads retain metadata version 1. Signed or zero-containing +//! Positive-only payloads use metadata version 1. Signed or zero-containing //! payloads use version 2 and append a negative-magnitude store and zero count. -//! Older native readers reject version 2 instead of silently dropping samples. use rmp_serde::{decode::Error as RmpDecodeError, encode::Error as RmpEncodeError, from_slice}; use serde::{Deserialize, Serialize}; diff --git a/tests/PBT_COVERAGE.md b/tests/PBT_COVERAGE.md index cddee44f..2758a99c 100644 --- a/tests/PBT_COVERAGE.md +++ b/tests/PBT_COVERAGE.md @@ -2,8 +2,8 @@ One `proptest` law per line. Each compares the sketch against an answer computed without it; what that answer is comes first in every section. One -file per sketch under `tests/pbt/`. `cargo test --test pbt` runs 367 laws; -with `--all-features`, 435, adding the four **experimental** modules. Every +file per sketch under `tests/pbt/`. `cargo test --test pbt` runs 365 laws; +with `--all-features`, 433, adding the four **experimental** modules. Every law was mutation-checked: break the implementation, the law goes red. Shared generators, `grid` and `round_trip!` live in `support.rs`. @@ -105,25 +105,24 @@ Compared against: a key's `(col, sign)` per row, read off a one-key sketch; the - msgpack keeps cells and estimates - ASAPv1 keeps 64 estimates; `edge_rows` x `edge_dimension` -## ddsketch.rs — 14 +## ddsketch.rs — 12 -Compared against: the sorted stream. `DDSketch::get_value_at_quantile` targets the `ceil(q*n)`-th value, portable `DdSketch::quantile` the `floor(q*(n-1))`-th. Streams carry negatives, zeros and positives inside the indexable band. +Compared against: the sorted stream. `DDSketch::get_value_at_quantile` targets the `ceil(q*n)`-th value. Streams carry negatives, zeros and positives inside the indexable band. - placement - negative store, zero counter, positive store hold exactly the three class counts; `get_count() == arrivals` - - portable `DdSketch`: the same three counts - `sum() == stream sum`, bit for bit - - first dense allocation: one 128-bucket chunk, seed value at its center; both impls, both signed stores + - first dense allocation: one 128-bucket chunk, seed value at its center; both signed stores - relative error `<= alpha` - - `|quantile(q) - exact| <= alpha * |exact|`, across the sign boundary, both impls + - `|quantile(q) - exact| <= alpha * |exact|`, across the sign boundary - `quantile(q)` non-decreasing in `q`, across the sign boundary - single value `v`: `quantile(q) == v` for every `q`; `min`, `max`, `q=0`, `q=1` equal the tracked extremes - merge - counts add; both stores, both offsets, zero counter identical either way round - merged `quantile(q)` within `alpha` of the combined stream's exact value - - different `alpha`: refused, receiver count unchanged, both impls; same `alpha`: merges + - different `alpha`: refused, receiver count unchanged; same `alpha`: merges - wire - - msgpack keeps stores, offsets, zero count, quantiles + - ASAPv1 keeps stores, offsets, zero count, `sum`, quantiles - ASAPv1 keeps count, `alpha`, `sum`, quantiles; stream of NaN, ±inf, ±0, f64 extremes ## elastic.rs — 12 diff --git a/tests/README.md b/tests/README.md index f18d2bfb..1c1baeb8 100644 --- a/tests/README.md +++ b/tests/README.md @@ -15,7 +15,7 @@ its theorem. | Path | Purpose | | --- | --- | | `common/mod.rs` | Seeded stream generators (`zipf_u64`, `uniform_u64`, `normal_f64`, `exponential_f64`, adversarial `log_uniform_f64`, `duplicate_heavy_f64`, `monotonic_f64`, `outside_in_ordering`), exact truth trackers (`FreqTruth`, `NumericTruth`), assertion helpers | -| `common/specs.rs` | One error model per metric — `CountMinSpec`, `CountSketchSpec`, `SecondMomentSpec`, `KllRankSpec`, `RelativeQuantileSpec` (+ `DdRankConvention`), `CardinalityConfidenceSpec`, `SamplingConfidenceSpec`, `PrioritySampleSpec` — plus the three acceptance rules and the trial-unit rules that say which one applies | +| `common/specs.rs` | One error model per metric — `CountMinSpec`, `CountSketchSpec`, `SecondMomentSpec`, `KllRankSpec`, `RelativeQuantileSpec` (+ `dd_rank_index`), `CardinalityConfidenceSpec`, `SamplingConfidenceSpec`, `PrioritySampleSpec` — plus the three acceptance rules and the trial-unit rules that say which one applies | | `common/conformance.rs` | Capability traits + standard conformance batteries (the floor) | | `conformance_kit.rs` | Reference adapters: established sketches running through the kit (copy these) | | `e2e/` | Every end-to-end suite, one target (`cargo test --test e2e`), with `common/` compiled once for all of them | diff --git a/tests/asapv1_golden.rs b/tests/asapv1_golden.rs index 30b41651..43e3fc2d 100644 --- a/tests/asapv1_golden.rs +++ b/tests/asapv1_golden.rs @@ -9,9 +9,10 @@ //! See `asapv1_golden/README.md`. use asap_sketchlib::{ - Classic, Count, CountMin, ErtlMLE, FastPath, HllSketch, HllVariant, HyperLogLogHIPP12, - HyperLogLogP12, KLL, MessagePackCodec, RegularPath, Vector2D, + Classic, Count, CountMin, DDSketch, ErtlMLE, FastPath, HllSketch, HllVariant, + HyperLogLogHIPP12, HyperLogLogP12, KLL, MessagePackCodec, RegularPath, Vector1D, Vector2D, }; +use serde::Serialize; fn decode_hex(s: &str) -> Vec { let s = s.trim(); @@ -32,6 +33,9 @@ const GOLDEN_CS_FAST: &str = include_str!("../asapv1_golden/cs_i64_fast_2x4.hex" const GOLDEN_CS_I32: &str = include_str!("../asapv1_golden/cs_i32_regular_2x4.hex"); const GOLDEN_KLL_F64: &str = include_str!("../asapv1_golden/kll_f64_k200.hex"); const GOLDEN_KLL_I64: &str = include_str!("../asapv1_golden/kll_i64_k200.hex"); +const GOLDEN_DD_POSITIVE: &str = include_str!("../asapv1_golden/ddsketch_positive_a001.hex"); +const GOLDEN_DD_SIGNED: &str = include_str!("../asapv1_golden/ddsketch_signed_a001.hex"); +const GOLDEN_DD_EMPTY: &str = include_str!("../asapv1_golden/ddsketch_empty_a001.hex"); /// The known P12 register pattern shared by all three HLL fixtures. fn p12_registers() -> Vec { @@ -287,3 +291,162 @@ fn kll_i64_k200_matches_golden() { assert_eq!(decoded.quantile(0.0), 1.0); assert_eq!(decoded.quantile(1.0), 50.0); } + +// --------------------------------------------------------------------------- +// DDSketch: bucket stores, offsets, zero count and scalars set directly +// (DDSketch never hashes). The positive-only and empty states are metadata +// version 1; the signed state is version 2 and adds the negative store and +// zero count. +// --------------------------------------------------------------------------- + +#[derive(Serialize)] +struct DdStore { + counts: Vector1D, + offset: i32, +} + +/// The field names of `DDSketch`'s serde form. +#[derive(Serialize)] +struct DdState { + alpha: f64, + store: DdStore, + sum: f64, + min: f64, + max: f64, + negative_store: DdStore, + zero_count: u64, +} + +/// Positive-only: `uint` widths fixint / uint8 / uint16 / uint32 / uint64 in +/// the counts and an int8 offset. +const DD_POSITIVE_COUNTS: [u64; 7] = [1, 0, 127, 128, 300, 65536, 4294967296]; +const DD_POSITIVE_OFFSET: i32 = -40; +const DD_POSITIVE_SCALARS: (f64, f64, f64) = (2181071000.0, 0.453125, 0.5078125); + +/// Signed: uint16 positive offset, int16 negative offset, nonzero zero count. +const DD_SIGNED_COUNTS: [u64; 3] = [3, 0, 2]; +const DD_SIGNED_OFFSET: i32 = 310; +const DD_SIGNED_NEGATIVE_COUNTS: [u64; 2] = [5, 1]; +const DD_SIGNED_NEGATIVE_OFFSET: i32 = -208; +const DD_SIGNED_ZERO_COUNT: u64 = 7; +const DD_SIGNED_SCALARS: (f64, f64, f64) = (2523.90625, -0.016, 515.0); + +fn dd_sketch( + (counts, offset): (&[u64], i32), + (negative_counts, negative_offset): (&[u64], i32), + zero_count: u64, + (sum, min, max): (f64, f64, f64), +) -> DDSketch { + let state = DdState { + alpha: 0.01, + store: DdStore { + counts: Vector1D::from_vec(counts.to_vec()), + offset, + }, + sum, + min, + max, + negative_store: DdStore { + counts: Vector1D::from_vec(negative_counts.to_vec()), + offset: negative_offset, + }, + zero_count, + }; + rmp_serde::from_slice(&rmp_serde::to_vec_named(&state).expect("encode state")) + .expect("a valid DDSketch state") +} + +fn dd_positive() -> DDSketch { + dd_sketch( + (&DD_POSITIVE_COUNTS, DD_POSITIVE_OFFSET), + (&[], 0), + 0, + DD_POSITIVE_SCALARS, + ) +} + +fn dd_signed() -> DDSketch { + dd_sketch( + (&DD_SIGNED_COUNTS, DD_SIGNED_OFFSET), + (&DD_SIGNED_NEGATIVE_COUNTS, DD_SIGNED_NEGATIVE_OFFSET), + DD_SIGNED_ZERO_COUNT, + DD_SIGNED_SCALARS, + ) +} + +#[test] +fn ddsketch_positive_a001_matches_golden() { + let want = decode_hex(GOLDEN_DD_POSITIVE); + + let got = dd_positive().serialize_to_bytes().expect("serialize"); + assert_eq!(got, want, "DDSketch positive bytes diverge from golden"); + + let decoded = DDSketch::deserialize_from_bytes(&want).expect("decode"); + assert_eq!(decoded.alpha(), 0.01); + assert_eq!(decoded.store_counts(), DD_POSITIVE_COUNTS.as_slice()); + assert_eq!(decoded.store_offset(), DD_POSITIVE_OFFSET); + assert!(decoded.negative_store_counts().is_empty()); + assert_eq!(decoded.zero_count(), 0); + let (sum, min, max) = DD_POSITIVE_SCALARS; + assert_eq!(decoded.sum(), sum); + assert_eq!(decoded.min(), Some(min)); + assert_eq!(decoded.max(), Some(max)); + assert_eq!(decoded.get_count(), DD_POSITIVE_COUNTS.iter().sum::()); + assert_eq!(decoded.serialize_to_bytes().expect("re-serialize"), want); +} + +#[test] +fn ddsketch_signed_a001_matches_golden() { + let want = decode_hex(GOLDEN_DD_SIGNED); + + let got = dd_signed().serialize_to_bytes().expect("serialize"); + assert_eq!(got, want, "DDSketch signed bytes diverge from golden"); + + let decoded = DDSketch::deserialize_from_bytes(&want).expect("decode"); + assert_eq!(decoded.alpha(), 0.01); + assert_eq!(decoded.store_counts(), DD_SIGNED_COUNTS.as_slice()); + assert_eq!(decoded.store_offset(), DD_SIGNED_OFFSET); + assert_eq!( + decoded.negative_store_counts(), + DD_SIGNED_NEGATIVE_COUNTS.as_slice() + ); + assert_eq!(decoded.negative_store_offset(), DD_SIGNED_NEGATIVE_OFFSET); + assert_eq!(decoded.zero_count(), DD_SIGNED_ZERO_COUNT); + let (sum, min, max) = DD_SIGNED_SCALARS; + assert_eq!(decoded.sum(), sum); + assert_eq!(decoded.min(), Some(min)); + assert_eq!(decoded.max(), Some(max)); + assert_eq!(decoded.get_count(), 18); + assert_eq!(decoded.serialize_to_bytes().expect("re-serialize"), want); +} + +#[test] +fn ddsketch_empty_a001_matches_golden() { + let want = decode_hex(GOLDEN_DD_EMPTY); + + let state = dd_sketch( + (&[], 0), + (&[], 0), + 0, + (0.0, f64::INFINITY, f64::NEG_INFINITY), + ); + let got = state.serialize_to_bytes().expect("serialize"); + assert_eq!(got, want, "DDSketch empty bytes diverge from golden"); + assert_eq!( + DDSketch::new(0.01).serialize_to_bytes().expect("serialize"), + want, + "a fresh sketch is the empty state" + ); + + let decoded = DDSketch::deserialize_from_bytes(&want).expect("decode"); + assert_eq!(decoded.alpha(), 0.01); + assert!(decoded.store_counts().is_empty()); + assert_eq!(decoded.store_offset(), 0); + assert!(decoded.negative_store_counts().is_empty()); + assert_eq!(decoded.zero_count(), 0); + assert_eq!(decoded.get_count(), 0); + assert_eq!(decoded.sum(), 0.0); + assert_eq!(decoded.min(), None); + assert_eq!(decoded.max(), None); + assert_eq!(decoded.serialize_to_bytes().expect("re-serialize"), want); +} diff --git a/tests/bug_verification.rs b/tests/bug_verification.rs index 1de70dd6..d5b69ca7 100644 --- a/tests/bug_verification.rs +++ b/tests/bug_verification.rs @@ -10,51 +10,7 @@ //! in `tests/e2e/nitro.rs`, which names both failure modes on the same run //! that already exercises every ingestion path. -use asap_sketchlib::message_pack_format::portable::ddsketch::DdSketch as PortableDds; -use asap_sketchlib::{CountL2HH, DDSketch, DataInput, DefaultXxHasher}; - -// --------------------------------------------------------------------------- -// Bug 3: portable DdSketch reports gamma^(k+0.5) (bucket log-midpoint) as the -// representative (portable/ddsketch.rs:426). For values sitting at the lower -// edge of bucket k the relative error is sqrt(gamma)-1 ~= alpha + alpha^2/2, -// violating the advertised relative-accuracy guarantee. Core DDSketch already -// uses gamma^k * (1 + alpha) (ddsketch.rs:383-392), so the two disagree on -// identical data. -// -// Synthetic data: value gamma^k * (1 + 1e-6) repeated 10_000x, alpha = 0.05. -// Truth for every quantile: the value itself. Current behavior: 5.13% error. -// --------------------------------------------------------------------------- -#[test] -fn portable_ddsketch_respects_alpha_at_bucket_edges() { - let alpha = 0.05; - let gamma = (1.0f64 + alpha) / (1.0 - alpha); - let v = gamma.powi(20) * (1.0 + 1e-6); - - let mut port = PortableDds::new(alpha); - let mut core = DDSketch::new(alpha); - for _ in 0..10_000 { - port.update(v); - core.add(&v); - } - - let q_port = port.quantile(0.5).unwrap(); - let rel_port = ((q_port - v) / v).abs(); - assert!( - rel_port <= alpha * (1.0 + 1e-6), - "portable DdSketch median off by {rel_port:.5} (> alpha={alpha}) at a \ - bucket-edge value" - ); - - // Both implementations must sit within alpha of the truth on identical - // data. (Exact equality is impossible: core clamps representatives to its - // observed min/max, which the portable wire format does not carry.) - let q_core = core.get_value_at_quantile(0.5).unwrap(); - let rel_core = ((q_core - v) / v).abs(); - assert!( - rel_core <= alpha * (1.0 + 1e-6), - "core DDSketch median off by {rel_core:.5} (> alpha={alpha})" - ); -} +use asap_sketchlib::{CountL2HH, DataInput, DefaultXxHasher}; // --------------------------------------------------------------------------- // Bug 4: CountL2HH's hot-path L2 accumulation ran in plain i64 diff --git a/tests/common/conformance.rs b/tests/common/conformance.rs index e52705b4..2b013dbd 100644 --- a/tests/common/conformance.rs +++ b/tests/common/conformance.rs @@ -429,10 +429,7 @@ where /// interchangeable: a rank band says nothing about DDSketch's promise, and a /// relative-value band says nothing about KLL's. /// -/// The order statistic compared against comes from the spec's own -/// `DdRankConvention`, so a core `DDSketch` (`ceil(q*n)`) and the portable -/// `DdSketch` (`floor(q*(n-1))`) are each scored on the question they actually -/// answer. +/// The order statistic compared against is `specs::dd_order_statistic`. pub fn relative_quantile_battery( sketch: &str, new_sketch: F, @@ -456,7 +453,7 @@ where sk.update(*v); } for &q in qs { - let truth = spec.convention.order_statistic(&sorted, q); + let truth = super::specs::dd_order_statistic(&sorted, q); let est = sk.quantile(q); let outcome = spec.check(q, est, truth); report.record( diff --git a/tests/common/specs.rs b/tests/common/specs.rs index 54f4aebf..1cc4b03d 100644 --- a/tests/common/specs.rs +++ b/tests/common/specs.rs @@ -996,57 +996,26 @@ pub fn breakpoint_rank_interval_distance(estimated: &[(f64, f64)], sorted_truth: // DDSketch: relative value error // --------------------------------------------------------------------------- -/// Which order statistic a DDSketch implementation answers a quantile query -/// with. The two shipped implementations do not agree, and the truth a test -/// compares against has to follow the implementation it is testing. -#[derive(Clone, Copy, Debug, PartialEq, Eq)] -pub enum DdRankConvention { - /// `DDSketch::get_value_at_quantile`: `rank = ceil(q * n)`, 1-based, so the - /// answer is `sorted[ceil(q*n) - 1]`. `q = 0` and `q = 1` short-circuit to - /// the exact stored minimum and maximum. - CeilNearestRank, - /// Portable `DdSketch::quantile`: `target = floor(q * (n - 1))`, 0-based, - /// so the answer is `sorted[floor(q*(n-1))]` — the lower-quantile - /// convention of the DDSketch paper and of DataDog's reference - /// implementation. No exact minimum or maximum is retained, so the - /// endpoints are bucket representatives like any other rank. - LowerFloor, +/// Zero-based index into an ascending slice of the order statistic +/// `DDSketch::get_value_at_quantile` answers `q` with: `rank = ceil(q * n)`, +/// 1-based, so `sorted[ceil(q*n) - 1]`. `q = 0` and `q = 1` are the exact +/// minimum and maximum. +pub fn dd_rank_index(n: usize, q: f64) -> usize { + assert!(n > 0, "a quantile rank is undefined on an empty stream"); + let q = q.clamp(0.0, 1.0); + (((q * n as f64).ceil() as usize).clamp(1, n)) - 1 } -impl DdRankConvention { - /// The zero-based index into an ascending `sorted` slice this convention - /// answers `q` with. - pub fn index(self, n: usize, q: f64) -> usize { - assert!( - n > 0, - "a quantile convention is undefined on an empty stream" - ); - let q = q.clamp(0.0, 1.0); - match self { - DdRankConvention::CeilNearestRank => (((q * n as f64).ceil() as usize).clamp(1, n)) - 1, - DdRankConvention::LowerFloor => ((q * (n - 1) as f64).floor() as usize).min(n - 1), - } - } - - /// The exact order statistic this convention answers `q` with. - pub fn order_statistic(self, sorted: &[f64], q: f64) -> f64 { - sorted[self.index(sorted.len(), q)] - } - - pub fn name(self) -> &'static str { - match self { - DdRankConvention::CeilNearestRank => "ceil(q*n) nearest-rank", - DdRankConvention::LowerFloor => "floor(q*(n-1)) lower-quantile", - } - } +/// The exact order statistic `DDSketch` answers `q` with. +pub fn dd_order_statistic(sorted: &[f64], q: f64) -> f64 { + sorted[dd_rank_index(sorted.len(), q)] } /// Relative-value-error contract for a DDSketch of accuracy parameter `alpha`. /// /// DDSketch's guarantee is on the *value*, not the rank: the returned estimate /// for the `q`-quantile must be within `alpha` relative error of the exact -/// order statistic **at the same `q` under that implementation's own rank -/// convention**: +/// `ceil(q*n)` order statistic at the same `q` (see [`dd_rank_index`]): /// /// ```text /// |est - true| / |true| <= alpha + numerical_slack(true) @@ -1069,24 +1038,12 @@ impl DdRankConvention { #[derive(Clone, Copy, Debug)] pub struct RelativeQuantileSpec { pub alpha: f64, - pub convention: DdRankConvention, } impl RelativeQuantileSpec { - /// A spec for the core `DDSketch`. + /// A spec for `DDSketch` at accuracy `alpha`. pub fn core(alpha: f64) -> Self { - Self { - alpha, - convention: DdRankConvention::CeilNearestRank, - } - } - - /// A spec for the portable `DdSketch`. - pub fn portable(alpha: f64) -> Self { - Self { - alpha, - convention: DdRankConvention::LowerFloor, - } + Self { alpha } } /// A few ULP of headroom, scaled by the magnitude of the logarithm because @@ -1116,24 +1073,21 @@ impl RelativeQuantileSpec { } else { Err(format!( "q={q}: est {estimate:.10e} vs exact {true_value:.10e} \ - -> relative error {rel:.3e} > alpha + slack = {tol:.3e} (alpha={}, {})", + -> relative error {rel:.3e} > alpha + slack = {tol:.3e} \ + (alpha={}, ceil(q*n) nearest-rank)", self.alpha, - self.convention.name(), )) } } - /// Tallies a sketch's answers over a q grid against the exact order - /// statistics **of this spec's own rank convention**. - /// - /// `sorted` must be ascending. Using one truth helper for both - /// implementations would compare each against the other's question. + /// Tallies a sketch's answers over a q grid against the exact `ceil(q*n)` + /// order statistics. `sorted` must be ascending. pub fn tally_into(&self, tally: &mut Tally, sorted: &[f64], qs: &[f64], quantile: F) where F: Fn(f64) -> Option, { for &q in qs { - let truth = self.convention.order_statistic(sorted, q); + let truth = dd_order_statistic(sorted, q); match quantile(q) { None => tally.record(false, || format!("q={q}: sketch returned None")), Some(est) => { diff --git a/tests/conformance_kit.rs b/tests/conformance_kit.rs index dd57a219..f82ff7d3 100644 --- a/tests/conformance_kit.rs +++ b/tests/conformance_kit.rs @@ -10,7 +10,6 @@ use common::conformance::{ }; use common::{FreqTruth, zipf_u64}; -use asap_sketchlib::message_pack_format::portable::ddsketch::DdSketch as PortableDds; use asap_sketchlib::message_pack_format::portable::hll::{HllSketch, HllVariant}; use asap_sketchlib::{ Bloom, CMSHeap, CSHeap, CountL2HH, CountMin, DDSketch, DataInput, FastPath, FoldCMS, FoldCS, @@ -175,17 +174,6 @@ impl QuantileOps for KllDynamicAdapter { } } -struct PortableDdsAdapter(PortableDds); - -impl QuantileOps for PortableDdsAdapter { - fn update(&mut self, value: f64) { - self.0.update(value); - } - fn quantile(&self, q: f64) -> f64 { - self.0.quantile(q).expect("non-empty sketch") - } -} - struct UnivMonQAdapter(UnivMonQ); impl QuantileOps for UnivMonQAdapter { @@ -364,26 +352,6 @@ fn kll_family_passes_quantile_conformance() { .assert_ok(); } -/// DDSketch through the **relative-value-error** battery, against exact -/// nearest-rank order statistics — the guarantee it actually makes. -#[test] -fn ddsketch_passes_relative_quantile_conformance() { - const ALPHA: f64 = 0.01; - let values: Vec = common::normal_f64(40_000, 500.0, 80.0, 7001) - .into_iter() - .filter(|v| *v > 0.0) - .collect(); - - conformance::relative_quantile_battery( - "PortableDds", - || PortableDdsAdapter(PortableDds::new(ALPHA)), - &values, - common::specs::RelativeQuantileSpec::portable(ALPHA), - &conformance::DEFAULT_QUANTILE_QS, - ) - .assert_ok(); -} - #[test] fn univmonq_passes_quantile_conformance() { let values: Vec = common::uniform_u64(30_000, 50_000, 7002) diff --git a/tests/ddsketch_signed.rs b/tests/ddsketch_signed.rs index 3cb591c6..0bfa1e2e 100644 --- a/tests/ddsketch_signed.rs +++ b/tests/ddsketch_signed.rs @@ -1,73 +1,31 @@ -use asap_sketchlib::message_pack_format::MessagePackCodec; -use asap_sketchlib::{DDSketch, DdSketch}; +use asap_sketchlib::DDSketch; -// Both public implementations must count negative and zero observations. +// Negative and zero observations are counted. #[test] fn signed_samples_and_zero_are_retained() { - let mut portable = DdSketch::new(0.01); let mut core = DDSketch::new(0.01); for v in [-20.0, -2.0, 0.0, 0.0, 2.0, 20.0] { - portable.update(v); core.add(&v); } - assert_eq!(portable.total_count(), 6); assert_eq!(core.get_count(), 6); - assert_eq!(portable.quantile(0.5), Some(0.0)); assert_eq!(core.get_value_at_quantile(0.5), Some(0.0)); - assert!(portable.quantile(0.0).unwrap() < -19.0); - assert!(portable.quantile(1.0).unwrap() > 19.0); + assert!(core.get_value_at_quantile(0.0).unwrap() < -19.0); + assert!(core.get_value_at_quantile(1.0).unwrap() > 19.0); } -// Signed snapshots must survive both codecs and merge with positive-only state. +// Signed snapshots survive ASAPv1 and merge. #[test] fn signed_merge_and_serialization() { - let mut p = DdSketch::new(0.01); let mut c = DDSketch::new(0.01); for v in [-4.0, 0.0, 4.0] { - p.update(v); c.add(&v); } - let mut p2 = DdSketch::from_msgpack(&p.to_msgpack().unwrap()).unwrap(); let mut c2 = DDSketch::deserialize_from_bytes(&c.serialize_to_bytes().unwrap()).unwrap(); - p2.merge(&p).unwrap(); c2.merge(&c).unwrap(); - assert_eq!(p2.total_count(), 6); assert_eq!(c2.get_count(), 6); - assert_eq!(p2.quantile_interpolated(0.5), Some(0.0)); assert_eq!(c2.get_value_at_quantile(0.5), Some(0.0)); } -#[test] -fn signed_delta_and_proto_round_trips() { - use prost::Message; - let mut base = DdSketch::new(0.01); - for v in [-100., -1., 0., 1., 100.] { - base.update(v); - } - let mut current = base.clone(); - for v in [-1000., -0.1, 0., 0., 0.1, 1000.] { - current.update(v); - } - let mut protobuf = base.clone(); - protobuf - .apply_delta_bytes(¤t.compute_delta(&base, 1)) - .unwrap(); - let mut msgpack = base; - msgpack - .apply_delta_msgpack_bytes(¤t.compute_delta_msgpack(&msgpack, 1)) - .unwrap(); - let bytes = current.to_proto().encode_to_vec(); - let snapshot = DdSketch::from_proto( - asap_sketchlib::proto::sketchlib::DdSketchState::decode(bytes.as_slice()).unwrap(), - ); - for restored in [protobuf, msgpack, snapshot] { - assert_eq!(restored.total_count(), 11); - for q in [0., 0.1, 0.5, 0.9, 1.] { - assert!((restored.quantile(q).unwrap() - current.quantile(q).unwrap()).abs() < 1e-9); - } - } -} - #[test] fn signed_order_statistics_obey_relative_bound() { let mut values: Vec = (-300i32..=300) @@ -80,44 +38,32 @@ fn signed_order_statistics_obey_relative_bound() { }) .collect(); values.sort_by(f64::total_cmp); - let mut portable = DdSketch::new(0.01); let mut core = DDSketch::new(0.01); for v in &values { - portable.try_update(*v).unwrap(); core.try_add(v).unwrap(); } for i in 0..=1000 { let q = i as f64 / 1000.; - let pi = (q * (values.len() - 1) as f64).floor() as usize; let ci = ((q * values.len() as f64).ceil() as usize).max(1) - 1; - for (got, want) in [ - (portable.quantile(q).unwrap(), values[pi]), - (core.get_value_at_quantile(q).unwrap(), values[ci]), - ] { - assert!( - (got - want).abs() <= 0.010000001 * want.abs(), - "q={q}, got={got}, want={want}" - ); - } + let (got, want) = (core.get_value_at_quantile(q).unwrap(), values[ci]); + assert!( + (got - want).abs() <= 0.010000001 * want.abs(), + "q={q}, got={got}, want={want}" + ); } } #[test] fn zero_only_special_values_and_octo_flush() { use asap_sketchlib::sketch_framework::octo::DdWorkerSketch; - let mut portable = DdSketch::new(0.01); let mut core = DDSketch::new(0.01); for v in [0., -0., 0.] { - portable.try_update(v).unwrap(); core.try_add(&v).unwrap(); } for v in [f64::NAN, f64::INFINITY, f64::NEG_INFINITY] { - assert!(portable.try_update(v).is_err()); assert!(core.try_add(&v).is_err()); } - assert_eq!(portable.total_count(), 3); assert_eq!(core.get_count(), 3); - assert_eq!(portable.quantile_interpolated(0.9), Some(0.)); let mut worker = DdWorkerSketch::new(0.01); let mut replay = DDSketch::new(0.01); for v in [-10., -10., -10., -0.1, 0., 0., 0.1, 10.] { @@ -134,20 +80,12 @@ fn zero_only_special_values_and_octo_flush() { #[test] fn tiny_values_are_counted_in_zero_bucket_and_signed_serde_survives() { let mut core = DDSketch::new(0.01); - let mut portable = DdSketch::new(0.01); let tiny = f64::MIN_POSITIVE / 2.; for v in [-tiny, tiny, 0.] { core.try_add(&v).unwrap(); - portable.try_update(v).unwrap(); } assert_eq!(core.get_count(), 3); assert_eq!(core.zero_count(), 3); - assert_eq!(portable.total_count(), 3); - assert_eq!(portable.quantile_interpolated(0.5), Some(0.)); - let named: DdSketch = - rmp_serde::from_slice(&rmp_serde::to_vec_named(&portable).unwrap()).unwrap(); - assert_eq!(named.total_count(), 3); - assert_eq!(named.quantile(0.5), Some(0.)); core.try_add(&-3.).unwrap(); let decoded: DDSketch = rmp_serde::from_slice(&rmp_serde::to_vec(&core).unwrap()).unwrap(); assert_eq!(decoded.get_count(), 4); @@ -155,15 +93,3 @@ fn tiny_values_are_counted_in_zero_bucket_and_signed_serde_survives() { assert_eq!(decoded.min(), core.min()); assert_eq!(decoded.max(), core.max()); } - -#[test] -fn cross_zero_interpolation_is_not_a_relative_error_certificate() { - let mut sketch = DdSketch::new(0.01); - sketch.update(-1.); - sketch.update(1.001); - let exact = (-1. + 1.001) / 2.; - let estimated = sketch.quantile_interpolated(0.5).unwrap(); - // Both magnitudes land in the same bucket, so their representatives cancel. - assert_eq!(estimated, 0.); - assert!((estimated - exact).abs() > 0.01 * exact.abs()); -} diff --git a/tests/e2e/quantiles.rs b/tests/e2e/quantiles.rs index 637a27c4..9ae1be2e 100644 --- a/tests/e2e/quantiles.rs +++ b/tests/e2e/quantiles.rs @@ -31,19 +31,17 @@ //! //! DDSketch's guarantee is deterministic — bucket width alone, no hash and no //! sampling — so its batteries tolerate zero violations and no statistical -//! model applies. The two shipped implementations answer a quantile query with -//! **different order statistics**, so each is compared against the truth for -//! its own convention; see `DdRankConvention`. +//! model applies. Its truth is the `ceil(q*n)` order statistic; see +//! `dd_rank_index`. use crate::common; -use common::specs::{DdRankConvention, KllRankSpec, RelativeQuantileSpec, Tally, rank_error}; +use common::specs::{KllRankSpec, RelativeQuantileSpec, Tally, dd_rank_index, rank_error}; use common::{ NumericTruth, assert_between, duplicate_heavy_f64, exponential_f64, log_uniform_f64, monotonic_f64, normal_f64, outside_in_ordering, uniform_u64, zipf_f64, }; -use asap_sketchlib::message_pack_format::portable::ddsketch::DdSketch as PortableDds; use asap_sketchlib::message_pack_format::portable::hydra_kll::HydraKllSketch; use asap_sketchlib::{ DDSketch, DataInput, KLL, KLLConfig, KLLDynamic, TumblingWindow, UnivMonQ, UnivMonQConfig, @@ -587,37 +585,26 @@ fn dds_streams(alpha: f64, n: usize, seed: u64) -> Vec<(&'static str, Vec)> ] } -/// DDSketch's relative-value-error guarantee, for the core sketch and the -/// portable wire twin, at every supported alpha. -/// -/// Each implementation is compared against the exact order statistic **of its -/// own rank convention**: `DDSketch::get_value_at_quantile` answers -/// `sorted[ceil(q*n) - 1]`, while the portable `DdSketch::quantile` answers -/// `sorted[floor(q*(n-1))]`. They are different questions (see -/// `ddsketch_core_and_portable_answer_different_order_statistics`), so a single -/// truth helper would score one of them against the other one's question and -/// quietly absorb the difference into `alpha`. +/// DDSketch's relative-value-error guarantee at every supported alpha, +/// against the exact order statistic `DDSketch::get_value_at_quantile` +/// answers: `sorted[ceil(q*n) - 1]`. /// /// The tolerance is `alpha + numerical_slack`, where the slack is a few ULP of /// the logarithmic mapping — never a percentage of alpha, which would license /// breaking the advertised guarantee by that percentage. #[test] -fn ddsketch_core_and_portable_satisfy_the_relative_value_error_contract() { +fn ddsketch_satisfies_the_relative_value_error_contract() { const SAMPLE_SIZES: [usize; 3] = [1_000, 20_000, 100_000]; for &alpha in &DDS_ALPHAS { let core_spec = RelativeQuantileSpec::core(alpha); - let port_spec = RelativeQuantileSpec::portable(alpha); let mut core_tally = Tally::default(); - let mut port_tally = Tally::default(); for (i, &n) in SAMPLE_SIZES.iter().enumerate() { let seed = 3_005_000u64 + i as u64 * 101 + (alpha * 1e6) as u64; for (label, values) in dds_streams(alpha, n, seed) { let mut core = DDSketch::new(alpha); - let mut port = PortableDds::new(alpha); for v in &values { core.add(v); - port.update(*v); } let truth = NumericTruth::new(values.clone()); assert_eq!( @@ -625,17 +612,9 @@ fn ddsketch_core_and_portable_satisfy_the_relative_value_error_contract() { truth.len(), "{label} alpha={alpha} n={n} seed={seed}: core dropped samples" ); - assert_eq!( - port.total_count() as usize, - truth.len(), - "{label} alpha={alpha} n={n} seed={seed}: portable dropped samples" - ); core_spec.tally_into(&mut core_tally, truth.sorted(), &DDS_QS, |q| { core.get_value_at_quantile(q) }); - port_spec.tally_into(&mut port_tally, truth.sorted(), &DDS_QS, |q| { - port.quantile(q) - }); } } // The guarantee is deterministic, not probabilistic: DDSketch's error @@ -649,126 +628,63 @@ fn ddsketch_core_and_portable_satisfy_the_relative_value_error_contract() { .collect::>() ); core_tally.assert_none(&format!("core DDSketch alpha={alpha}"), &context); - port_tally.assert_none(&format!("portable DdSketch alpha={alpha}"), &context); } } -/// The two implementations answer a quantile query with **different order -/// statistics**, and this pins the divergence rather than letting `alpha` -/// absorb it. -/// -/// - `DDSketch::get_value_at_quantile` uses `rank = ceil(q * n)`, 1-based. -/// - Portable `DdSketch::quantile` uses `target = floor(q * (n - 1))`, 0-based -/// — the lower-quantile convention of the DDSketch paper and of DataDog's -/// reference implementation, which is also what the wire format's Go twin -/// answers. -/// -/// **The decision taken here is to keep both.** The portable type exists to be -/// byte- and answer-compatible with `sketchlib-go`, so its convention is fixed -/// by an external contract; the core type's `ceil` convention is what its own -/// callers have been reading for the life of the API, and it is what lets `q=0` -/// and `q=1` return the exactly retained minimum and maximum. Changing either -/// silently moves numbers under existing callers, and the divergence is only -/// observable at small `n` or ragged `q` — precisely the cases pinned below. -/// What is *not* acceptable is leaving it undocumented, or scoring both against -/// one truth helper, which is what the previous revision did. -/// -/// The probes are chosen so the two formulas disagree: at `n = 3, q = 0.4` the -/// core answers `sorted[1]` and the portable `sorted[0]`. +/// At small `n` and ragged `q` the `ceil(q*n)` order statistic differs from +/// the `floor(q*(n-1))` one, and with values three buckets apart the two are +/// more than `alpha` apart, so these probes tell the conventions apart. #[test] -fn ddsketch_core_and_portable_answer_different_order_statistics() { - // (n, q, expected core 0-based index, expected portable 0-based index) - const PROBES: [(usize, f64, usize, usize); 8] = [ - (3, 0.4, 1, 0), - (4, 0.34, 1, 1), - (4, 0.3, 1, 0), - (7, 0.2, 1, 1), - (7, 0.6, 4, 3), - (5, 0.5, 2, 2), - (10, 0.25, 2, 2), - (10, 0.15, 1, 1), +fn ddsketch_answers_the_ceil_nearest_rank_order_statistic_at_small_n() { + // (n, q, expected 0-based index sorted[ceil(q*n) - 1]) + const PROBES: [(usize, f64, usize); 8] = [ + (3, 0.4, 1), + (4, 0.34, 1), + (4, 0.3, 1), + (7, 0.2, 1), + (7, 0.6, 4), + (5, 0.5, 2), + (10, 0.25, 2), + (10, 0.15, 1), ]; - - // Values one bucket apart at the coarsest alpha, so a one-rank difference - // is a different bucket and therefore a different answer — not two ranks - // that happen to share a representative. const ALPHA: f64 = 0.01; let gamma = (1.0 + ALPHA) / (1.0 - ALPHA); + let spec = RelativeQuantileSpec::core(ALPHA); - let mut disagreements = 0usize; - for &(n, q, core_idx, port_idx) in &PROBES { - let values: Vec = (0..n).map(|i| 100.0 * gamma.powi(3 * i as i32)).collect(); - let sorted = values.clone(); - - assert_eq!( - DdRankConvention::CeilNearestRank.index(n, q), - core_idx, - "core convention ceil(q*n)-1 at n={n} q={q}" - ); - assert_eq!( - DdRankConvention::LowerFloor.index(n, q), - port_idx, - "portable convention floor(q*(n-1)) at n={n} q={q}" - ); - - let mut core = DDSketch::new(ALPHA); - let mut port = PortableDds::new(ALPHA); - for v in &values { - core.add(v); - port.update(*v); + let mut distinguishing = 0usize; + for &(n, q, idx) in &PROBES { + assert_eq!(dd_rank_index(n, q), idx, "ceil(q*n)-1 at n={n} q={q}"); + if (q * (n - 1) as f64).floor() as usize != idx { + distinguishing += 1; } - let core_spec = RelativeQuantileSpec::core(ALPHA); - let port_spec = RelativeQuantileSpec::portable(ALPHA); - let core_est = core.get_value_at_quantile(q).expect("non-empty"); - let port_est = port.quantile(q).expect("non-empty"); - - if let Err(detail) = core_spec.check(q, core_est, sorted[core_idx]) { - panic!("core DDSketch n={n} q={q}: {detail}"); - } - if let Err(detail) = port_spec.check(q, port_est, sorted[port_idx]) { - panic!("portable DdSketch n={n} q={q}: {detail}"); + let sorted: Vec = (0..n).map(|i| 100.0 * gamma.powi(3 * i as i32)).collect(); + let mut sketch = DDSketch::new(ALPHA); + for v in &sorted { + sketch.add(v); } - - if core_idx != port_idx { - disagreements += 1; - // Each implementation must be answering *its own* order statistic, - // so the two answers must differ here: if they agreed, one of them - // would have silently changed convention. - assert!( - core_est != port_est, - "n={n} q={q}: the conventions pick different order statistics \ - ({core_idx} vs {port_idx}) but both returned {core_est}" - ); + let est = sketch.get_value_at_quantile(q).expect("non-empty"); + if let Err(detail) = spec.check(q, est, sorted[idx]) { + panic!("DDSketch n={n} q={q}: {detail}"); } } assert!( - disagreements >= 3, - "the probe set must contain cases where the two conventions genuinely \ - disagree; only {disagreements} did" + distinguishing >= 3, + "only {distinguishing} probes separate ceil(q*n) from floor(q*(n-1))" ); } -/// Endpoint behaviour, which is **not** the same on the two implementations. -/// -/// - The core sketch tracks the exact minimum and maximum beside the bucket -/// store, and `get_value_at_quantile` short-circuits `q <= 0` and `q >= 1` to -/// them. Its endpoints are therefore *exact*, with zero error. -/// - The portable sketch carries no min/max scalars at all — they were removed -/// from the wire — so its endpoints are ordinary bucket representatives and -/// are only guaranteed within `alpha`. The previous revision of this test -/// claimed it "clamps its bucket representative into [min, max], so its -/// endpoints are exact too", which is not what the code does. +/// The sketch tracks the exact minimum and maximum beside the bucket store, and +/// `get_value_at_quantile` short-circuits `q <= 0` and `q >= 1` to them, so its +/// endpoints are exact. #[test] -fn ddsketch_core_endpoints_are_exact_and_portable_endpoints_are_alpha_relative() { +fn ddsketch_endpoints_are_exact() { for &alpha in &DDS_ALPHAS { let values = log_uniform_f64(5_000, (1.0 + alpha) / (1.0 - alpha), 3..30, 4_242); let truth = NumericTruth::new(values.clone()); let mut core = DDSketch::new(alpha); - let mut port = PortableDds::new(alpha); for v in &values { core.add(v); - port.update(*v); } assert_eq!( core.get_value_at_quantile(0.0), @@ -790,17 +706,6 @@ fn ddsketch_core_endpoints_are_exact_and_portable_endpoints_are_alpha_relative() Some(truth.max()), "core alpha={alpha}: max() must be exact" ); - - // The portable twin holds only buckets, so its endpoints get the same - // relative-value guarantee as any other quantile and nothing more. - let spec = RelativeQuantileSpec::portable(alpha); - let (p0, p1) = (port.quantile(0.0).unwrap(), port.quantile(1.0).unwrap()); - if let Err(detail) = spec.check(0.0, p0, truth.min()) { - panic!("portable alpha={alpha} at q=0: {detail}"); - } - if let Err(detail) = spec.check(1.0, p1, truth.max()) { - panic!("portable alpha={alpha} at q=1: {detail}"); - } } } @@ -813,11 +718,7 @@ fn ddsketch_core_endpoints_are_exact_and_portable_endpoints_are_alpha_relative() fn ddsketch_satisfies_the_relative_error_contract_at_bucket_boundaries() { for &alpha in &DDS_ALPHAS { let gamma = (1.0 + alpha) / (1.0 - alpha); - // A one-sample sketch answers every q with that one sample, so the two - // rank conventions coincide here by construction (both index 0) and the - // probe isolates the mapping from any rank effect. let core_spec = RelativeQuantileSpec::core(alpha); - let port_spec = RelativeQuantileSpec::portable(alpha); for k in [-40i32, -7, 0, 1, 13, 60, 200] { let edge = gamma.powi(k); if edge <= 0.0 || !edge.is_finite() { @@ -832,9 +733,7 @@ fn ddsketch_satisfies_the_relative_error_contract_at_bucket_boundaries() { ]; for probe in probes { let mut core = DDSketch::new(alpha); - let mut port = PortableDds::new(alpha); core.add(&probe); - port.update(probe); if core.get_count() == 0 { continue; // outside the indexable range at this alpha } @@ -844,10 +743,6 @@ fn ddsketch_satisfies_the_relative_error_contract_at_bucket_boundaries() { if let Err(detail) = core_spec.check(0.5, est, probe) { panic!("core DDSketch alpha={alpha} gamma^{k} boundary probe: {detail}"); } - let pest = port.quantile(0.5).unwrap(); - if let Err(detail) = port_spec.check(0.5, pest, probe) { - panic!("portable DdSketch alpha={alpha} gamma^{k} boundary probe: {detail}"); - } } } } @@ -858,13 +753,11 @@ fn ddsketch_satisfies_the_relative_error_contract_at_bucket_boundaries() { /// guarantee is stated over. #[test] fn ddsketch_merge_and_delta_replay_preserve_the_relative_error_contract() { - use asap_sketchlib::message_pack_format::portable::ddsketch::DdSketchDelta; use asap_sketchlib::octo_delta::DdDelta; const N: usize = 40_000; for &alpha in &DDS_ALPHAS { let spec = RelativeQuantileSpec::core(alpha); - let port_spec = RelativeQuantileSpec::portable(alpha); let values = zipf_f64(N, 8_192, 1.1, 1e3, 1e7, 7_654_321 + (alpha * 1e5) as u64); let truth = NumericTruth::new(values.clone()); @@ -929,49 +822,6 @@ fn ddsketch_merge_and_delta_replay_preserve_the_relative_error_contract() { &DDS_QS[1..6] ), ); - - // Portable side: merge, then a delta carrying the same buckets. - let mut pa = PortableDds::new(alpha); - let mut pb = PortableDds::new(alpha); - for (i, v) in values.iter().enumerate() { - if i % 2 == 0 { - pa.update(*v); - } else { - pb.update(*v); - } - } - pa.merge(&pb).expect("same-alpha portable merge"); - let mut port_tally = Tally::default(); - port_spec.tally_into(&mut port_tally, truth.sorted(), &DDS_QS, |q| pa.quantile(q)); - port_tally.assert_none( - &format!("portable DdSketch alpha={alpha} after merge"), - &format!("zipf n={N} over [1e3, 1e7]"), - ); - - let mut pr = PortableDds::new(alpha); - let delta = DdSketchDelta { - buckets: pa - .store_counts - .iter() - .enumerate() - .filter(|(_, c)| **c > 0) - .map(|(i, c)| (pa.store_offset + i as i32, *c)) - .collect(), - d_count: pa.total_count() as i64, - ..DdSketchDelta::default() - }; - pr.apply_delta(&delta).expect("benign delta"); - let mut pr_tally = Tally::default(); - port_spec.tally_into(&mut pr_tally, truth.sorted(), &DDS_QS[1..6], |q| { - pr.quantile(q) - }); - pr_tally.assert_none( - &format!("portable DdSketch alpha={alpha} rebuilt from a delta"), - &format!( - "zipf n={N} over [1e3, 1e7], interior q grid {:?}", - &DDS_QS[1..6] - ), - ); } } @@ -1622,19 +1472,17 @@ fn portable_hydra_kll_per_key_medians_satisfy_the_rank_characterization() { // ------------------------------------------- DDSketch input rejection /// Structural guards, not accuracy: values the mapping cannot index and -/// deltas spanning an implausible bucket range must be rejected without -/// corrupting state. No error bound applies to any of it. +/// merges across mappings must be rejected without corrupting state. No error +/// bound applies to any of it. #[test] fn ddsketch_rejects_untrackable_values_and_mapping_mismatches() { let alpha = 0.01; let mut core = DDSketch::new(alpha); - let mut port = PortableDds::new(alpha); - // Non-finite / above-maximum-magnitude values must be - // dropped by BOTH implementations: silently, without corrupting bucket 0 - // (NaN floor-casts to 0) and without letting one sample force a distant- - // bucket allocation (unguarded, f64::MAX maps ~35k buckets away at - // alpha=0.01 — ~277 KiB of amplification per sample, scaling with 1/lnγ). + // Non-finite / above-maximum-magnitude values are dropped silently, + // without corrupting bucket 0 (NaN floor-casts to 0) and without letting + // one sample force a distant-bucket allocation (unguarded, f64::MAX maps + // ~35k buckets away at alpha=0.01). for v in [ f64::NAN, f64::NEG_INFINITY, @@ -1643,31 +1491,27 @@ fn ddsketch_rejects_untrackable_values_and_mapping_mismatches() { 1e308, ] { core.add(&v); - port.update(v); } assert_eq!(core.get_count(), 0, "core must drop untrackable extremes"); - assert_eq!( - port.total_count(), - 0, - "portable must drop untrackable extremes" - ); assert!( - port.store_counts.len() < 10_000, - "portable store grew to {} buckets from untrackable input", - port.store_counts.len() + core.store_counts().is_empty(), + "store grew to {} buckets from untrackable input", + core.store_counts().len() + ); + assert_eq!( + core.get_value_at_quantile(0.5), + None, + "nothing trackable was added" ); - assert_eq!(port.quantile(0.5), None, "nothing trackable was added"); // Normal samples still work after the rejected inputs. let good = [1.0f64, 2.0, 4.0, 8.0, 16.0]; for v in good { core.add(&v); - port.update(v); } assert_eq!(core.get_count(), 5); - assert_eq!(port.total_count(), 5); assert_between( - port.store_counts.len() as f64, + core.store_counts().len() as f64, 1.0, 2.0 * 256.0, // initial GROW_CHUNK seed + at most one more "store stays compact", @@ -1675,46 +1519,30 @@ fn ddsketch_rejects_untrackable_values_and_mapping_mismatches() { // Boundary acceptance: values just INSIDE the indexable range must still // be tracked, so the guards reject only genuinely unmappable extremes. - // Bounds come from the shared production helper — core and portable MUST - // agree because they compute from the same function. let (min_idx, max_idx) = asap_sketchlib::sketches::ddsketch::ddsketch_indexable_bounds(alpha); - let mut port_boundary = PortableDds::new(alpha); let mut core_boundary = DDSketch::new(alpha); let just_inside_min = min_idx * (1.0 + 1e-9); // a hair above the floor let just_inside_max = max_idx * (1.0 - 1e-9); // a hair below the ceiling - port_boundary.update(just_inside_min); - port_boundary.update(just_inside_max); core_boundary.add(&just_inside_min); core_boundary.add(&just_inside_max); assert_eq!( - port_boundary.total_count(), + core_boundary.get_count(), 2, "in-range extremes near boundaries must be kept" ); - assert_eq!(core_boundary.get_count(), 2, "core boundary agreement"); - assert_eq!(port_boundary.total_count(), core_boundary.get_count()); // Below the minimum counts as zero; above the maximum is rejected. - port_boundary.update(min_idx * 0.5); - port_boundary.update(max_idx * (1.0 + 1e-6)); core_boundary.add(&(min_idx * 0.5)); - assert_eq!( - port_boundary.total_count(), - 3, - "sub-indexable neighbor enters zero bucket" - ); + core_boundary.add(&(max_idx * (1.0 + 1e-6))); assert_eq!( core_boundary.get_count(), 3, "core retains sub-indexable neighbor as zero" ); - - assert_eq!(port_boundary.zero_count, 1); assert_eq!(core_boundary.zero_count(), 1); - // Item 2 contract: mismatched mappings are a runtime error in BOTH types, - // not a debug-only assertion. + // Mismatched mappings are a runtime error, not a debug-only assertion. let other_alpha = 0.05; let mut core_other = DDSketch::new(other_alpha); core_other.add(&1.0); @@ -1722,79 +1550,20 @@ fn ddsketch_rejects_untrackable_values_and_mapping_mismatches() { core.merge(&core_other).is_err(), "core merge must reject alpha mismatch" ); - let mut port_other = PortableDds::new(other_alpha); - port_other.update(1.0); - assert!( - port.merge(&port_other).is_err(), - "portable merge must reject alpha mismatch" - ); - // Tiny-alpha regression: at alpha=1e-9 ln(gamma) ~ 2e-9, and naive - // reciprocal-multiplied guard formulas diverge from core's — admitting - // v=1e-300 whose bucket index saturates i32 and overflows ensure_bucket. - // Both implementations count it as zero without allocating a bucket. - let mut tiny = PortableDds::new(1e-9); + // Tiny-alpha regression: at alpha=1e-9 ln(gamma) ~ 2e-9, and v=1e-300 + // would saturate an i32 bucket index. It is counted as zero without + // allocating a bucket. let mut tiny_core = DDSketch::new(1e-9); - tiny.update(1e-300); tiny_core.add(&1e-300); - assert_eq!( - tiny.total_count(), - 1, - "portable counts sub-indexable value at tiny alpha" - ); assert_eq!( tiny_core.get_count(), 1, "core counts sub-indexable value at tiny alpha" ); - assert_eq!( - tiny.store_counts.len(), - 0, - "no allocation may occur for zero-mapped values" - ); -} - -#[test] -fn portable_ddsketch_rejects_hostile_delta_spans() { - use asap_sketchlib::message_pack_format::portable::ddsketch::DdSketchDelta; - - let alpha = 0.01; - let mut base = PortableDds::new(alpha); - base.update(1.0); - let (len_before, offset_before) = (base.store_counts.len(), base.store_offset); - - // A corrupt/hostile delta pointing near i32::MAX must be rejected with an - // error BEFORE any allocation: the naive pad would be ~2e9 buckets. - let hostile = DdSketchDelta { - buckets: vec![(i32::MAX - 1, 1), (7, 3)], - ..DdSketchDelta::default() - }; assert!( - base.apply_delta(&hostile).is_err(), - "hostile far-span delta must be rejected" - ); - assert_eq!( - base.store_counts.len(), - len_before, - "state untouched on rejection" - ); - assert_eq!( - base.store_offset, offset_before, - "offset untouched on rejection" - ); - - // Benign deltas still apply: bucket index 6 carries count 5 afterward, - // and its representative gamma^6*(1+alpha) becomes visible in queries. - let benign = DdSketchDelta { - buckets: vec![(6, 5)], - ..DdSketchDelta::default() - }; - base.apply_delta(&benign).expect("benign delta"); - let gamma = (1.0 + alpha) / (1.0 - alpha); - assert_eq!( - base.quantile(0.9), - Some(gamma.powf(6.0) * (1.0 + alpha)), - "applied delta count visible through queries" + tiny_core.store_counts().is_empty(), + "no allocation may occur for zero-mapped values" ); } @@ -2020,9 +1789,7 @@ fn ddsketch_satisfies_the_relative_value_error_contract_at_extreme_accuracy_para for (i, &alpha) in DDS_EXTREME_ALPHAS.iter().enumerate() { let core_spec = RelativeQuantileSpec::core(alpha); - let port_spec = RelativeQuantileSpec::portable(alpha); let mut core_tally = Tally::default(); - let mut port_tally = Tally::default(); let seed = 0x0DDA_0000u64 + i as u64 * 7919; let (min_indexable, max_indexable) = asap_sketchlib::sketches::ddsketch::ddsketch_indexable_bounds(alpha); @@ -2035,10 +1802,8 @@ fn ddsketch_satisfies_the_relative_value_error_contract_at_extreme_accuracy_para continue; } let mut core = DDSketch::new(alpha); - let mut port = PortableDds::new(alpha); for v in &values { core.add(v); - port.update(*v); } let truth = NumericTruth::new(values.clone()); assert_eq!( @@ -2046,24 +1811,15 @@ fn ddsketch_satisfies_the_relative_value_error_contract_at_extreme_accuracy_para truth.len(), "{label} alpha={alpha}: core dropped an indexable sample" ); - assert_eq!( - port.total_count() as usize, - truth.len(), - "{label} alpha={alpha}: portable dropped an indexable sample" - ); core_spec.tally_into(&mut core_tally, truth.sorted(), &DDS_QS, |q| { core.get_value_at_quantile(q) }); - port_spec.tally_into(&mut port_tally, truth.sorted(), &DDS_QS, |q| { - port.quantile(q) - }); } let context = format!( "alpha={alpha} n={N} indexable=[{min_indexable:.3e}, {max_indexable:.3e}] \ q grid {DDS_QS:?}" ); core_tally.assert_none(&format!("core DDSketch alpha={alpha}"), &context); - port_tally.assert_none(&format!("portable DdSketch alpha={alpha}"), &context); } } diff --git a/tests/msgpack_compat.rs b/tests/msgpack_compat.rs index 267a51c6..bd110fb3 100644 --- a/tests/msgpack_compat.rs +++ b/tests/msgpack_compat.rs @@ -31,8 +31,8 @@ use asap_sketchlib::message_pack_format::portable::delta_set_aggregator::DeltaRe use asap_sketchlib::message_pack_format::portable::hydra_kll::HydraKllSketchWire; use asap_sketchlib::message_pack_format::portable::kll::KllSketchData; use asap_sketchlib::{ - CountMinSketch, CountMinSketchWithHeap, CountSketch, DdSketch, HllSketch, HllVariant, - HydraKllSketch, KllSketch, SetAggregator, + CountMinSketch, CountMinSketchWithHeap, CountSketch, HllSketch, HllVariant, HydraKllSketch, + KllSketch, SetAggregator, }; // ===== round-trip: every wire-format-aligned type ===== @@ -72,18 +72,6 @@ fn count_sketch_round_trip() { assert_eq!(restored.cols, 64); } -#[test] -fn dd_sketch_round_trip() { - let mut s = DdSketch::new(0.01); - s.update(1.0); - s.update(10.0); - s.update(100.0); - let bytes = s.to_msgpack().expect("encode"); - let restored = DdSketch::from_msgpack(&bytes).expect("decode"); - // `count` is not on the wire; recover it by summing the bucket array. - assert_eq!(restored.total_count(), 3); -} - #[test] fn hll_sketch_round_trip() { let mut s = HllSketch::new(HllVariant::Regular, 8); @@ -210,7 +198,7 @@ fn hydra_kll_wire_shape() { // ===== golden-bytes placeholders: an ignored, uncovered gap ===== // -// The five tests below are **empty and ignored**. They verify nothing today. +// The four tests below are **empty and ignored**. They verify nothing today. // // What they need is a msgpack payload produced by `sketchlib-go` and checked in // under `tests/fixtures/msgpack/`. That fixture cannot be generated here: this @@ -237,10 +225,6 @@ fn count_min_decodes_go_bytes() { // assert_eq!(s.rows, EXPECTED_ROWS); } -#[ignore = "gap: needs a sketchlib-go-produced msgpack fixture; none can be generated in this repo"] -#[test] -fn dd_sketch_decodes_go_bytes() {} - #[ignore = "gap: needs a sketchlib-go-produced msgpack fixture; none can be generated in this repo"] #[test] fn hll_sketch_decodes_go_bytes() {} diff --git a/tests/pbt/ddsketch.rs b/tests/pbt/ddsketch.rs index b389ef9d..d52ba205 100644 --- a/tests/pbt/ddsketch.rs +++ b/tests/pbt/ddsketch.rs @@ -1,13 +1,11 @@ -//! Property tests for DDSketch: the native sketch and the portable `DdSketch` -//! that carries it over the wire. +//! Property tests for DDSketch. //! //! Masson, Rim, Lee, VLDB '19. The model is the paper's relative-error //! guarantee, not a transcription of the bucket arithmetic: an answer sits //! within a relative alpha of the exact order statistic of the stream. use crate::support::{QUANTILES, extreme_values}; -use asap_sketchlib::message_pack_format::MessagePackCodec; -use asap_sketchlib::{DDSketch, DdSketch}; +use asap_sketchlib::DDSketch; use proptest::prelude::*; const QUANTILE_PROBES: [f64; 7] = [0.0, 0.01, 0.25, 0.5, 0.75, 0.99, 1.0]; @@ -74,14 +72,6 @@ fn dd_of(alpha: f64, values: &[f64]) -> DDSketch { s } -fn portable_of(alpha: f64, values: &[f64]) -> DdSketch { - let mut s = DdSketch::new(alpha); - for v in values { - s.update(*v); - } - s -} - fn sorted(values: &[f64]) -> Vec { let mut v = values.to_vec(); v.sort_by(|a, b| a.partial_cmp(b).expect("the generator excludes NaN")); @@ -123,24 +113,6 @@ proptest! { prop_assert_eq!(sketch.get_count(), values.len() as u64, "total count"); } - #[test] - fn ddsketch_portable_keeps_negative_and_zero_observations( - alpha in alpha(), - values in signed_values(120), - ) { - let sketch = portable_of(alpha, &values); - let negatives = values.iter().filter(|v| **v < 0.0).count() as u64; - let zeros = values.iter().filter(|v| **v == 0.0).count() as u64; - let positives = values.iter().filter(|v| **v > 0.0).count() as u64; - - prop_assert_eq!( - sketch.negative_store_counts.iter().sum::(), negatives, "negative store" - ); - prop_assert_eq!(sketch.zero_count, zeros, "zero bucket"); - prop_assert_eq!(sketch.store_counts.iter().sum::(), positives, "positive store"); - prop_assert_eq!(sketch.total_count(), values.len() as u64, "total count"); - } - /// `sum` is carried on the wire and read back through `sum()`, so it is /// the running total of the stream, not an estimate off the buckets. #[test] @@ -159,7 +131,7 @@ proptest! { } /// The first dense allocation is one `GROW_CHUNK` centred on the value's - /// bucket, in both implementations and in both signed stores. The layout + /// bucket, in both signed stores. The layout /// reaches the wire as `store_counts` / `store_offset`, where the Go /// producer's bytes have to match it for the same input. #[test] @@ -168,13 +140,10 @@ proptest! { value in 1e-6f64..1e9, ) { let native = dd_of(alpha, &[value, -value]); - let portable = portable_of(alpha, &[value, -value]); for (label, counts) in [ - ("native positive", native.store_counts()), - ("native negative", native.negative_store_counts()), - ("portable positive", portable.store_counts.as_slice()), - ("portable negative", portable.negative_store_counts.as_slice()), + ("positive", native.store_counts()), + ("negative", native.negative_store_counts()), ] { prop_assert_eq!(counts.len(), GROW_CHUNK, "{} store width", label); prop_assert_eq!(counts[GROW_CHUNK_CENTRE], 1, "{} store centre", label); @@ -259,27 +228,6 @@ proptest! { } } - /// The portable twin answers the floor(q*(n-1)) order statistic, and it - /// carries no min/max, so the bound applies at every q including 0 and 1. - #[test] - fn ddsketch_portable_signed_answers_stay_within_alpha_of_the_exact_order_statistic( - alpha in alpha(), - values in signed_values(50), - ) { - let sketch = portable_of(alpha, &values); - let ordered = sorted(&values); - - for q in probe_quantiles(40) { - let want = ordered[(q * (ordered.len() - 1) as f64).floor() as usize]; - let got = sketch.quantile(q).expect("a non-empty sketch answers"); - prop_assert!( - (got - want).abs() <= want.abs() * (alpha + 1e-12), - "alpha={} q={} n={}: got {}, exact order statistic {}", - alpha, q, ordered.len(), got, want - ); - } - } - // ===== Merge ===== #[test] @@ -288,25 +236,25 @@ proptest! { a in maybe_signed_values(40), b in maybe_signed_values(40), ) { - let mut ab = portable_of(alpha, &a); - ab.merge(&portable_of(alpha, &b)).expect("merge"); - let mut ba = portable_of(alpha, &b); - ba.merge(&portable_of(alpha, &a)).expect("merge"); - - prop_assert_eq!(ab.total_count(), (a.len() + b.len()) as u64); - prop_assert_eq!(ab.total_count(), ba.total_count()); - prop_assert_eq!(&ab.store_counts, &ba.store_counts); - prop_assert_eq!(ab.store_offset, ba.store_offset); - prop_assert_eq!(&ab.negative_store_counts, &ba.negative_store_counts); - prop_assert_eq!(ab.negative_store_offset, ba.negative_store_offset); - prop_assert_eq!(ab.zero_count, ba.zero_count); + let mut ab = dd_of(alpha, &a); + ab.merge(&dd_of(alpha, &b)).expect("merge"); + let mut ba = dd_of(alpha, &b); + ba.merge(&dd_of(alpha, &a)).expect("merge"); + + prop_assert_eq!(ab.get_count(), (a.len() + b.len()) as u64); + prop_assert_eq!(ab.get_count(), ba.get_count()); + prop_assert_eq!(ab.store_counts(), ba.store_counts()); + prop_assert_eq!(ab.store_offset(), ba.store_offset()); + prop_assert_eq!(ab.negative_store_counts(), ba.negative_store_counts()); + prop_assert_eq!(ab.negative_store_offset(), ba.negative_store_offset()); + prop_assert_eq!(ab.zero_count(), ba.zero_count()); prop_assert_eq!( - ab.zero_count, + ab.zero_count(), a.iter().chain(&b).filter(|v| **v == 0.0).count() as u64, "the merged zero bucket" ); prop_assert_eq!( - ab.negative_store_counts.iter().sum::(), + ab.negative_store_counts().iter().sum::(), a.iter().chain(&b).filter(|v| **v < 0.0).count() as u64, "the merged negative store" ); @@ -340,8 +288,8 @@ proptest! { } /// Two alphas are two index mappings, so the bucket indices of one sketch - /// mean nothing under the other's gamma. Both implementations refuse the - /// merge and leave the receiver as it was. + /// mean nothing under the other's gamma. The merge is refused and leaves + /// the receiver as it was. #[test] fn ddsketch_merge_refuses_a_different_index_mapping( alpha in alpha(), @@ -357,13 +305,6 @@ proptest! { ); prop_assert_eq!(native.get_count(), values.len() as u64, "refused merge changed the count"); - let mut portable = portable_of(alpha, &values); - prop_assert!( - portable.merge(&portable_of(other_alpha, &values)).is_err(), - "portable alpha {} merged a sketch built with alpha {}", alpha, other_alpha - ); - prop_assert_eq!(portable.total_count(), values.len() as u64, "refused merge changed the count"); - prop_assert!( dd_of(alpha, &values).merge(&dd_of(alpha, &values)).is_ok(), "a shared mapping must still merge" @@ -377,19 +318,20 @@ proptest! { alpha in alpha(), values in maybe_signed_values(120), ) { - let s = portable_of(alpha, &values); - - let bytes = s.to_msgpack().expect("encode"); - let restored = DdSketch::from_msgpack(&bytes).expect("decode"); - - prop_assert_eq!(restored.total_count(), s.total_count()); - prop_assert_eq!(&restored.store_counts, &s.store_counts); - prop_assert_eq!(restored.store_offset, s.store_offset); - prop_assert_eq!(&restored.negative_store_counts, &s.negative_store_counts); - prop_assert_eq!(restored.negative_store_offset, s.negative_store_offset); - prop_assert_eq!(restored.zero_count, s.zero_count); + let s = dd_of(alpha, &values); + + let bytes = s.serialize_to_bytes().expect("encode"); + let restored = DDSketch::deserialize_from_bytes(&bytes).expect("decode"); + + prop_assert_eq!(restored.get_count(), s.get_count()); + prop_assert_eq!(restored.store_counts(), s.store_counts()); + prop_assert_eq!(restored.store_offset(), s.store_offset()); + prop_assert_eq!(restored.negative_store_counts(), s.negative_store_counts()); + prop_assert_eq!(restored.negative_store_offset(), s.negative_store_offset()); + prop_assert_eq!(restored.zero_count(), s.zero_count()); + prop_assert_eq!(restored.sum().to_bits(), s.sum().to_bits()); for q in QUANTILE_PROBES { - prop_assert_eq!(restored.quantile(q), s.quantile(q), "q={}", q); + prop_assert_eq!(restored.get_value_at_quantile(q), s.get_value_at_quantile(q), "q={}", q); } } diff --git a/tests/sketches_go_parity_probe.rs b/tests/sketches_go_parity_probe.rs index 2997e486..3ad693a8 100644 --- a/tests/sketches_go_parity_probe.rs +++ b/tests/sketches_go_parity_probe.rs @@ -12,12 +12,11 @@ use asap_sketchlib::common::DataInput; use asap_sketchlib::common::hash::CANONICAL_HASH_SEED; use asap_sketchlib::common::structures::HllBucketListP14; use asap_sketchlib::proto::sketchlib::{ - CountMinState, CountSketchState, CounterType, DdSketchState, HyperLogLogState, SketchEnvelope, + CountMinState, CountSketchState, CounterType, HyperLogLogState, SketchEnvelope, sketch_envelope::SketchState, }; use asap_sketchlib::sketches::countminsketch::CountMin; use asap_sketchlib::sketches::countsketch::Count; -use asap_sketchlib::sketches::ddsketch::DDSketch; use asap_sketchlib::sketches::hll::{ErtlMLE, HyperLogLogImpl}; use asap_sketchlib::{DefaultXxHasher, FastPath, Vector2D}; use prost::Message; @@ -39,15 +38,6 @@ const COUNTMIN_GOLDEN_HEX: &str = include_str!("../src/sketches/testdata/cms_env /// the wrapper test already uses. const HLL_GOLDEN_HEX: &str = include_str!("../src/sketches/testdata/hll_envelope_golden.hex"); -/// 403-byte DDSketch envelope for `alpha=0.01`, `(1..=50)` integer-as-f64 -/// input, without the DataPoint-level METRIC scalars (count/sum/min/max → -/// proto tags 4-7 reserved). -/// -/// NOTE: this is currently the RUST-produced value; it MUST be reconciled -/// against the parallel `sketchlib-go` golden regeneration before declaring -/// cross-language byte parity. -const DDSKETCH_GOLDEN_HEX: &str = "0801728e03096214ae47e17a843f128003000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000000000001000000000000000000000000000000000000000100000000000000000000000000000100000000000000000000010000000000000000010000000000000001000000000001000000000001000000000001000000010000000001000000010000010000000100000100000100000100000100010000010001000100010001000100010001000100010100010100010100010101000101010100010101010101010100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000187f"; - #[test] fn sketches_count_fastpath_matches_go_count_sketch_envelope() { let rows = 3usize; @@ -218,51 +208,6 @@ fn sketches_hll_classic_matches_go_envelope() { ); } -#[test] -fn sketches_ddsketch_matches_go_envelope() { - let mut sk = DDSketch::new(0.01); - for i in 1..=50i32 { - sk.add(&(i as f64)); - } - - // Go serializes `alpha` round-tripped through gamma to mirror its - // internal storage: `alpha_wire = (γ-1)/(γ+1)`, `γ = (1+α)/(1-α)`. - // This is a 25-ULP shift from the user-supplied α=0.01. - let alpha = sk.alpha(); - let gamma = (1.0 + alpha) / (1.0 - alpha); - let alpha_wire = (gamma - 1.0) / (gamma + 1.0); - // The DataPoint-level METRIC scalars (count/sum/min/max) are not on the - // wire; only alpha + the bucket array. - let state = DdSketchState { - alpha: alpha_wire, - store_counts: sk.store_counts().to_vec(), - store_offset: sk.store_offset(), - ..Default::default() - }; - let envelope = SketchEnvelope { - format_version: 1, - producer: None, - hash_spec: None, - sample_p: 0.0, - sketch_state: Some(SketchState::Ddsketch(state)), - }; - let mut got = Vec::with_capacity(envelope.encoded_len()); - envelope.encode(&mut got).expect("prost encode"); - - let want = decode_hex(DDSKETCH_GOLDEN_HEX); - assert_eq!( - got.len(), - want.len(), - "DDSketch envelope length: got {} want {}", - got.len(), - want.len(), - ); - assert_eq!( - got, want, - "sketches::DDSketch envelope diverges from Go golden", - ); -} - fn decode_hex(s: &str) -> Vec { s.trim() .as_bytes() diff --git a/tests/xtest_consumer.rs b/tests/xtest_consumer.rs index 0113ff45..7d0069f3 100644 --- a/tests/xtest_consumer.rs +++ b/tests/xtest_consumer.rs @@ -7,7 +7,6 @@ //! Files consumed (from $XTEST_DIR/): //! countmin.pb — CountMinState with float64 counters //! kll.pb — KLLState with quantile items and coin RNG state -//! ddsketch.pb — DDSketchState with alpha + bucket array //! hll.pb — HyperLogLogState (ErtlMLE variant) //! countsketch.pb — CountSketchState with float64 signed counters //! coco.pb — CocoSketchState (hash+val+hasKey buckets) @@ -159,57 +158,6 @@ fn cross_language_proto() { all_ok = false; } - // ----------------------------------------------------------------------- - // DDSketch - // ----------------------------------------------------------------------- - println!(); - println!("[DDSketch] Step 1/3 — Read ddsketch.pb"); - let bytes = read_file(in_dir.join("ddsketch.pb")); - let env = SketchEnvelope::decode(bytes.as_slice()).expect("decode ddsketch envelope"); - - println!( - "[DDSketch] Step 2/3 — Validate envelope (format_version={}, producer={})", - env.format_version, - env.producer.as_ref().map_or("?", |p| &p.library) - ); - - let dd_state = match env.sketch_state { - Some(sketch_envelope::SketchState::Ddsketch(ref s)) => s.clone(), - other => panic!("expected DDSketch sketch_state, got {other:?}"), - }; - - // `count` is not on the wire; recover it by summing the bucket counts. - let dd_total_count: u64 = dd_state.store_counts.iter().copied().sum(); - println!( - "[DDSketch] alpha={:.4} count={} buckets={} offset={}", - dd_state.alpha, - dd_total_count, - dd_state.store_counts.len(), - dd_state.store_offset - ); - - let dd = DdFromProto::from_state(&dd_state); - let p50_dd = dd.quantile(0.50); - let p99_dd = dd.quantile(0.99); - println!( - "[DDSketch] Step 3/3 — p50 ≈ {:.2} (expect ~5000) p99 ≈ {:.2} (expect ~9900)", - p50_dd.unwrap_or(f64::NAN), - p99_dd.unwrap_or(f64::NAN) - ); - - let p50_ok = p50_dd - .map(|v| (v - 5000.0).abs() / 5000.0 < 0.02) - .unwrap_or(false); - let p99_ok = p99_dd - .map(|v| (v - 9900.0).abs() / 9900.0 < 0.02) - .unwrap_or(false); - if p50_ok && p99_ok { - println!("[DDSketch] PASS"); - } else { - eprintln!("[DDSketch] FAIL: p50={p50_dd:?} p99={p99_dd:?}"); - all_ok = false; - } - // ----------------------------------------------------------------------- // HLL (ErtlMLE estimator) // ----------------------------------------------------------------------- @@ -1164,82 +1112,3 @@ impl KllFromProto { samples.last().unwrap().0 } } - -// --------------------------------------------------------------------------- -// Minimal DDSketch deserialization + quantile query -// --------------------------------------------------------------------------- - -struct DdFromProto { - gamma: f64, - #[allow(dead_code)] - inv_log_gamma: f64, - store_counts: Vec, - store_offset: i32, - // Total count is recovered by summing the bucket array; the - // DataPoint-level count/min/max scalars are not on the wire. - count: u64, -} - -impl DdFromProto { - fn from_state(s: &DdSketchState) -> Self { - let alpha = s.alpha; - let gamma = (1.0 + alpha) / (1.0 - alpha); - let log_gamma = gamma.ln(); - let inv_log_gamma = 1.0 / log_gamma; - Self { - gamma, - inv_log_gamma, - store_counts: s.store_counts.clone(), - store_offset: s.store_offset, - count: s.store_counts.iter().copied().sum(), - } - } - - fn bin_representative(&self, k: i32) -> f64 { - self.gamma.powf(k as f64 + 0.5) - } - - /// Representative value of the lowest non-empty bucket — the - /// α-bounded estimate of the minimum now that the `min` scalar is - /// gone from the wire. - fn min_estimate(&self) -> Option { - self.store_counts - .iter() - .position(|&c| c > 0) - .map(|i| self.bin_representative(self.store_offset + i as i32)) - } - - /// Representative value of the highest non-empty bucket — the - /// α-bounded estimate of the maximum. - fn max_estimate(&self) -> Option { - self.store_counts - .iter() - .rposition(|&c| c > 0) - .map(|i| self.bin_representative(self.store_offset + i as i32)) - } - - fn quantile(&self, q: f64) -> Option { - if self.count == 0 { - return None; - } - if q <= 0.0 { - return self.min_estimate(); - } - if q >= 1.0 { - return self.max_estimate(); - } - let rank = (q * self.count as f64).ceil() as u64; - let mut seen = 0u64; - for (i, &c) in self.store_counts.iter().enumerate() { - if c == 0 { - continue; - } - seen += c; - if seen >= rank { - let bin = self.store_offset + i as i32; - return Some(self.bin_representative(bin)); - } - } - self.max_estimate() - } -} diff --git a/tools/gen-proto/src/main.rs b/tools/gen-proto/src/main.rs index 1de6ad15..9c293417 100644 --- a/tools/gen-proto/src/main.rs +++ b/tools/gen-proto/src/main.rs @@ -32,7 +32,6 @@ const PROTO_FILES: &[&str] = &[ "countsketch/countsketch.proto", "hll/hll.proto", "kll/kll.proto", - "ddsketch/ddsketch.proto", "univmon/univmon.proto", "hydra/hydra.proto", "cocosketch/cocosketch.proto",