Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 16 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -66,6 +66,10 @@ signals a backwards-compatible change.
worker's seed and the aggregator's from one seed. The unseeded constructors
draw from the thread generator. The seed is not serialized, and the ASAPv1
payload is unchanged.
- **DDSketch ASAPv1 golden fixtures**: `ddsketch_positive_a001` and
`ddsketch_empty_a001` (metadata version 1) and `ddsketch_signed_a001`
(metadata version 2, negative store and zero count), checked by
`tests/asapv1_golden.rs`.

### Changed

Expand All @@ -88,6 +92,18 @@ signals a backwards-compatible change.
as an empty window like any other, and that window takes a place in
`max_windows`.

### Removed

- **DDSketch's protobuf and portable formats.** DDSketch serializes through
ASAPv1 only (`DDSketch::serialize_to_bytes` / `deserialize_from_bytes`).
Removed: `proto/ddsketch/ddsketch.proto` and the generated
`proto::sketchlib::{DdSketchState, DdSketchDelta, DdSketchBucketDelta}`;
`SketchEnvelope`'s `ddsketch` field (14), now reserved;
`message_pack_format::portable::ddsketch` (`DdSketch`, `DdSketchDelta`,
`DDSKETCH_GROW_CHUNK`, `MAX_APPLY_DELTA_SPAN_BUCKETS`) and the crate-root
re-exports of the first three; the `MessagePackCodec` impl for `DDSketch`.
There is no ASAPv1 DDSketch delta.

### Fixed

- `KLLDynamic` panicked on every quantile query once a NaN had entered the
Expand Down
10 changes: 0 additions & 10 deletions docs/api/api_ddsketch.md
Original file line number Diff line number Diff line change
Expand Up @@ -95,16 +95,6 @@ assert!(p50 >= 1.0);
retained minimum and maximum rather than to a bucket representative, so the
endpoints carry zero error.

This differs from the portable wire twin
`message_pack_format::portable::ddsketch::DdSketch::quantile`, which uses the
lower-quantile convention `floor(q * (n - 1))` of the DDSketch paper and of the
Go reference implementation, and which retains no min/max scalars — its
endpoints are ordinary bucket representatives, accurate to `alpha` like any
other rank. The two agree at most `q` and diverge at small `n` or ragged `q`
(at `n = 3, q = 0.4` this type answers `sorted[1]` and the portable one
`sorted[0]`). Both conventions are deliberate and are pinned by
`ddsketch_core_and_portable_answer_different_order_statistics`.

## Caveats

- Inputs must be positive values.
Expand Down
10 changes: 5 additions & 5 deletions docs/asapv1_wire_format.md
Original file line number Diff line number Diff line change
Expand Up @@ -673,17 +673,17 @@ Wire counter types are the base Count-Min's: **`"i32"`, `"i64"` and `"f64"`**. `
| 3 | `min` | f64 | smallest value as ingested; `+inf` when empty |
| 4 | `max` | f64 | largest value as ingested; `-inf` when empty |

**Metadata vs payload.** `alpha` shapes every bucket index and is chosen at construction, so per the config-to-metadata rule it is a structural param, like HLL's `precision`. `gamma`, `log_gamma` and `inv_log_gamma` are derived from it (`gamma = (1 + alpha) / (1 - alpha)`) and appear nowhere: the decoder re-derives them exactly as `DDSketch::new` does. The store's dimensions are not configuration — the store grows with the data — so there is no `rows`/`cols` analogue; `counts` is self-delimiting and `offset` positions it. The metadata is `metadata_version` then `alpha`.
**Metadata vs payload.** `alpha` shapes every bucket index and is chosen at construction, so per the config-to-metadata rule it is a structural param, like HLL's `precision`. `gamma`, `log_gamma` and `inv_log_gamma` are derived from it (`gamma = (1 + alpha) / (1 - alpha)`) and appear nowhere: the decoder re-derives them exactly as `DDSketch::new` does. The store's dimensions are not configuration — the store grows with the data — so there is no `rows`/`cols` analogue; `counts` is self-delimiting and `offset` positions it. The metadata is `metadata_version` then `alpha`. `alpha` is the constructor's argument bit for bit, never re-derived from `gamma`.

**DDSketch carries no hash-spec group.** It never hashes: it maps a raw `f64` through a logarithm and carries no hasher type parameter at all. The hash spec answers "how were keys hashed", and a sketch that does not hash has no truthful answer, so the group is omitted entirely. This is the Q-KLL precedent, and the reason DDSketch's metadata schema is not `HashProfile`-derived the way HLL's and Count-Min's are.

**"As ingested" is not always "exact".** On the `add` path the three scalars see the caller's own `f64`, so they are exact. On the `apply_delta` path they do not: a delta names a bucket index rather than a value, so the sketch folds in that bucket's **representative** — `sum` accumulates `representative * delta.value`, and `min` / `max` compare against the representative. A sketch fed only deltas reports representative-based extrema. For a signed sum, cancellation can amplify relative error; it does not inherit the per-value relative accuracy `alpha`. They are still carried rather than recomputed: a sketch fed by both paths mixes exact and representative contributions, and no walk of the buckets reproduces the mix.

**`sum` saturates at signed `f64::MAX`.** All three paths that advance it — `add`, `apply_delta` and `merge` — go through one guarded step: the new total is taken when it is finite and `f64::MAX` with the overflow sign is stored otherwise. So a sketch that ingests enough mass to overflow the accumulator still serializes, and only `sum` degrades, only past `f64::MAX`. `count`, the bucket store, `min` and `max` do not pass through the guard and are unaffected. This is what keeps rule 7's finiteness requirement satisfiable: an unguarded `+=` would leave a legally-ingested sketch carrying `+inf`, which the same rule refuses on both doors, so a sketch nothing did wrong to would have no encoding.

**Signed extension (metadata version 2).** Positive-only states retain the five-field payload and version 1. States with negative observations or a zero count use version 2 and append three fields: `negative_counts: array<u64>`, `negative_offset: int`, and `zero_count: u64`. Negative values use the same mapping on their magnitudes; quantile traversal visits their buckets in descending index order, then zeros, then positive buckets in ascending order. Values smaller in magnitude than the mapping minimum count as zero; nonfinite and too-large magnitudes are rejected by `try_add`. The legacy `add` wrapper ignores those errors.
**Signed extension (metadata version 2).** Positive-only states retain the five-field payload and version 1. States with negative observations or a zero count use version 2 and append three fields: `negative_counts: array<u64>`, `negative_offset: int`, and `zero_count: u64`. Negative values use the same mapping on their magnitudes; quantile traversal visits their buckets in descending index order, then zeros, then positive buckets in ascending order. Values smaller in magnitude than the mapping minimum count as zero; nonfinite and too-large magnitudes are rejected by `try_add` and ignored by `add`.

For version 2 the total is `sum(counts) + sum(negative_counts) + zero_count`, checked for overflow. Both stores obey the span checks below. A nonempty signed state requires finite scalars and `min <= max`; the positive-only `min > 0` and `sum >= min` checks do not apply. Empty-state scalars remain unchanged. Version 1 readers reject version 2. The existing positive-only golden bytes remain unchanged.
For version 2 the total is `sum(counts) + sum(negative_counts) + zero_count`, checked for overflow. Both stores obey the span checks below. A nonempty signed state requires finite scalars and `min <= max`; the positive-only `min > 0` and `sum >= min` checks do not apply. Empty-state scalars are as in version 1. A decoder that knows only version 1 rejects version 2.

**Emitted order (cross-language contract).** `counts` is the dense store in its own bucket-index order, ascending from `offset`, with the `GROW_CHUNK` padding zeros left in place and `offset` left where the sketch put it. Nothing is trimmed and nothing is sorted, so a decoded sketch re-serializes byte-identically and `store_counts()` / `store_offset()` mean the same thing on both sides.

Expand Down Expand Up @@ -1341,14 +1341,14 @@ Good direction (more compact, higher fidelity, less Rust-internal duplication),
2. **Golden byte-vector fixtures** in one shared repo, [`sketchlib-golden-bytes`](https://github.com/ProjectASAP/sketchlib-golden-bytes), which each implementation mounts at `asapv1_golden/`; both languages decode and re-encode them byte-identically. These replace the `portable`-as-oracle round-trip test.
3. **This registry**, mirrored, never independently allocated.

Fixtures exist for six `kind_id`s — HLL's three estimators, Count-Min, Count Sketch and compact KLL — and `sketchlib-go` mirrors those. Every other kind here has a payload and **no fixture**, so this document is its only contract; `asapv1_golden/README.md` lists the gap. `portable` carries its own Go goldens.
`asapv1_golden/README.md` ("Coverage") lists which `kind_id`s have fixtures; a kind without one has this document as its only contract. `portable` carries its own Go goldens.

**Hash profile on the Go side.**
Rust derives the hash spec from a generic `HashProfile` bound on the hasher type; Go has no generic hasher type, so there is nothing to derive from.
On the Go side the profile is simply **written into** the metadata on encode and **read from** it on decode.
Go MUST validate the profile it reads (same fail-closed intent as Rust): a sketch is only mergeable/queryable if its `hash_profile_id` + seeds match the profile Go is prepared to reproduce.

Sequencing: (2) covers six `kind_id`s, and the one `native bytes == portable bytes` test is HLL-only, since no other `portable` type emits ASAPv1 bytes.
Sequencing: (2) covers the kinds the golden README lists, and the one `native bytes == portable bytes` test is HLL-only, since no other `portable` type emits ASAPv1 bytes.
Retire `portable` once the fixtures cover the rest.

---
Expand Down
18 changes: 6 additions & 12 deletions docs/ddsketch_signed_values.md
Original file line number Diff line number Diff line change
@@ -1,18 +1,12 @@
# Signed DDSketch

Both `DDSketch` (core) and `DdSketch` (portable) retain positive samples, negative samples, and zeros. The design follows [DataDog DDSketch](https://github.com/DataDog/sketches-go/blob/master/ddsketch/ddsketch.go): separate positive and negative magnitude stores, plus a zero count. Bucket indices can be negative in either store; an index's sign does not encode an observation's sign.
`DDSketch` retains positive samples, negative samples, and zeros. The design follows [DataDog DDSketch](https://github.com/DataDog/sketches-go/blob/master/ddsketch/ddsketch.go): separate positive and negative magnitude stores, plus a zero count. Bucket indices can be negative in either store; an index's sign does not encode an observation's sign.

`try_add` / `try_update` return errors for NaN, infinities, or magnitudes above the mapping's maximum. Existing `add` / `update` wrappers keep their return types and ignore those errors. Magnitudes below the mapping's minimum are counted as zero, including subnormal inputs. Such nonzero tiny observations are outside the relative-error guarantee. Original sample values are still compressed into buckets, not retained individually.
`try_add` returns an error for NaN, infinities, or magnitudes above the mapping's maximum. `add` ignores those errors. Magnitudes below the mapping's minimum are counted as zero, including subnormal inputs. Such nonzero tiny observations are outside the relative-error guarantee. Original sample values are still compressed into buckets, not retained individually.

Core preserves its existing ceil(q*n) order-statistic convention and exact ingested extrema. Portable preserves floor(q*(n-1)); `quantile_interpolated` preserves the PromQL interpolation convention. For indexable nonzero order statistics the estimate has absolute error at most `alpha * abs(value)`, for either sign.
Quantiles use the ceil(q*n) order-statistic convention and exact ingested extrema. For indexable nonzero order statistics the estimate has absolute error at most `alpha * abs(value)`, for either sign.

If interpolation combines endpoints x and y with weight t, the absolute error is bounded by `alpha * ((1-t)*abs(x) + t*abs(y))`, excluding zero-mapped tiny values. Opposite signs can cancel, so the interpolated result need not have relative error alpha. Dividing quantiles additionally requires a finite nonzero true denominator and valid relative-error bounds for both operands. This change does not establish those planner preconditions.
## Wire and OctoSketch

## Wire and downstream upgrade

- Native ASAPv1: positive-only states keep metadata version 1 and their existing bytes. Signed/zero states use metadata version 2 and append negative counts, negative offset, and zero count. Old native readers reject version 2.
- Portable MessagePack: positive-only snapshots keep their three fields; signed snapshots append the same three fields. Positive-only deltas keep their two arrays; signed deltas append negative indices, negative counts, and zero-count increment. New readers accept both layouts.
- Protobuf: `DdSketchState` adds fields 16–18; `DdSketchDelta` adds fields 16–17. `DdSketch::to_proto` / `from_proto` preserve all stores. Old protobuf readers silently ignore unknown fields: upgrade Collector, backend, and Go producer schema/mapping together before emitting signed protobuf. This is not DataDog's binary wire format.
- Octo: `DdDelta` now requires a `store: DdStore` discriminator. Existing positive delta constructors should use `DdStore::Positive`. Worker residual keys are `(DdStore, i32)`. Both promotion and flush preserve all three categories; partial counts still need flushing before complete queries.

Go implementations of the ASAP wire formats and downstream dependency pins are not changed by this Rust library patch.
- ASAPv1: positive-only states use metadata version 1. States with negative samples or zeros use version 2, which appends negative counts, negative offset, and zero count.
- OctoSketch: `DdDelta` carries a `store: DdStore` discriminator, and worker residual keys are `(DdStore, i32)`. Promotion and flush preserve all three categories; partial counts need flushing before complete queries.
9 changes: 3 additions & 6 deletions docs/e2e_testing_harness.md
Original file line number Diff line number Diff line change
Expand Up @@ -101,7 +101,7 @@ One spec per *metric*, not per sketch, because the metric is what differs:
| `CountSketchSpec` | two-sided L2, rank-independent | marginal `sqrt(3/w)·‖f₋ᵢ‖₂` at `P[Bin(d, 1/3) ≥ ⌈d/2⌉]`; simultaneous at the smallest κ with `P[Bin(d,1/κ) ≥ ⌈d/2⌉] ≤ δ/D` |
| `SecondMomentSpec` | F2 from a Count Sketch matrix | `sqrt(2κ/w)`, same median amplification |
| `KllRankSpec` | KLL **maximum** rank error over a q grid | `ε(k) = 2.446 / k^0.9433` — an Apache DataSketches *characterization fit*, not a theorem about this code |
| `RelativeQuantileSpec` + `DdRankConvention` | DDSketch relative value error | `α + ULP slack` vs the exact order statistic **of that implementation's own rank convention** |
| `RelativeQuantileSpec` + `dd_rank_index` | DDSketch relative value error | `α + ULP slack` vs the exact `ceil(q·n)` order statistic |
| `CardinalityConfidenceSpec` | HLL / KMV | `z · σ_rel`, σ derived exactly, the tail a normal approximation |
| `SamplingConfidenceSpec` | Nitro | `z · sqrt(f(p·r(1−r) + (1−p)/p))`, `r = frac(1/p)` |
| `PrioritySampleSpec` | `UniformSampling` | `len = ⌈n·rate⌉` exactly; `Var[mean] = (σ_N²/m)(N−m)/(N−1)` |
Expand Down Expand Up @@ -141,11 +141,8 @@ Keeping these apart is deliberate. There is no shared
`QuantileSpec { rank_tol }` that KLL and DDSketch both use, because they do not
promise the same thing: a correct KLL can return a value 100× off on a
heavy-tailed stream and still be within its rank guarantee, and a correct
DDSketch has no rank guarantee at all. For the same reason the two DDSketch
implementations do not share a truth helper: `DDSketch::get_value_at_quantile`
answers `sorted[ceil(q·n) − 1]` while the portable `DdSketch::quantile` answers
`sorted[floor(q·(n−1))]`, so `DdRankConvention` carries the choice and each is
scored on the question it actually answers.
DDSketch has no rank guarantee at all. `DDSketch::get_value_at_quantile`
answers `sorted[ceil(q·n) − 1]`, and `dd_rank_index` computes that truth.

Two distinctions are easy to lose:

Expand Down
2 changes: 1 addition & 1 deletion docs/features.md
Original file line number Diff line number Diff line change
Expand Up @@ -112,7 +112,7 @@ MessagePack (`rmp-serde`) support. **serde support** means the type derives `Ser
| EHSketchList | Yes | Yes |
| ExponentialHistogram | No | Yes |

Protobuf (prost): `.proto` definitions exist for CountMin, Count, HLL, DDSketch, KLL, Elastic, Coco, Hydra, and UnivMon. Rust conversion code is in progress.
Protobuf (prost): `.proto` definitions exist for CountMin, Count, HLL, KLL, Elastic, Coco, Hydra, and UnivMon. Rust conversion code is in progress.

### API Stability

Expand Down
8 changes: 4 additions & 4 deletions docs/message_pack_format.md
Original file line number Diff line number Diff line change
Expand Up @@ -169,10 +169,10 @@ decode → re-encode them byte-identically. The `kind_id` registry is mirrored
verbatim with Go's `wire/asapmsgpack/magic_ids.go`, never independently
allocated.

That proof covers the six `kind_id`s a fixture exists for — HLL's three
estimators, Count-Min, Count Sketch and compact KLL. Every other implemented
kind has **no ASAPv1 golden and therefore no ASAPv1 drift guard**;
[its README](https://github.com/ProjectASAP/sketchlib-golden-bytes#coverage) lists what is covered.
That proof covers only the `kind_id`s a fixture exists for; a kind without one
has **no ASAPv1 drift guard**.
[The golden README's Coverage section](https://github.com/ProjectASAP/sketchlib-golden-bytes#coverage)
lists them.
The older `portable` path is guarded separately, by the `sketchlib-go` goldens
of its own listed above.

Expand Down
Loading
Loading