Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 27 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -66,6 +66,9 @@ signals a backwards-compatible change.
worker's seed and the aggregator's from one seed. The unseeded constructors
draw from the thread generator. The seed is not serialized, and the ASAPv1
payload is unchanged.
- **ASAPv1 golden fixtures for `KLLDynamic`** (kind_id `0x06 0x01`):
`kll_dynamic_f64_k200` and `kll_dynamic_i64_k200`, the i64 items crossing
every msgpack integer width in both signs, checked in `tests/asapv1_golden.rs`.

### Changed

Expand All @@ -88,6 +91,30 @@ signals a backwards-compatible change.
as an empty window like any other, and that window takes a place in
`max_windows`.

### Removed

- **KLL's proto and portable formats.** `KLL` and `KLLDynamic` serialize only
as ASAPv1 (`serialize_to_bytes` / `deserialize_from_bytes`, kind_ids
`0x06 0x00` / `0x06 0x01`). Removed:
- the `kll` field (13) of `SketchEnvelope` (now reserved) and with it
`sketch_envelope::SketchState::Kll`; `KLLState` stays in
`proto/kll/kll.proto` only as `HydraCell.kll`;
- the crate-root `KllSketch` and `KllSketchData`, and the public module
`message_pack_format::portable::kll`: `SketchlibKll`, `new_sketchlib_kll`,
`new_sketchlib_kll_with_seed`, `sketchlib_kll_update`,
`sketchlib_kll_quantile`, `sketchlib_kll_merge`,
`bytes_from_sketchlib_kll`, `sketchlib_kll_from_bytes`,
`KllSketch::{merge_refs, aggregate_kll, from_portable_state}`, the
`MessagePackCodec` impls for `KllSketch` and `KllSketchData`, the
value-offset codec (`KLL_SCALE_SWEEP`, `encode_value_offset`,
`decode_value_offset`, `KllProtoItems`), and the `unsafe impl Send/Sync
for KllSketch`. `HydraKllSketch`'s per-cell type stays reachable through
the public fields `HydraKllSketch.sketch` and
`HydraKllSketchWire.sketches`, but can no longer be named;
- `message_pack_format::native::{kll, kll_dynamic}`, the `MessagePackCodec`
impls for `KLL<T>` and `KLLDynamic<T>`;
- `KLL::from_portable_state`.

### Fixed

- `KLLDynamic` panicked on every quantile query once a NaN had entered the
Expand Down
10 changes: 5 additions & 5 deletions docs/asapv1_wire_format.md
Original file line number Diff line number Diff line change
Expand Up @@ -458,7 +458,7 @@ Both KLL variants (the compact fixed-buffer `KLL` and the growable `KLLDynamic`)

`num_levels` is `levels.len() - 1` (derived, so not stored). The `coin` is the randomized-compaction RNG; it is carried so a decoded sketch can keep compacting deterministically (a query-only consumer may ignore it). It is a nested 3-element array mirroring `sketchlib-go`'s `CoinState`.

**Item order (cross-language contract).** `levels` / `items` use the **top-most-level-first** layout, byte-for-byte matching `sketchlib-go`'s `KLLState`: index `i` in `levels` maps to compactor level `num_levels - 1 - i`, and level 0's run is in **input order**. The compact `KLL` grows its buffer leftward and stores level 0 reverse-input, so its encoder reverses level 0 back to input order (and its decoder reverses it in); `KLLDynamic` already stores this layout natively. Within a level, order past the first compaction is not guaranteed byte-identical across the two Rust variants (or across languages), but the retained set and quantiles agree — see the caveat on `KLL::wire_items`.
**Item order (cross-language contract).** `levels` / `items` use the **top-most-level-first** layout, byte-for-byte matching `sketchlib-go`: index `i` in `levels` maps to compactor level `num_levels - 1 - i`, and level 0's run is in **input order**. The compact `KLL` grows its buffer leftward and stores level 0 reverse-input, so its encoder reverses level 0 back to input order (and its decoder reverses it in); `KLLDynamic` already stores this layout natively. Within a level, order past the first compaction is not guaranteed byte-identical across the two Rust variants (or across languages), but the retained set and quantiles agree — see the caveat on `KLL::wire_items`.

**Wire widths.** `levels` is `u32` on the wire, widened to `usize` in memory. The coin's `remaining_bits` is `u32` on the wire but `u8` in memory, so rule 8 below is what makes the narrowing safe; `state` and `bit_cache` are `u64` on both sides.

Expand Down Expand Up @@ -1341,14 +1341,14 @@ Good direction (more compact, higher fidelity, less Rust-internal duplication),
2. **Golden byte-vector fixtures** in one shared repo, [`sketchlib-golden-bytes`](https://github.com/ProjectASAP/sketchlib-golden-bytes), which each implementation mounts at `asapv1_golden/`; both languages decode and re-encode them byte-identically. These replace the `portable`-as-oracle round-trip test.
3. **This registry**, mirrored, never independently allocated.

Fixtures exist for six `kind_id`s — HLL's three estimators, Count-Min, Count Sketch and compact KLL — and `sketchlib-go` mirrors those. Every other kind here has a payload and **no fixture**, so this document is its only contract; `asapv1_golden/README.md` lists the gap. `portable` carries its own Go goldens.
Fixtures exist for seven `kind_id`s — HLL's three estimators, Count-Min, Count Sketch and both KLL variants — and `sketchlib-go` mirrors those. Every other kind here has a payload and **no fixture**, so this document is its only contract; `asapv1_golden/README.md` lists the gap. `portable` carries its own Go goldens.

**Hash profile on the Go side.**
Rust derives the hash spec from a generic `HashProfile` bound on the hasher type; Go has no generic hasher type, so there is nothing to derive from.
On the Go side the profile is simply **written into** the metadata on encode and **read from** it on decode.
Go MUST validate the profile it reads (same fail-closed intent as Rust): a sketch is only mergeable/queryable if its `hash_profile_id` + seeds match the profile Go is prepared to reproduce.

Sequencing: (2) covers six `kind_id`s, and the one `native bytes == portable bytes` test is HLL-only, since no other `portable` type emits ASAPv1 bytes.
Sequencing: (2) covers seven `kind_id`s, and the one `native bytes == portable bytes` test is HLL-only, since no other `portable` type emits ASAPv1 bytes.
Retire `portable` once the fixtures cover the rest.

---
Expand All @@ -1360,8 +1360,8 @@ Retire `portable` once the fixtures cover the rest.
- **Q-SEEDS**: `seed_list` is **inlined** in v1 so the bytes self-describe the hash. Resolving seeds from `hash_profile_id` alone is a v2 space optimization. Each sketch still carries only the seed *index* it uses.
- **Q-PROFILE**: the hash-spec metadata is **derived from the hasher's `HashProfile`** (`hll_metadata::<H>` / `cms_metadata::<H>`) and never hardcoded, so it is always truthful to the hasher. Custom hash profiles are **supported and self-describing**. Merge compatibility is hash-spec equality, so a custom-profile sketch is not mergeable with a standard one.
- **Q-CMS**: Count-Min is one `kind_id` (`0x02 0x00`); counter type and mode live in the metadata, so the id stays single. The counter-type domain is `"i32"`, `"i64"` and `"f64"`, with no `i128` (no msgpack integer form); `i32` is recorded at its own width and pinned on decode.
- **Q-KLL**: KLL metadata carries **no hash-spec group** — KLL is comparison-based and never hashes, so those fields have no truthful value. Its metadata is structural-only (`metadata_version`, `k`, `m`, `item_type`, optional `seed`) and is *not* `HashProfile`-derived. The two KLL variants (compact `0x06 0x00`, dynamic `0x06 0x01`) share one payload `[levels, items, coin]` and differ only by `kind_id`. `item_type` (`"f64"`/`"i64"`) is a metadata param, not a separate `kind_id` (mirrors Q-CMS's `counter_type`). Retained samples use the top-most-level-first layout that matches `sketchlib-go`'s `KLLState`.
- **Q-KLL-SEED**: KLL records its reproducible compaction `seed` as an **optional** metadata key. It is construction config (so metadata, not payload, per the config→metadata rule), and it is the first optional key in v1: present only when the sketch carries a seed, omitted otherwise. Rationale: the payload's `coin` already carries the RNG's *current* position (enough to resume compaction), but `seed` is what a later `clear()` re-seeds from — so serializing it lets a decoded sketch keep `clear()`-reproducibility instead of falling back to wall-clock. Cost of omitting it is bounded (only a decoded-then-`clear()`ed sketch loses cross-run byte reproducibility — never correctness), but it is cheap to carry and future-proofs the checkpoint/restore path. `KLLDynamic` has no seed concept and never emits the key; the two variants are deliberately **not** forced to be symmetric here. Go carries and preserves the key without interpreting it.
- **Q-KLL**: KLL metadata carries **no hash-spec group** — KLL is comparison-based and never hashes, so those fields have no truthful value. Its metadata is structural-only (`metadata_version`, `k`, `m`, `item_type`, optional `seed`) and is *not* `HashProfile`-derived. The two KLL variants (compact `0x06 0x00`, dynamic `0x06 0x01`) share one payload `[levels, items, coin]` and differ only by `kind_id`. `item_type` (`"f64"`/`"i64"`) is a metadata param, not a separate `kind_id` (mirrors Q-CMS's `counter_type`). Retained samples use the top-most-level-first layout that matches `sketchlib-go`.
- **Q-KLL-SEED**: KLL records its reproducible compaction `seed` as an **optional** metadata key. It is construction config (so metadata, not payload, per the config→metadata rule), and it is the first optional key in v1: present only when the sketch carries a seed, omitted otherwise. Rationale: the payload's `coin` already carries the RNG's *current* position (enough to resume compaction), but `seed` is what a later `clear()` re-seeds from — so serializing it lets a decoded sketch keep `clear()`-reproducibility instead of falling back to wall-clock. Cost of omitting it is bounded (only a decoded-then-`clear()`ed sketch loses cross-run byte reproducibility — never correctness), but it is cheap to carry and future-proofs the checkpoint/restore path. `KLLDynamic` never emits the key, even when built with a seed; the two variants are deliberately **not** forced to be symmetric here. Go carries and preserves the key without interpreting it.
- **Q-CS**: Count Sketch is one `kind_id` (`0x04 0x00`) and mirrors Count-Min's metadata and `[counts]` payload, including `counter_type`. Its type domain differs: `"i32"` and `"i64"`, with no `"f64"` — not because a float fails a bound, but because **no `CountSketchCounter` impl exists for it**: the trait covers `i32`, `i64` and `i128` and nothing else, so Count-Min's `f64` has no Count Sketch counterpart to serialize — and no `i128` (no msgpack integer form), which leaves the two `CsWireCounter` covers. Structural-param order matches Q-CMS: `... matrix_seed_index, rows, cols, counter_type, mode`. **`i32` is not widened to `i64`** — unlike Count-Min, whose counters are plain numbers, a Count Sketch appears *nested* inside `HydraCounter` and `EHSketchList` as `Vector2D<i32>` and must decode back into that variant, so the width is identity and is recorded exactly. Cells are signed, so a decoder must not assume monotonicity.
- **Q-CMS-DIMS**: Count-Min `rows`/`cols` are **metadata** and the payload omits them. They are configuration that shapes the payload (like HLL's `precision`), so per the config-to-metadata rule they belong in the descriptor. The payload is then just `[counts]`. Canonical structural-param order: `... matrix_seed_index, rows, cols, counter_type, mode`.
- **Q-VER**: no payload version field. A new incompatible encoding gets a **new `kind_id`**; retired ids are reserved forever and never recycled.
Expand Down
4 changes: 2 additions & 2 deletions docs/e2e_testing_harness.md
Original file line number Diff line number Diff line change
Expand Up @@ -325,8 +325,8 @@ error against its bound.

- No assertion depends on unseeded randomness — no `rand::rng()`, no
wall-clock seeding, no implicit RNG inside a constructor. An unseeded
sketch appears only where the assertion holds for every draw. `KLL::init_kll_with_seed`,
`KLLDynamic::init_kll_with_seed`, `KllSketch::with_seed`,
sketch appears only where the assertion holds for every draw.
`KLL::init_kll_with_seed`, `KLLDynamic::init_kll_with_seed`,
`NitroBatch::with_target_and_seed`, `UniformSampling::with_seed`,
`Coco::init_with_size_and_seed` and `CocoOctoPlan::with_seed` exist for this.
- An estimator whose guarantee is unbiasedness gets no per-key band: its
Expand Down
2 changes: 1 addition & 1 deletion docs/features.md
Original file line number Diff line number Diff line change
Expand Up @@ -112,7 +112,7 @@ MessagePack (`rmp-serde`) support. **serde support** means the type derives `Ser
| EHSketchList | Yes | Yes |
| ExponentialHistogram | No | Yes |

Protobuf (prost): `.proto` definitions exist for CountMin, Count, HLL, DDSketch, KLL, Elastic, Coco, Hydra, and UnivMon. Rust conversion code is in progress.
Protobuf (prost): `.proto` definitions exist for CountMin, Count, HLL, DDSketch, Elastic, Coco, Hydra, and UnivMon. Rust conversion code is in progress.

### API Stability

Expand Down
10 changes: 5 additions & 5 deletions docs/message_pack_format.md
Original file line number Diff line number Diff line change
Expand Up @@ -150,8 +150,8 @@ being **phased out** in favor of the per-sketch `wire.rs` + the shared
`envelope.rs`.

- `portable/` holds per-sketch wire types that predate the envelope. They
carry their own byte-parity goldens against `sketchlib-go` — HLL, Count-Min,
Count Sketch and KLL (both directions) — separate from `asapv1_golden/`.
carry their own byte-parity goldens against `sketchlib-go` — HLL, Count-Min
and Count Sketch — separate from `asapv1_golden/`.
- `native/` is a set of thin `MessagePackCodec` shims over the sketches'
`serialize_to_bytes` / `deserialize_from_bytes`. Every type it wraps emits the
ASAPv1 envelope, so each shim is a pass-through.
Expand All @@ -169,9 +169,9 @@ decode → re-encode them byte-identically. The `kind_id` registry is mirrored
verbatim with Go's `wire/asapmsgpack/magic_ids.go`, never independently
allocated.

That proof covers the six `kind_id`s a fixture exists for — HLL's three
estimators, Count-Min, Count Sketch and compact KLL. Every other implemented
kind has **no ASAPv1 golden and therefore no ASAPv1 drift guard**;
That proof covers the seven `kind_id`s a fixture exists for — HLL's three
estimators, Count-Min, Count Sketch and both KLL variants. Every other
implemented kind has **no ASAPv1 golden and therefore no ASAPv1 drift guard**;
[its README](https://github.com/ProjectASAP/sketchlib-golden-bytes#coverage) lists what is covered.
The older `portable` path is guarded separately, by the `sketchlib-go` goldens
of its own listed above.
Expand Down
Loading
Loading