Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 19 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -66,6 +66,10 @@ signals a backwards-compatible change.
worker's seed and the aggregator's from one seed. The unseeded constructors
draw from the thread generator. The seed is not serialized, and the ASAPv1
payload is unchanged.
- **ASAPv1 golden fixtures for Hydra's five counter variants** (`0x07 0x00`
KLL, `0x07 0x01` Count-Min, `0x07 0x02` Count Sketch, `0x07 0x03` HLL,
`0x07 0x04` UnivMon), each a small grid whose cell states are set directly,
checked by `tests/asapv1_golden.rs`.

### Changed

Expand All @@ -88,6 +92,21 @@ signals a backwards-compatible change.
as an empty window like any other, and that window takes a place in
`max_windows`.

### Removed

- **Hydra's proto and portable formats.** `Hydra` serializes as ASAPv1 only
(`serialize_to_bytes` / `deserialize_from_bytes`, kinds `0x07 0x00`-`0x07
0x04`). Removed: the portable `HydraKllSketch` and `HydraKllSketchWire`
(module `message_pack_format::portable::hydra_kll`, and the crate-root
`HydraKllSketch` re-export), with `HydraKllSketch`'s `MessagePackCodec` impl,
`with_seed`, `merge_refs` and `aggregate_hydrakll`; the proto file
`hydra/hydra.proto` with `HydraState`, `HydraCell` and `HydraCounterType`
(and their generated `proto::sketchlib` types); and `SketchEnvelope`'s
`hydra` field (16, now reserved). The nearest replacement is `Hydra` with a
one-column schema and `HydraCounter::KLL`, but it routes `label:value`
subkeys through the matrix hash at `HYDRA_SEED`, not `xxh32(key, row) %
cols`, so `HydraKllSketch` cells do not map onto the new grid.

### Fixed

- `KLLDynamic` panicked on every quantile query once a NaN had entered the
Expand Down
6 changes: 2 additions & 4 deletions docs/asapv1_wire_format.md
Original file line number Diff line number Diff line change
Expand Up @@ -707,8 +707,6 @@ A Hydra is a `rows x cols` grid of counters over a fixed set of named key column

**Counters are inlined, not nested.** A cell's raw state goes straight into Hydra's positional array in the shape that counter's own section fixes; no cell carries an envelope, a magic or a metadata map of its own.

**Not this: `portable/hydra_kll.rs`.** `HydraKllSketch` is a legacy Go-interop type, not `0x07 0x00`, and it is the counter-example this rule exists to rule out: every cell of its matrix carries a **full ASAPv1 KLL envelope** — magic, version, kind_id, both length prefixes and a complete metadata map — plus `k` repeated once per cell beside it, in a msgpack shape of its own with no ASAPv1 envelope around the whole. That is exactly what Q-NEST forbids, and `0x07 0x00` does none of it: one envelope for the grid, `counter_k` / `counter_m` / `counter_item_type` carried once in its metadata, and each cell just the `[levels, items, coin]` array §3.3 fixes.

**`0x07 0x01` Count-Min counter, `0x07 0x02` Count Sketch counter** — the fixed-size matrix counters tile one array:

| Pos | Field | Type | Notes |
Expand Down Expand Up @@ -1341,14 +1339,14 @@ Good direction (more compact, higher fidelity, less Rust-internal duplication),
2. **Golden byte-vector fixtures** in one shared repo, [`sketchlib-golden-bytes`](https://github.com/ProjectASAP/sketchlib-golden-bytes), which each implementation mounts at `asapv1_golden/`; both languages decode and re-encode them byte-identically. These replace the `portable`-as-oracle round-trip test.
3. **This registry**, mirrored, never independently allocated.

Fixtures exist for six `kind_id`s — HLL's three estimators, Count-Min, Count Sketch and compact KLL — and `sketchlib-go` mirrors those. Every other kind here has a payload and **no fixture**, so this document is its only contract; `asapv1_golden/README.md` lists the gap. `portable` carries its own Go goldens.
`asapv1_golden/README.md`'s Coverage section lists which `kind_id`s have fixtures. Every other kind here has a payload and **no fixture**, so this document is its only contract. `portable` carries its own Go goldens.

**Hash profile on the Go side.**
Rust derives the hash spec from a generic `HashProfile` bound on the hasher type; Go has no generic hasher type, so there is nothing to derive from.
On the Go side the profile is simply **written into** the metadata on encode and **read from** it on decode.
Go MUST validate the profile it reads (same fail-closed intent as Rust): a sketch is only mergeable/queryable if its `hash_profile_id` + seeds match the profile Go is prepared to reproduce.

Sequencing: (2) covers six `kind_id`s, and the one `native bytes == portable bytes` test is HLL-only, since no other `portable` type emits ASAPv1 bytes.
Sequencing: (2) covers the `kind_id`s that README lists, and the one `native bytes == portable bytes` test is HLL-only, since no other `portable` type emits ASAPv1 bytes.
Retire `portable` once the fixtures cover the rest.

---
Expand Down
2 changes: 1 addition & 1 deletion docs/features.md
Original file line number Diff line number Diff line change
Expand Up @@ -112,7 +112,7 @@ MessagePack (`rmp-serde`) support. **serde support** means the type derives `Ser
| EHSketchList | Yes | Yes |
| ExponentialHistogram | No | Yes |

Protobuf (prost): `.proto` definitions exist for CountMin, Count, HLL, DDSketch, KLL, Elastic, Coco, Hydra, and UnivMon. Rust conversion code is in progress.
Protobuf (prost): `.proto` definitions exist for CountMin, Count, HLL, DDSketch, KLL, Elastic, Coco, and UnivMon. Rust conversion code is in progress.

### API Stability

Expand Down
8 changes: 4 additions & 4 deletions docs/message_pack_format.md
Original file line number Diff line number Diff line change
Expand Up @@ -169,10 +169,10 @@ decode → re-encode them byte-identically. The `kind_id` registry is mirrored
verbatim with Go's `wire/asapmsgpack/magic_ids.go`, never independently
allocated.

That proof covers the six `kind_id`s a fixture exists for — HLL's three
estimators, Count-Min, Count Sketch and compact KLL. Every other implemented
kind has **no ASAPv1 golden and therefore no ASAPv1 drift guard**;
[its README](https://github.com/ProjectASAP/sketchlib-golden-bytes#coverage) lists what is covered.
`asapv1_golden/README.md`'s
[Coverage section](https://github.com/ProjectASAP/sketchlib-golden-bytes#coverage)
lists which `kind_id`s have fixtures. Every other implemented kind has **no
ASAPv1 golden and therefore no ASAPv1 drift guard**.
The older `portable` path is guarded separately, by the `sketchlib-go` goldens
of its own listed above.

Expand Down
Loading
Loading