Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
148 changes: 95 additions & 53 deletions CHANGELOG.md

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -112,7 +112,7 @@ for &(event, true_count) in &events {
```rust
use asap_sketchlib::Kll;

// Kll is a quantile sketch — it tracks the distribution of values so you can
// KLL is a quantile sketch — it tracks the distribution of values so you can
// ask questions like "what is the median?" without storing every data point.
let mut sketch = Kll::<f64>::default();

Expand Down
2 changes: 1 addition & 1 deletion docs/advanced_use_cases.md
Original file line number Diff line number Diff line change
Expand Up @@ -86,4 +86,4 @@ sketch answers, while `MicroCm` answers frequency only. `MicroCm` is
| Sliding window, frequency only, tight memory | `MicroCm` (Experimental) |
| Sliding window + universal monitoring | `ExponentialHistogramUnivMon` (Unstable) |

API references: [`docs/api/api_exponential_histogram.md`](./api/api_exponential_histogram.md), [`docs/api/api_ehsketchlist.md`](./api/api_ehsketchlist.md), [`docs/api/api_microscope.md`](./api/api_microscope.md)
API references: [`docs/api/api_exponential_histogram.md`](./api/api_exponential_histogram.md), [`docs/api/api_eh_bucket_sketch.md`](./api/api_eh_bucket_sketch.md), [`docs/api/api_microscope.md`](./api/api_microscope.md)
38 changes: 19 additions & 19 deletions docs/api/api_coco.md → docs/api/api_coco_sketch.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# API: Coco
# API: CocoSketch

Status: `Ready`

Expand All @@ -18,32 +18,32 @@ subset aggregation.

```rust
fn new() -> Self
fn init_with_size(w: usize, d: usize) -> Self
fn init_with_size_and_seed(w: usize, d: usize, seed: u64) -> Self
fn with_dimensions(rows: usize, cols: usize) -> Self
fn with_dimensions_and_seed(rows: usize, cols: usize, seed: u64) -> Self
```

`new()` builds a `1024 x 4` table: 4096 buckets, 128 KiB, plus a heap allocation
`new()` builds a `4 x 1024` table: 4096 buckets, 128 KiB, plus a heap allocation
per occupied bucket for its stored key.

`new` and `init_with_size` draw the tie-break and the election from the thread
generator. `init_with_size_and_seed` draws them from a generator seeded with
`new` and `with_dimensions` draw the tie-break and the election from the thread
generator. `with_dimensions_and_seed` draws them from a generator seeded with
`seed`, so the table is a deterministic function of the seed and the insert
sequence; a clone copies the generator's state along with the table.

Sizing: `d` is the number of arrays an insert scans, and the paper recommends
2 to 4 — larger `d` spreads small flows better at the cost of a longer scan per
packet. `w` is the buckets per array, and it is the parameter to raise: bucket
count drives the collision rate, and the sketch can attribute mass to at most
`w * d` distinct keys at once, so size it against the number of heavy flows you
expect to resolve.
Sizing: `rows` is the number of arrays an update scans (the paper's `d`), and the
paper recommends 2 to 4 — more rows spread small flows better at the cost of a
longer scan per packet. `cols` is the buckets per array (the paper's `w`), and it
is the parameter to raise: bucket count drives the collision rate, and the
sketch can attribute mass to at most `rows * cols` distinct keys at once, so size
it against the number of heavy flows you expect to resolve.

## Insert/Update

```rust
fn update_weighted(&mut self, key: &str, weight: u64)
```

The SIGCOMM '21 update: the `d` mapped buckets are scanned for `key` first and a
The SIGCOMM '21 update: the `rows` mapped buckets are scanned for `key` first and a
match absorbs `v` directly; otherwise the whole increment lands in the smallest
of them and that bucket's key is replaced with `key` with probability `v / val`.
When several buckets share that smallest value the paper draws one of them
Expand All @@ -69,19 +69,19 @@ leaves a key in at most one bucket, so no key comes back twice. Step 4 is
`SELECT g(k_F), SUM(Size) ... GROUP BY g(k_F)`, which is `group_by(g)`.

Reach for `group_by` when you want the whole result table: it folds every
recorded flow onto its partial key in one `O(w * d)` pass, where asking
recorded flow onto its partial key in one `O(rows * cols)` pass, where asking
`estimate_projected` for each partial key separately costs one such pass per
group. A single known partial key is still cheaper through `estimate_projected`,
which allocates nothing.

`estimate` is the paper's point query: the sum of the `d` mapped buckets
that currently hold `key`, in `O(d)`. Because every increment is attributed to
`estimate` is the paper's point query: the sum of the `rows` mapped buckets
that currently hold `key`, in `O(rows)`. Because every increment is attributed to
exactly one bucket, summing it over the observed keys returns the total inserted
mass.

`estimate_projected` is the paper's partial-key query. Section 4.3 defines it as
`SELECT g(k_F), SUM(Size) ... GROUP BY g(k_F)`, where `g` maps a full key to a
partial key; `project` is that `g`. The table is scanned in `O(w * d)` and a
partial key; `project` is that `g`. The table is scanned in `O(rows * cols)` and a
bucket counts only when its stored full key projects exactly to `partial_key`.

`estimate_with_udf` is the same scan behind a general predicate over the
Expand Down Expand Up @@ -124,7 +124,7 @@ map every key to the same bucket. A wider table is refused on both sides.
```rust
use asap_sketchlib::CocoSketch;

let mut sk = CocoSketch::init_with_size(1024, 4);
let mut sk = CocoSketch::with_dimensions(4, 1024);
sk.update_weighted("region=us|id=1", 3);
let _ = sk.estimate("region=us|id=1");
let _ = sk.estimate_projected("region=us", |full| {
Expand All @@ -146,7 +146,7 @@ let _ = sk.estimate_projected("region=us", |full| {
> **Feature gate:** requires `octo-runtime` for the runtime itself.

```rust
let plan = CocoOctoPlan::new(1024, 2); // or CocoOctoPlan::with_seed(1024, 2, seed)
let plan = CocoOctoPlan::new(2, 1024); // or CocoOctoPlan::with_seed(2, 1024, seed)
let config = OctoConfig { threshold: plan.threshold().clone(), ..OctoConfig::default() };
let out = run_octo(&inputs, &config, plan.clone(), || plan.aggregator());
out.parent.sketch.estimate("flow::7");
Expand Down
5 changes: 1 addition & 4 deletions docs/api/api_common.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,6 @@ src/common/
├── structure_utils.rs # Nitro and median utility
├── precompute_hash.rs # PRECOMPUTED_HASH
├── precompute_sample.rs # LN_ONE_MINUS_U_TABLE
├── precompute_sample2.rs # PRECOMPUTED_SAMPLE_RATE_1PERCENT
└── structures/
├── vector1d.rs
├── vector2d.rs
Expand All @@ -37,8 +36,7 @@ src/common/
```rust
use asap_sketchlib::common::{
// Input layer
DataInput, OwnedDataInput, HeapEntry, L2HH,
input_to_owned, heap_item_to_sketch_input,
DataInput, OwnedDataInput, HeapEntry,

// Hash layer
SketchHasher, DefaultXxHasher,
Expand Down Expand Up @@ -69,7 +67,6 @@ use asap_sketchlib::common::{
// Precomputed assets
PRECOMPUTED_HASH,
LN_ONE_MINUS_U_TABLE,
PRECOMPUTED_SAMPLE_RATE_1PERCENT,
};
```

Expand Down
2 changes: 1 addition & 1 deletion docs/api/api_common_heap.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ A capacity-bounded min-heap of `HeapEntry<C>` beside an index from key digest to
heap position, so a keyed update is a lookup and a sift rather than a scan. The
index is patched through each sift: `update` costs one hash, one map probe and
`O(log k)` index writes, and its rate does not fall as `k` grows. See
[TopKHeap acceleration](../hhheap_acceleration.md) for the measurements.
[TopKHeap acceleration](../topk_heap_acceleration.md) for the measurements.

## Constructors

Expand Down
26 changes: 8 additions & 18 deletions docs/api/api_common_input.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,30 +15,26 @@ Canonical input and ownership model shared across sketches/frameworks.
- `HeapEntry<C: HeapCount = i64>`
- `HydraQuery<'a>`
- `HydraCounter`
- `L2HH`

`HydraQuery` and `HydraCounter` are defined in `asap_sketchlib::common::input`.

## Constructors / Conversions

```rust
fn heap_item_to_sketch_input(item: &OwnedDataInput) -> DataInput<'_>
fn input_to_owned(input: &DataInput<'_>) -> OwnedDataInput
// OwnedDataInput
fn as_data_input(&self) -> DataInput<'_>

// DataInput
fn to_owned_input(&self) -> OwnedDataInput

// HeapEntry<C>
fn from_input(k: DataInput, count: C) -> Self
fn new(key: OwnedDataInput, count: C) -> Self
fn init_item(k: DataInput, count: C) -> Self
```

## Insert/Update

```rust
// L2HH
fn update_and_est(&mut self, key: &DataInput, value: i64) -> f64
fn update_and_est_without_l2(&mut self, key: &DataInput, value: i64) -> f64
fn clear(&mut self)

// HydraCounter
fn update(&mut self, value: &DataInput, weight: Option<i32>)
fn update_with_hash(
Expand All @@ -52,26 +48,20 @@ fn update_with_hash(
## Query

```rust
// L2HH
fn l2(&self) -> f64

// HydraCounter
fn query(&self, query: &HydraQuery) -> Result<f64, String>
```

## Merge

```rust
// L2HH
fn merge(&mut self, other: &L2HH)

// HydraCounter
fn merge(&mut self, other: &HydraCounter) -> Result<(), String>
```

## Serialization

- `DataInput`, `OwnedDataInput`, `HeapEntry`, `HydraCounter`, and `L2HH` derive serde traits.
- `DataInput`, `OwnedDataInput`, `HeapEntry`, and `HydraCounter` derive serde traits.
- `HydraQuery` is query-only and does not derive serde in current code.

## Variants (Core)
Expand Down Expand Up @@ -116,14 +106,14 @@ follow the bytes: `-0.0 == 0.0` and `NaN == NaN`.
## Examples

```rust
use asap_sketchlib::{DataInput, input_to_owned};
use asap_sketchlib::DataInput;
use asap_sketchlib::common::input::HydraQuery;

let q = HydraQuery::Frequency(DataInput::Str("key"));
let _ = q;

let key = DataInput::String("flow".to_string());
let _owned = input_to_owned(&key);
let _owned = key.to_owned_input();
```

## See Also
Expand Down
1 change: 1 addition & 0 deletions docs/api/api_common_structures.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@ Shared matrix, vector, and bit storage plus utility structures used by sketch im
```rust
// Vector2D
fn unfilled(rows: usize, cols: usize) -> Self
fn filled(rows: usize, cols: usize, value: T) -> Self
fn from_fn<F>(rows: usize, cols: usize, f: F) -> Self

// Vector3D
Expand Down
File renamed without changes.
File renamed without changes.
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ Every update reaches both; a top-k query ranks the summary's monitored keys by
their Count-Min estimate. The summary picks the candidates and the matrix gives
each its frequency. The summary's `capacity` is independent of `k`.

[`CountMinSketchWithHeap`](./api_cms_heap.md) keeps an estimate heap instead: a key's count is
[`CountMinSketchWithHeap`](./api_count_min_sketch_with_heap.md) keeps an estimate heap instead: a key's count is
its matrix estimate and a newcomer enters only by beating the heap minimum.

## Type/Struct
Expand Down
3 changes: 1 addition & 2 deletions docs/api/api_count_sketch.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# API: Count
# API: CountSketch

Status: `Ready`

Expand Down Expand Up @@ -47,7 +47,6 @@ fn estimate_with_hash(&self, hashed_val: &H::HashType) -> f64
fn estimate(&self, val: &DataInput) -> f64
fn estimate_with_hash(&self, hashed_val: u128) -> f64
fn update_weighted_and_estimate(&mut self, val: &DataInput, weight: i64) -> f64
fn fast_update_and_est_without_l2(&mut self, val: &DataInput, c: i64) -> f64
fn l2(&self) -> f64
fn l2_squared(&self) -> f64
```
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -131,7 +131,7 @@ assert!(sk.estimate(&DataInput::Str("flow")) >= 1.0);
- Estimate semantics follow Count Sketch and may be non-integer.
- Merge requires matching dimensions and compatible type parameters.
- **Over integer counters, heap counts are a saturating, truncating view of the
estimate.** The insert path stores `heap_count_from_estimate(estimate)`, where the
estimate.** The insert path stores `CountSketchHeapCounter::heap_count(estimate)`, where the
estimate is the row median as `f64` and the heap holds `i64`. The conversion
saturates at `i64::MAX` / `i64::MIN` (and maps `NaN` to `0`) rather than
wrapping, because a wrapped negative count would corrupt the heap's ordering.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ their Count Sketch estimate (row median). The summary picks the candidates and
the matrix gives each its frequency. The summary's `capacity` is independent
of `k`.

[`CountSketchWithHeap`](./api_cs_heap.md) keeps an estimate heap instead.
[`CountSketchWithHeap`](./api_count_sketch_with_heap.md) keeps an estimate heap instead.

## Type/Struct

Expand Down Expand Up @@ -69,7 +69,7 @@ fn from_asapv1(bytes: &[u8]) -> Result<Self, RmpDecodeError>
The **ASAPv1** envelope, kind `0x1f 0x00` (spec §3.25), for
`CountSketchWithSpaceSaving<Vector2D<T>, Mode, H>` with `T` `i32`, `i64` or `f64`, `Mode`
`FastPath` or `RegularPath`, and `H: HashProfile`. Same layout as
[CountMinSketchWithSpaceSaving](./api_cms_space_saving.md). There is no delta kind.
[CountMinSketchWithSpaceSaving](./api_count_min_sketch_with_space_saving.md). There is no delta kind.

## Examples

Expand Down
38 changes: 19 additions & 19 deletions docs/api/api_ehsketchlist.md → docs/api/api_eh_bucket_sketch.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,17 +10,17 @@ Unified enum wrapper for sketch payloads used by EH-style frameworks.

```rust
enum EhBucketSketch {
CM(CountMinSketch<Vector2D<i32>, FastPath>),
COCO(CocoSketch),
COUNTL2HH(CountSketchWithL2Norm),
CS(CountSketch<Vector2D<i32>, FastPath>),
DDS(DdSketch),
ELASTIC(ElasticSketch),
HLL(HyperLogLog<Ertl>),
CountMinSketch(CountMinSketch<Vector2D<i32>, FastPath>),
CocoSketch(CocoSketch),
CountSketchWithL2Norm(CountSketchWithL2Norm),
CountSketch(CountSketch<Vector2D<i32>, FastPath>),
DdSketch(DdSketch),
ElasticSketch(ElasticSketch),
Hll(HyperLogLog<Ertl>),
Kll(Kll),
#[cfg(feature = "experimental")]
UNIFORM(UniformSampling),
UNIVMON(UnivMon),
Uniform(UniformSampling),
UnivMon(UnivMon),
}
```

Expand Down Expand Up @@ -67,18 +67,18 @@ The ten nested kind_ids:

| Variant | Nested kind_id | Registry name | Feature |
| ------- | -------------- | ------------- | ------- |
| `CM` | `0x02 0x00` | Count-Min | default |
| `COCO` | `0x0c 0x00` | Coco | default |
| `COUNTL2HH` | `0x19 0x00` | CountSketchWithL2Norm | default |
| `CS` | `0x04 0x00` | Count Sketch | default |
| `DDS` | `0x05 0x00` | DDSketch | default |
| `ELASTIC` | `0x0b 0x00` | Elastic | default |
| `HLL` | `0x01 0x02` | HLL Ertl-MLE | default |
| `CountMinSketch` | `0x02 0x00` | Count-Min | default |
| `CocoSketch` | `0x0c 0x00` | Coco | default |
| `CountSketchWithL2Norm` | `0x19 0x00` | CountSketchWithL2Norm | default |
| `CountSketch` | `0x04 0x00` | Count Sketch | default |
| `DdSketch` | `0x05 0x00` | DDSketch | default |
| `ElasticSketch` | `0x0b 0x00` | Elastic | default |
| `Hll` | `0x01 0x02` | HLL Ertl-MLE | default |
| `Kll` | `0x06 0x00` | KLL compact | default |
| `UNIFORM` | `0x0d 0x00` | UniformSampling | `experimental` |
| `UNIVMON` | `0x10 0x00` | UnivMon | default |
| `Uniform` | `0x0d 0x00` | UniformSampling | `experimental` |
| `UnivMon` | `0x10 0x00` | UnivMon | default |

Each id is pinned to one algorithm: `HLL` is Ertl-MLE, so `0x01 0x01` (Classic)
Each id is pinned to one algorithm: `Hll` is Ertl-MLE, so `0x01 0x01` (Classic)
and `0x01 0x03` (HIP) are rejected, and `Kll` is compact, so `0x06 0x01`
(dynamic) is rejected. The dispatch is the same in every build: a decoder built
without `experimental` rejects `0x0d 0x00` with an error naming the variant and
Expand Down
2 changes: 1 addition & 1 deletion docs/api/api_elastic.md → docs/api/api_elastic_sketch.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# API: Elastic
# API: ElasticSketch

Status: `Ready`

Expand Down
4 changes: 2 additions & 2 deletions docs/api/api_hashlayer.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ Status: `Ready`

Group multiple compatible sketches that share a hash layout so the hash is
computed once per insert and fanned out to every sketch in the ensemble.
Supports both frequency sketches (CountMinSketch, Count) and cardinality sketches
Supports both frequency sketches (CountMinSketch, CountSketch) and cardinality sketches
(HyperLogLog variants).

### Quick Example
Expand Down Expand Up @@ -79,7 +79,7 @@ Sketches are converted into `EnsembleSketch` via `From` impls, so you can use `.

```rust
use asap_sketchlib::{
CountMinSketch, Count, FastPath, Vector2D, DefaultXxHasher,
CountMinSketch, CountSketch, FastPath, Vector2D, DefaultXxHasher,
HyperLogLog, Ertl,
};
use asap_sketchlib::sketch_framework::hashlayer::{EnsembleSketch, HashSketchEnsemble};
Expand Down
2 changes: 2 additions & 0 deletions docs/api/api_kll.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,8 @@ fn with_k_and_seed(k: i32, seed: u64) -> Self
fn with_seed(k: usize, m: usize, seed: u64) -> Self
```

`with_k` and `with_k_and_seed` set `m` to `DEFAULT_M` (8).

The unseeded constructors draw the compaction coin from the wall clock, so two
runs over the same input produce different sketches. The `*_with_seed` forms
fix the coin and store the seed, so `clear()` re-seeds from it too — use them
Expand Down
2 changes: 1 addition & 1 deletion docs/api/api_microscope.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# API: MicroCM (MicroscopeSketch)
# API: MicroCm (Microscope sketch)

Status: `Experimental`

Expand Down
8 changes: 5 additions & 3 deletions docs/api/api_octo.md
Original file line number Diff line number Diff line change
Expand Up @@ -432,8 +432,8 @@ copy, made once at the hand-off.
CocoSketch and Elastic key on a `String`, so their plans render the input
with `flow_key_string`, which renders every `DataInput` variant: numbers by
`to_string`, bytes as lowercase hex. `EhBucketSketch` converts differently —
its `ELASTIC` arm renders bytes with `String::from_utf8_lossy`, and its
`COCO` arm drops every input that is not `Str` or `String`. The
its `ElasticSketch` arm renders bytes with `String::from_utf8_lossy`, and its
`CocoSketch` arm drops every input that is not `Str` or `String`. The
`flow_key_string` rendering is what the aggregator stores, so it is what
`CocoSketch::estimate` or `ElasticSketch::estimate` must be asked for. Their payloads
carry no hashes: the heavy half of an Elastic insert can evict a key the
Expand Down Expand Up @@ -542,7 +542,9 @@ heap, which is Algorithm 2.
The two keyed-bucket pairs instead replay the key, per §4.4.
`CocoOctoAggregator` calls the weighted `CocoSketch::update_weighted`, so the promoted
mass contests the parent's own buckets and runs the parent's own `v/val`
election. Both halves of the Coco pair draw from the thread generator;
election. `CocoOctoWorker`, `CocoOctoAggregator` and `CocoOctoPlan` take
`(rows, cols)`, as `CocoSketch::with_dimensions` does. Both halves of the Coco
pair draw from the thread generator;
`CocoSketchWorker::with_seed`, `CocoOctoWorker::with_threshold_and_seed`,
`CocoOctoAggregator::with_seed` and `CocoOctoPlan::with_seed` /
`with_threshold_and_seed` draw from a seeded one instead. A seeded plan
Expand Down
Loading
Loading