Skip to content

fix(univmon_optimized): unbiased distinct count and direct L2 for UnivMonPyramid #222

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
zzylol wants to merge 1 commit into from
Closed

fix(univmon_optimized): unbiased distinct count and direct L2 for UnivMonPyramid #222

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions docs/api/api_univmon_optimized.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,7 @@ fn calc_l2(&self) -> f64
fn calc_entropy(&self) -> f64
fn calc_card(&self) -> f64
fn calc_g_sum<F>(&self, g: F, is_card: bool) -> f64
fn top_layer_complete(&self) -> bool

// UnivSketchPool
fn available(&self) -> usize
Expand All @@ -56,6 +57,15 @@ For the supported non-negative update stream, `calc_l1()` returns the exact
tracked total weight rather than estimating the linear sum through the
hierarchy.

The pyramid's estimators are `UnivMon`'s (see [UnivMon](api_univmon.md)).
`calc_l2()` reads layer 0's Count Sketch directly in standard mode, so its
`(ε, δ)` bound uses layer 0's dimensions (the elephant ones whenever
`elephant_layers ≥ 1`); in terminal mode it sums the strata, each held to its
own layer's dimensions. `calc_card()` counts every heap key and, when
`top_layer_complete()` holds, misses by more than `ε F0` with probability at
most `2/(kε²)` for `k = heap_size ≥ 8`, independent of the elephant/mouse
split.

## Merge

```rust
Expand Down
8 changes: 7 additions & 1 deletion docs/tests.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -875,6 +875,12 @@ Wire tests: [`src/sketch_framework/univmon_optimized/wire.rs`](../src/sketch_fra
| `pyramid_accuracy_zipf` | Pyramid accuracy Zipf. | On heavy/medium/light Zipf-like workload, requires relative error `<15%` for `L1`, `L2`, cardinality, and entropy. |
| `pyramid_fast_insert_accuracy` | Pyramid fast insert accuracy. | Using `fast_insert` only, requires relative error `<15%` for `L1`, `L2`, cardinality, and entropy versus exact frequency map. |
| `pyramid_memory_savings_vs_uniform` | Pyramid memory savings vs uniform. | Verifies pyramid column budget is smaller than uniform baseline and computed memory savings exceed `30%`. |
| `pyramid_calc_card_counts_heap_keys_with_nonpositive_estimates` | A heap key counts toward the distinct count whatever its estimate. | In a one-layer `1x1` pyramid, inserts a `+1`-signed key with weight `5` and a `-1`-signed key with weight `1`, verifies the light key's estimate is not positive and the top layer is complete, and that `calc_card()` is `2.0`. |
| `pyramid_calc_l2_reads_layer_zero_directly_in_standard_mode` | Standard-mode L2 is layer 0's own Count Sketch estimate. | After 2000 weighted keys into `UnivMonPyramid::new(16, 2, 5, 256, 3, 64, 8)` through `insert`, verifies `calc_l2()` equals layer 0's `get_l2()` within `1e-9`. |
| `pyramid_calc_l2_sums_stratum_estimates_in_terminal_mode` | Terminal-mode L2 sums the disjoint strata. | After the same keys through `fast_insert`, verifies `calc_l2()` equals `sqrt` of the summed per-layer `get_l2_sqr()` within `1e-9`. |
| `pyramid_top_layer_complete_tracks_the_top_heap` | The distinct-count precondition is readable. | On a one-layer heap-4 pyramid, verifies `top_layer_complete()` holds after 4 keys, fails after a fifth, and holds again after `free()`. |
| `pyramid_calc_l2_meets_its_documented_bound` | `calc_l2` meets the `(ε, δ)` bound on its doc comment. | 150 seeded trials of 400 fresh random Zipf-weighted keys into `UnivMonPyramid::new(8, 1, 5, 64, 3, 16, 3)`: trials with `calc_l2()^2` off by more than `ε = 0.5` relative stay within `n·δ + 4σ + 1` for `δ = P(Bin(5, 0.125) >= 3)`. `_long` (ignored) runs 3000 trials. |
| `pyramid_calc_card_meets_its_documented_bound` | `calc_card` meets its bound when the top layer is complete. | 60 seeded trials of 2000 fresh random keys into a heap-32 pyramid with `3x256` elephant and `2x32` mouse layers, `8 = ceil(log2(2000/32)) + 2` deep: incomplete top layers stay rare, misses beyond `ε = 0.5` stay within the Chebyshev `2/(kε²)` bound, the variance stays under `2 F0²/k`, and the mean error is within four standard errors of zero. `_long` (ignored) runs 2000 trials. |
| `pyramid_round_trip_serialization` | The envelope frames a pyramid under kind_id `0x11 0x00`, and the bytes are stable across a round trip. | For `UnivMonPyramid::new(4, 2, 3, 16, 2, 8, 4)` fed four weighted string keys, verifies the bytes open with the ASAPv1 magic, a `kind_id_len` of `2`, and `0x11 0x00`, that the metadata carries that layout and `key_type` `string`, and that the decode matches on `bucket_size`, L1, L2, cardinality, entropy, and the candidate flags, re-serializing identically. |
| `pyramid_two_tier_geometry_survives` | The two tiers are derived from the layer's position. | Verifies every decoded layer keeps its tier's dimensions - `3x16` for the two elephant layers and `2x8` for the mouse layers - its own seed index, and its counter array. |
| `pyramid_layers_with_different_heap_loads_round_trip` | Each layer's heap contents survive the round trip. | For `UnivMonPyramid::new(8, 2, 3, 32, 2, 16, 4)` fed 40 weighted keys, verifies the layers carry different heap loads, that every layer's length and per-key counts match after a decode, and that the bytes re-serialize identically. |
Expand Down Expand Up @@ -1858,7 +1864,7 @@ Test file: [`tests/e2e/frameworks.rs`](../tests/e2e/frameworks.rs)
| test_name | test_description | what_is_tested |
| --- | --- | --- |
| `univmon_weighted_metrics_and_fast_insert_parity` | UnivMon's weighted L1 is exact and its L2, entropy and cardinality hold their bands, with `fast_insert` tracking the standard path. | Drives two `UnivMon::init_univmon(32, 5, 2048, 8)` sketches over `zipf_u64(20_000, 1000, 1.2, 4003)` with weight `1 + i % 7`, one through `insert` and one through `fast_insert`, then asserts both `calc_l1()` equal the exact weighted total with `assert_eq!`; `calc_l2()` inside `univmon_l2_band(exact, 5, 2048)` (`b = sqrt(2*3/2048)`, about `0.054`); `calc_entropy()` in `[0.88, 1.12]` times the exact bits; `calc_card()` in `[0.94, 1.06]` times the exact distinct count; and the fast path's `calc_l2()` in `[0.85, 1.15]` times exact. |
| `univmon_pyramid_weighted_metrics` | UnivMonPyramid's weighted L1 is exact and its L2 and cardinality hold their bands. | Drives `UnivMonPyramid::with_defaults()` over `zipf_u64(15_000, 800, 1.3, 4004)` with weight `1 + i % 3` and asserts `calc_l1()` equals the exact weighted total with `assert_eq!`, `calc_l2()` is in `[0.85, 1.15]` times exact, and `calc_card()` in `[0.80, 1.20]` times the exact distinct count; entropy is not checked. |
| `univmon_pyramid_weighted_metrics` | UnivMonPyramid's weighted L1 is exact and its L2 and cardinality hold their bands. | Drives `UnivMonPyramid::new(128, 8, 3, 2048, 3, 512, 16)` (the default tiers with heap 128, so the 20% band is over 1.6 of `calc_card`'s documented standard deviations) over `zipf_u64(15_000, 800, 1.3, 4004)` with weight `1 + i % 3` and asserts `calc_l1()` equals the exact weighted total with `assert_eq!`, `calc_l2()` is in `[0.85, 1.15]` times exact, that `top_layer_complete()` holds, and `calc_card()` in `[0.80, 1.20]` times the exact distinct count; entropy is not checked. |
| `univmon_generic_g_sum_reproduces_every_named_estimator` | The generic g-sum reproduces the cardinality and entropy estimators and its own heuristic delegate, and tracks L2. | Over `UnivMon::init_univmon(32, 5, 2_048, 8)` fed `zipf_u64(20_000, 1_000, 1.2, 4_201)` with weight `1 + i % 7`, `assert_eq!`s `calc_l2()` against layer 0's `get_l2()` (the L2 estimator reads layer 0 directly), requires `calc_g_sum(\|x\| x*x, false).sqrt()` within `[0.85, 1.15]` of the exact L2, `assert_eq!`s `calc_g_sum(\|_\| 1.0, true)` against `calc_card()`, `calc_g_sum(x^2, false)` against `calc_g_sum_heuristic(x^2, false)`, and `calc_g_sum(\|_\| 0.0, false)` against `0.0`; requires `calc_entropy()` to equal `log2(L1) - gsum(x*log2 x)/L1` within `1e-9`; and checks `calc_g_sum(\|x\| x, false)` only against `[0.50, 1.50]` times the exact L1, so the identity g-sum's agreement with L1 is not pinned tightly. |
| `univmon_g_sum_is_zero_on_an_empty_sketch` | An empty UnivMon reports zero for every estimator. | On a fresh `UnivMon::init_univmon(32, 5, 256, 4)`, `assert_eq!`s `calc_l1()`, `calc_entropy()`, `calc_g_sum(\|x\| x*x, false)` and `calc_card()` each against `0.0`. |
| `a_univmon_pool_recycles_a_returned_sketch_and_hands_back_a_cleared_one` | A returned sketch becomes available again and comes back cleared without a new allocation. | On `UnivSketchPool::new(2, 16, 3, 256, 4)`: asserts `available() == 2` and `total_allocated() == 2` when fresh, `available() == 0` after two `take()`s with `total_allocated()` still `2`, and `total_allocated() == 3` after a third `take()` past capacity; then fills the first sketch with `zipf_u64(5_000, 512, 1.1, 4_202)` inserts (checking `calc_l1() > 0.0`), `put()`s it back and asserts `available() == 1`, and requires the next `take()` to report `calc_l1() == 0.0` with `total_allocated()` still `3`. |
Expand Down
Loading