Repository navigation
Conversation
Add Hydra::query_group_total, answering a subpopulation's total mass (e.g. count by (g)) as the minimum over rows of the hashed cell's total mass: with non-negative counts every row overestimates, so the min is the Count-Min estimator. HydraCounter::total_mass reads a cell's mass from the counter's own state: a Count-Min inner row sum or the KLL retained weight. No new state, so the wire format is unchanged. The doc comment states the error bound, and tests check exact totals, merge, misuse, min-over-rows, and the bound on a narrow weighted grid.
zzylol
force-pushed
the
feat/hydra-group-total
branch
from
October 15, 2026 03:03
6fc50f1 to
d4f0412
Compare
GordonYuanyc
added a commit
that referenced
this pull request
Oct 20, 2026
…223, #228, #229, #233, #238) (#242) * feat(univmon): check_shape refuses a pyramid shape without building it `UnivMon::check_shape(heap_size, rows, cols, layers)` accepts exactly the shapes `UnivMon::new` builds: positive dimensions, at most UNIVMON_MAX_LAYERS layers, at most MATRIX_MAX_ROWS rows, rows within the 128-bit hash budget and a counter count that fits usize. `new` panics exactly when it fails, so it no longer builds a pyramid the ASAPv1 wire refuses. Ports #229. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(univmon): count every cardinality candidate, direct L2, documented error bounds `estimate_cardinality` counts every heap key, including zero-weight keys and keys whose Count Sketch estimate is not positive. `estimate_l2` reads layer 0's Count Sketch F2 estimate in standard mode and sums the strata's F2 estimates in terminal mode; `estimate_g_sum` keeps the paper's recurrence, and its flag is now `count_every_key` (`true` counts every heap key). New `top_layer_complete()`. The L2 bound and the distinct-count bound, stated jointly with `top_layer_complete()`, are derived in docs/api/api_univmon.md, with Monte Carlo tests. Ports #221. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(univmon_two_tier): count every cardinality candidate and direct L2 `UnivMonTwoTier` gets the estimators `UnivMon` has: `estimate_cardinality` counts every heap key whatever its estimate, zero-weight keys included; `estimate_l2` reads layer 0's Count Sketch F2 estimate in standard mode and sums the strata's in terminal mode; new `top_layer_complete()`. The API doc states the bounds in the two-tier layout's terms, with Monte Carlo tests. Ports #222. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(hydra): group-total query from per-cell mass `Hydra::query_group_total(key)` returns a subpopulation's estimated total mass: the minimum over rows of the subkey's cell `HydraCounter::total_mass()`, a Count-Min cell's inner row sum or a KLL cell's retained weight. Count Sketch, HyperLogLog and UnivMon counters return an error. The total is read from existing cell state, so the wire format is unchanged. `query_key` and the new query share one subkey hash helper. Ports #223. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * perf(kll): allocate the item buffer as items arrive Kll's item buffer starts empty, then holds 16 items and doubles up to max_capacity; a full buffer compacts only at max_capacity, so retained items, coin tosses, quantiles, merges and ASAPv1 bytes match the eagerly allocated buffer for the same input and seed. The merge scratch buffer starts empty; merge, from_portable_state and the ASAPv1 decoder size the buffer to what they hold. lazy_allocation_matches_eager compares every level and the coin with a test-only eager copy (k = 8 to 1024, up to 50,000 items, with a merge). Ports #238. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(kll): Kll::memory_bytes and KllDynamic::memory_bytes Both report the bytes the sketch holds now: Kll its struct, item buffer, level array, merge scratch space and memoized CDF; KllDynamic its struct, item buffer and level bounds. Ports #238. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(kmv): estimate and merge by shared reference, memory_bytes, early reject past the k-th hash Kmv::estimate takes &self and Kmv::merge takes &Kmv. Kmv::memory_bytes is the struct plus the hash slots the heap has allocated (CommonHeap::allocated_slots, crate-private). A full sketch turns away a hash at or above its largest retained one before scanning the retained hashes for a duplicate; the retained set and every estimate are unchanged. Ports #228. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(heap)!: TopKHeap::estimated_memory_bytes; UnivMonQ::memory_bytes TopKHeap::estimated_memory_bytes(heap_size, key) is the bytes a heap of that capacity holds in steady state once keys shaped like key churn through it: the struct, entry and digest arrays, the digest index (std HashMap layout, grown once more when tombstones force a resize of an index over half full) and the owned key bytes. A heap holds at most this. UnivMonQ::estimated_memory_bytes(&self) becomes memory_bytes(&self) and counts what the instance holds: its counters, candidate tables (bucket count from the largest capacity() seen), candidate heaps and ordered-sample heap. tests/heap_memory_bytes.rs checks both against a counting global allocator: the heap estimate as an upper bound after a fill and exactly after churn, UnivMon-Q exactly. Ports #233 (the heap half). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(cms,cs)!: check_shape, clear and memory accounting for the matrix sketches CountMinSketch, CountSketch and their WithHeap and WithSpaceSaving forms gain check_shape(rows, cols), which accepts the ASAPv1 geometry, 1 <= rows <= MATRIX_MAX_ROWS and 1 <= cols <= u32::MAX, on every hashing path. with_dimensions, new and from_storage panic on any other shape. Past 20 rows the regular path and the fast path's per-row (Rows) layout hash row r + 20 with row r's seed; the packed layouts reuse no seed. Tests that built such a shape to watch the encoder refuse it check check_shape instead, Hydra's included. clear() zeroes the counters (and empties the heap for the WithHeap types) and returns the Nitro sampler to its start (Nitro::reset, through the new MatrixStorage::reset_sampler), keeping shape, allocation and admission rate. TopKHeap::memory_bytes reports the bytes a heap holds, taking the digest index's bucket count from the largest capacity() it has reported, which evictions' tombstones cannot lower. The WithHeap types add memory_bytes and estimated_memory_bytes(rows, cols, heap_size, key). tests/heap_memory_bytes.rs checks all of them against a counting allocator, after a fill and after 50 times heap_size evicting keys. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(hash)!: a seed for every seed index; matrix sketches take any row count Seed index i hashes with seed_for_index(i) = SEED_LIST[i % 20] * (i / 20 + 1) + i / 20 in wrapping u64 arithmetic; indexes below 20 keep their seed. The matrix hash passes seed indexes through unreduced, so rows past 20 on the regular path and the fast path's per-row layout hash independently, and UnivMon-Q's occurrence priorities hash at their own seed. check_shape on Count-Min, Count Sketch and their WithHeap and WithSpaceSaving forms refuses only a zero dimension and a counter count past usize; Elastic, Hydra and UnivMon build any row count. to_asapv1 and from_asapv1 keep the MATRIX_MAX_ROWS bound. The UnivMon-Q residual-CDF accuracy test hashes each of its 32 sketches at its own seed index. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(cms,cs)!: memory_bytes, CountMinF64 at the root, check_shape without type parameters CountMinSketch and CountSketch gain memory_bytes(&self): the struct plus its rows * cols counters, exact for Vector2D storage, checked against a counting allocator in tests/heap_memory_bytes.rs. CountMinF64 is re-exported from sketches and the crate root. check_shape on CountMinSketch, CountSketch, their WithHeap and WithSpaceSaving forms is defined on the type with its default parameters, so CountMinSketch::check_shape(3, 64) resolves without naming storage, path or hasher. A call naming non-default type parameters no longer compiles. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(asapv1)!: seed_derivation is "seed_list_index_scaled" The ASAPv1 seed_derivation string names seed_for_index. Bytes carrying "seed_list_index_wrap" no longer decode. The golden submodule moves to the regenerated fixtures (sketchlib-golden-bytes#3). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: zz_y <zeyingz@umd.edu> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Hydra answers a subpopulation's item frequency (
query_key/query_frequency), but not the subpopulation's total mass. Queries likecount by (g)therefore cannot be answered from a Hydra grid, and a caller holding a Count-Min Hydra has to reject them.What
Hydra::query_group_total(key)returns a subpopulation's estimated total mass. Thekeyargument is the same positional, full-width key asquery_key.HydraCounter::total_mass()returns one cell's total mass, read from the counter's own state:KLL::count), which compaction preserves.HydraQuery::L1Normreturns.How
The subpopulation hashes to one cell per row, as in
query_key. Each row's estimate is that cell'stotal_mass(), and the answer is the median over rows, the same combinationquery_keyuses. The subkey hashing is factored into a privatesubkey_hashhelper that both queries share.The doc comment states this error bound. Let
n_Qbe the true total,Nthe total inserted mass,Dthe number of key columns,wthe column count andrthe row count:A cell holds
n_Qplus the mass of every other subkey hashed to it. The expected excess is at mostM/w, whereM = N(2^D - 1)is the mass after fan-out. By Markov's inequality, a row exceeds4M/wwith probability at most 1/4, so the median fails only when a majority of rows fail. The lower bound assumes non-negative update counts.Alternative considered: a dedicated total cell per subkey, updated on insert. It would need new grid state and a wire-format change, and it would add nothing for Count-Min or KLL cells, which already hold their exact mass.
Before this PR
After this PR
Verification
The tests were written first; they did not compile before the method was added.
hydra_group_total_matches_exact_totals: weighted rows with distinct items per row. Count-Min and KLL grids return exact totals for every one-column and two-column group, return 0 for an unseen group, and return exactly twice the totals after a self-merge.hydra_group_total_rejects_misuse: wrong arity, an unconstrained key, and Count Sketch, HyperLogLog and UnivMon counters all return errors.hydra_group_total_obeys_documented_bound: a narrow 5 x 64 grid with D = 2, 180 weighted rows and 243 groups. Every estimate is at least the exact total, all 243 groups overestimate (so collisions are exercised), and 0 groups exceed the bound4M/w = 202.5. The test asserts that the violation rate is at mostP[Bin(5, 1/4) >= 3] = 0.104.cargo fmt --check,clippy -D warningswith and without--all-features,cargo testwith and without--all-features,cargo doc) was run on a separate Linux machine.Limitations and follow-up
e^-r-style guarantee. This PR keeps the median for consistency withquery_key; switching would be a small follow-up.i64and returned asf64, so they are exact up to 2^53.