Repository navigation
Conversation
The portable CountSketch sliced every row's column and sign bits from one 64-bit hash whatever the shape. Once rows * (ceil_log2(cols) + 1) passed 64, deep rows reused shallow rows' bits: at 30000 columns (15 column bits and a sign bit per row) only about four rows were independent, and debug builds panicked on the shift. update and estimate now take the key's hash from hash_for_matrix_seeded, which the native fast-path sketches use: one 64-bit hash while the rows fit, then one 128-bit hash, then one hash per row. A shape that fits 64 bits gets the same hash as before, so its cells and the Go parity probe are unchanged.
This was referenced Oct 15, 2026
Contributor
Author
|
Closing: the portable CountSketch it fixed was removed from main (#241); the remaining CountSketch already lays out its row hash by shape (hash_layout_for_matrix). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The portable
CountSketchsliced every row's column and sign bits from one 64-bit hash whatever the shape. Oncerows * (ceil_log2(cols) + 1)passes 64, deep rows reuse shallow rows' bits: at 30000 columns (15 column bits and a sign bit per row) only about four rows are independent, so the median over rows does not have the error bound its depth promises. Debug builds panic on the shift (attempt to shift right with overflow). ASAPPlanner hit both and moved off this type (ASAPPlanner #706).What
CountSketch::updateandestimatelay out the key's hash by shape, as the native fast-path sketches do: one 64-bit hash while every row fits, then one 128-bit hash, then one hash per row. A shape that fits 64 bits gets the same hash as before, so its cells and the Go parity probe are unchanged.How
Both methods call a private
row_hash, which returnshash_for_matrix_seeded(0, rows, cols, key): the same seed-0 xxHash the old code used, inMatrixHashType::Packed64when the shape fits,Packed128when it fits 128 bits, andRowsotherwise.Key code interfaces added / modified
src/message_pack_format/portable/countsketch.rsUsage, end to end (checked by a scratch test at the head commit):
Before this PR
A 15 x 30000 sketch panicked in debug builds and, in release, read deep rows' columns from bits the shallow rows' signs and columns already used.
After this PR
The same sketch hashes every row independently and reads back its weights. Shapes up to 64 bits are byte-for-byte unchanged.
Verification
deep_shapes_hash_every_row_independently(15 x 30000: one update reads back; over 200 keys no two rows share more than 2 columns). It failed first with the shift-overflow panic.packed_shapes_keep_their_cells(5 x 1024: each row's column equals the packed 64-bit hash's) and the existingtests/sketches_go_parity_probe.rs(4 tests) pass.cargo fmt --all -- --check; clippy with-D warningsfor--all-featuresand default features;cargo test --all-features --locked countsketchpasses.Limitations
r's seed from the 20-entry seed list, so pastMATRIX_MAX_ROWS(20) rows, rowr + 20repeats rowr. That wrap is shared by every per-row-seeded matrix sketch (nativeCountMin/CountonRegularPath,CMSHeap,CSHeap). The wire refuses such states, but the constructors still build them. Whether constructors should refuse more than 20 rows (as fix(univmon): reject layer seeds and row geometries that break hash independence #220 did for UnivMon) or derive further seeds is left as an open question; this PR does not change it.