Skip to content

fix(portable): lay out the CountSketch row hash by shape - #230

Closed
zzylol wants to merge 1 commit into
mainfrom
fix/matrix-rows-hash-independently
Closed

zzylol wants to merge 1 commit into
mainfrom
fix/matrix-rows-hash-independently

Conversation

@zzylol

@zzylol zzylol commented Oct 15, 2026

Copy link
Copy Markdown
Contributor

Why

The portable CountSketch sliced every row's column and sign bits from one 64-bit hash whatever the shape. Once rows * (ceil_log2(cols) + 1) passes 64, deep rows reuse shallow rows' bits: at 30000 columns (15 column bits and a sign bit per row) only about four rows are independent, so the median over rows does not have the error bound its depth promises. Debug builds panic on the shift (attempt to shift right with overflow). ASAPPlanner hit both and moved off this type (ASAPPlanner #706).

What

CountSketch::update and estimate lay out the key's hash by shape, as the native fast-path sketches do: one 64-bit hash while every row fits, then one 128-bit hash, then one hash per row. A shape that fits 64 bits gets the same hash as before, so its cells and the Go parity probe are unchanged.

How

Both methods call a private row_hash, which returns hash_for_matrix_seeded(0, rows, cols, key): the same seed-0 xxHash the old code used, in MatrixHashType::Packed64 when the shape fits, Packed128 when it fits 128 bits, and Rows otherwise.

Key code interfaces added / modified

src/message_pack_format/portable/countsketch.rs

impl CountSketch {
    /// Adds `value` to `key`'s cell in every row, signed per row. Used per sample.
    pub fn update(&mut self, key: &str, value: f64);   // was: one Packed64 hash for every shape
    /// Median over rows of the sign-corrected cells. Used to read one key.
    pub fn estimate(&self, key: &str) -> f64;           // was: one Packed64 hash for every shape
}

Usage, end to end (checked by a scratch test at the head commit):

use asap_sketchlib::CountSketch;

// A collector sizes a CountSketch for a 2e-4 point error: 30000 columns, 15 rows. `sketch`
// is the window's state; each sample adds its value to its key.
let mut sketch = CountSketch::new(15, 30_000);   // panicked on the first update in debug builds
sketch.update("api", 2.5);
sketch.update("db", 4.0);
// What the query reads back: 2.5 and 4.0, each the median of 15 independently hashed rows.
let api = sketch.estimate("api");
let db = sketch.estimate("db");

Before this PR

A 15 x 30000 sketch panicked in debug builds and, in release, read deep rows' columns from bits the shallow rows' signs and columns already used.

After this PR

The same sketch hashes every row independently and reads back its weights. Shapes up to 64 bits are byte-for-byte unchanged.

Verification

  • New deep_shapes_hash_every_row_independently (15 x 30000: one update reads back; over 200 keys no two rows share more than 2 columns). It failed first with the shift-overflow panic.
  • New packed_shapes_keep_their_cells (5 x 1024: each row's column equals the packed 64-bit hash's) and the existing tests/sketches_go_parity_probe.rs (4 tests) pass.
  • cargo fmt --all -- --check; clippy with -D warnings for --all-features and default features; cargo test --all-features --locked countsketch passes.

Limitations

The portable CountSketch sliced every row's column and sign bits from one
64-bit hash whatever the shape. Once rows * (ceil_log2(cols) + 1) passed
64, deep rows reused shallow rows' bits: at 30000 columns (15 column
bits and a sign bit per row) only about four rows were independent, and
debug builds panicked on the shift.

update and estimate now take the key's hash from hash_for_matrix_seeded,
which the native fast-path sketches use: one 64-bit hash while the rows
fit, then one 128-bit hash, then one hash per row. A shape that fits 64
bits gets the same hash as before, so its cells and the Go parity probe
are unchanged.
@zzylol

zzylol commented Oct 20, 2026

Copy link
Copy Markdown
Contributor Author

Closing: the portable CountSketch it fixed was removed from main (#241); the remaining CountSketch already lays out its row hash by shape (hash_layout_for_matrix).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant