[zzylol]

Why

Every Count Sketch estimate collects its per-row estimates in a fresh Vec and then takes the median, so each query pays a heap allocation and a free. On small shapes that is most of the gap between an estimate and an update. On main eb04e7f, 3×1024 with u64 keys costs 59 ns per fast-path estimate against 21 ns per update.

What

The per-row estimates are collected on the stack:

• RowEstimates (a crate-private SmallVec<[f64; MATRIX_MAX_ROWS]>) for:
  • Vector2D (fast_query_median, fast_query_median_with_key), which serves the CountSketch fast path;
  • the CountSketch regular path;
  • BitMatrix::fast_query_median;
  • CountSketchWithL2Norm::estimate_with_hash and l2_squared;
  • CountSketchFold::estimate.
• A [f64; ROWS] array for the fixed-size matrices (QuickMatrixI32/I64/I128, FixedMatrix, in impl_fixed_matrix!).

No Count Sketch median allocates for 20 rows or fewer. Wider in-memory Vector2D or BitMatrix matrices spill as before. Results are unchanged; the median code is untouched.

How

The collected Vecs become RowEstimates, and the fixed-matrix macro fills a [f64; $rows] array. compute_median_inline_f64 and the regular path's sort take the slice as before.

Before / After

Benchmark: a local example, not committed. Release build, pinned to one core (taskset -c 50), 2^20 keys per case, the two builds alternated over 7 rounds, median ns/op. The host was shared (load 40–60), so differences under ~5 ns are noise. The update rows are unchanged code. The regular-path update reads 7–9 ns slower in this table. A separate example that times only that update (3×1024, 5×2048, 7×2048) runs at the same speed on both builds (best of 7: 83.8 vs 84.0, 122.6 vs 122.7 and 160.6 vs 160.7 ns), and its disassembly is the same length on both. So the slowdown comes from how this all-in-one bench compiles (LTO inlines every case into one function), not from the library.

u64	3x1024	5x2048	7x2048	5x30000	9x30000	15x65536
hash.matrix	8 → 7	8 → 7	8 → 8	8 → 8	100 → 102	142 → 147
cs.fast.update	21 → 22	28 → 29	56 → 54	85 → 83	202 → 221	265 → 284
cs.fast.estimate	59 → 46	77 → 66	186 → 178	126 → 113	387 → 378	568 → 601
cs.regular.update	40 → 47	62 → 69	84 → 91	84 → 87	148 → 152	247 → 252
cs.regular.estimate	102 → 94	165 → 153	229 → 225	168 → 163	313 → 316	581 → 579
cm.fast.update	18 → 17	23 → 22	37 → 36	66 → 65	194 → 201	225 → 251
cm.fast.estimate	15 → 16	21 → 21	35 → 35	64 → 64	190 → 186	230 → 234

str	3x1024	5x2048	7x2048	5x30000	9x30000	15x65536
hash.matrix	16 → 16	16 → 16	20 → 19	20 → 19	168 → 165	268 → 259
cs.fast.update	31 → 30	37 → 37	65 → 65	99 → 96	285 → 300	399 → 449
cs.fast.estimate	68 → 54	87 → 75	202 → 190	138 → 130	482 → 476	737 → 739
cs.regular.update	63 → 72	101 → 110	140 → 148	155 → 153	266 → 264	446 → 441
cs.regular.estimate	115 → 108	218 → 183	265 → 258	229 → 222	411 → 404	731 → 728
cm.fast.update	26 → 25	31 → 29	50 → 47	80 → 79	273 → 280	371 → 421
cm.fast.estimate	24 → 24	29 → 28	44 → 45	78 → 80	279 → 267	379 → 406

Not in this PR (follow-ups)

• Holding up to 20 row hashes inline in MatrixHash::Rows (was SmallVec<[u64; 8]>). Measured, and not a pure speedup. It grows MatrixHash from about 80 to about 176 bytes, and the packed layouts paid for it:
  • hash alone 8 → 11 ns;
  • CS update at 7×2048 55 → 70 ns;
  • CS update at 5×30000 (string keys) 97 → 139 ns.
    The Rows layout saved only 4–10 ns, because the row hashing (9–15 xxh3 calls) dominates, not the allocation. A Rows path that hashes each row when it is read, instead of materializing the hashes inside MatrixHash, would avoid both costs.
• One 128-bit hash per pair of rows, 64 bits each. This roughly halves the Rows hashing, since a 128-bit xxh3 costs about the same as a 64-bit one. It changes the independence assumption: two rows share one hash.
• Hash the key once, then apply a cheap per-row finalizer. About 1 ns per row, and string keys are not rehashed per row. Rows then depend on one base hash.

Verification

• cargo fmt --all -- --check
• clippy -D warnings with --all-features, default features, octo-runtime and experimental
• cargo test --all-features --locked (2,233 passed) and cargo test --locked (1,977 passed); the existing estimate, median and golden tests cover the unchanged results
• RUSTDOCFLAGS="-D warnings" cargo doc --no-deps --all-features --locked