Preserve Boolean buffer handles during mask reduction - #10018
Conversation
d0dc4b9 to
d3f98b9
Compare
Merging this PR will degrade performance by 21.38%
|
| Mode | Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|---|
| ❌ | Simulation | take_fsl_random[128, 10] |
32.7 µs | 59.4 µs | -44.98% |
| ❌ | Simulation | density_sweep_single_slice[0.9] |
28.5 µs | 45.8 µs | -37.74% |
| ❌ | WallTime | filtered_sink_i64_avx512[OneNullInEight] |
22.3 µs | 26.4 µs | -15.53% |
| ❌ | WallTime | dict_canonicalize_gt_u8_neon[1000000] |
488.5 µs | 547.5 µs | -10.77% |
| ⚡ | WallTime | filtered_owned_i64_avx512[OneNullInEight] |
26 µs | 22.4 µs | +16.36% |
| Simulation | take_fsl_u32_random[256, 10] |
< 1 ns | < 1 ns | N/A | |
| Simulation | fixed_16_advancing_ptr_safe[100] |
< 1 ns | < 1 ns | N/A | |
| Simulation | preverify_advancing_ptr_unchecked[1000] |
< 1 ns | < 1 ns | N/A | |
| Simulation | preverify_advancing_ptr_unchecked[10000] |
< 1 ns | < 1 ns | N/A |
Tip
Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.
Comparing ct/bool-mask-buffer-handle (5b30168) with develop (515e61e)
Footnotes
-
329 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
d3f98b9 to
325f6b4
Compare
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
325f6b4 to
5b30168
Compare
## Summary
Allows lazy masks to attach directly as validity when an encoding can
inspect validity through metadata and the input is `AllValid` or
`NonNullable`. Array-backed validity and encodings that have not opted
in retain the existing lazy-mask fallback.
The reduction rule, in pseudocode, is:
```text
Mask(Array(values, validity = AllValid), m)
-> Array(values, validity = m)
Mask(Array(values, validity = NonNullable), m)
-> Array(values, validity = m)
```
This applies when the input encoding opts into
`VALIDITY_IS_METADATA_ONLY` and supports mask reduction. `m` is a
non-nullable Boolean array with the same length as the input, and it may
be lazy. The reduction reuses the values and attaches `m` as validity
without executing it. The output dtype is nullable.
The identity is `AllValid AND m = m`, so no intermediate validity array
is needed.
## Changes
Adds `MaskReduce::VALIDITY_IS_METADATA_ONLY`, defaulting to false, and
enables it for Boolean, primitive, decimal, string, list, struct, map,
and ByteBool encodings. String, decimal, and list reducers preserve
existing buffers and children when rebuilding. List-view masking also
preserves its metadata without revalidating the zero-copy flag.
Two focused tests cover direct attachment to an all-valid primitive
array and the fallback for array-backed validity, including an all-true
bitmap. Existing constant-mask tests remain unchanged, and all tests
stay inline. The Boolean device-buffer fix landed in #10018.
Validation before the rebase: 168 focused comparison, mask, and RowFn
tests passed on the combined stack through #9979, along with `cargo
clippy -p vortex-array --all-targets --all-features -- -D warnings`. The
rebase preserves this PR's patches. Tests, formatting, and benchmarks
were not rerun locally after the rebase.
---------
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Summary
Preserves the buffer handle and bit offset when masking a Boolean array. The previous reducer called
to_bit_buffer(), which requires host memory and panics for device-backed values.Changes
Rebuilds through
try_new_from_handleand adds one device-buffer regression that checks buffer identity, bit offset, length, and attached validity.