Reduce lazy masks into arrays with all-valid metadata

[connortsui20]

Summary

Allows lazy masks to attach directly as validity when an encoding can inspect validity through metadata and the input is AllValid or NonNullable. Array-backed validity and encodings that have not opted in retain the existing lazy-mask fallback.

The reduction rule, in pseudocode, is:

Mask(Array(values, validity = AllValid), m)
    -> Array(values, validity = m)

Mask(Array(values, validity = NonNullable), m)
    -> Array(values, validity = m)

This applies when the input encoding opts into VALIDITY_IS_METADATA_ONLY and supports mask reduction. m is a non-nullable Boolean array with the same length as the input, and it may be lazy. The reduction reuses the values and attaches m as validity without executing it. The output dtype is nullable.

The identity is AllValid AND m = m, so no intermediate validity array is needed.

Changes

Adds MaskReduce::VALIDITY_IS_METADATA_ONLY, defaulting to false, and enables it for Boolean, primitive, decimal, string, list, struct, map, and ByteBool encodings. String, decimal, and list reducers preserve existing buffers and children when rebuilding. List-view masking also preserves its metadata without revalidating the zero-copy flag.

Two focused tests cover direct attachment to an all-valid primitive array and the fallback for array-backed validity, including an all-true bitmap. Existing constant-mask tests remain unchanged, and all tests stay inline. The Boolean device-buffer fix landed in #10018.

Validation before the rebase: 168 focused comparison, mask, and RowFn tests passed on the combined stack through #9979, along with cargo clippy -p vortex-array --all-targets --all-features -- -D warnings. The rebase preserves this PR's patches. Tests, formatting, and benchmarks were not rerun locally after the rebase.

[codspeed]

Merging this PR will regress 4 benchmarks

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

⚠️ 3 benchmarks measured no execution time

Nothing ran under measurement, usually because the compiler removed the code under test. These results are not comparable, so they count as unchanged.

Preventing compiler optimizations

⚡ 12 improved benchmarks
❌ 4 regressed benchmarks
✅ 2160 untouched benchmarks
⏩ 385 skipped benchmarks¹

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

	Mode	Benchmark	BASE	HEAD	Efficiency
❌	Simulation	take_chunked_fsl_sorted[32, 64]	195.7 µs	269.6 µs	-27.42%
❌	WallTime	decode_avx2[8192, (Inline, OneNullInEight)]	86 µs	104.7 µs	-17.86%
❌	WallTime	filtered_owned_i64_avx512[OneNullInEight]	22.2 µs	26.2 µs	-15.14%
❌	WallTime	filtered_owned_i64_avx2[OneNullInEight]	22 µs	25.5 µs	-13.88%
⚡	WallTime	deferred_bool_avx2[16384, (ConstantLhs, PartialAccepted)]	205.8 µs	33.2 µs	×6.2
⚡	WallTime	deferred_bool_avx512[16384, (ConstantLhs, PartialAccepted)]	206.2 µs	38.2 µs	×5.4
⚡	WallTime	deferred_bool_neon[16384, (ConstantLhs, PartialAccepted)]	169.5 µs	54.3 µs	×3.1
⚡	WallTime	deferred_bool_avx2[16384, (Columns, PartialAccepted)]	109.8 µs	42.2 µs	×2.6
⚡	WallTime	deferred_bool_avx512[16384, (ConstantLhs, NullOnlyFailure)]	470.8 µs	286.3 µs	+64.45%
⚡	WallTime	deferred_bool_avx2[16384, (ConstantLhs, NullOnlyFailure)]	454.5 µs	281.4 µs	+61.5%
⚡	Simulation	take_fsl_f16_force_per_index[256, 10]	57 µs	38.8 µs	+46.85%
⚡	WallTime	mul_u64_nonnull_neon	40.4 µs	29.1 µs	+38.76%
⚡	WallTime	deferred_bool_neon[16384, (ConstantLhs, NullOnlyFailure)]	478.6 µs	377.5 µs	+26.79%
⚡	WallTime	deferred_bool_avx2[16384, (Columns, NullOnlyFailure)]	344.3 µs	278.7 µs	+23.53%
⚡	WallTime	multiply_shapes_neon[(32768, PerRowPerRow)]	39 µs	32.7 µs	+19.03%
⚡	WallTime	mul_i64_nonnull_neon	38.9 µs	32.9 µs	+18.38%
⚠️	Simulation	fixed_16_advancing_ptr_safe[100]	< 1 ns	< 1 ns	N/A
⚠️	Simulation	preverify_advancing_ptr_unchecked[1000]	< 1 ns	< 1 ns	N/A
⚠️	Simulation	preverify_advancing_ptr_unchecked[10000]	< 1 ns	< 1 ns	N/A

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.

---

_{Comparing ct/all-valid-mask-reduction (a8d283b) with develop (26231bb)²}

Footnotes

1. 385 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

2. No successful run was found on develop (da48008) during the generation of this report, so 26231bb was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

[joseph-isaacs]

should this be on the main vtable?

[connortsui20]

honestly I would like that