Skip to content

fix(array): honor the execution allocator for RowFn outputs - #10014

Draft
connortsui20 wants to merge 5 commits into
developfrom
ct/row-fn-output-allocator
Draft

connortsui20 wants to merge 5 commits into
developfrom
ct/row-fn-output-allocator

Conversation

@connortsui20

@connortsui20 connortsui20 commented Sep 23, 2026

Copy link
Copy Markdown
Member

Depends on #10016.

RowFn output payloads bypass the execution allocator. This change allocates them directly through ctx.allocator() across owned, deferred-retry, selected, filtered, constant, and sink execution, while preserving zero-copy primitive publication and empty-output paths.

OutputElement chooses its collection storage through an associated buffer type and an allocation hook. The executor writes through OutputBuffer slots, and the buffer implementation constructs the array. Vortex primitive and Boolean implementations use BufferMut, while scalar and fixed-size-list sinks use the same storage contract. UTF-8 descriptors, external bytes, and polygon payloads also use the execution allocator. Physical sink parameters remain separate from allocation resources.

Regressions check ownership of returned payloads using canonical inputs prepared before allocation tracking, including a context override, constant UTF-8 output, retry execution, and zero-copy reuse. A zero-sized output with Vec storage exercises the owned execution paths without requiring BufferMut. Boolean collector selection is preserved.

The Boolean dense-retry path from merged #9986 also uses the execution allocator, with coverage in the existing packed-output allocator test.

Validation before the rebase: 168 focused comparison, mask, and RowFn tests passed on the combined stack through #9979, along with cargo clippy -p vortex-array --all-targets --all-features -- -D warnings. Tests, formatting, and benchmarks were not rerun locally after the rebase.

@connortsui20 connortsui20 added the changelog/break A breaking API change label Sep 23, 2026
@codspeed

codspeed Bot commented Sep 23, 2026

Copy link
Copy Markdown

Merging this PR will degrade performance by 11.73%

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

⚠️ 3 benchmarks measured no execution time

Nothing ran under measurement, usually because the compiler removed the code under test. These results are not comparable, so they count as unchanged.

Preventing compiler optimizations

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 4 improved benchmarks
❌ 7 regressed benchmarks
✅ 2165 untouched benchmarks
⏩ 385 skipped benchmarks1

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Mode Benchmark BASE HEAD Efficiency
Simulation take[duplicates/repeated/primitive/nonnull/chunks=16/indices=1000] 28.7 µs 102.8 µs -72.13%
WallTime filtered_owned_i64_avx512[OneNullInEight] 22.3 µs 27.7 µs -19.29%
WallTime filtered_owned_i64_avx2[OneNullInEight] 22 µs 25.6 µs -13.91%
WallTime decode_avx2[8192, (External, AllValid)] 151.5 µs 173.5 µs -12.7%
WallTime decode_avx2[8192, (External, OneNullInEight)] 141.5 µs 160.6 µs -11.86%
Simulation allocate_drop_arrow[65536] 90.7 µs 101.9 µs -11%
Simulation non_nullable[256] 195.5 µs 218.4 µs -10.46%
WallTime mul_u64_nonnull_neon 40 µs 29.1 µs +37.75%
WallTime mul_i64_nonnull_neon 38.5 µs 32.8 µs +17.45%
WallTime multiply_shapes_neon[(32768, PerRowPerRow)] 38.6 µs 32.9 µs +17.32%
WallTime filtered_sink_i64_avx512[OneNullInEight] 26.4 µs 23.5 µs +12.44%
⚠️ Simulation fixed_16_advancing_ptr_safe[100] < 1 ns < 1 ns N/A
⚠️ Simulation preverify_advancing_ptr_unchecked[1000] < 1 ns < 1 ns N/A
⚠️ Simulation preverify_advancing_ptr_unchecked[10000] < 1 ns < 1 ns N/A

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.


Comparing ct/row-fn-output-allocator (e1ca00c) with develop (2ab66d7)2

Open in CodSpeed

Footnotes

  1. 385 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports.

  2. No successful run was found on develop (ad6ee9d) during the generation of this report, so 2ab66d7 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report.

@connortsui20
connortsui20 force-pushed the ct/row-fn-output-allocator branch from 7042d42 to 573b85f Compare September 23, 2026 21:19
@connortsui20
connortsui20 added this pull request to stack #10021 September 23, 2026 21:19
@connortsui20
connortsui20 removed this pull request from stack #10021 September 23, 2026 21:23
@connortsui20
connortsui20 force-pushed the ct/row-fn-output-allocator branch from 573b85f to 4aef610 Compare September 23, 2026 21:23
@connortsui20
connortsui20 changed the base branch from develop to ct/all-valid-mask-reduction September 23, 2026 21:23
@connortsui20
connortsui20 added this pull request to stack #10022 September 23, 2026 21:23
@connortsui20
connortsui20 force-pushed the ct/row-fn-output-allocator branch 2 times, most recently from 6bf049e to a200209 Compare September 23, 2026 21:39
@connortsui20
connortsui20 force-pushed the ct/row-fn-output-allocator branch from a200209 to 7da447a Compare September 23, 2026 21:53
Base automatically changed from ct/all-valid-mask-reduction to develop September 24, 2026 02:36
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
@connortsui20
connortsui20 force-pushed the ct/row-fn-output-allocator branch from 7da447a to e1ca00c Compare September 24, 2026 02:37

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/break A breaking API change

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant