SQL style null-on-empty semantics for Sum and Mean

[mhk197]

This changes the existing vortex.sum to use SQL-style null-on-empty semantics.

New Sum semantics

Input	Result
No rows	null
All values null	null
Valid values summing to zero	0
Integer or decimal overflow	null
All NaNs with the default skip_nans	0
Any NaN with include_nans	NaN

NaNs count as valid values even when skipped, which is why an all-NaN Sum is zero rather than empty.

Partial state

Every new Sum partial is:

{
    sum: T,
    is_overflow: bool,
    is_empty: bool,
}

The sum field is non-nullable. The outer struct is nullable so grouped or null rows can still be represented. skip_nans remains aggregate configuration and is not part of the partial.

A fresh accumulator begins as:

{ sum: 0, is_overflow: false, is_empty: true }

Observing a valid value clears is_empty. Empty states are merge identities, overflow is absorbing, and finalization returns null when either is_empty or is_overflow is true.

Grouped Sum kernels produce this same state directly. As a result, list_sum no longer needs a separate empty-list mask: null, empty, and all-null lists naturally finalize to null.

Mean

Mean remains a combined Sum + Count aggregate and now uses the canonical Sum state. It still finalizes as sum / count and retains its count == 0 guard.

• Empty or all-null Mean returns null.
• All-NaN Mean with skip_nans returns null because count is zero.
• An overflowed Sum produces a null Mean.
• Normal input produces sum / count.

Backwards compatibility

Legacy scalar partials are normalized into the canonical struct:

Incoming partial	Canonical interpretation	Final behavior
Legacy scalar x	{ sum: x, is_overflow: false, is_empty: false }	x
Legacy scalar 0	{ sum: 0, is_overflow: false, is_empty: false }	0
Legacy scalar null	{ sum: 0, is_overflow: true, is_empty: false }	null
New struct partial	Used directly	New semantics

This preserves zero-on-empty when an old partial appears. The legacy scalar representation could not distinguish an empty Sum from a genuine zero, so every legacy zero remains a nonempty zero.

Both scalar and array partials are normalized. Cached scalar Stat::Sum values also pass through this compatibility path.

Sum options and zone-map deserialization

Sum now uses SumAggregateOpts { skip_nans, struct_partial }. New/default Sum options set and serialize struct_partial = true, selecting the canonical struct partial.

Historical Sum options serialized only skip_nans. The new options decoder is wire-compatible with those bytes and treats a missing struct_partial field as false. That selects the legacy nullable scalar partial dtype when an existing zone map is deserialized.

This is needed because a zoned layout derives its stored stats-table schema from the deserialized aggregate options. The option therefore lets old zone maps validate and read their scalar Sum fields, while newly written zone maps derive the struct Sum field. Legacy scalar fields are normalized once after reading, and the rest of pruning uses the canonical representation.

No separate Sum-specific zoned metadata version or schema branch is needed.

Forward compatibility

New files store struct Sum partials in zoned stats tables. Readers from before this change ignore the new protobuf option field, derive the historical scalar Sum dtype, and reject the struct stats field during dtype validation.

This is intentional under Vortex's current compatibility contract: new readers remain able to read older files, but older readers are not yet guaranteed to read newer files. The zoned metadata version remains unchanged because struct_partial disambiguates the stored schema for new readers; bumping the version would only make older readers reject the layout earlier.

Closes #9084.

[codspeed-hq]

Merging this PR will regress 10 benchmarks

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 9 improved benchmarks
❌ 10 regressed benchmarks
✅ 1918 untouched benchmarks
🆕 4 new benchmarks
⏩ 85 skipped benchmarks¹

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

	Mode	Benchmark	BASE	HEAD	Efficiency
❌	Simulation	sum_f64_all_valid	87.7 µs	194.5 µs	-54.91%
❌	Simulation	sum_i32_nullable_all_valid	95.2 µs	185.9 µs	-48.77%
❌	Simulation	fsl_sum_small	136.3 µs	212.8 µs	-35.94%
❌	Simulation	sum_i32_clustered_nulls	150.2 µs	230.8 µs	-34.9%
❌	Simulation	sum_f64_clustered_nulls	158.1 µs	238.8 µs	-33.77%
❌	Simulation	listview_sum_small	174.7 µs	223.7 µs	-21.92%
❌	Simulation	cold_misaligned[(64, 256)]	4.4 ms	5 ms	-12.62%
❌	Simulation	search_index_in_range_chunked	5.6 ms	6.4 ms	-11.64%
❌	Simulation	list_sum_small	279.2 µs	315.4 µs	-11.49%
❌	Simulation	sparse_null_count	77.3 µs	86.2 µs	-10.37%
⚡	Simulation	list_sum_nullable_elements_medium	11.6 ms	4.7 ms	×2.5
⚡	Simulation	list_sum_nullable_elements_large	1,093.1 ms	464.4 ms	×2.4
⚡	Simulation	listview_sum_large	231.7 ms	150.5 ms	+53.93%
⚡	Simulation	fsl_sum_medium	1.7 ms	1.2 ms	+49.74%
⚡	Simulation	list_sum_large	275.4 ms	185.4 ms	+48.56%
⚡	Simulation	fsl_sum_large	140 ms	96.5 ms	+45%
⚡	Simulation	listview_sum_medium	2.1 ms	1.6 ms	+30.48%
⚡	Simulation	list_sum_medium	2.5 ms	2 ms	+27.38%
⚡	Simulation	count_i32_clustered_nulls	113.9 µs	102.6 µs	+11.02%
🆕	Simulation	canonical_sum_f64_all_valid	N/A	211.6 µs	N/A
...	...	...	...	...	...

ℹ️ Only the first 20 benchmarks are displayed. Go to the app to view all benchmarks.

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.

---

_{Comparing mk/unify-sums-v2 (e2b15cb) with develop (19f771f)}

Footnotes

1. 85 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

[myrrc]

LGTM for vortex-duckdb/vortex-sqllogictest changes

[robert3005]

thanks a lot for handling this

[gatesn]

I'm thinking we should revert this and instead just introduce a new sum aggregate function with a new ID.